This paper gives precise and comprehensive detail along with a proposed system for malware detection using ML and Deep Learning techniques by integrating both behavior-based detection methods and signature-based methods. The primary purpose of this paper is (A) Outline difficulty identified with malware detection. (B) Represent detail and categorized ML technique for malware detection. (C) Investigating the structure of basic strategies in malware discovery. (D) Inspecting the essential deep learning approach for malware detection using a grouping of malware inside the data mining. The point of interest and downside of various malware detection approaches were analyzed based on evaluation strategy and their capability. The proposed model uses random forest for making an end-to-end pipeline for malware detection. During comparative study with five other state of the art models, the proposed model obtained accuracy of 99.7% on the dataset. The experimental results show the proposed model outperformed other five state of the art techniques. This research paper encourages the researcher to think about the best approach for malware detection. © 2020 River Publishers.