Malware is the one which frequently growing day by day and becomes major threats to the Internet Security. The are several methods for classifying of new malware from the existing signatures or code. The traditional approaches are not much effective to compete the new arriving malware samples. More antivirus softwares provides defense mechanism against malwares but still zero-day attack is not achieved. To enhance in mechanisms machine learning algorithms are used and provide good experimental results accordingly. While the traditional signature approaches are also failed to compete the new malwares. In this paper, we define malware and types of malware as an overview, as well we define the new mechanism of using machine learning algorithms how effective and efficient in classification of malware detection and we presented the existing works related to malware detection classification using machine learning algorithms and it is discussed about main important challenges that are facing in malware detection classification. © IJSTR 2020.