The detection and classification of malwares in windows executables is an important and demanding task in the field of data mining. The malwares can easily damage the system by creating harm in the user's system, so some of the existing techniques are developed in the traditional works for an accurate malware detection. But, it lacks some major drawbacks such as inaccurate detection, not highly efficient, requires a large amount of time to classify the malware type, and an increased computational complexity. To solve these issues, this article develops an efficient system for detecting the malwares in an Application Programmable Interfaces (APIs), and classifying its type as worms, virus, Trojans, or normal. Initially, the input dataset is preprocessed by normalizing the data, then its upper and lower boundaries are estimated during feature extraction. Furthermore, the Rete algorithm is implemented to generate the rules based on the pattern matching process. Here, the Multi-Dimensional Naïve Bayes Classification (MDNBS) is implemented to classify the malware that occurred in an API call sequences. In experiments, the performance results of the existing and proposed techniques are evaluated and compared based on the measures of True Positive Rate (TPR), False Positive Rate (FPR), precision, recall, f-measure and, accuracy. © 2018 Taylor & Francis Group, LLC.