Cloud Network has emerged as one of the most adopted technologies both among the end-users and the developers. Despite cloud networks being popular, security in cloud remains a pivotal research challenge and a topic that is much discussed about. Denial of service (DoS) attack is carried out in cloud by one or more perpetrators using multiple compromised nodes to flood a specific target and thereby resulting in unavailability of services. Classification methods can be used effectively to identify attack signature or recurring patterns of such DoS attacks. Therefore, classification using machine learning techniques have been used in this work for feature selection and classification in order to identify the DoS attacks. For this purpose, a new rule based approach for detecting the DoS attacks which uses a domain expert’s knowledge has been proposed in this paper. Moreover, two new algorithms namely Feature Selection Algorithm using Scoring and Ranking and Rule based Classification Algorithm for detecting DoS Attacks are proposed in this paper in which the final classification is carried out by applying the rules from the rule base and is validated using a domain-expert. We have evaluated the proposed system on an experimental set-up on cloud and used real time DoS tools and observed that the proposed method achieved better DoS attack detection accuracy than the existing classification algorithms used for security. © 2018, Springer Science+Business Media, LLC, part of Springer Nature.