The poor security and larger number of IoT devices are highly possible to be snatched and results in Distributed Denial of Service attack. The IoT attacks corrupt the availability of particular nodes or the whole network by jamming the signal or exhausting the resource nodes’ battery. The devices of IoT are facing security problems because of the increase in attacks which are launched by violators during data transmission via internet. To detect such attacks, a hybrid optimization approach usinga hybrid Metaheuristic lion optimization algorithm and Firefly optimization algorithm (ML-F) is proposed. The NSL-KDD and NBaIoT datasets are used where the input data is pre-processed to remove the noises and the missing data. After preprocessing the data, feature extraction is undergone using Recursive feature elimination (RFE).The low rate attacks are selected after splitting the data using a hybrid ML-F optimization algorithm. After selecting the features,a random forest classifier is used for the process of classifying the attacks. The proposed hybrid ML-F method achieves higher performance than the existing gradient boost classifier method,in classifying the attacks. The proposed hybrid ML-F method achieved an accuracy of 99.98%, precision of 99.87%, recall of 100% and f-measure of 99.73%. The existing gradient boosting classifier method shows the Accuracy of 50.93%, Precision of 54.87%, Recall of 77.67% and F-measure of 64.34%. © 2021, The Society for Reliability Engineering, Quality and Operations Management (SREQOM), India and The Division of Operation and Maintenance, Lulea University of Technology, Sweden.