IoT are prone to vulnerabilities as a result of a lack of centralized management, dynamic topologies, and predefined boundary. There are diverse attacks that affect the performance of IoT network. Flooding attack is a DoS attack that brings down the network by flooding with a huge count of HELLO packets, routed to a destination that does not exist. The main intent of this paper is to develop a novel robust model for detecting and preventing HELLO flooding attacks using optimized deep learning approach. In this proposed research model, the steps like Cluster head selection, k-paths generation, HELLO flooding attack detection and prevention, and optimal shortest path selection are employed. Once after the random cluster head selection and k-paths generation, few Route Discovery Frequency Vectors like Route Discover Time and Inter Route Discovery Time of each node is determined for detecting the HELLO flooding attack. Initially, a threshold function is used to match with the computed Received Signal Strength (RSS) of each node to detect the stranger node. Further, the HELLO flood attack is confirmed by the optimized Deep Belief Network (DBN), which is removed from the network subsequently. Once after securing the network, the shortest route path selection is done optimally by the improved meta-heuristic algorithm. Here, improved Rider Optimization Algorithm (ROA) termed as Bypass-Linked Attacker Update-based ROA (BAU-ROA) is used for performing the optimal DBN as well as optimal shortest path selection. The objective constraints like node trust, distance between the nodes, delay of transmission, and packet loss ratio are considered for performing the optimal shortest path selection. Finally, the experimental evaluation of various performance measures validates the fruitful performance of the proposed model. Based on the analysis, the const function of proposed BAU-ROA is 28.1% superior to D-DHOA, 34.1% superior to DHOA, and 39.2% superior to WOA at 5th iteration. When considering the 10th iteration, the developed BAU-ROA is 12.2% superior to DHOA, 19.3% superior to D-DHOA, 28.1% superior to ROA, and 39.2% superior to WOA. © 2020 Elsevier B.V.