Among all the reasons for the occurrence of road accidents, the human state of drowsiness and Underage driving contributes a major share. With the rigid implementation of traffic rules and national schemes, it does not result in decreasing the accidents. Hence, there is a need for automation of surveillance which strictly restricts the teen driving and fatigue driving. In this paper, we introduce a face image descriptor-based combination of deep learning model i.e., convolution neural network (CNN) with ResNet50 architecture to predict age and a recurrent neural network (RNN) with LSTM architecture to detect the drowsiness in driver and alert them when they are drowsy. The algorithm is based on face recognition for age prediction and the blink frequency for detecting the fatigue. Image processing techniques are utilized to obtain the feature-based extraction for prediction. The proposed model developed could give a validation accuracy of 96% thus providing the promising results. This automation model thus could help the road safety authorities in their work and also decreases the occurrence of road accidents.