In recent days, the number of road accidents are greatly increased in world wide. The people around the world died due to the rapid road crush. The driver behavior must be detected to neglect the occurrence of road accidents. In this work, the three stage deep learning-based techniques are developed to detect the distracted driver action. The developed model is implemented in three phase to detect the distracted behavior of the driver during the driving process. In the first stage, the images are gathered from standard public resources regarding various drivers while driving. Next, the deep features retrieved using VGG16 and Inception and Xception networks. Then, the weighted fusedfusion is done by gathered features, where the weight parameter is optimized through the Adaptive Gradient-based Optimizer (AGBO) algorithm. Further, the driver behavior classification is done through a Deep Belief Network (DBN) with Radial Basis Functions Neural Networks (RBF), where the hyperparameters of DBN and RBF are optimized through the same AGBO algorithm. Finally, the first set of classified outcomes is attained from the images. In the second stage, the gathered images are fed to the feature extraction stage, where the texture features and motion features are gathered from the images. These collected features are fed to the same optimized DBN with RBF and the second set of classified outcomes with human behaviors has been attained.In the third stage, the gathered image fralgoriom the dataset is fed to the PCA for extracting the features. The gathered features are given to the same optimized DBN with the RBF classifier to get the classified outcome. Later, the classified outcomes from the three stages are fed to the ranking procedure for detecting the distracted driver action. When considering the evaluation indices, the suggested distracted driver prediction model attains 97.67% of accuracy, 97.68% of sensitivity, 97.67% of specificity, and 97.55% of precision value. Thus, the performance of the developed model in the distracted driver prediction is enhanced than the existing approaches.