The use of mobile phones has become one of the major threats to road safety, especially in young novice drivers. To avoid crashes induced by distraction, adaptive distraction mitigation systems have been developed that can determine how to detect a driver's distraction state. A driving simulator experiment was conducted in this paper to better explore the relationship between drivers' cognitive distractions and traffic safety, and to better analyze the mechanism of distracting effects on young drivers during the driving process. A total of 36 participants were recruited and asked to complete an n-back memory task while following the lead vehicle. Drivers' vehicle control behavior was collected, and an ANOVA was conducted on both lateral driving performance and longitudinal driving performance. Indicators from three aspects, i.e., lateral indicators only, longitudinal indicators only, and combined lateral and longitudinal indicators, were inputted into both SVM and random forest models, respectively. Results demonstrated that the SVM model with parameter optimization outperformed the random forest model in all aspects, among which the genetic algorithm had the best parameter optimization effect. For both lateral and longitudinal indicators, the identification effect of lateral indicators was better than that of longitudinal indicators, probably because drivers are more inclined to control the vehicle in lateral operation when they were cognitively distracted. Overall, the comprehensive model built in this paper can effectively identify the distracted state of drivers and provide theoretical support for control strategies of driving distraction.