Driver Emotion Research Articles

A Comprehensive Review: Multisensory and Cross-Cultural Approaches to Driver Emotion Modulation in Vehicle Systems

Road accidents are caused by multiple factors. Aggressive driving and traffic violations account for 74% of road traffic accidents. In total, 92% of fatalities occur in low- and middle-income countries. Drivers’ emotions significantly influence driving performance, making emotional modulation critical during vehicle interaction. With the rise of smart vehicles, in-vehicle affective computing and human-centered design have gained importance. This review analyzes 802 studies related to driver emotional regulation, focusing on 74 studies regarding sensory stimuli and cultural contexts. The results show that single-sensory methods dominate, yet multisensory approaches using auditory and visual elements are more effective. Most studies overlook cultural factors, particularly the differences in East–West cultural values, indicating a need to tailor modulation methods based on cultural preferences. Designs must emphasize adaptability and cultural consistency. This review aims to analyze driver emotional modulation thoroughly, providing key insights for developing vehicle systems that meet the diverse emotional and cultural needs of global drivers. Future research should focus on creating multisensory emotional modulation systems that offer positive reinforcement without causing excessive relaxation or aggression, accommodating subtle cultural and individual differences, thus enhancing the safety of autonomous driving.

Brain-Inspired Driver Emotion Detection for Intelligent Cockpits Based on Real Driving Data

Brain-Inspired Driver Emotion Detection for Intelligent Cockpits Based on Real Driving Data

P‐63: In‐Cabin Immersiveness Enhancement Based on Driving Environments and Driver's Emotion

This paper proposes an in‐car immersiveness enhancement framework based on driving environments and driver's emotional status, thereby increasing the driver's driving performance and safety. Specifically, the framework is proposed to generate new content for immersive displays and systems, providing enhanced immersiveness across different displays, systems, and driving environments. The immersiveness‐enhanced contents are generated based on the color/style transfer, analysis of the driving environment, and driver's emotional status recognition. The qualitative and quantitative evaluation proved the advantages and practicability of the proposed framework in enhancing in‐cabin immersiveness.

Driver emotion recognition based on attentional convolutional network

Unstable emotions, particularly anger, have been identified as significant contributors to traffic accidents. To address this issue, driver emotion recognition emerges as a promising solution within the realm of cyber-physical-social systems (CPSS). In this paper, we introduce SVGG, an emotion recognition model that leverages the attention mechanism. We validate our approach through comprehensive experiments on two distinct datasets, assessing the model’s performance using a range of evaluation metrics. The results suggest that the proposed model exhibits improved performance across both datasets.

Influence of emotions on the aggressive driving behavior of online ­car-hailing drivers based on association rule mining

Emotion is an important factor that can lead to the occurrence of aggressive driving. This paper proposes an association rule mining-based method for analysing contributing factors associated with aggressive driving behaviour among online car-hailing drivers. We collected drivers’ emotion data in real time in a natural driving setting. The findings show that 29 of the top 50 association rules for aggressive driving are related to emotions, revealing a strong relationship between driver emotions and aggressive driving behaviour. The emotions of anger, surprised, happy and disgusted are frequently associated with aggressive driving behaviour. Negative emotions combined with other factors (for example, driving at high speeds and high acceleration rates and with no passengers in the vehicle) are more likely to lead to aggressive driving behaviour than negative emotions alone. The results of this study provide practical implications for the supervision and training of car-hailing drivers.

Effects of a color gradient and emoji in AR-HUD warning interfaces in autonomous vehicles on takeover performance and driver emotions

Objective This study examined the effects of color gradients and emojis in an augmented reality-head-up display (AR-HUD) warning interface on driver emotions and takeover performance. Methods A total of 48 participants were grouped into four different warning interfaces for a simulated self-driving takeover experiment. Two-way analysis of variance and the Kruskal–Wallis test was used to analyze takeover time, mood, task load, and system availability. Results Takeover efficiency and task load did not significantly differ among the interfaces, but the interfaces with a color gradient and emoji positively affected drivers’ emotions. Emojis also positively affected emotional valence, and the color gradient had a high emotional arousal effect. Both the color gradient and the emoji interfaces had an inhibitory effect on negative emotions. The emoji interface was easier to learn, reducing driver learning costs. Conclusions These findings offer valuable insights for designing safer and more user-friendly AR-HUD interfaces for self-driving cars.

A Dual Channel Cyber–Physical Transportation Network for Detecting Traffic Incidents and Driver Emotion

A Dual Channel Cyber–Physical Transportation Network for Detecting Traffic Incidents and Driver Emotion

Cognitive Aspect of Emotion Estimation of a Driver

Despite rapid advancements in the automotive industry, traffic safety risks persist. Addressing this challenge requires innovative driver assistance technologies. Common accidents result from driver inattention, fatigue, and stress, leading to issues like falling asleep at the wheel and improper acceleration and braking. Our study aims to contribute to advanced driver assistance systems that adapt to drivers' emotional needs, ultimately enhancing road safety. In this paper, we mention the result of our research to estimate drivers' emotions using sensors. For that purpose, we developed a sensor network containing sensors such as EEG, eye tracker, and driving simulator. We explored the relationship. As a result, we confirmed the relation between the driver's emotions, especially sleep conditions, driving speed, duration, and brain wave behavior.

Driver Emotion Recognition Involving Multimodal Signals: Electrophysiological Response, Nasal-Tip Temperature, and Vehicle Behavior

Driver Emotion Recognition Involving Multimodal Signals: Electrophysiological Response, Nasal-Tip Temperature, and Vehicle Behavior

A Multimodal Driver Emotion Recognition Algorithm Based on the Audio and Video Signals in Internet of Vehicles Platform

A Multimodal Driver Emotion Recognition Algorithm Based on the Audio and Video Signals in Internet of Vehicles Platform

Boosting Weak Learners With Multi-Agent Reinforcement Learning for Enhanced Stacking Models: An Application on Driver Emotion Classification

Boosting Weak Learners With Multi-Agent Reinforcement Learning for Enhanced Stacking Models: An Application on Driver Emotion Classification

A multi-modal driver emotion dataset and study: Including facial expressions and synchronized physiological signals

To address the limitations of databases in the field of emotion recognition and to cater to the trend of integrating data from multiple sources, we have established a multi-modal emotional dataset based on spontaneous expression of drivers. By selecting emotional induction materials and inducing emotions before each driving task, facial expression videos and synchronous physiological signals of the drivers during driving were collected. The dataset includes records of 64 participants under five different emotions (neutral, happy, angry, sad, and fear), and the emotional valence, arousal, and peak time of all participants in each driving task were recorded. To analyze the dataset, spatio-temporal convolutional neural networks were designed to analyze the different modalities of data with varying durations in the dataset, aiming to investigate their performance in emotion recognition. The results demonstrate that the fusion of multi-modal data significantly improves the accuracy of driver's emotion recognition, with accuracy increases of 11.28% and 6.83% compared to using only facial video signals or physiological signals, respectively. Therefore, the publication and analysis of multi-modal emotional data for driving scenarios is crucial to support further research in the fields of multimodal perception and intelligent transportation engineering.

A Robust Driver Emotion Recognition Method Based on High-Purity Feature Separation

A Robust Driver Emotion Recognition Method Based on High-Purity Feature Separation

Evaluating emotionalizing effects of active sound designs

Numerous results in the field of musicological research and traffic psychology research show that emotions can be altered by certain sets of pitches or sound characteristics, and emotions on the other hand influence our driving behaviour. Regardless, there is no research on how different artificial vehicle sounds could influence the driving behaviour via emotions. We want to create an active sound design that can alter the driver's emotions to increase traffic safety in certain driving scenarios. To evaluate which harmonic compositions and sound characteristics could be used as stimuli, we first extracted sound characteristics from different music pieces or sounds that have a proven emotionalizing effect, and generated new sounds based on said characteristics that also could work in the context of an active sound design for electrical vehicles. With these sounds, we conducted a subject study with 45 participants to evaluate if people perceive the intended emotions, and further determine the best sounds to use in future subject studies to evaluate their impact on drivers' behaviour. This paper describes the extracted sounds, their characteristics and psychoacoustical properties. Furthermore, it discusses the correlation between the sound properties and the perception of and emotional effect on the subjects.

Drivers' Comprehensive Emotion Recognition Based on HAM.

Negative emotions of drivers may lead to some dangerous driving behaviors, which in turn lead to serious traffic accidents. However, most of the current studies on driver emotions use a single modality, such as EEG, eye trackers, and driving data. In complex situations, a single modality may not be able to fully consider a driver's complete emotional characteristics and provides poor robustness. In recent years, some studies have used multimodal thinking to monitor single emotions such as driver fatigue and anger, but in actual driving environments, negative emotions such as sadness, anger, fear, and fatigue all have a significant impact on driving safety. However, there are very few research cases using multimodal data to accurately predict drivers' comprehensive emotions. Therefore, based on the multi-modal idea, this paper aims to improve drivers' comprehensive emotion recognition. By combining the three modalities of a driver's voice, facial image, and video sequence, the six classification tasks of drivers' emotions are performed as follows: sadness, anger, fear, fatigue, happiness, and emotional neutrality. In order to accurately identify drivers' negative emotions to improve driving safety, this paper proposes a multi-modal fusion framework based on the CNN + Bi-LSTM + HAM to identify driver emotions. The framework fuses feature vectors of driver audio, facial expressions, and video sequences for comprehensive driver emotion recognition. Experiments have proved the effectiveness of the multi-modal data proposed in this paper for driver emotion recognition, and its recognition accuracy has reached 85.52%. At the same time, the validity of this method is verified by comparing experiments and evaluation indicators such as accuracy and F1 score.

Identification of aggressive driving behavior of online car‐hailing drivers based on association classification

AbstractWith the rapid development of online car‐hailing, the related crashes have become a key issue with public concern. Identifying and predicting aggressive driving behaviors is critical to reduce traffic crashes. In this study, we propose a method to recognize aggressive driving behavior based on association classification, with multisource features being employed, including driver emotion, vehicle kinematic characteristics, and road environment. The model performs best in a 10‐fold cross‐test when the minimum support and minimum confidence are set as 0.01 and 0.8, respectively. Besides, we also compare the performance of aggressive driving behavior recognition classifiers constructed using association classification with other rule‐based classification methods, including ID3, C4.5, CART, and Random Forest. The results show that association classification performs better than other classification competitors. Thirty‐six if–then rules generated by the association classification are used to analyze the influencing factors and associated mechanisms of aggressive driving behavior. It is found that aggressive driving behavior is highly correlated with driver anger and disgust emotions. Aggressive driving behavior is more likely to occur when no passengers are in the car than the case with passengers. Driver entertainment behavior and passenger interference also affect driving behavior. Moreover, drivers are prone to aggressive driving when making a U‐turn. This research not only proposed a new identification method for aggressive driving behavior but also provided a comprehensive understanding of the associated influencing factors which thus benefit the further research and development of safety assistance driving devices.

Driver Emotion Recognition With a Hybrid Attentional Multimodal Fusion Framework

Negative emotions may induce dangerous driving behaviors leading to extremely serious traffic accidents. Therefore, it is necessary to establish a system that can automatically recognize driver emotions so that some actions can be taken to avoid traffic accidents. Existing studies on driver emotion recognition have mainly used facial data and physiological data. However, there are fewer studies on multimodal data with contextual characteristics of driving. In addition, fully fusing multimodal data in the feature fusion layer to improve the performance of emotion recognition is still a challenge. To this end, we propose to recognize driver emotion using a novel multimodal fusion framework based on convolutional long-short term memory network (ConvLSTM), and hybrid attention mechanism to fuse non-invasive multimodal data of eye, vehicle, and environment. In order to verify the effectiveness of the proposed method, extensive experiments have been carried out on a dataset collected using an advanced driving simulator. The experimental results demonstrate the effectiveness of the proposed method. Finally, a preliminary exploration on the correlation between driver emotion and stress is performed.

Targeting young driver emotions can reduce their cellphone distractions

ObjectiveThis study sought to evaluate the relationship between young drivers’ intention to engage in cellphone distractions while driving and their emotions towards the associated risks. First, we assessed whether the emotions of guilt, shame, and fear are associated with young drivers’ intention to engage in cellphone distractions, through an extended Theory of Planned Behavior (TPB) model. Second, we evaluated whether road signs that may evoke these negative emotions reduce cellphone use intentions among young drivers. MethodsAn online survey was conducted with young drivers (18 to 25 years old) from Ontario, Canada. 403 responses were collected, of which, 99 responses were used to evaluate the first objective and all 403 responses were used to evaluate the second objective. ResultsAnticipating feelings of guilt, shame, and fear negatively predicted the intention to engage in cellphone distractions, above and beyond the standard TPB constructs (i.e., attitudes, subjective norms, and perceived behavioral control). When prompted with potentially emotion-evoking road signs (children crossing, my mom/dad works here), an increase in anticipated feelings of these emotions corresponded with lower intention to engage in cellphone distractions. ConclusionCountermeasures that target young driver emotions toward distracted driving risks may be effective in reducing their distraction engagement. Future studies in more controlled environments can investigate causal relationships between emotions and distracted driving among young drivers.

A novel driver emotion recognition system based on deep ensemble classification

Driver emotion classification is an important topic that can raise awareness of driving habits because many drivers are overconfident and unaware of their bad driving habits. Drivers will acquire insight into their poor driving behaviors and be better able to avoid future accidents if their behavior is automatically identified. In this paper, we use different models such as convolutional neural networks, recurrent neural networks, and multi-layer perceptron classification models to construct an ensemble convolutional neural network-based enhanced driver facial expression recognition model. First, the faces of the drivers are discovered using the faster region-based convolutional neural network (R-CNN) model, which can recognize faces in real-time and offline video reliably and effectively. The feature-fusing technique is utilized to integrate the features extracted from three CNN models, and the fused features are then used to train the suggested ensemble classification model. To increase the accuracy and efficiency of face detection, a new convolutional neural network block (InceptionV3) replaces the improved Faster R-CNN feature-learning block. To evaluate the proposed face detection and driver facial expression recognition (DFER) datasets, we achieved an accuracy of 98.01%, 99.53%, 99.27%, 96.81%, and 99.90% on the JAFFE, CK+, FER-2013, AffectNet, and custom-developed datasets, respectively. The custom-developed dataset has been recorded as the best among all under the simulation environment.

Real Time Drowsiness Detection System Using CNN

Abstract: Driver fatigue and rash driving are the leading causes of road accidents, which result in the loss of valued life and decrease road traffic safety. Driver drowsiness solutions that are reliable and precise are essential to prevent accidents and increase road traffic safety. Various driver drowsiness detection systems have been developed using various technologies that are geared at the specific parameter of detecting the driver's tiredness. This research offers a unique multi-level distribution model for detecting driver drowsiness utilising Convolution Neural Networks (CNN) and. To detect the driver's behaviour and emotion, the driver's face pattern is handled with a 2D Convolution Neural Network (CNN). The suggested model is built with OpenCV, and the experimental findings show that it recognises the driver's emotion and tiredness more efficiently than existing technologies