Driver Emotion Research Articles

Study on Driver Cross-Subject Emotion Recognition Based on Raw Multi-Channels EEG Data

In our life, emotions often have a profound impact on human behavior, especially for drivers, as negative emotions can increase the risk of traffic accidents. As such, it is imperative to accurately discern the emotional states of drivers in order to preemptively address and mitigate any negative emotions that may otherwise manifest and compromise driving behavior. In contrast to many current studies that rely on complex and deep neural network models to achieve high accuracy, this research aims to explore the potential of achieving high recognition accuracy using shallow neural networks through restructuring the structure and dimensions of the data. In this study, we propose an end-to-end convolutional neural network (CNN) model called simply ameliorated CNN (SACNN) to address the issue of low accuracy in cross-subject emotion recognition. We extracted features and converted dimensions of EEG signals from the SEED dataset from the BCMI Laboratory to construct 62-dimensional data, and obtained the optimal model configuration through ablation experiments. To further improve recognition accuracy, we selected the top 10 channels with the highest accuracy by separately training the EEG data of each of the 62 channels. The results showed that the SACNN model achieved an accuracy of 88.16% based on raw cross-subject data, and an accuracy of 91.85% based on EEG channel data from the top 10 channels. In addition, we explored the impact of the position of the BN and dropout layers on the model through experiments, and found that a targeted shallow CNN model performed better than deeper and larger perceptual field CNN models. Furthermore, we discuss herein the future issues and challenges of driver emotion recognition in promising smart city applications.

Global-Local-Feature-Fused Driver Speech Emotion Detection for Intelligent Cockpit in Automated Driving

Affective interaction between the intelligent cockpit and humans is becoming an emerging topic full of opportunities. Robust recognition of the driver's emotions is the first step for affective interaction, and the intelligent cockpit recognizes emotions through the driver's speech, which has a wide range of technical application potential. In this paper, we first proposed a multi-feature fusion parallel structure speech emotion recognition network, which complementarily fuses the global acoustic features and local spectral features of the entire speech. Second, we designed and conducted the speech data collection under the driver's emotion and established the driver's speech emotion (SpeechEmo) dataset in the dynamic driving environment including 40 participants. Finally, the proposed model was validated on the SpeechEmo and public datasets, and quantitative analysis was carried out. It was found that the proposed model achieved advanced recognition performance, and the ablation experiments verified the importance of different components of the model. The proposed model and dataset are beneficial to the realization of human-vehicle affective interaction in intelligent cockpits in the future toward a better human experience.

Happiness and high reliability develop affective trust in in-vehicle agents.

The advancement of Conditionally Automated Vehicles (CAVs) requires research into critical factors to achieve an optimal interaction between drivers and vehicles. The present study investigated the impact of driver emotions and in-vehicle agent (IVA) reliability on drivers' perceptions, trust, perceived workload, situation awareness (SA), and driving performance toward a Level 3 automated vehicle system. Two humanoid robots acted as the in-vehicle intelligent agents to guide and communicate with the drivers during the experiment. Forty-eight college students participated in the driving simulator study. The participants each experienced a 12-min writing task to induce their designated emotion (happy, angry, or neutral) prior to the driving task. Their affective states were measured before the induction, after the induction, and after the experiment by completing an emotion assessment questionnaire. During the driving scenarios, IVAs informed the participants about five upcoming driving events and three of them asked for the participants to take over control. Participants' SA and takeover driving performance were measured during driving; in addition, participants reported their subjective judgment ratings, trust, and perceived workload (NASA-TLX) toward the Level 3 automated vehicle system after each driving scenario. The results suggested that there was an interaction between emotions and agent reliability contributing to the part of affective trust and the jerk rate in takeover performance. Participants in the happy and high reliability conditions were shown to have a higher affective trust and a lower jerk rate than other emotions in the low reliability condition; however, no significant difference was found in the cognitive trust and other driving performance measures. We suggested that affective trust can be achieved only when both conditions met, including drivers' happy emotion and high reliability. Happy participants also perceived more physical demand than angry and neutral participants. Our results indicated that trust depends on driver emotional states interacting with reliability of the system, which suggested future research and design should consider the impact of driver emotions and system reliability on automated vehicles.

An automated hyperparameter tuned deep learning model enabled facial emotion recognition for autonomous vehicle drivers

The progress of autonomous driving cars is a difficult movement that causes problems regarding safety, ethics, social acceptance, and cybersecurity. Currently, the automotive industry is utilizing these technologies to assist drivers with advanced driver assistance systems. This system helps different functions to careful driving and predict drivers' ability of stable driving behavior and road safety. A great number of researches have shown that the driver's emotion is the major factor that handles the emotions, resulting in serious vehicle collisions. As a result, continuous monitoring of drivers' behavior could assist to evaluate their behavior to prevent accidents. The study proposes a new Squirrel Search Optimization with Deep Learning Enabled Facial Emotion Recognition (SSO-DLFER) technique for Autonomous Vehicle Drivers. The proposed SSO-DLFER technique focuses mainly on the identification of driver facial emotions in the AVs. The proposed SSO-DLFER technique follows two major processes namely face detection and emotion recognition. The RetinaNet model is employed at the initial phase of the face detection process. For emotion recognition, the SSO-DLFER technique applied the Neural Architectural Search (NASNet) Large feature extractor with a gated recurrent unit (GRU) model as a classifier. For improving the emotion recognition performance, the SSO-based hyperparameter tuning procedure is performed. The simulation analysis of the SSO-DLFER technique is tested under benchmark datasets and the experimental outcome was investigated under various aspects. The comparative analysis reported the enhanced performance of the SSO-DLFER algorithm on recent approaches.

On-road driver facial expression emotion recognition with parallel multi-verse optimizer (PMVO) and optical flow reconstruction for partial occlusion in internet of things (IoT)

In recent years, many large-scale information systems in the Internet of Things (IoT) can be converted into interdependent sensor networks, such as smart cities, smart medical systems, and industrial Internet systems. The successful application of Facial Expression Recognition in the IoT will make our algorithms faster, more convenient, lower overall costs, providing better business practices, and enhance sustainability. Facial Expression Recognition (FER)is essential to effectively communicate between human and machines. Video facial expression detection is a crucial component for gauging the driver's mood for driver assistance system. The emotions of the driver have a significant role in dictating the behavior of the driver, according to several studies, which can lead to disastrous car crashes. However, criteria affecting the identification of driver emotions with the right kind of monitoring include changes in stance, lighting, and occlusions. Based on the restoration of the blurred facial region, the Driver Facial Expression Emotion Recognition (DFEER) system was developed to address these issues. One of the first things to do when dealing with a sequence of blurred faces is to calculate the optical fluxes between the frames. After that, an optical flow reconstruction using a trained PMVO is performed to fix the occlusion-induced damage (Parallel Multi-Verse Optimizer). The major solutions are randomly split into groups of occluded and non-occluded optical flows using a parallel technique, and the groups discuss their findings after a set number of iterations. After optical flows have been rebuilt, they are used immediately in the classification phase of expression prediction. In this study, Very Deep Convolution Networks (VGGNet) proposed a method for recognising human emotions. Both the CK+ (Cohn-Kanade database) and the KMU-FED (Keimyung University Facial Expression of Drivers) databases are used to carry out the assessments of the classification model's efficacy. Accuracy, recall, precision, and f-measure the effectiveness of the suggested strategy is then evaluated using the findings.

Real Time Drowsiness Detection System Using CNN

Abstract: Driver fatigue and rash driving are the leading causes of road accidents, which result in the loss of valued life and decrease road traffic safety. Driver drowsiness solutions that are reliable and precise are essential to prevent accidents and increase road traffic safety. Various driver drowsiness detection systems have been developed using various technologies that are geared at the specific parameter of detecting the driver's tiredness. This research offers a unique multi-level distribution model for detecting driver drowsiness utilising Convolution Neural Networks (CNN) and. To detect the driver's behaviour and emotion, the driver's face pattern is handled with a 2D Convolution Neural Network (CNN). The suggested model is built with OpenCV, and the experimental findings show that it recognises the driver's emotion and tiredness more efficiently than existing technologies.

Effect of emotion on galvanic skin response and vehicle control data during simulated driving

PurposeThis study aims to investigate the effect of driver emotion on the physiological and vehicle control data and the possibility of predicting the driver emotion to enhance the driving experience. MethodThe driver emotions in the driving context were classified into eight categories depending on high and low levels of arousal and valence: happiness, surprise, fear, anger, depression, boredom, relief, and neutrality. Fourteen male and female volunteers with ages between 22 and 34 y participated in the test, and approximately 540 min of image, physiological, and vehicle data were collected. After inducing the participants’ target emotion through the viewing of a film and writing of passages, we asked the participants to drive on a highway through a driving simulator and self-evaluate their emotion. After the test, the participants were allowed to go home after emotion neutralization. ResultThe participants’ self-evaluated emotions correlated highly with the intended induced emotions. High arousal and negative valence emotions such as anger and low arousal and positive valence emotions such as relief exhibited a statistically significant elevation for the following indicators: galvanic skin response amplitude, longitudinal vehicle control data such as throttling and braking, and lateral vehicle control data. ConclusionsThe test results confirmed that a driver’s emotional state could be reflected in the differences between the biometric data or vehicle control data. In particular, the emotion associated with a high arousal and negative valence could be clearly distinguished from that associated with a low arousal and positive valence. Therefore, the driver's emotional state affects the traffic condition, and the detection of potentially risky emotions such as those associated with road rage and development of a suitable driving mode can help enhance the driving safety. ApplicationsDrivers’ emotions can be identified based on physiological data and vehicle control data and integrated into the system to formulate appropriate responses. For instance, anger anticipated from a driver can be alleviated by a preemptive measure, thereby enhancing the traffic safety.

A Spontaneous Driver Emotion Facial Expression (DEFE) Dataset for Intelligent Vehicles: Emotions Triggered by Video-Audio Clips in Driving Scenarios

In this article, a new dataset, the driver emotion facial expression (DEFE) dataset for drivers’ spontaneous emotions analysis is introduced. The dataset includes facial expression recordings from 60 participants during driving. After watching a selected video-audio clip to elicit a specific emotion, each participant completed the driving tasks in the same driving scenario and rated his/her emotional responses during the driving processes from the aspects of dimensional emotion method and discrete emotion method. The study also conducted classification experiments to recognize the scales of arousal, valence, dominance, as well as the emotion category and intensity to establish baseline results for the proposed dataset. Furthermore, this paper compared emotion recognition results difference through facial expressions between dynamic driving and static life scenarios. The results showed that dynamic driving and static life datasets were different in emotion recognition results. To further explore the reasons for the difference in emotion recognition results, the analysis from the AU (action unit) presence perspective was studied. The results showed significant differences in the AUs presence of facial expressions between dynamic driving and static life scenarios, indicating that drivers’ facial expressions may be affected by the driving task to influence the recognition of drivers’ emotions through facial expressions. Therefore, to accurately recognize the drivers’ emotions to establish a reliable emotion-aware human-machine interaction system, thereby improving driving safety and comfort, publishing a human emotion dataset specifically for the driver is necessary. The proposed dataset will be publicly available so that researchers worldwide can use it to develop and examine their driver emotion analysis methods. To the best of our knowledge, this is currently the only public driver facial expression dataset.

The Analysis of Driver's Behavioral Tendency Under Different Emotional States Based on a Bayesian Network

Affective factors have been associated with an array of driving behaviors. However, the mechanism by which emotion influences driving behaviors remains largely unknown. In the present study, a probabilistic approach for characterizing the emotional influence on driving behavioral decision-making was proposed. Firstly, a series of experiments were designed to obtain the human-vehicle-environment data when the drivers, whose emotional states were effectively induced, were driving a vehicle in the car-following scene. Next, a Bayesian Network (BN) was developed to model the causal relationships between driving behavioral tendency, driver's emotion and other indicators related to vehicle driving. Finally, the driver's different tendencies in driving behavior caused by emotional factors were analyzed through calculating the probability distribution of driving behavioral tendency under different emotional conditions in the BN. The research results are not only beneficial to improve the accuracy of driving behavior identification and prediction which is of great significance for vehicle active safety, but also have a potential promoting effect on human-computer harmonious interaction of vehicle system.

Estimating and Visualizing Drivers’ Emotions Using the Internet of Digital Reality

Recently, the development of self-driving technology has progressed rapidly. However, self-driving cars have not yet become widespread. Thus, with an aging population, accidents such as road rage and acceleration and brake accidents are likely to continue. Stress is one key reason for such dangerous driving. Thus, technologies must be developed to provide mental support to drivers as required. In this study, we considered estimating driver emotions as a first step along these lines. To this end, we developed a technology to estimate emotions by collecting data on biological signals such as brain waves, heart rate, body movement, and data on a driver's operating status while they are driving. In addition, we introduce a Positive and Negative Affect Schedule (PANAS) to express the psychological states experienced by drivers. We further present the results of an analysis of data on a driver's emotions from PANAS and data obtained from electroencephalogram (EEG) readings and other biological signals from a car. In addition, the relationship between this experimental environment and the Internet of Digital Reality (IoD) is described.

Research on Driver Emotion Classification Based on ECG Signals

Traffic includes factors that affect people, vehicles, and the environment. According to official data, traffic accidents caused directly or indirectly by human factors account for over 90% of the total number of traffic accidents, of which 70% are caused by vehicle drivers. Studying driver characteristics and considering driver factors in traffic safety has become an increasingly popular topic in the field of traffic safety. Driving emotions, as one of the driver characteristics, have a strong correlation with driving behavior. This article designs a simulated driving experiment and collects the electrocardiogram (ECG) indicators of drivers in driving with pleasure and anger during the experiment. The KNN clustering algorithm is used to classify and analyze the driver’s emotions.

Driver Emotion and Fatigue State Detection Based on Time Series Fusion

Studies have shown that driver fatigue or unpleasant emotions significantly increase driving risks. Detecting driver emotions and fatigue states and providing timely warnings can effectively minimize the incidence of traffic accidents. However, existing models rarely combine driver emotion and fatigue detection, and there is space to improve the accuracy of recognition. In this paper, we propose a non-invasive and efficient detection method for driver fatigue and emotional state, which is the first time to combine them in the detection of driver state. Firstly, the captured video image sequences are preprocessed, and Dlib (image open source processing library) is used to locate face regions and mark key points; secondly, facial features are extracted, and fatigue indicators, such as driver eye closure time (PERCLOS) and yawn frequency are calculated using the dual-threshold method and fused by mathematical methods; thirdly, an improved lightweight RM-Xception convolutional neural network is introduced to identify the driver’s emotional state; finally, the two indicators are fused based on time series to obtain a comprehensive score for evaluating the driver’s state. The results show that the fatigue detection algorithm proposed in this paper has high accuracy, and the accuracy of the emotion recognition network reaches an accuracy rate of 73.32% on the Fer2013 dataset. The composite score calculated based on time series fusion can comprehensively and accurately reflect the driver state in different environments and make a contribution to future research in the field of assisted safe driving.

Technical Design Space Analysis for Unobtrusive Driver Emotion Assessment Using Multi-Domain Context

Driver emotions play a vital role in driving safety and performance. Consequently, regulating driver emotions through empathic interfaces have been investigated thoroughly. However, the prerequisite - driver emotion sensing - is a challenging endeavor: Body-worn physiological sensors are intrusive, while facial and speech recognition only capture overt emotions. In a user study (N=27), we investigate how emotions can be unobtrusively predicted by analyzing a rich set of contextual features captured by a smartphone, including road and traffic conditions, visual scene analysis, audio, weather information, and car speed. We derive a technical design space to inform practitioners and researchers about the most indicative sensing modalities, the corresponding impact on users' privacy, and the computational cost associated with processing this data. Our analysis shows that contextual emotion recognition is significantly more robust than facial recognition, leading to an overall improvement of 7% using a leave-one-participant-out cross-validation.

Upaya Peningkatan Kualitas Kerja Moda Trasnportasi Online Gojek di Kota Padang

This study aims to determine the work quality of Gojek online transportation mode drivers, the inhibiting factors in improving the quality of work and the solutions offered by Gojek online transportation in minimizing obstacles. This research is a descriptive qualitative research, this research uses data collection techniques using interviews, observations and document studies. The results of this study are: 1) in carrying out work, the Gojek online transportation driver is quite good and the driver is assessed based on the points on the Gojek online transportation driver account. 2) In improving the work quality of Gojek online transportation drivers, there are several obstacles, such as the driver's education level, pressure, to the driver's emotions. 3) The solution provided by Gojek online transportation is to provide training every month with different materials and rewards for each driver who excels.

A Lightweight Two-Stream Model for Driver Emotion Recognition

In recent years, more and more attention has been paid to the traffic safety of drivers. Studies show that the motor vehicle speed will affect the incidence of traffic accidents and traffic accident casualty rate. In this paper, machine learning vision technology is used for real-time recognition of driver emotions, and a lightweight SIC-NET+RC-NET dual-stream recognition model is designed to cope with different scenarios of vehicle speed during driving. SIC-NET is a lightweight CNN network model, which greatly reduces the number of model parameters and computational power requirements, sacrifices part of accuracy, and greatly improves the real-time performance of model recognition. To improve the residual convolutional network model of inception blocks, RC-NET nested multiple micro-convolutional neural networks are used to identity mapping between different inception blocks to enhance the network training effect, expand network width and convolutional depth, and alleviate network degradation. This article stipulates that when the vehicle speed is greater than 60km/h, SIC-NET is used. When the vehicle speed is less than or equal to 60km/h, SIC-NET+RC-NRT is used. The FER2013 dataset was used for testing and validation in this paper. When SIC-NET is used, the model has better real-time performance and can provide a more timely warning when the vehicle speed is higher. When SIC-NET+RC-NET is used, the accuracy of the model in the FER2013 dataset is 69.65%, and the model can better cope with the scenario of a low vehicle speed. The experiment shows that the lightweight dual-flow model has good robustness and identification effect, which solves the problem of using the same model for different vehicle speeds and can effectively ensure the safety of the driver’s life and property.

Manta Ray Foraging Optimization with Transfer Learning Driven Facial Emotion Recognition

Recently, facial expression-based emotion recognition techniques obtained excellent outcomes in several real-time applications such as healthcare, surveillance, etc. Machine-learning (ML) and deep-learning (DL) approaches can be widely employed for facial image analysis and emotion recognition problems. Therefore, this study develops a Transfer Learning Driven Facial Emotion Recognition for Advanced Driver Assistance System (TLDFER-ADAS) technique. The TLDFER-ADAS technique helps proper driving and determines the different types of drivers’ emotions. The TLDFER-ADAS technique initially performs contrast enhancement procedures to enhance image quality. In the TLDFER-ADAS technique, the Xception model was applied to derive feature vectors. For driver emotion classification, manta ray foraging optimization (MRFO) with the quantum dot neural network (QDNN) model was exploited in this work. The experimental result analysis of the TLDFER-ADAS technique was performed on FER-2013 and CK+ datasets. The comparison study demonstrated the promising performance of the proposed model, with maximum accuracy of 99.31% and 99.29% on FER-2013 and CK+ datasets, respectively.

Personalized EV Driving Sound Design Based on the Driver's Total Emotion Recognition

<div class="section abstract"><div class="htmlview paragraph">An active sound design (ASD) technique enables the implementation of a specific sound in addition to the real engine/e-motor sound in a vehicle. However, it is difficult to satisfy the various needs of customers because it can provide only a few sounds designed by the manufacturer. This paper presents the method of providing the appropriate driving sound and soundscape in an electric vehicle according to the driver’s emotion and driving environment in real-time. For this purpose, it is studied how to construct a driving sound library from the various sound sources and how to recognize a driver's total emotion from the multi-modal data such as facial expression, heart rate, and electrodermal activity using the CNN and support vector machine algorithms. Then it is discussed how to generate the driving sound of electric vehicle according to the driver’s emotion. Using these methods, a personalized driving sound suitable to the driver's total emotion is provided by using the ASD system of electric vehicle in real-time. Additionally, it is studied how to recognize the driving environment from the outside image and match the soundscape (e.g. effect sound, background music and so on) to playback in an audio amplifier using the CNN and machine learning algorithms. Finally, it shows the demonstration of a prototype system and the people's response in a real driving situation. It is expected that this system can provide a new user experience through the personalized sound in electric vehicle by understanding the customer's feeling and driving situation.</div></div>

Multimodal Data Collection System for Driver Emotion Recognition Based on Self-Reporting in Real-World Driving.

As vehicles provide various services to drivers, research on driver emotion recognition has been expanding. However, current driver emotion datasets are limited by inconsistencies in collected data and inferred emotional state annotations by others. To overcome this limitation, we propose a data collection system that collects multimodal datasets during real-world driving. The proposed system includes a self-reportable HMI application into which a driver directly inputs their current emotion state. Data collection was completed without any accidents for over 122 h of real-world driving using the system, which also considers the minimization of behavioral and cognitive disturbances. To demonstrate the validity of our collected dataset, we also provide case studies for statistical analysis, driver face detection, and personalized driver emotion recognition. The proposed data collection system enables the construction of reliable large-scale datasets on real-world driving and facilitates research on driver emotion recognition. The proposed system is avaliable on GitHub.

CogEmoNet: A Cognitive-Feature-Augmented Driver Emotion Recognition Model for Smart Cockpit

Driver’s emotion recognition is vital to improving driving safety, comfort, and acceptance of intelligent vehicles. This article presents a cognitive-feature-augmented driver emotion detection method that is based on emotional cognitive process theory and deep networks. Different from the traditional methods, both the driver’s facial expression and cognitive process characteristics (age, gender, and driving age) were used as the inputs of the proposed model. Convolutional techniques were adopted to construct the model for driver’s emotion detection simultaneously considering the driver’s facial expression and cognitive process characteristics. A driver’s emotion data collection was carried out to validate the performance of the proposed method. The collected dataset consists of 40 drivers’ frontal facial videos, their cognitive process characteristics, and self-reported assessments of driver emotions. Another two deep networks were also used to compare recognition performance. The results prove that the proposed method can achieve well detection results for different databases on the discrete emotion model and dimensional emotion model, respectively.

Driver Drowsiness Detection System

Abstract: Recently, in addition to autonomous vehicle technology research and development, machine learning methods have been used to predict a driver's condition and emotions in order to provide information that will improve road safety. A driver's condition can be estimated not only by basic characteristics such as gender, age, and driving experience, but also by a driver's facial expressions, bio-signals, and driving behaviours. Recent developments in video processing using machine learning have enabled images obtained from cameras to be analysed with high accuracy. Therefore, based on the relationship between facial features and a driver’s drowsy state, variables that reflect facial features have been established. In this paper, we proposed a method for extracting detailed features of the eyes, the mouth, and positions of the head using OpenCV and Dlib library in order to estimate a driver’s level of drowsiness. Keywords: Drowsiness, OpenCV, Dlib, facial features, video processing