Driver State Research Articles

Effect of emotion on galvanic skin response and vehicle control data during simulated driving

PurposeThis study aims to investigate the effect of driver emotion on the physiological and vehicle control data and the possibility of predicting the driver emotion to enhance the driving experience. MethodThe driver emotions in the driving context were classified into eight categories depending on high and low levels of arousal and valence: happiness, surprise, fear, anger, depression, boredom, relief, and neutrality. Fourteen male and female volunteers with ages between 22 and 34 y participated in the test, and approximately 540 min of image, physiological, and vehicle data were collected. After inducing the participants’ target emotion through the viewing of a film and writing of passages, we asked the participants to drive on a highway through a driving simulator and self-evaluate their emotion. After the test, the participants were allowed to go home after emotion neutralization. ResultThe participants’ self-evaluated emotions correlated highly with the intended induced emotions. High arousal and negative valence emotions such as anger and low arousal and positive valence emotions such as relief exhibited a statistically significant elevation for the following indicators: galvanic skin response amplitude, longitudinal vehicle control data such as throttling and braking, and lateral vehicle control data. ConclusionsThe test results confirmed that a driver’s emotional state could be reflected in the differences between the biometric data or vehicle control data. In particular, the emotion associated with a high arousal and negative valence could be clearly distinguished from that associated with a low arousal and positive valence. Therefore, the driver's emotional state affects the traffic condition, and the detection of potentially risky emotions such as those associated with road rage and development of a suitable driving mode can help enhance the driving safety. ApplicationsDrivers’ emotions can be identified based on physiological data and vehicle control data and integrated into the system to formulate appropriate responses. For instance, anger anticipated from a driver can be alleviated by a preemptive measure, thereby enhancing the traffic safety.

A methodology to explore the road safety impact of fitness to drive solutions for commercial drivers: The PANACEA project

In Europe, one in four road deaths occurred in an accident involving a goods vehicle in 2018 (ETSC 2020). Commercial drivers are at higher risk for suffering from physiological, psychological and prescribed medication and illicit drug use, including alcohol misuse. Fitness to drive or driver state monitoring systems integrate technologies able to detect altered driver states and provide them feedback. They constitute an emerging phenomenon, and their effects on changing people's behaviour to drive more safely, and in general, their impact on road safety should be better investigated.The scope of this paper is to present a methodology able to simulate different scenarios to understand how a driver state monitoring system can support improving road safety in the European Union. A conceptual framework is presented to support the definition of the impact assessment methodology and is applied to the PANACEA European research project. The project develops an integrated solution for driving ability assessment of commercial drivers, paired with a countermeasure and coaching solution. The PANACEA system uses algorithms and technologies for detecting, monitoring and assessing alcohol consumption, licit (barbituric) and illicit (methadone substitute) drugs, fatigue and cognitive load (Commercial Health Toolkits (CHTs)). It also provides strategic, tactical and operational countermeasures that will be tested and evaluated to assess their effectiveness and acceptance by the system's users.The methodology presented is able to assess both a single and multiple countermeasures among those developed within the project. Different scenarios have been considered by modifying the variables according to the screening prevalence, solution acceptance level, driving context and time. The methodology uses the results from the project pilot studies in terms of accuracy, sensitivity, and specificity of CHTs and countermeasures results in combination with evidence from the existing literature.

A Contextual Multimodal System for Increasing Situation Awareness and Takeover Quality in Conditionally Automated Driving

Advanced driver assistances are becoming increasingly common in commercial cars, not only to assist but also to free drivers from manual driving whenever possible. Soon, drivers should be allowed to engage in non-driving-related tasks. The fact that responsibility for driving is shifting from humans to machines must be considered in the development of these assistances in order to guarantee safety and trust. In this article, we introduce AdVitam (for Advanced Driver-Vehicle Interaction to Make future driving safer), an autonomous system aiming at maintaining driver’s situation awareness and optimizing takeover quality during conditionally automated driving. The information conveyed to drivers is dynamically adapted to achieve these goals, depending on the driving environment and the driver’s physiological state. This system consists of three connected modules. The first module (Driver State) predicts the driver’s state with machine learning and physiological signals as inputs. The second module (Supervision) uses different interfaces (a haptic seat, a personal device, and ambient lights) to maintain the drivers’ situation awareness during the autonomous driving phases. The third module (Intervention) is a machine learning model that chooses the most appropriate combination among haptic, auditory, and visual modalities to request the driver to take over control and thus optimize takeover quality. To evaluate the system and each module independently, a preliminary user study with 35 drivers was conducted in a fixed-base driving simulator. All participants drove in two different environments (rural and urban). In addition, the activation of the Supervision and Intervention modules were manipulated as two between-subject factors. Results show that conveying information on the driving environment status through multimodal interfaces increases drivers’ situation awareness (i.e., better identification of potential problems in the environment) and trust in the automated vehicle. However, the system does not show positive outcomes on takeover quality. Besides, the Driver State module provided consistent predictions with the experimental manipulation. The system proposed in this paper could lead to better acceptance and safety when conditionally automated vehicles will be released by increasing drivers’ trust during phases of automated driving.

Application of smart watches for monitoring the health state of professional drivers

The article discusses the possibility of using smartwatches in transport companies. The popularity of the introduction of wearables among professional drivers will gradually eliminate the dangers associated with the use of smartphones while driving buses and trucks. Attention was also drawn to the possibilities of integrating the smartwatch with smartphones and using additional functions allowing to monitor the condition and health of a professional driver. Smartwatch will thus contribute to improving the quality of work performed among professional drivers by minimizing the risk of stress, falling asleep or dehydration, which in the past led to accidents and road collisions. The aim of the article is to present the potential of smartwatches in transport companies in Poland. The gradual introduction of smartwatches in companies and the popularization of smartwatches among professional drivers raises the question: Will the introduction of smartwatches in transport companies that enable monitoring of fitness and health improve the safety, efficiency and quality of work of professional drivers? The survey was conducted in 2012-2022 among non-financial companies, including companies from the TSL industry (small and medium-sized enterprises). The survey showed that Polish entrepreneurs are willing to equip their employees with mobile devices. For professional drivers, the smartwatch will become an economical and user-friendly solution that improves safety, efficiency and productivity of their field work.

Using fNIRS to Verify Trust in Highly Automated Driving

Trust in automation is crucial for the safe and appropriate adoption of automated driving technology. Current research methods to measure trust mainly rely on subjective scales, with several intrinsic limitations. This empirical experiment proposes a novel method to measure trust objectively, using functional near-infrared spectroscopy (fNIRS). Through manipulating participants’ expectations regarding driving automation credibility, we have induced and successfully measured opposing levels of trust in automation. Most notably, our results evidence two separate yet interrelated cortical mechanisms for trust and distrust. Trust is demonstrably linked to decreased monitoring and working memory, whereas distrust is event-related and strongly tied to affective (or emotional) mechanisms. This paper evidence that trust in automation and situation awareness are strongly interrelated during driving automation usage. Our findings are crucial for developing future driver state monitoring technology that mitigates the impact of inappropriate reliance, or over trust, in automated driving systems.

Real-World Application of Early Detection of Sudden Changes in Driver’s Physical Condition Using Machine Learning and Interpretation Method

In order to decrease fatal traffic accidents, early detection of sudden changes in driver’s physical condition is desired. In this paper, we reported an example of modeling process of "driver’s normal behavior model," which is a part of driver state sensing technology we are developing. By combining machine learning and domain knowledge, we have achieved the model for normal steering operation that had both high prediction accuracy and high interpretability/explainability under real-world conditions.

Assessing the physiological effect of non-driving-related task performance in conditionally automated driving systems: A systematic review and meta-analysis protocol.

Level 3 automated driving systems involve the continuous performance of the driving task by artificial intelligence within set environmental conditions, such as a straight highway. The driver's role in Level 3 is to resume responsibility of the driving task in response to any departure from these conditions. As automation increases, a driver's attention may divert towards non-driving-related tasks (NDRTs), making transitions of control between the system and user more challenging. Safety features such as physiological monitoring thus become important with increasing vehicle automation. However, to date there has been no attempt to synthesise the evidence for the effect of NDRT engagement on drivers' physiological responses in Level 3 automation. A comprehensive search of the electronic databases MEDLINE, EMBASE, Web of Science, PsycINFO, and IEEE Explore will be conducted. Empirical studies assessing the effect of NDRT engagement on at least one physiological parameter during Level 3 automation, in comparison with a control group or baseline condition will be included. Screening will take place in two stages, and the process will be outlined within a PRISMA flow diagram. Relevant physiological data will be extracted from studies and analysed using a series of meta-analyses by outcome. A risk of bias assessment will also be completed on the sample. This review will be the first to appraise the evidence for the physiological effect of NDRT engagement during Level 3 automation, and will have implications for future empirical research and the development of driver state monitoring systems.

Driver State Monitoring Technology for Conditionally Automated Vehicles: Review and Future Prospects

Driver State Monitoring Technology for Conditionally Automated Vehicles: Review and Future Prospects

How to define a safety tolerance zone for speed? Insights from the i-DREAMS project

Several factors of driver state negatively impact road safety, such as distraction (in-vehicle or external), fatigue and drowsiness, health issues and extreme emotions. The aim of the current study is to define a Safety Tolerance Zone (STZ) for speed, and integrate crash prediction and risk assessment. A naturalistic driving experiment was conducted and data from a representative sample (N=20 drivers) was utilized. Explanatory variables of risk and the most reliable indicators were assessed. A feature importance algorithm extracted from Extreme Gradient Boosting (XGBoost) was used to evaluate the significance of variables on forecasting STZ. Additionally, a Neural Network model was implemented for real-time data prediction. Results indicated a strong relationship between the STZ level for speed and the independent variables of headway, distance travelled and medium harsh braking events.

Bringing a Vehicle to a Controlled Stop: Effectiveness of a Dual-Control Scheme for Identifying Driver Drowsiness and Executing Safety Control under Hands-off Partial Driving Automation

This study proposes a strategy to bring the vehicle to a controlled stop if the driver fails to supervise the partial driving automation. In scenarios where the driving context changes, the system provides information on the object/event detection and response, and it observes the driver's response (i.e., pressing a button). If the driver fails to respond to the given information within 15 seconds, the system implements a first-stage control that reduces the speed to 50 km/h. If the driver does not perform the accelerator action within 30 seconds after first-stage control, the system determines that “the driver failed to supervise the partial driving automation system”. To demonstrate the effectiveness of the proposed system for identifying driver drowsiness and implementing deceleration control, a total of 26 drivers participated in an experiment. In 12 among 23 instances in which first-stage control was implemented, the system could successfully determine that “the driver failed to supervise the partial driving automation.” In 8 of the 12 instances, the vehicle was brought to a complete stop. We conclude that the system could be used effectively for safety control and for identifying the driver state.

Detection of Driver Cognitive Distraction Using Machine Learning Methods

Driver distraction is one of the primary causes of crashes. As a result, there is a great need to continuously observe driver state and provide appropriate interventions to distracted drivers. Cognitive distraction refers to the “look but not see” situations when the drivers’ eyes are focused on the forward roadway, but their mind is not. Typically, cognitive distractions can result from fatigue, conversation with a co-passenger, listening to the radio, or other similarly loading secondary tasks that do not necessarily take a driver’s eyes off the roadway. This makes it one of the hardest distractions to detect as there are no visible clues of driver distraction. In this study, we have identified features from different sources including eye-tracking, physiological, and vehicle kinematics data that are relevant towards the classification of distracted and non-distracted drivers via the analysis of data collected from a driving simulator study involving 40 drivers across multiple driving scenarios. The key classification algorithms implemented include Random Forest, Decision Trees and Support Vector Machines. A reduced feature set including pupil area, pupil vertical and horizontal motion was found to be predictive of driver distraction while maintaining an average accuracy of 90% across various road types. Additionally, the impact of road types on driver behaviour was also identified. The findings of the study has practical application towards the design of driver distraction monitoring systems.

Driver Emotion and Fatigue State Detection Based on Time Series Fusion

Studies have shown that driver fatigue or unpleasant emotions significantly increase driving risks. Detecting driver emotions and fatigue states and providing timely warnings can effectively minimize the incidence of traffic accidents. However, existing models rarely combine driver emotion and fatigue detection, and there is space to improve the accuracy of recognition. In this paper, we propose a non-invasive and efficient detection method for driver fatigue and emotional state, which is the first time to combine them in the detection of driver state. Firstly, the captured video image sequences are preprocessed, and Dlib (image open source processing library) is used to locate face regions and mark key points; secondly, facial features are extracted, and fatigue indicators, such as driver eye closure time (PERCLOS) and yawn frequency are calculated using the dual-threshold method and fused by mathematical methods; thirdly, an improved lightweight RM-Xception convolutional neural network is introduced to identify the driver’s emotional state; finally, the two indicators are fused based on time series to obtain a comprehensive score for evaluating the driver’s state. The results show that the fatigue detection algorithm proposed in this paper has high accuracy, and the accuracy of the emotion recognition network reaches an accuracy rate of 73.32% on the Fer2013 dataset. The composite score calculated based on time series fusion can comprehensively and accurately reflect the driver state in different environments and make a contribution to future research in the field of assisted safe driving.

Driver Assisted Lane Keeping with Conflict Management Using Robust Sliding Mode Controller

Lane-keeping assistance design for road vehicles is a multi-objective design problem that needs to simultaneously maintain lane tracking, ensure driver comfort, provide vehicle stability, and minimize conflict between the driver and the autonomous controller. In this work, a cooperative control strategy is proposed for lane-keeping keeping by integrating driving monitoring, variable level of assistance allocation, and human-in-the-loop control. In the first stage, a time-varying physical driver loading pattern is identified based on a relationship between lateral acceleration, road curvature, and the measured maximum driver torque. Together with the monitored driver state that indicates driver mental loading, an adaptive driver activity function is then formulated that replicates the levels of assistance required for the driver in the next stage. To smoothly transition authority between various modes (from manual to autonomous and vice versa) based on the generated levels of assistance, a novel higher-order sliding mode controller is proposed and closed-loop stability is established. Further, a novel sharing parameter (which is proportional to the torques coming from the driver and from the autonomous controller) is used to minimize the conflict. Experimental results on the SHERPA high-fidelity vehicle simulator show the real-time implementation feasibility. Extensive experimental results provided on the Satory test track show improvement in cooperative driving quality by 9.4%, reduction in steering workload by 86.13%, and reduced conflict by 65.38% when compared with the existing design (no sharing parameter). These results on the cooperative performance highlight the significance of the proposed controller for various road transportation challenges.

DriverSVT: Smartphone-Measured Vehicle Telemetry Data for Driver State Identification

One of the key functions of driver monitoring systems is the evaluation of the driver’s state, which is a key factor in improving driving safety. Currently, such systems heavily rely on the technology of deep learning, that in turn requires corresponding high-quality datasets to achieve the required level of accuracy. In this paper, we introduce a dataset that includes information about the driver’s state synchronized with the vehicle telemetry data. The dataset contains more than 17.56 million entries obtained from 633 drivers with the following data: the driver drowsiness and distraction states, smartphone-measured vehicle speed and acceleration, data from magnetometer and gyroscope sensors, g-force, lighting level, and smartphone battery level. The proposed dataset can be used for analyzing driver behavior and detecting aggressive driving styles, which can help to reduce accidents and increase safety on the roads. In addition, we applied the K-means clustering algorithm based on the 11 least-correlated features to label the data. The elbow method showed that the optimal number of clusters could be either two or three clusters. We chose to proceed with the three clusters to label the data into three main scenarios: parking and starting driving, driving in the city, and driving on highways. The result of the clustering was then analyzed to see what the most frequent critical actions inside the cabin in each scenario were. According to our analysis, an unfastened seat belt was the most frequent critical case in driving in the city scenario, while drowsiness was more frequent when driving on the highway.

Prediction of the Ability to Develop the Relaxation Skill in Drivers of Locomotive Crews

The effectiveness of biofeedback training for the development of relaxation skills using the NeuroDog hardware-software complex is investigated. The relaxation skill is one of the conditions for maintaining the functional state of drivers and assistant drivers of locomotive crews at an optimal level during a working trip, and it is also one of the factors for a successful inter-trip rest. A multiple regression analysis has been carried out, which allows to predict the indicators characterizing the success of developing the relaxation skill, depending on the severity of psychophysiological and personal indicators of drivers and assistant drivers of locomotive crews. Two regression models have been constructed to explain the dependence of the indicators of developing the skill of achieving relaxation on the psychophysiological and personal characteristics of drivers and assistant drivers of locomotive crews. Models are characterized by sufficient quality. Quality indicators: of the first regression model – coefficient of determination R2 = 0.71 (with F(40, 65) = 4.06, p = 0.00000027), Durbin-Watson statistics d = 1.81, of the second regression model – R2 = 0.75 (with F(49, 56) = 3.51, p = 0.0000043), d = 2.09. To predict the indicators of developing the skill of achieving relaxation, regression equations were obtained depending on the severity of psychophysiological and personal indicators of locomotive crew drivers.

Exploring the influence of driver affective state and auditory display urgency on takeover performance in semi-automated vehicles: Experiment and modelling

As semi-automated vehicles become more available to the general public, it is important to investigate human factors, including both the driver side and the interface side. Despite much research on semi-automated vehicles, little research has conducted considering both driver states and takeover request display design. The present study investigated the effects of drivers’ affective states and auditory display urgency on takeover response time and performance quality. Thirty-six participants experienced takeover scenarios in a semi-automated vehicle using a driving simulator, while playing an online game. For takeover quality, angry drivers drove faster, took longer to change lanes and had lower steering wheel angles than neutral drivers, which made riskier driving. However, there was no difference in eye glance behaviors. Higher frequency and more repetitions of the auditory displays led to faster takeover reaction times, but there was no time difference between angry and neutral drivers. Drivers’ response time to takeover displays from both affect groups was modelled using the QN-MHP framework, which resulted in a R2 of 0.505 with the empirical data collected. In sum, results suggest that drivers’ anger state influenced takeover quality, while display urgency influenced takeover response time. This study is expected to make a significant contribution to research on the influence of emotion, specifically, anger on takeover performance in semi-automated vehicles as well as to the takeover display design.

Comparing the Performances of Ensemble classifiers to Detect Eye Stat

Brain signals required for the brain-computer interface are obtained through the electroencephalography (EEG) method. EEG data is used in the analysis of many problems such as epileptic seizure detection, bipolar mood disorder, attention deficit, and detection of the sleep state of the vehicle driver. It is very important to determine whether the eye is open or closed, which is a substantial organ for the determination of the cognitive state of the person. The aim of this paper is to present a stable and successful model for detecting the eye states that are opened or closed. In this context, the performances of several ensemble classifiers were examined on the Emotiv EEG Neuroheadset dataset, which has 14 features excluding the target variable, 14980 records that have 8225 eye states opened and 6755 eye states closed. In the experiments, firstly the min-max normalization process was applied to the dataset, and then the classification performances of these classifiers were evaluated via a 5-fold cross-validation technique. The performance of each model was measured using accuracy, sensitivity, and specificity metrics. The obtained results show that the Random Forest algorithm is an acceptable level with 92.61% value of accuracy, 94.31% value of sensitivity and 91.36% value of specificity for detecting the eye state.

Two-Stream Spatial-Temporal Graph Convolutional Networks for Driver Drowsiness Detection.

Convolutional neural networks (CNNs) have achieved remarkable performance in driver drowsiness detection based on the extraction of deep features of drivers' faces. However, the performance of driver drowsiness detection methods decreases sharply when complications, such as illumination changes in the cab, occlusions and shadows on the driver's face, and variations in the driver's head pose, occur. In addition, current driver drowsiness detection methods are not capable of distinguishing between driver states, such as talking versus yawning or blinking versus closing eyes. Therefore, technical challenges remain in driver drowsiness detection. In this article, we propose a novel and robust two-stream spatial-temporal graph convolutional network (2s-STGCN) for driver drowsiness detection to solve the above-mentioned challenges. To take advantage of the spatial and temporal features of the input data, we use a facial landmark detection method to extract the driver's facial landmarks from real-time videos and then obtain the driver drowsiness detection result by 2s-STGCN. Unlike existing methods, our proposed method uses videos rather than consecutive video frames as processing units. This is the first effort to exploit these processing units in the field of driver drowsiness detection. Moreover, the two-stream framework not only models both the spatial and temporal features but also models both the first-order and second-order information simultaneously, thereby notably improving driver drowsiness detection. Extensive experiments have been performed on the yawn detection dataset (YawDD) and the National TsingHua University drowsy driver detection (NTHU-DDD) dataset. The experimental results validate the feasibility of the proposed method. This method achieves an average accuracy of 93.4% on the YawDD dataset and an average accuracy of 92.7% on the evaluation set of the NTHU-DDD dataset.

Is My Driver Observation Model Overconfident? Input-Guided Calibration Networks for Reliable and Interpretable Confidence Estimates

Driver observation models are rarely deployed under perfect conditions. In practice, illumination, camera placement and type differ from the ones present during training and unforeseen behaviours may occur at any time. While observing the human behind the steering wheel leads to more intuitive human-vehicle-interaction and safer driving, it requires recognition algorithms which do not only predict the correct driver state, but also determine their prediction quality through realistic and interpretable confidence measures. Reliable uncertainty estimates are crucial for building trust and are a serious obstacle for deploying activity recognition networks in real driving systems. In this work, we for the first time examine how well the confidence values of modern driver observation models indeed match the probability of the correct outcome and show that raw neural network-based approaches tend to significantly overestimate their prediction quality. To correct this misalignment between the confidence values and the actual uncertainty, we consider two strategies. First, we enhance two activity recognition models often used for driver observation with temperature scaling – an off-the-shelf method for confidence calibration in image classification. Then, we introduce Calibrated Action Recognition with Input Guidance (CARING) – a novel approach leveraging an additional neural network to learn scaling the confidences depending on the video representation. Extensive experiments on the Drive&Act dataset demonstrate that both strategies drastically improve the quality of model confidences, while our CARING model outperforms both, the original architectures and their temperature scaling enhancement, leading to best uncertainty estimates.

A Method of Car Driver's Phone Call Recognition Based on Human Joint Points

The distracted driving state of the car driver has caused serious harm to traffic safety and the personal safety of the driver. The most frequent and common distraction is the distracted driving behavior of the driver on the phone. In order to accurately identify the car driver's phone call is distracting driving behavior. A driver's phone call behavior recognition method based on the joint point information of the driver's body was proposed in this manuscript. After using the OpenPose network structure to extract the joint points of the human body, the driver’s angle of the driver's upper and lower arms and the normalized distance between the joint points are calculated and compared to relatively accurate identification of car drivers' phone calls and distracted driving behaviors The experimental results show that the method proposed in this paper has an accuracy rate of 97.23% for the distracted driving state of the car driver's phone call.