Driver State Monitoring Systems Research Articles

Using driver monitoring to estimate readiness in automation: a conceptual model based on simulator experimental data

This paper provides a theoretical overview of how the concept of driver readiness can be objectively measured, using controlled experimental data. First, a literature review regarding the concept of driver readiness is provided. Then, it highlights challenges for a standardized readiness estimation model. A conceptual readiness estimation model is presented, and a methodology is proposed for defining readiness thresholds for use by Driver State Monitoring (DSM) systems. The paper then explores how this model can be used to estimate readiness thresholds. A proof of concept for the model application is presented, using previously collected experimental involving SAE Level 2 automation. This paper contributes to the state of the art in DSM-development, by providing a methodology for estimating driver readiness, while considering variabilities across individual drivers. The model also allows readiness thresholds to be defined with data from driving simulator experiments, without relying on subjective assessment of readiness as its ground truth.

Differences in driver takeover performance and physiological responses in conditionally automated driving: Links to emotional instability

When the automated driving system reaches its technical limit, drivers will have to take over control from the vehicle within a limited time. The drivers’ emotional instability may affect whether they can safely control the vehicle during takeover transitions. Accordingly, this paper explores the association between emotional instability and physiological responses and driver takeover performance. 42 drivers engaged in negative emotion induced by movie clips were involved in a sequence of takeover events in a driving simulator. Personality questionnaires were used to assess the driver’s level of emotional instability. Their physiological data before the takeover request (TOR) and operational behavior data fragments for the takeover process were extracted. The results demonstrated that integration of subjective evaluations with electroencephalogram (EEG) and electrocardiogram (ECG) signals allows for more objective measurement of drivers’ level of emotional instability. The statistical analyses indicated that emotional instability has a significant effect on drivers’ heartbeat interval, heart rate, the frequency bands of θ(4–8 Hz), α(8–12 Hz), β(13–30 Hz), γ (30–50 Hz) and lateral takeover performance. Those findings of differences in driver physiological responses and takeover performance under emotional instability may provide additional support for the design of driver state monitoring and adaptive warning systems.

An Eye-Fixation Related Electroencephalography Technique for Predicting Situation Awareness: Implications for Driver State Monitoring Systems.

This study developed a fixation-related electroencephalography band power (FRBP) approach for situation awareness (SA) assessment in automated driving. Maintaining good SA in Level 3 automated vehicles is crucial to drivers' takeover performance when the automated system fails. A multimodal fusion approach that enables the analysis of the visual behavioral and cognitive processes of SA can facilitate real-time assessment of SA in future driver state monitoring systems. Thirty participants performed three simulated automated driving tasks. After each task, the Situation Awareness Global Assessment Technique (SAGAT) was deployed to capture their SA about key elements that could affect their takeover task performance. Participants eye movements and brain activities were recorded. Data on their brain activity after each eye fixation on the key elements were extracted and labeled according to the correctness of the SAGAT. Mixed-effects models were used to identify brain regions that were indicative of SA, and machine learning models for SA assessment were developed based on the identified brain regions. Participants' alpha and theta oscillation at frontal and temporal areas are indicative of SA. In addition, the FRBP technique can be used to predict drivers' SA with an accuracy of 88% using a neural network model. The FRBP technique, which incorporates eye movements and brain activities, can provide more comprehensive evaluation of SA. Findings highlight the potential of utilizing FRBP to monitor drivers' SA in real-time. The proposed framework can be expanded and applied to driver state monitoring systems to measure human SA in real-world driving.

The Effect of Vehicle Automation Styles on Drivers’ Emotional State

Self-driving vehicles promise many safety, mobility, and environmental benefits. However, users’ lack of trust and acceptance may threaten the success and potential of this technology. Monitoring the driver’s emotional state is one way to address this challenge. Empathetic automation can respond to the driver’s state and improve the experience and acceptance of self-driving vehicle drivers. In this study, 24 participants rode in a self-driving vehicle simulator and experienced three automation styles (aggressive, moderate, conservative) and four intersection types (with and without a stop sign, and with and without traffic.) We identified the observed drivers’ emotions from the video data and labeled the video frames using the dimensional and discrete emotion models to examine how automation behavior affects the driver’s emotional state. We used multilevel Bayesian linear regression and multilevel Dirichlet regression to model the continuous and discrete emotions, respectively. The automation driving style effect varied for each participant. The same conditions provoked positive responses for some participants, and negative for others. Furthermore, the results showed that intersection type, the position within the intersection, and their interaction affected the driver’s emotional state. This indicates that personalized driver state monitoring systems might enhance drivers’ experience in self-driving vehicles.

Using pupillometry and gaze-based metrics for understanding drivers’ mental workload during automated driving

This Horizon2020-funded driving simulator-based study on automated driving investigated the effect of different car-following scenarios, and takeover situations, on drivers’ mental workload, as measured by eye tracking-based metrics of pupil diameter and self-reported workload ratings. This study incorporated a mixed design format, with 16 drivers recruited for the SAE Level 2 (L2; SAE International, 2021) automation group, who were asked to monitor the driving and road environment during automation, and 16 drivers in the Level 3 (L3) automation group, who engaged in a non-driving related task (NDRT; Arrows task) during automation. Drivers in each group undertook two experimental drives, lasting about 18 min each. To manipulate perceived workload, difficulty of the driving task was controlled by incorporating a lead vehicle which maintained either a Short (0.5 s) or Long (1.5 s) Time Headway (THW) condition during automated car-following (ACF). Each ACF session was followed by a subsequent request to takeover, which happened either in the presence or absence of a lead vehicle. Results from standard deviation of pupil diameter values indicated that drivers’ mental workload levels fluctuated significantly more when monitoring the drive during L2 ACF, compared to manual car-following (MCF). Additionally, we found that drivers’ mental workload, as indicated by their mean pupil diameter, increased steeply around takeovers, and was further exacerbated by the presence of a lead vehicle during the takeovers, especially in the Short THW condition, for both groups. Pupil diameter was found to be sensitive to subtle variations in mental workload, and closely resembled the trend seen in self-reported workload ratings. Further research is warranted to assess the feasibility of using eye-tracking-based metrics along with other physiological sensors, especially in real-world settings, to understand whether they can be used as real-time indicators of drivers’ mental workload, in future driver state monitoring systems.

A methodology to explore the road safety impact of fitness to drive solutions for commercial drivers: The PANACEA project

In Europe, one in four road deaths occurred in an accident involving a goods vehicle in 2018 (ETSC 2020). Commercial drivers are at higher risk for suffering from physiological, psychological and prescribed medication and illicit drug use, including alcohol misuse. Fitness to drive or driver state monitoring systems integrate technologies able to detect altered driver states and provide them feedback. They constitute an emerging phenomenon, and their effects on changing people's behaviour to drive more safely, and in general, their impact on road safety should be better investigated.The scope of this paper is to present a methodology able to simulate different scenarios to understand how a driver state monitoring system can support improving road safety in the European Union. A conceptual framework is presented to support the definition of the impact assessment methodology and is applied to the PANACEA European research project. The project develops an integrated solution for driving ability assessment of commercial drivers, paired with a countermeasure and coaching solution. The PANACEA system uses algorithms and technologies for detecting, monitoring and assessing alcohol consumption, licit (barbituric) and illicit (methadone substitute) drugs, fatigue and cognitive load (Commercial Health Toolkits (CHTs)). It also provides strategic, tactical and operational countermeasures that will be tested and evaluated to assess their effectiveness and acceptance by the system's users.The methodology presented is able to assess both a single and multiple countermeasures among those developed within the project. Different scenarios have been considered by modifying the variables according to the screening prevalence, solution acceptance level, driving context and time. The methodology uses the results from the project pilot studies in terms of accuracy, sensitivity, and specificity of CHTs and countermeasures results in combination with evidence from the existing literature.

Assessing the physiological effect of non-driving-related task performance in conditionally automated driving systems: A systematic review and meta-analysis protocol.

Level 3 automated driving systems involve the continuous performance of the driving task by artificial intelligence within set environmental conditions, such as a straight highway. The driver's role in Level 3 is to resume responsibility of the driving task in response to any departure from these conditions. As automation increases, a driver's attention may divert towards non-driving-related tasks (NDRTs), making transitions of control between the system and user more challenging. Safety features such as physiological monitoring thus become important with increasing vehicle automation. However, to date there has been no attempt to synthesise the evidence for the effect of NDRT engagement on drivers' physiological responses in Level 3 automation. A comprehensive search of the electronic databases MEDLINE, EMBASE, Web of Science, PsycINFO, and IEEE Explore will be conducted. Empirical studies assessing the effect of NDRT engagement on at least one physiological parameter during Level 3 automation, in comparison with a control group or baseline condition will be included. Screening will take place in two stages, and the process will be outlined within a PRISMA flow diagram. Relevant physiological data will be extracted from studies and analysed using a series of meta-analyses by outcome. A risk of bias assessment will also be completed on the sample. This review will be the first to appraise the evidence for the physiological effect of NDRT engagement during Level 3 automation, and will have implications for future empirical research and the development of driver state monitoring systems.

Classifying Older Drivers’ Gaze Behaviour during Automated versus Non-Automated Driving: A Preliminary Step towards Detecting Mode Confusion

Mode confusion occurs when the driver of an automated vehicle (AV) is confused about the active operating mode of the AV and therefore, their responsibilities as the driver. Mode confusion is a serious safety concern, especially for cohorts who are less familiar with AVs and/or who are more likely to have poorer situational awareness during automated driving such as older adults. In this article, we propose a design framework for driver state monitoring systems that can potentially be used to detect older drivers’ mode confusion by inferring drivers’ perceived AV mode using gaze behaviour data. As a proof-of-concept for an AV with two modes, the efficacy of the proposed framework is tested by applying it on a gaze behaviour dataset collected from 29 older drivers (65+) during simulated non-automated and simulated fully automated drives. The proposed framework utilizes classification models trained on features extracted from the gaze behaviour data. Among 25 features, the mRMR (maximum relevance minimum redundancy) feature ranking framework ranked our proposed feature of weighted static gaze entropy as having the highest relevance with the driver’s perceived AV modes while having the least redundancy with the rest of the selected features. An ensemble stacking model achieved the highest classification performance with an average accuracy of 73% and an average AUC score of 80%. The results indicate that gaze behaviour features can distinguish between the driving scenarios of automated and non-automated as perceived by the drivers. While the dataset does not include confirmed instances of driver’s mode confusion and therefore, the framework testing provides preliminary results towards a proof of concept, this work provides a foundational model for future studies in which actual data from confirmed mode confusions are intentionally introduced or measured. In turn, this study can inform future designs of driver state monitoring systems aimed to detect and mitigate the safety risks of driver’s mode confusions in automated vehicles.

Physiological Measures of Risk Perception in Highly Automated Driving

Highly automated driving will likely result in drivers being out-of-the-loop during specific scenarios and engaging in a wide range of non-driving related tasks. Manifesting in lower levels of risk perception to emerging events, and thus affect drivers’ availability to take-over manual control in safety-critical scenarios. In this empirical research, we measured drivers’ (N = 20) risk perception with cardiac and skin conductance indicators through a series of high-fidelity, simulated highly automated driving scenarios. By manipulating the presence of surrounding traffic and changing driving conditions as long-term risk modulators, and including a driving hazard event as a short-term risk modulator, we hypothesised that an increase in risk perception would induce greater physiological arousal. Our results demonstrate that heart rate variability features are superior at capturing arousal variations from these long-term, low to moderate risk scenarios. In contrast, skin conductance responses are more sensitive to rapidly evolving situations associated with moderate to high risk. Based on this research, future driver state monitoring systems should adopt multiple physiological measures to capture changes in the long and short term, modulation of risk perception. This will enable enhanced perception of driver readiness and improved availability to safely deal with take-over events when requested by an automated vehicle.

Public acceptance of driver state monitoring for automated vehicles: Applying the UTAUT framework

Driver state monitoring (DSM) systems aim to measure driver/occupant state, considering factors such as fatigue, workload, attentiveness, and wellbeing. They are influential for some vehicles on the road today, but as we move towards higher levels of automation their use is expected to become even more important. Uncertainty around public perception of these systems is a potentially limiting factor, with potential concerns for privacy, data storage, perceived utility, and reliability amongst other things posing potential threats. A survey was conducted based on the Universal Theory of Acceptance and Use of Technology (UTAUT) framework, which collected 311 individual responses to explore the acceptability of this technology in specific relation to conditionally and partially automated vehicles. This paper identified that the factors of effort expectancy, performance expectancy, social influence, and attitudes towards using new technology are all positively related to overall behavioural intention, whereas anxiety is negatively related. Other demographic influences are discussed, and recommendations made for future research and marketing/implementation of DSM systems to ensure acceptance by future users.

Eye movements predict driver reaction time to takeover request in automated driving: A real-vehicle study

For automated driving at SAE level 3 or lower, driver performance in responding to takeover requests (TORs) is decisive in providing system safety. A driver state monitoring system that can predict a driver’s performance in a TOR event will facilitate a safer control transition from vehicle to driver. This experimental study investigated whether driver eye-movement measured before a TOR can predict driving performance in a subsequent TOR event. We recruited participants (N = 36) to obtain realistic results in a real-vehicle study. In the experiment, drivers rode in an automated vehicle on a test track for about 32 min, and a critical TOR event occurred at the end of the drive. Eye movements were measured by a camera-based driver monitoring system, and five measures were extracted from the last 2-min epoch prior to the TOR event. The correlations between each eye-movement measure and driver reaction time were examined, and a multiple regression model was built using a stepwise procedure. The results showed that longer reaction time could be significantly predicted by a smaller number of large saccades, a greater number of medium saccades, and lower saccadic velocity. The implications of these relationships are consistent with previous studies. The present real-vehicle study can provide insights to the automotive industry in the search for a safer and more flexible interface between the automated vehicle and the driver.

Efficient and compact face descriptor for driver drowsiness detection

Current advances in driver drowsiness detection consist of a variety of innovative technologies generally based on driver state monitoring systems. Extracting effective and relevant features to characterize drowsy symptoms in images and videos is still an open topic. In this work, we introduce a face monitoring system based on a compact face texture descriptor able to cover the most discriminant drowsy features. The compactness has been achieved by both a multi-scale pyramidal face representation that capture the main characteristics of local and global information, and the feature selection process applied on the raw extracted features. The proposed framework is rolled out in four phases: (i) face detection and alignment; (ii) Pyramid-Multi Level (PML) face representation; (iii) face description using a multi-level multi scale feature extraction; and (vi) feature subset selection and classification. Experiments conducted on the public dataset NTH Drowsy Driver Detection (NTHUDDD) show the effectiveness of the proposed face descriptor and the associated selection schemes. The results show that the proposed method compares favorably with several approaches including those based on deep Convolutional Neural Networks.

Effects of an Unexpected and Expected Event on Older Adults' Autonomic Arousal and Eye Fixations During Autonomous Driving.

Driving cessation for some older adults can exacerbate physical, cognitive, and mental health challenges due to loss of independence and social isolation. Fully autonomous vehicles may offer an alternative transport solution, increasing social contact and encouraging independence. However, there are gaps in understanding the impact of older adults’ passive role on safe human–vehicle interaction, and on their well-being. 37 older adults (mean age ± SD = 68.35 ± 8.49 years) participated in an experiment where they experienced fully autonomous journeys consisting of a distinct stop (an unexpected event versus an expected event). The autonomous behavior of the vehicle was achieved using the Wizard of Oz approach. Subjective ratings of trust and reliability, and driver state monitoring including visual attention strategies (fixation duration and count) and physiological arousal (skin conductance and heart rate), were captured during the journeys. Results revealed that subjective trust and reliability ratings were high after journeys for both types of events. During an unexpected stop, overt visual attention was allocated toward the event, whereas during an expected stop, visual attention was directed toward the human–machine interface (HMI) and distributed across the central and peripheral driving environment. Elevated skin conductance level reflecting increased arousal persisted only after the unexpected event. These results suggest that safety-critical events occurring during passive fully automated driving may narrow visual attention and elevate arousal mechanisms. To improve in-vehicle user experience for older adults, a driver state monitoring system could examine such psychophysiological indices to evaluate functional state and well-being. This information could then be used to make informed decisions on vehicle behavior and offer reassurance during elevated arousal during unexpected events.

Matters of State: Examining the effectiveness of lane departure warnings as a function of driver distraction

Lane departure warnings (LDW) have the potential to mitigate a significant number of lane departure crashes. Such safety benefits have yet been realized, in part due to drivers deactivating LDW systems. Perceived false alarms—where drivers receive a warning but feel the warning was unnecessary or incorrect—could lead to system disuse. In part, this could be a failure of LDW systems to account for the state of the driver. The current study investigated whether LDWs were more effective for drivers when they were distracted compared to when they were undistracted, using a high-fidelity driving simulator. During distracted lane departures, drivers with LDW responded faster and had less severe lane departures than drivers without LDW. During undistracted lane departures, there was no evidence of a benefit of LDW over no warning. These results suggest that lane departure warnings are most effective when drivers are distracted. This study suggests a need for driver state monitoring systems to enable adaptive automation.

Assessment of Drivers’ Comprehensive Driving Capability Under Man-Computer Cooperative Driving Conditions

The effective assessment of drivers' driving capability under the condition of an advanced driver assistance system is of great significance to the precise switching of driving rights between human and machine and the promotion of the development of man-computer collaboration. In this study, real-time collected warning data from bus driver state monitoring system (DSMS) and advanced driver assistance system (ADAS) were utilized to determine the drivers' comprehensive driving capability indicators. The information utility and interaction of the indicators were considered, and an integrated weight method based on standard deviations was proposed. This method was used to combine the entropy weight method and improved analytic network process (ANP), to evaluate the drivers' comprehensive driving capability under man-computer cooperative driving conditions in real time. The results show that the entropy weight method and improved ANP algorithm have good consistency and are significantly correlated and that the integrated weight method is effective and dependable. The top four indicators in the integrated weighting results were eye closure (0.241), yawn (0.210), rapid deceleration (0.186), and lane departure (0.159). Drivers' comprehensive driving capability scores were concentrated in the score range of 1 to 6, with the lowest scores in zones A and B for stages 2, 11 and 21. Therefore, it is necessary to further explore the relationship between driver behavior, vehicle status and road traffic environment within the score range of 1 to 6 so that the man-computer interaction can be optimized and the driver's comprehensive driving capability can be improved.

Video-Based Contactless Heart-Rate Detection and Counting via Joint Blind Source Separation with Adaptive Noise Canceller

Driver assistance systems are a major focus of the automotive industry. Although technological functions that help drivers are improving, the monitoring of driver state functions receives less attention. In this respect, the human heart rate (HR) is one of the most important bio-signals, and it can be detected remotely using consumer-grade cameras. Based on this, a video-based driver state monitoring system using HR signals is proposed in this paper. In a practical automotive environment, monitoring the HR is very challenging due to changes in illumination, vibrations, and human motion. In order to overcome these problems, source separation strategies were employed using joint blind source separation, and feature combination was adopted to maximize HR variation. Noise-assisted data analysis was then adopted using ensemble empirical mode decomposition to extract the pure HR. Finally, power spectral density analysis was conducted in the frequency domain, and a post-processing smoothing filter was applied. The performance of the proposed approach was tested based on commonly employed metrics using the MAHNOB-HCI public dataset and compared with recently proposed competing methods. The experimental results proved that our method is robust for a variety of driving conditions based on testing using a driving dataset and static indoor environments.

Standardized evaluation of camera-based driver state monitoring systems

Driver state sensing technologies, such as vehicular systems, start to be widely considered by automotive manufacturers. To reduce the cost and minimize the intrusiveness towards driving, the majority of these systems rely on the in-cabin camera &#x0028 s &#x0029 and other optical sensors. With their great capabilities in detecting and intervening of driver distraction and inattention, these technologies may become key components in future vehicle safety and control systems. However, to the best of our knowledge, currently, there is no common standard available to objectively compare the performance of these technologies. Thus, it is imperative to develop one standardized process for evaluation purposes. In this paper, we propose one systematic and standardized evaluation process after successfully addressing three difficulties: 1&#x0029 defining and selecting the important influential individual and environmental factors, 2&#x0029 countering the effects of individual differences and randomness in driver behaviors, and 3&#x0029 building a reliable in-vehicle driver head motion tracking tool to collect ground-truth motion data. We have collected data on a large scale on a commercial driver state-sensing platform. For each subject, 30 to 40 minutes of head motion data was collected and included variables, such as lighting conditions, head &#x002F face features, and camera locations. The collected data was analyzed based on a proposed performance measure. The results show that the developed process can efficiently evaluate an individual camera-based driver state sensing product, which builds a common base for comparing the performance of different systems.

Driver Facial Landmark Detection in Real Driving Situations

This paper proposes a novel facial landmark detection (FLD) algorithm for use in real driving situations. The proposed algorithm is based on an ensemble of local weighted random forest regressor (WRFR) with random sampling consensus (RANSAC) and explicit global shape models, and considers the dynamic and irregular characteristics of driving. In this paper, to estimate the offset distance from a landmark and a reference point, we first detect the nose region as a reference point. Next, we propose a local WRFR to maintain the generality with a small number of regression trees. With the WRFR, we adopt RANSAC instead of the averaging or median of offsets to handle the problem of sensitivity to outlier offsets, and we estimate the accurate 2D offset vector. To identify the erroneous positions of local landmarks and rearrange the overall landmark layout, we adopt the global face models based on the spatial relation between landmarks. Using the unified framework of the proposed FLD, our proposed algorithm is robust to large head poses and partial occlusions caused by a driver’s hair or sunglasses. For a benchmark data set considering real driving situations, we construct a data set called a face alignment data set used in driving (FADID) using a near-infrared camera for FLD under real driving situations. We apply the proposed algorithm to various driving sequences in FADID, and the results show that its FLD performance is better than that of other state-of-the-art methods, while the computational speed is high for real-time applications such as driver-state monitoring systems.

Gaze position modulates the effectiveness of forward collision warnings for drowsy drivers

Advanced driver assistance systems (ADAS) have the potential to prevent crashes and reduce their severity. Forward collision warnings (FCW) are quickly becoming standard across vehicle lineups and may prevent frontal crashes by alerting drivers. Previous research has demonstrated the effectiveness of FCW for distracted drivers, but their effectiveness for other types of impairment remains unknown. Like distraction, drowsiness can impair driver response time and lead to crashes. The goal of the present study was to evaluate the effectiveness of FCW for moderately and severely drowsy drivers using a high-fidelity driving simulator. Drowsy drivers were divided into three warning conditions during a revealed stop vehicle forward collision event: An auditory alert, a haptic seat vibration, and a no warning baseline. Results indicate that FCW were effective at speeding drowsy driver response, but only when the drowsy drivers were looking away from the forward roadway at the onset of the event. These results have important implications for ADAS technology and driver state monitoring systems.

Towards testing auditory–vocal interfaces and detecting distraction while driving: A comparison of eye-movement measures in the assessment of cognitive workload

Recently, there has been a growing need among researchers to understand the problem of cognitive workload induced by auditory–verbal–vocal tasks while driving in realistic conditions. This is due to the fact that we need (a) valid methods to evaluate in-vehicle electronic devices using voice control systems and (b) experimental data to build more reliable driver state monitoring systems. In this study, we examined the effects of cognitive workload induced by the delayed digit recall task (n-back) while driving. We used a high-fidelity driving simulator and a highway scenario with moderate traffic to study eye movements in realistic driving conditions. This study included 46 participants, and the results indicate that a change in pupil size is most sensitive for measuring changes in cognitive demand in auditory–verbal–vocal tasks. Less sensitive measures included changes in fixation location and blink rate. Fixation durations and the driving performance metrics did not provide sensitive measures of graded levels of cognitive demand.