Driver Monitoring System Research Articles

Real Time Indian Traffic Sign Detection using Image Processing and CNN

Driver Assistance and Monitoring System plays a very important role in traffic management especially in Indian roads. It eventually reduces the accidents and major injuries. DAMS (Driver Assistance and Monitoring System) give the safety and driving comfort. The main motto of our work is to design the effective methodology for the assistance and driver monitoring system which alerts the driver when it detects the road signs so that driver can take the appropriate action. The proposed methodology detects a road signs which is present in the dataset under cluttered background and different lighting conditions. The proposed work detecting the road sign based on colour and shape. The edge of the road sign is detected using canny edge operator. The images are enhanced and removed the noise using median filters. The images are classified as stop, no entry, speed limit using Convolutional Neural Network (CNN) classifier.

Train Driver Fatigue Detection Using Eye Feature Vector and Support Vector Machine

Fatigue driving is one of the main causes of traffic accidents. The eye features are the important cues of fatigue detection. In order to improve the accuracy and robustness of detection based on a single eye feature, we propose a fatigue detection algorithm based on the eye feature (EFV) vector. Firstly, the coordinates of the eye region were localized with facial landmarks detector and the landmarks geometric relation (LGR) was calculated as a feature value. Secondly, a deep transfer learning network was designed to classify the driver eye state on a small dataset. The probability value of the eyes being open state was calculated. Then an eye feature vector was constructed to overcome the limitations of a single fixed threshold and a support vector machine (SVM) model was trained for eye state classification recognition. Finally, the performance of the proposed detection model was evaluated by the percentage of eyelid closure over time (PERCLOS) criterion. The results show that the accuracy of this model can reach 91.67% on the test database, which is higher than the single-feature-based method. This work lays a foundation for the online fatigue detection of train drivers and the deployment of the train driving monitoring system.

Real-time Driver Drowsiness Detection with Android

Abstract: Major causes for the road accidents are Driver Fatigue and Distraction. Recently many driver monitoring systems have been proposed for monitoring driver activities to avoid accidents. Almost all the systems which exists till date are in the specialized embedded or hardware form, which are present in expensive and luxurious cars. This research paper presents a cost efficient approach to detect driver fatigue and monitor dizziness/drowsiness in real-time. An intelligent android app to monitor driver fatigue and distraction during travel using Adaptive Template Matching and Adaptive Boosting is achieved and implemented. Developing of technologies for preventing drowsiness is quite challenging in the area of accident avoiding systems. Preventing drowsiness during driving requires a method to accurately detect lowering of driver attentiveness and a method for alerting and to forewarn the driver. In this detection approach, we developed an android app that utilizes image processing technology and analyse all the images captured through the user’s phone camera. Drop of alertness is detected based on the degree to which the driver's eyes are open or closed. This application provides a non-contact method for judging various levels of driver awareness and enables an early detection of a reduction in attentiveness while driving. Keywords: Drowsiness Detection, Google vision API, Face detection, Eye Detection, Android application

DriverMVT: In-Cabin Dataset for Driver Monitoring including Video and Vehicle Telemetry Information

Developing a driver monitoring system that can assess the driver’s state is a prerequisite and a key to improving the road safety. With the success of deep learning, such systems can achieve a high accuracy if corresponding high-quality datasets are available. In this paper, we introduce DriverMVT (Driver Monitoring dataset with Videos and Telemetry). The dataset contains information about the driver head pose, heart rate, and driver behaviour inside the cabin like drowsiness and unfastened belt. This dataset can be used to train and evaluate deep learning models to estimate the driver’s health state, mental state, concentration level, and his/her activity in the cabin. Developing such systems that can alert the driver in case of drowsiness or distraction can reduce the number of accidents and increase the safety on the road. The dataset contains 1506 videos for 9 different drivers (7 males and 2 females) with total number of frames equal 5119k and total time over 36 h. In addition, evaluated the dataset with multi-task temporal shift convolutional attention network (MTTS-CAN) algorithm. The algorithm mean average error on our dataset is 16.375 heartbeats per minute.

Driver’s Face Pose Estimation Using Fine-Grained Wi-Fi Signals for Next-Generation Internet of Vehicles

Driver’s behavior and gesture recognition are most significant in the emerging next-generation vehicular technology. Driver’s face may provide important cues about his/her attention and fatigue behavior. Therefore, driver’s face pose is one of the key indicators to be considered for automatic driver monitoring system in next-generation Internet of Vehicles (IoV) technology. Driver behavior monitoring is most significant in order to reduce road accidents. This paper aims to address the problem of driver’s attentiveness monitoring using face pose estimation in a nonintrusive manner. The proposed system is based on wireless sensing, leveraging channel state information (CSI) of WiFi signals. In this paper, we present a novel classification algorithm that is based on the combination of support vector machine (SVM) and K nearest neighbor (KNN) to enhance the classification accuracy. Experimental results demonstrate that the proposed device-free wireless implementation can localize a driver’s face very accurately with an average recognition rate of 91.8 % .

Driver Drowsiness Detection System Using Convolutional Neural Networks

In recent years, the rise of car accident fatalities has grown significantly around the world. Hence, road security has become a global concern and a challenging problem that needs to be solved. The deaths caused by road accidents are still increasing and currently viewed as a significant general medical issue. The most recent developments have made in advancing knowledge and scientific capacities of vehicles, enabling them to see and examine street situations to counteract mishaps and secure travelers. Therefore, the analysis of driver’s behaviors on the road has become one of the leading research subjects in recent years, particularly drowsiness, as it grants the most elevated factor of mishaps and is the primary source of death on roads. This project presents a way to analyze and anticipate driver drowsiness by applying a Convolutionl Neural Network over a sequence frame driver’s face. We used a dataset to shape and approve our model and implemented repetitive neural network architecture multi-layer model-based 3D Convolutional Networks to detect driver drowsiness. After a training session, we obtained a promising accuracy that approaches a 92% acceptance rate, which made it possible to develop a real-time driver monitoring system to reduce road accidents.

Contactless Vital Sign Monitoring System for In-Vehicle Driver Monitoring Using a Near-Infrared Time-of-Flight Camera

We demonstrate a Contactless Vital Sign Monitoring (CVSM) system and road-test the system for in-cabin driver monitoring using a near-infrared indirect Time-of-Flight (ToF) camera. The CVSM measures both heart rate (HR) and respiration rate (RR) by leveraging the simultaneously measured grayscale and depth information from a ToF camera. For a camera-based driver monitoring system (DMS), key challenges from varying background illumination and motion-induced artifacts need to be addressed. In this study, active illumination and depth-based motion compensation are used to mitigate these two challenges. For HR measurements, active illumination allows the system to work under various lighting conditions, while our depth-based motion compensation has the advantage of directly measuring the motion of the driver without making prior assumptions about the motion artifacts. In addition, we can extract RR directly from the chest wall motion, circumventing the challenge of acquiring RR from the near-infrared photoplethysmography (PPG) signal of low signal quality. We investigate the system’s performance in various scenarios, including monitoring both drivers and passengers while driving on highways and local roads. Our results show that our CVSM system is ambient light agnostic, and the success rates of HR measurements on the highway are 82% and 71.9% for the passenger and driver, respectively. At the same time, we show that the system can measure RR on users driving on a highway with a mean deviation of −1.4 breaths per minute (BPM). With reliable HR and RR measurement in the vehicle, the CVSM system could one day be a key enabler to sudden sickness or drowsiness detection in DMS.

Physiological indicators of driver workload during car-following scenarios and takeovers in highly automated driving

This driving simulator study, conducted as a part of Horizon2020-funded L3Pilot project, investigated how different car-following situations affected driver workload, within the context of vehicle automation. Electrocardiogram (ECG) and electrodermal activity (EDA)-based physiological metrics were used as objective indicators of workload, along with self-reported workload ratings. A total of 32 drivers were divided into two equal groups, based on whether they engaged in a non-driving related task (NDRT) during automation (SAE Level 3) or monitored the drive (SAE Level 2). Drivers in both groups were exposed to two counterbalanced experimental drives, lasting ∼ 18 min each, of Short (0.5 s) and Long (1.5 s) Time Headway conditions during automated car-following (ACF), which was followed by a takeover that happened with or without a lead vehicle. Results showed that driver workload due to the NDRT was significantly higher than both monitoring the drive during ACF and manual car-following (MCF). Furthermore, the results indicated that a lead vehicle maintain a shorter THW can significantly increase driver workload during takeover scenarios, potentially affecting driver safety. This warrants further research into understanding safe time headway thresholds to be maintained by automated vehicles, without placing additional cognitive or attentional demands on the driver. Our results indicated that ECG and EDA signals are sensitive to variations in workload, which warrants further investigation on the value of combining these two signals to assess driver workload in real-time, to help future driver monitoring systems respond appropriately to the limitations of the driver, and predict their performance in the driving task, if and when they have to resume manual control of the vehicle after a period of automated driving.

Challenges of Large-Scale Multi-Camera Datasets for Driver Monitoring Systems.

Tremendous advances in advanced driver assistance systems (ADAS) have been possible thanks to the emergence of deep neural networks (DNN) and Big Data (BD) technologies. Huge volumes of data can be managed and consumed as training material to create DNN models which feed functions such as lane keeping systems (LKS), automated emergency braking (AEB), lane change assistance (LCA), etc. In the ADAS/AD domain, these advances are only possible thanks to the creation and publication of large and complex datasets, which can be used by the scientific community to benchmark and leverage research and development activities. In particular, multi-modal datasets have the potential to feed DNN that fuse information from different sensors or input modalities, producing optimised models that exploit modality redundancy, correlation, complementariness and association. Creating such datasets pose a scientific and engineering challenge. The BD dimensions to cover are volume (large datasets), variety (wide range of scenarios and context), veracity (data labels are verified), visualization (data can be interpreted) and value (data is useful). In this paper, we explore the requirements and technical approach to build a multi-sensor, multi-modal dataset for video-based applications in the ADAS/AD domain. The Driver Monitoring Dataset (DMD) was created and partially released to foster research and development on driver monitoring systems (DMS), as it is a particular sub-case which receives less attention than exterior perception. Details on the preparation, construction, post-processing, labelling and publication of the dataset are presented in this paper, along with the announcement of a subsequent release of DMD material publicly available for the community.

Threats Detection during Human-Computer Interaction in Driver Monitoring Systems.

This paper presents an approach and a case study for threat detection during human–computer interaction, using the example of driver–vehicle interaction. We analyzed a driver monitoring system and identified two types of users: the driver and the operator. The proposed approach detects possible threats for the driver. We present a method for threat detection during human–system interactions that generalizes potential threats, as well as approaches for their detection. The originality of the method is that we frame the problem of threat detection in a holistic way: we build on the driver–ITS system analysis and generalize existing methods for driver state analysis into a threat detection method covering the identified threats. The developed reference model of the operator–computer interaction interface shows how the driver monitoring process is organized, and what information can be processed automatically, and what information related to the driver behavior has to be processed manually. In addition, the interface reference model includes mechanisms for operator behavior monitoring. We present experiments that included 14 drivers, as a case study. The experiments illustrated how the operator monitors and processes the information from the driver monitoring system. Based on the case study, we clarified that when the driver monitoring system detected the threats in the cabin and notified drivers about them, the number of threats was significantly decreased.

Road traffic conditions in Kenya: Exploring the policies and traffic cultures from unstructured user-generated data using NLP

Road traffic accidents (RTA) are a prevalent cause of fatality with African countries having the highest fatality index (25–34 per quota). The World Health Organization estimates Kenya's fatality rate due to RTA at 28 per quota. From literature, the country's fatality and injuries have increased by 26% and 46.5%, respectively, since the year 2015. The country is faced with incomplete RTA data capturing, hindering effective planning and policy adjustments to curb the menace. In this paper, we scrapped user-generated data (Twitter) and national transport and safety authority's (NTSA) reports to shed light on traffic safety, practices, and cultures in the country. To this end, we gathered 1,000,000 tweets and 8000 speeding entries between 2015 and 2021 and performed natural language processing (NLP) and quantitative study of the data. We applied NLP and n-gram search of keywords to categorize data into 8 topics: traffic, public service vehicle (PSVs), policing, accident, infrastructure, recklessness, robbery, and corruption. From the data, policing, which touches on all police and law-enforcement-related activity was found to be highly correlated with PSVs, recklessness, accidents, traffic congestion, robbery, infrastructure, and corruption with indices of r(76) = 0.92, 0. 91, 0.87, 0.82, 0.81, 0.76, and 0.70, respectively with p < 0.001. The topic modeling confirmed the identified topics to be the latent discussion issues affecting the public. From the study, PSVs, policing and traffic flow were isolated as key issues that ought to be addressed immediately. The research recommended the integration of driver monitoring systems to strengthen policing. The research, which utilized unstructured data, points to the utility of data mining which would greatly benefit traffic research, particularly African-based studies, that suffer from data inadequacy.

A Review of Recent Developments in Driver Drowsiness Detection Systems.

Continuous advancements in computing technology and artificial intelligence in the past decade have led to improvements in driver monitoring systems. Numerous experimental studies have collected real driver drowsiness data and applied various artificial intelligence algorithms and feature combinations with the goal of significantly enhancing the performance of these systems in real-time. This paper presents an up-to-date review of the driver drowsiness detection systems implemented over the last decade. The paper illustrates and reviews recent systems using different measures to track and detect drowsiness. Each system falls under one of four possible categories, based on the information used. Each system presented in this paper is associated with a detailed description of the features, classification algorithms, and used datasets. In addition, an evaluation of these systems is presented, in terms of the final classification accuracy, sensitivity, and precision. Furthermore, the paper highlights the recent challenges in the area of driver drowsiness detection, discusses the practicality and reliability of each of the four system types, and presents some of the future trends in the field.

Let Complexity Bring Clarity: A Multidimensional Assessment of Cognitive Load Using Physiological Measures.

The effects of cognitive load on driver behavior and traffic safety are unclear and in need of further investigation. Reliable measures of cognitive load for use in research and, subsequently, in the development and implementation of driver monitoring systems are therefore sought. Physiological measures are of interest since they can provide continuous recordings of driver state. Currently, however, a few issues related to their use in this context are not usually taken into consideration, despite being well-known. First, cognitive load is a multidimensional construct consisting of many mental responses (cognitive load components) to added task demand. Yet, researchers treat it as unidimensional. Second, cognitive load does not occur in isolation; rather, it is part of a complex response to task demands in a specific operational setting. Third, physiological measures typically correlate with more than one mental state, limiting the inferences that can be made from them individually. We suggest that acknowledging these issues and studying multiple mental responses using multiple physiological measures and independent variables will lead to greatly improved measurability of cognitive load. To demonstrate the potential of this approach, we used data from a driving simulator study in which a number of physiological measures (heart rate, heart rate variability, breathing rate, skin conductance, pupil diameter, eye blink rate, eye blink duration, EEG alpha power, and EEG theta power) were analyzed. Participants performed a cognitively loading n-back task at two levels of difficulty while driving through three different traffic scenarios, each repeated four times. Cognitive load components and other coinciding mental responses were assessed by considering response patterns of multiple physiological measures in relation to multiple independent variables. With this approach, the construct validity of cognitive load is improved, which is important for interpreting results accurately. Also, the use of multiple measures and independent variables makes the measurements (when analyzed jointly) more diagnostic-that is, better able to distinguish between different cognitive load components. This in turn improves the overall external validity. With more detailed, diagnostic, and valid measures of cognitive load, the effects of cognitive load on traffic safety can be better understood, and hence possibly mitigated.

In cars (are we really safest of all?): interior sensing and emotional opacity

ABSTRACT This paper analyses expert and regulatory perspectives on car driver-monitoring systems that measure bodies to infer and react to emotions, fatigue, and attentiveness. Developers of driver-monitoring systems promise increased safety on the road, alongside comfort for cabin occupants through personalisation and automation. The impetus is three-fold, namely: (1) European road safety policy seeks to vastly reduce road deaths using computational surveillance; (2) there is a growing interest around in cabin safety solutions that sense emotion and affective states of drivers and passengers; and (3) autonomous driving trends are changing the nature of interactions between vehicle and driver. Safety led applications are of special interest because they are backed by policy and standards initiatives including the European Union’s Vision Zero policy and the industry led New Car Assessment Programme (NCAP). Informed by 13 interviews with experts working in and around in-cabin sensing technologies, this paper first identifies and explores features of emergent in-cabin profiling through emotional artificial intelligence (AI) and biometric measures. It then examines how in-car sensing should be regulated by analysing data protection laws and the proposed EU AI Act. A deep ambivalence emerged from our participants around the emergence of emotional AI in cars, and how best to regulate these technologies.

Detection of yawning in driver behavior based a convolutional neural network

Among the factors that usually cause road accidents in the world is driver fatigue, which accumulates during the trip or is present even before it begins. One of the most common signs of fatigue or tiredness of a vehicle driver is yawning. The detection of signs of yawning in human behavior is potentially able to further characterize its state of fatigue. Computer image processing methods are actively used to detect the openness of the mouth and yawning for a person. However, this approach has many disadvantages, which include different environmental conditions and a variety of situational yawning options for different people. The paper presents a scheme of a detector for determining signs of yawning, which is focused on processing images of the driver’s face using data analysis methods, computer image processing, and a convolutional neural network model. The essence of the proposed method is to detect yawning in the driver’s behavior in the cabin of a vehicle based on the analysis of a sequence of images obtained from a video camera. It is shown that the driver’s yawning state is accompanied by a wide and prolonged openness of the mouth. Prolonged openness of the mouth signals the appearance of signs of yawning. A conceptual model for detecting the openness of the mouth for a vehicle driver is presented and a scheme for processing and labeling the YawDD and Kaggle Drowsiness Dataset datasets is developed. The developed convolutional neural network model showed an accuracy of 0.992 and recall of 0.871 on a test 10 % data set. The proposed scheme for detecting the yawning state has been validated on a test video subset extracted from the YawDD: Yawning Detection Dataset. This detection scheme successfully detected 124 yawns among all video files from the test dataset. The proportion of correctly classified objects is 98.2 % accuracy, precision is equal to 96.1 %, recall is 98.4 %, and F score is 97.3 % while detecting signs of yawning in driver behavior. Detecting signs of yawning in the driver’s behavior allows one to clarify information about the driver and thereby to increase the effectiveness of existing driver monitoring systems in the vehicle cabin, aimed at preventing and reducing the risk of road accidents. The proposed approach can be combined with other technologies for monitoring driver behavior when building an intelligent driver support system.

Efficient in-cabin monitoring solution using TI TDA2PxSOCs

In-Cabin Monitoring Systems (ICMS) are functional safety systems designed to monitor driver and/or passengers inside an automobile. DMS (Driver Monitoring System) and OMS (Occupant Monitoring System) are two variants of ICMS. DMS focusses solely on the driver to monitor events like fatigue, inattention and vital health signs. OMS monitor all the occupants, including unattended children. Besides safety, ICMS can also augment security and comfort through driver identification and personalization. In-Cabin monitoring brings a unique set of challenges from an imaging perspective in terms of higher analytics needs, smaller form factor, low light vision and color accuracy. This paper discusses these challenges and provides an efficient implementation on Texas Instrument's TDA2Px automotive processor. The paper also provides details on novel implementation of RGB+IR sensor format processing that commonly used in these systems enabling premium ICMS system using TDA2Px system-on-chip (SoC).

Trends and Future Prospects of the Drowsiness Detection and Estimation Technology.

Drowsiness is among the important factors that cause traffic accidents; therefore, a monitoring system is necessary to detect the state of a driver’s drowsiness. Driver monitoring systems usually detect three types of information: biometric information, vehicle behavior, and driver’s graphic information. This review summarizes the research and development trends of drowsiness detection systems based on various methods. Drowsiness detection methods based on the three types of information are discussed. A prospect for arousal level detection and estimation technology for autonomous driving is also presented. In the case of autonomous driving levels 4 and 5, where the driver is not the primary driving agent, the technology will not be used to detect and estimate wakefulness for accident prevention; rather, it can be used to ensure that the driver has enough sleep to arrive comfortably at the destination.

Predicting Driver Stress Levels with a Sensor-Equipped Steering Wheel and a Quality-Aware Heart Rate Measurement Algorithm.

Unobtrusive monitoring of driver mental states has been regarded as an important element in improving the safety of existing transportation systems. While many solutions exist relying on camera-based systems for e.g., drowsiness detection, these can be sensitive to varying lighting conditions and to driver facial accessories, such as eye/sunglasses. In this work, we evaluate the use of physiological signals derived from sensors embedded directly into the steering wheel. In particular, we are interested in monitoring driver stress levels. To achieve this goal, we first propose a modulation spectral signal representation to reliably extract electrocardiogram (ECG) signals from the steering wheel sensors, thus allowing for heart rate and heart rate variability features to be computed. When input to a simple logistic regression classifier, we show that up to 72% accuracy can be achieved when discriminating between stressful and non-stressful driving conditions. In particular, the proposed modulation spectral signal representation allows for direct quality assessment of the obtained heart rate information, thus can provide additional intelligence to autonomous driver monitoring systems.

Evaluating the impacts of situational awareness and mental stress on takeover performance under conditional automation

Several safety concerns emerge for the transition of control from the automated driving system to a human driver after the vehicle issues a takeover warning under conditional vehicle automation (SAE Level 3). In this context, recent advances in in-vehicle driver monitoring systems enable tracking drivers’ physiological indicators (e.g., eye-tracking and heart rate (HR) measures) to assess their real-time situational awareness (SA) and mental stress. This study seeks to analyze differences in driver’s SA and mental stress over time (i.e., successive experiment runs) using these physiological indicators to assess their impacts on takeover performance. We use eye-tracking measures (i.e., on-road glance rate and road attention ratio) as indicators of driver’s SA during automated driving. Further, we use the pre-warning normalized HR (NHR) and HR variability (HRV) as well as the change in NHR and HRV after the takeover warning as indicators of mental stress immediately before and the change in mental stress after the takeover warning, respectively. To analyze the effects of driver state (in terms of SA and mental stress) on the overall takeover performance, this study uses a comprehensive metric, Takeover Performance Index (TOPI), proposed in our previous work (Agrawal & Peeta, 2021). The TOPI combines multiple driving performance indicators while partly accounting for their interdependencies. Results from statistical analyses of data from 134 participants using driving simulator experiments illustrate significant differences in driver state over successive experiment runs, except for the change in mental stress after the takeover warning. Some significant correlations were found between the physiological indicators of SA and mental stress used in this study. Takeover performance model results illustrate a significant negative effect of change in NHR after the takeover warning on the TOPI. However, none of the other physiological indicators show significant impacts on takeover performance. The study findings provide valuable insights to auto manufacturers for designing integrated in-vehicle driver monitoring and warning systems that enhance road safety and user experience.

Анализ современных исследований в области детектирования утомления водителя в кабине транспортного средства

The article analyzes the methods of detecting driver fatigue which are described in modern literature. There are a great variety of methods for assessing the functional state of a person. A functional state is an integral set of characteristics of those functions and qualities of a person that directly or indirectly determine the performance of any activity. The physical and mental state of a person, the success of his work, training, creativity depends on the functional state of the organism. The assessment of dynamic driver behavior has become an increasingly popular area of research in recent years. Dynamic assessment of driver behavior includes continuous monitoring that allows you to determine functional states, in contrast to modern driver monitoring systems, which assess conditions such as drowsiness and impaired attention for a short (1-10 s) time interval. Such systems allow us to talk about physiological, but not neurophysiological monitoring, which allows monitoring the functional state of fatigue. Therefore, it makes sense to monitor the driver’s state of fatigue of, as well as to warn them in a timely manner to avoid collisions with other vehicles. In the article, a study was carried out and an analysis of the ways to obtain the appropriate characteristics from a person, with the help of which it is possible to determine his functional state of fatigue. As a result of the analysis of the sources, the most common methods for determining the functional state of the driver were selected. Further, the sources found were classified according to the most common methods for obtaining significant characteristics of the functional state of the driver. As a result, a comparative analysis was made, demonstrating the capabilities of modern systems of this class.