Driver's Face Research Articles

A driver drowsiness and distraction warning system based on raspberry Pi 3 Kit

In this article, a system to detect driver drowsiness and distraction based on image sensing technique is created. With a camera used to observe the face of driver, the image processing system embedded in the Raspberry Pi 3 Kit will generate a warning sound when the driver shows drowsiness based on the eye-closed state or a yawn. To detect the closed eye state, we use the ratio of the distance between the eyelids and the ratio of the distance between the upper lip and the lower lip when yawning. A trained data set to extract 68 facial features and “frontal face detectors” in Dlib are utilized to determine the eyes and mouth positions needed to carry out identification. Experimental data from the tests of the system on Vietnamese volunteers in our University laboratory show that the system can detect at realtime the common driver states of “Normal”, “Close eyes”, “Yawn” or “Distraction”

A driver drowsiness and distraction warning system based on raspberry Pi 3 Kit

In this article, a system to detect driver drowsiness and distraction based on image sensing technique is created. With a camera used to observe the face of driver, the image processing system embedded in the Raspberry Pi 3 Kit will generate a warning sound when the driver shows drowsiness based on the eye-closed state or a yawn. To detect the closed eye state, we use the ratio of the distance between the eyelids and the ratio of the distance between the upper lip and the lower lip when yawning. A trained data set to extract 68 facial features and “frontal face detectors” in Dlib are utilized to determine the eyes and mouth positions needed to carry out identification. Experimental data from the tests of the system on Vietnamese volunteers in our University laboratory show that the system can detect at realtime the common driver states of “Normal”, “Close eyes”, “Yawn” or “Distraction”

Toward Driver Face Recognition in the Intelligent Traffic Monitoring Systems

This paper models the driver face recognition problem under the intelligent traffic monitoring systems as severe illumination variation face recognition with single sample problem. Firstly, in the point of view of numerical value sign, the current illumination invariant unit is derived from the subtraction of two pixels in the face local region, which may be positive or negative, we propose a generalized illumination robust (GIR) model based on positive and negative illumination invariant units to tackle severe illumination variations. Then, the GIR model can be used to generate several GIR images based on the local edge-region or the local block-region, which results in the edge-region based GIR (EGIR) image or the block-region based GIR (BGIR) image. For single GIR image based classification, the GIR image utilizes the saturation function and the nearest neighbor classifier, which can develop EGIR-face and BGIR-face. For multi GIR images based classification, the GIR images employ the extended sparse representation classification (ESRC) as the classifier that can form the EGIR image based classification (GIRC) and the BGIR image based classification (BGIRC). Further, the GIR model is integrated with the pre-trained deep learning (PDL) model to construct the GIR-PDL model. Finally, the performances of the proposed methods are verified on the Extended Yale B, CMU PIE, AR, self-built Driver and VGGFace2 face databases. The experimental results indicate that the proposed methods are efficient to tackle severe illumination variations.

Car Driver Drowsiness Recognition Android-Based System

The growth of the number of vehicles in Indonesia especially in terrestrial mode is also offset by the increasing number of accidents. One of the main causes of traffic accidents is due to drivers who driving in drowsiness. This condition leads to loss of concentration and even awareness of the driver. The drowsiness recognition system built into android smartphone platforms which owned by almost all walks of life, can help the community and government in providing direct warning especially to motorists while driving in a drowsy state. The method used is the Haar-Cascade Classifier to detect the driver's face and utilize the detectable face size ratio to create ROI in the mouth area. Then the using of the FindContour algorithm to find contours in dark areas when the mouth is open because it evaporates. The size of the contour will be compared to the Mouth ROI size for the drowsy driver's decision. System creation using Android Studio with C ++ and Java programming languages, as well as OpenCV libraries for Haar-Cascade and FindContour. Based on the test to 10 respondents, the accuracy of the system recognizes sleepiness on the driver reached 88.7% during the day.

Driver Face Verification with Depth Maps.

Face verification is the task of checking if two provided images contain the face of the same person or not. In this work, we propose a fully-convolutional Siamese architecture to tackle this task, achieving state-of-the-art results on three publicly-released datasets, namely Pandora, High-Resolution Range-based Face Database (HRRFaceD), and CurtinFaces. The proposed method takes depth maps as the input, since depth cameras have been proven to be more reliable in different illumination conditions. Thus, the system is able to work even in the case of the total or partial absence of external light sources, which is a key feature for automotive applications. From the algorithmic point of view, we propose a fully-convolutional architecture with a limited number of parameters, capable of dealing with the small amount of depth data available for training and able to run in real time even on a CPU and embedded boards. The experimental results show acceptable accuracy to allow exploitation in real-world applications with in-board cameras. Finally, exploiting the presence of faces occluded by various head garments and extreme head poses available in the Pandora dataset, we successfully test the proposed system also during strong visual occlusions. The excellent results obtained confirm the efficacy of the proposed method.

General logarithm difference model for severe illumination variation face recognition

Logarithm difference is the subtraction of the center pixel and its neighbor in the face local region. The commonly-used assumption of the illumination invariant measure is that illumination intensities of neighbor pixels are approximately equal in the face local region. However, logarithm difference based illumination invariant measure performs unsatisfactorily under severe illumination variations, since severe varying illumination cause differences of illumination intensities are large in the face local region. In this paper, the general logarithm difference model (GLDM) is proposed to tackle severe illumination variations. In spired by the fact that a logarithm difference is the subtraction of two neighbor pixels, which may be a positive or negative numerical value, we divide a face local region into a positive region and a negative region, and the GLDM is developed by integrating positive and negative logarithm differences. Then, the multiscale logarithm difference edge-maps (MSLDE) [7] is employed as the test-bed, and the proposed GLDM is introduced into MSLDE to form General MSLDE (GMSLDE). Further, the proposed GMSLDE method is integrated with the advanced deep learning model VGG to obtain GMSLDE-VGG. Finally, the performance of GMSLDE and GMSLDE-VGG are verified not only on the Extended Yale B and CMU PIE face databases with severe illumination variations, but also on the LFW and our self-built Driver face databases with moderate illumination variations. The experimental results indicate that the proposed GLDM can efficiently improve the performance of logarithm difference edge-map against illumination variations, especially for severe illumination variations.

Autonomous Vehicle Using Various Machine Learning Algorithms

Today and possibly for a long time to come, the full driving task is too complex an activity to be fully formalized as a sensing-acting robotics system that can be explicitly solved through model based and learning-based approaches in order to achieve full unconstrained vehicles autonomy. Vehicle control, mapping, scene perception, trajectory optimization, and higher-level planning decisions are open challenges for autonomous vehicle development. This is especially true for real-world operation where the margin of allowable error is extremely small and the number of edge-cases is extremely large. Human beings will remain an integral part of monitoring the AI system as it performs anywhere from just over 0% to just under 100% of the driving until we solved these problems. We are continually developing new methods for analysis of the massive-scale dataset collected from various vehicle owners. Many recorded data has various messages, and high-de?nition video streams of the driver face, the driver cabin, the forward roadway, and the instrument cluster. The study is on- going and growing. Till date, we have 110 participants, 11,945 days of participation, 89,405,807 miles, and 10.9 billion video frames. Here we presents the design of the study, the data collection hardware, the processing of the data, and the computer vision algorithms currently being used to extract actionable knowledge from the data.

Shared control of highly automated vehicles using steer-by-wire systems

A shared control of highly automated Steer-by-Wire system is proposed for cooperative driving between the driver and vehicle in the face of driver's abnormal driving. A fault detection scheme is designed to detect the abnormal driving behaviour and transfer the control of the car to the automatic system designed based on a fault tolerant model predictive control (MPC) controller driving the vehicle along an optimal safe path. The proposed concept and control algorithm are tested in a number of scenarios representing intersection, lane change and different types of driver's abnormal behaviour. The simulation results show the feasibility and effectiveness of the proposed method.

IL-GAN: Illumination-invariant representation learning for single sample face recognition

Single sample per person face recognition influenced by varying illumination is a tricky issue. Conventional techniques for illumination-invariant face recognition either realize illumination normalization on the whole face, or learn the illumination-invariant representation from the face image. This paper holds the opinion that deep learning method, which is more similar to the behavior of primate brain, can leverage the advantages of both the conventional techniques. Motivated by the success of generative adversarial network in image representation, this paper proposes IL-GAN model based on the basic structures of variational auto-encoder and generative adversarial network, generating the Controlled Illumination-level Face Image while preserves identity character as well performing a powerful latent representation from the face image, which encodes illumination-invariant signatures. Moreover, this model can be adopted in single sample per person face recognition. Meanwhile, this research proposes an novel illumination level estimation method based on singular value decomposition to generate the Controlled Illumination-level Face Image optionally. Finally, the performances of the proposed method and other state-of-the-art techniques are verified on the Extended Yale B, CMU PIE, IJB-A and our Self-built Driver Face databases. The experimental results indicate that the IL-GAN model outperforms previous approaches for single sample per person face recognition under varying illumination.

MIT Advanced Vehicle Technology Study: Large-Scale Naturalistic Driving Study of Driver Behavior and Interaction With Automation

For the foreseeble future, human beings will likely remain an integral part of the driving task, monitoring the AI system as it performs anywhere from just over 0% to just under 100% of the driving. The governing objectives of the MIT Autonomous Vehicle Technology (MIT-AVT) study are to (1) undertake large-scale real-world driving data collection that includes high-definition video to fuel the development of deep learning based internal and external perception systems, (2) gain a holistic understanding of how human beings interact with vehicle automation technology by integrating video data with vehicle state data, driver characteristics, mental models, and self-reported experiences with technology, and (3) identify how technology and other factors related to automation adoption and use can be improved in ways that save lives. In pursuing these objectives, we have instrumented 23 Tesla Model S and Model X vehicles, 2 Volvo S90 vehicles, 2 Range Rover Evoque, and 2 Cadillac CT6 vehicles for both long-term (over a year per driver) and medium term (one month per driver) naturalistic driving data collection. Furthermore, we are continually developing new methods for analysis of the massive-scale dataset collected from the instrumented vehicle fleet. The recorded data streams include IMU, GPS, CAN messages, and high-definition video streams of the driver face, the driver cabin, the forward roadway, and the instrument cluster (on select vehicles). The study is on-going and growing. To date, we have 122 participants, 15,610 days of participation, 511,638 miles, and 7.1 billion video frames. This paper presents the design of the study, the data collection hardware, the processing of the data, and the computer vision algorithms currently being used to extract actionable knowledge from the data.

Single Sample Face Recognition under Varying Illumination via QRCP Decomposition.

In this paper, we present a novel high-frequency facial feature and a high-frequency based sparse representation classification to tackle single sample face recognition (SSFR) under varying illumination. Firstly, we propose the assumption that QRCP bases can represent intrinsic face surface features with different frequencies, and their corresponding energy coefficients describe illumination intensities. Based on this assumption, we take QRCP bases with corresponding weighting coefficients (i.e. the major components of energy coefficients) to develop the high-frequency facial feature of the face image, which is named as QRCP-face. The normalized QRCP-face (NQRCPface) is constructed to further constraint illumination effects by normalizing the weighting coefficients of QRCP-face. Moreover, we propose the adaptive QRCP-face (AQRCP-face) that assigns a special parameter to NQRCP-face via the illumination level estimated by the weighting coefficients. Secondly, we consider that the differences of pixel images cannot model the intraclass variations of generic faces with illumination variations, and the specific identification information of the generic face is redundant for the current SSFR with generic learning. To tackle above two issues, we develop a general high-frequency based sparse representation (GHSP) model. Two practical approaches separated high-frequency based sparse representation (SHSP) and unified high-frequency based sparse representation (UHSP) are developed. Finally, the performances of the proposed methods are verified on the Extended Yale B, CMU PIE, AR, LFW and our self-built Driver face databases. The experimental results indicate that the proposed methods outperform previous approaches for SSFR under varying illumination.

Video Analytic Based Health Monitoring for Driver in Moving Vehicle by Extracting Effective Heart Rate Inducing Features

We propose a novel remote heart rate (HR) estimation method using facial images based on video analytics. Most of previous methods have been demonstrated in well-controlled indoor environments. In contrast, this paper proposes a practical video analytic framework under actual driving conditions by extracting key HR inducing features. In particular, when cars are driven, effective and stable HR estimation becomes challenging as there are many dynamic elements, such as rapid illumination changes, vibrations, and ambient lighting that can exist in the vehicle interior. To overcome those disturbances of HR estimation, the driver face region is first detected and cropped to the region of interest (RoI). Second, the components related to HR are extracted from mixed noisy components using ensemble empirical mode decomposition (EEMD). Finally, the extracted signal is analyzed in frequency domain and smoothed with temporal filtering. To verify our approach, the proposed method is compared with recent prominent methods employing a public HCI dataset. It has been demonstrated that the proposed approach delivers superior performance under driving conditions using Bland-Altman plots.

A Deep Learning Approach for Face Detection and Location on Highway

Face detection and location technique is a hot research direction during recent years. Especially, driver face detection on highway is still a challenging problem in social safty deserving research. This paper proposes a novel algorithm based on the improved Multi-task Cascaded Convolutional Networks (MTCNN) and Support Vector Machine (SVM) to realize accurate face region detection and feature location of driver's face on highway, predicting face and feature location via a coarse-to-fine pattern. The proposed algorithm is verified under various complex highway conditions. Experimental results show that the proposed model shows satisfied performance compared to other state-of-the-art techniques used in driver face detection and alignment, keeping robust to the occlusions, varying pose and extreme illumination on highway.

Biometrical driver face verification

The paper discusses the problem of face verification in a driver monitoring system for the purpose of traffic safety. Two different methods of face verification were proposed. Both of them are based on a convolutional neural network and were developed with the use of a transfer learning technique. In the paper, the results produced by both proposed method have been presented and compared. Moreover, their advantages and disadvantages have been discussed.

Multi-Timescale Drowsiness Characterization Based on a Video of a Driver's Face.

Drowsiness is a major cause of fatal accidents, in particular in transportation. It is therefore crucial to develop automatic, real-time drowsiness characterization systems designed to issue accurate and timely warnings of drowsiness to the driver. In practice, the least intrusive, physiology-based approach is to remotely monitor, via cameras, facial expressions indicative of drowsiness such as slow and long eye closures. Since the system’s decisions are based upon facial expressions in a given time window, there exists a trade-off between accuracy (best achieved with long windows, i.e., at long timescales) and responsiveness (best achieved with short windows, i.e., at short timescales). To deal with this trade-off, we develop a multi-timescale drowsiness characterization system composed of four binary drowsiness classifiers operating at four distinct timescales (5 s, 15 s, 30 s, and 60 s) and trained jointly. We introduce a multi-timescale ground truth of drowsiness, based on the reaction times (RTs) performed during standard Psychomotor Vigilance Tasks (PVTs), that strategically enables our system to characterize drowsiness with diverse trade-offs between accuracy and responsiveness. We evaluated our system on 29 subjects via leave-one-subject-out cross-validation and obtained strong results, i.e., global accuracies of 70%, 85%, 89%, and 94% for the four classifiers operating at increasing timescales, respectively.

Driver Face Recognition: Anti-Theft System

At the present, vehicles security is now a concern for the owner because there is getting more and more vehicles being stolen or compromise by theft. Nowadays, security of the vehicles is not secure enough to withstand the compromise techniques that the theft have. In this project, a security method to implement in the driving environment, it is called the Driver Face Recognition-Anti Theft System. It involves face enrolment of the driver as the initial stage followed by facial recognition. This is a system is to assist current vehicles security to become better to prevent the vehicles being steal by the theft. Keywords: Anti-theft, Face Enrolment, Face Recognition, Security

Novel directional projection screen using divergent corner cube retroreflector with a three‐sided curved surface (D‐CCR)

AbstractWe devised and developed a novel directive projection screen for applications such as personal use, small group meeting, automotive pillar, and near‐eye. This screen reflects diffused light at an angle Δθview around a direction deviated by θview from the direction of the projector. By applying this screen to automobiles, for example, by installing a projector at a position slightly shifted from the driver, it is possible to reflect the projected light with high efficiency to a certain range around the face of driver. To realize such a characteristic, a hollow corner cube retroreflector (CCR) was modified as follows. The bottom surface of the CCR was tilted by θview/2 to change the direction of the reflected light by θview from the retroreflection direction. In addition, three sides' surfaces of the CCR were curved, so that the light to be diffused within a designed angle (θview ± Δθview/2). The novel retroreflector CCR (D‐CCR) has a single divergent surface with a three‐sided curved surface. We fabricated a prototype with θview = 10° and Δθview = 16° (i.e., the viewing angle for a single D‐CCR, 10° ± 8°), simulated, and measured its optical properties. We closely confirmed the desired characteristics in which an optical gain of 16 was obtained.

Convolutional Neural Network-Based Classification of Driver's Emotion during Aggressive and Smooth Driving Using Multi-Modal Camera Sensors.

Because aggressive driving often causes large-scale loss of life and property, techniques for advance detection of adverse driver emotional states have become important for the prevention of aggressive driving behaviors. Previous studies have primarily focused on systems for detecting aggressive driver emotion via smart-phone accelerometers and gyro-sensors, or they focused on methods of detecting physiological signals using electroencephalography (EEG) or electrocardiogram (ECG) sensors. Because EEG and ECG sensors cause discomfort to drivers and can be detached from the driver’s body, it becomes difficult to focus on bio-signals to determine their emotional state. Gyro-sensors and accelerometers depend on the performance of GPS receivers and cannot be used in areas where GPS signals are blocked. Moreover, if driving on a mountain road with many quick turns, a driver’s emotional state can easily be misrecognized as that of an aggressive driver. To resolve these problems, we propose a convolutional neural network (CNN)-based method of detecting emotion to identify aggressive driving using input images of the driver’s face, obtained using near-infrared (NIR) light and thermal camera sensors. In this research, we conducted an experiment using our own database, which provides a high classification accuracy for detecting driver emotion leading to either aggressive or smooth (i.e., relaxed) driving. Our proposed method demonstrates better performance than existing methods.

Nighttime Driving Safety Improvement via Image Enhancement for Driver Face Detection

Insufficient illumination makes driver face detection at night challenging. This paper proposes an adaptive attenuation quantification retinex (AAQR) method to enhance the details of nighttime images. There are three phases in this method: attenuation restriction, attenuation prediction, and adaptive quantification. The performance of the proposed method was evaluated by employing a robust face detection method via sparse representation. The collected driver face images at night were categorized into three groups (up-down, left-right, and mixed) according to the illumination distribution in each image. Results have shown that the detection rates of the images enhanced by the proposed AAQR method were 82%, 84%, and 91% for the up-down, left-right, and mixed illumination groups, respectively. In comparison to other image enhancement methods, the detection rate of the AAQR method was 2%–36% greater. Furthermore, the mean computing time for a single $640\times480$ nighttime image using AAQR was less than most of the compared advanced methods. Thus, the AAQR method is recommended for applications in driver face detection tasks at night.

Driver behavior analysis for right-turn drivers at signalized intersections using SHRP 2 naturalistic driving study data

IntroductionUnderstanding driver behavior is important for traffic safety and operation, especially at intersections where different traffic movements conflict. While most driver-behavior studies are based on simulation, this paper documents the analysis of driver-behavior at signalized intersections with the SHRP 2 Naturalistic Driving Study (NDS) data. This study analyzes the different influencing factors on the operation (speed control) and observation of right-turn drivers. MethodA total of 300 NDS trips at six signalized intersections were used, including the NDS time-series sensor data, the forward videos and driver face videos. Different factors of drivers, vehicles, roads and environments were studied for their influence on driver behavior. An influencing index function was developed and the index was calculated for each influencing factor to quantitatively describe its influencing level. The influencing index was applied to prioritize the factors, which facilitates development and selection of safety countermeasures to improve intersection safety. Drivers' speed control was analyzed under different conditions with consideration of the prioritized influencing factors. ResultsVehicle type, traffic signal status, conflicting traffic, conflicting pedestrian and driver age group were identified as the five major influencing factors on driver observation. ConclusionsThis research revealed that drivers have high acceleration and low observation frequency under Right-Turn-On-Red (RTOR), which constituted potential danger for other roadway users, especially for pedestrians. Practical applicationsAs speed has a direct influence on crash rates and severities, the revealed speed patterns of the different situations also benefit selection of safety countermeasures at signalized intersections.