Driver Assistance Systems Research Articles

Driver's visual fixation attention prediction in dynamic scenes using hybrid neural networks

The traffic driving scenario is dynamically varying and experienced drivers can allocate visual attention effectively in brief time intervals, focusing on prominent targets and areas in advance, thus ensuring safe driving. Therefore, it is significant to research the behavior of driver's visual attention allocation in the development of driver assistance systems and autonomous vehicles. Recently, inspired by top-down and bottom-up attention mechanisms, most visual attention models based on neural networks have been proposed. However, these models tend to yield highly distributed predictions for the specific driving scenario, and have relatively low prediction accuracy, making them unfeasible for being used in practice. Hence, to overcome these challenges, we propose a driver's visual attention model (DVAM) built on the encoder-decoder architecture with the utilization of the convolutional neural network (CNN) and the recurrent neural networks (RNN). Specifically, the model leverages the temporal and spatial dimension information to better mimic realistic dynamic driving, and extract abundant feature information, thus ensuring that the final predictions are authentic and effective. Then, extensive experiments demonstrate that our suggested DVAM achieves better robustness and precision in predicting driver attention to areas or targets compared to state-of-the-art methods on the DR(eye)VE dataset. Finally, our model is deployed on the TDV dataset to validate its generalization capabilities, which prove the potential for field adaptation.

Predictive Analysis of Vehicular Lane Changes: An Integrated LSTM Approach

In the rapidly advancing domain of vehicular traffic management and autonomous driving, accurate lane change predictions are paramount for ensuring safety and optimizing traffic flow. This study introduces a comprehensive two-stage prediction model that harnesses the capabilities of long short-term memory (LSTM) for anticipating vehicular lane changes. Initially, we employed a variety of models, such as regression methods, SVMs, and a multilayer perceptron, to categorize lane change behaviors. The dataset was then segmented based on vehicle trajectories and lane change patterns. In the subsequent phase, we utilized the superior classification outcomes from LinearSVC to curate our training data. We developed two dedicated LSTM networks tailored to specific datasets: the lane-keeping LSTM (LK-LSTM) and the lane-changing LSTM (LC-LSTM). By integrating insights from both models, we achieved a comprehensive prediction of vehicular lane changes. Our results indicate that the unified prediction model markedly enhances prediction precision. Accurate lane change predictions offer valuable contributions to advanced driver-assistance systems (ADAS), with the potential to minimize traffic mishaps and enhance traffic fluidity. As we transition to a more autonomous automotive era, refining these predictions becomes essential in seamlessly merging human and automated driving experiences.

Transferable driver facial expression recognition based on joint discriminative correlation alignment network with enhanced feature attention

AbstractFacial expression is closely related to the emotions of drivers, thus facilitating safe driving detection in advanced driving assistance system (ADAS). Recently, deep learning techniques have become prevalent for facial expression recognition. However, for driving scenarios, the facial expression recognition is mainly challenged by the problems of small sample size as well as effective feature representation. To address the above issues, this paper puts forward a transfer learning‐based method for driver facial expression recognition by fully exploiting the other sources of facial expression data that may obey different distributions. Specifically, an enhanced feature attention module is firstly devised such that affluent features with multi‐scales can be extracted and refined based on the spatial and channel attention mechanisms. Then, a joint correlation alignment loss is presented ensuring that the samples in the source and target domains are transformed into the shared common subspace to reduce difference of both the marginal and conditional distributions. Multiple transfer learning tasks on real‐world data are carried out to evaluate the proposed method. The experimental results show that our model achieves better recognition accuracy for driver facial expression compared with several traditional and deep learning‐based transfer learning algorithms.

A Network Analysis Approach to Quantifying Changes in ADAS from Over-the-Air Updates

Advanced driver assistance systems are changing the way that drivers interact with vehicles and may benefit from changes to consumer education that aim to improve the driver’s understanding of changes to their role. System updates will increasingly be applied over the air, which may lead to confusion about system capabilities if there is a substantial change, and there is a need to understand when changes need additional education. This paper uses a network analysis to quantify the change in driver assistance systems and also extends the approach to include a human user in the network. This paper describes the approach and presents an example comparing two adaptive cruise control systems, before and after an update. The network analysis quantified the connections within the network, with components of the driving assistance systems showing different connection strength with an increase in complexity of the system and with the addition of the user as a node in the network. The network analysis approach may be a useful tool for determining when additional consumer education is necessary for advanced driver assistance systems updates.

Driver Head Pose Detection From Naturalistic Driving Data

Driver behavior analysis plays an important role in driver assistance systems. A driver’s face and head pose hold the key towards understanding whether the driver’s attention and concentration are on the road while driving. Naturalistic driving studies (NDS) allow observing drivers in real-time under naturalistic traffic conditions. Yet, data collected in NDS often comprise low-resolution videos usually with more challenging camera positions compared to controlled studies. For instance, when the camera is not directly facing the driver, classifying head pose becomes more challenging, since the variation between different classes becomes much smaller. In this paper, we propose three different approaches to classify a driver’s head pose from naturalistic videos, which were captured by a camera providing a side view, instead of directly facing the driver. These approaches employ a sequence of five key points on the driver’s face. We compare these three proposed approaches with each other as well as with three different baselines by using leave-one-driver-out cross-validation on nine different drivers. Results show that our proposed method employing a Bidirectional Gated Recurrent Unit (BiGRU) outperforms the best performing baseline by 11% in terms of overall accuracy.

Evaluating the dissociation between drivers’ self-perceived and objective need for vehicle assistance during obstacle avoidance tasks

Driver-assistance systems are becoming more commonplace; however, the realized safety benefits of these technologies depend on whether a person accepts and adopts automated driving aids. One challenge to adoption could be a preference-performance dissociation (PPD), which is a mismatch between a self-perceived desire and an objective need for assistance. Research has reported PPD in driving but has not extensively leveraged driving performance data to confirm its existence. Thus, the goal of this study was to compare drivers’ self-reported need for vehicle assistance to their objective driving performance. Twenty-one participants drove on a simulated road and traversed challenging, real-world roadway obstacles. Afterwards, they were asked about their preference for automated vehicle assistance (e.g., steering and braking) during their drive. Overall, some participants exhibited PPD that included both over- and underestimating their need for a particular type of automated assistance. Findings can be used to develop shared control and adaptive automation strategies tailored to particular users and contexts across various safety-critical environments.

Do Drivers Require Education After Over-The-Air Adas Updates?

Over-the-air (OTA) advanced driver assistance systems (ADAS) updates may require drivers to adjust their mental models to maintain appropriate system use. Based on the magnitude of change to the ADAS, hands-on experience with the updated system may not require consumer education or training. In this study, 96 drivers experienced adaptive cruise control in the NADS-1 and received an OTA update with either pedestrian detection (PD) or traffic jam (TJ). Drivers that experienced PD had a higher confidence in their understanding of PD whereas drivers that experienced TJ, had a better understanding of TJ. However, their confidence was not correlated with their understanding of the system for either TJ or PD. Although hands-on experience with a new system may improve driver’s understanding or confidence in their understanding of the system, consumer education may strengthen the relationship between their understanding and confidence in their understanding of ADAS.

Object Detection Based on Lightweight YOLOX for Autonomous Driving.

Accurate and rapid response in complex driving scenarios is a challenging problem in autonomous driving. If a target is detected, the vehicle will not be able to react in time, resulting in fatal safety accidents. Therefore, the application of driver assistance systems requires a model that can accurately detect targets in complex scenes and respond quickly. In this paper, a lightweight feature extraction model, ShuffDet, is proposed to replace the CSPDark53 model used by YOLOX by improving the YOLOX algorithm. At the same time, an attention mechanism is introduced into the path aggregation feature pyramid network (PAFPN) to make the network focus more on important information in the network, thereby improving the accuracy of the model. This model, which combines two methods, is called ShuffYOLOX, and it can improve the accuracy of the model while keeping it lightweight. The performance of the ShuffYOLOX model on the KITTI dataset is tested in this paper, and the experimental results show that compared to the original network, the mean average precision (mAP) of the ShuffYOLOX model on the KITTI dataset reaches 92.20%. In addition, the number of parameters of the ShuffYOLOX model is reduced by 34.57%, the Gflops are reduced by 42.19%, and the FPS is increased by 65%. Therefore, the ShuffYOLOX model is very suitable for autonomous driving applications.

Investigating the Effects of Downhill Roads on Driver’s Eye Movements

Despite the increasing popularity of self-driving cars, researchers in the transportation industry continues to prioritize accident prevention and safety improvement. This study inspects the influence of driving experience and gender on the driver’s gaze and pupil behaviors while descending a downhill. The data in this paper was collected by equipping ten subjects Tobii Pro glasses II in a series of experiments designed to track their gaze and pupil behavior in a downhill road section. The statistical and Python-based data analysis shows that novice drivers pay more attention to the instrument panel on the downhill road, and their pupils’ diameter is larger at both the top and bottom of the hill. In addition, female drivers pay more attention to the instrument panel than male drivers do during downhill tasks. Finally, more accurate formulas can be obtained by considering the experience and gender of drivers through logistics regression. This study has important theoretical significance for discovering the driver’s target search process and improving the driver assistance system.

Factors Influencing Trust in Advanced Driver Assistance Systems for Current Users.

Understanding the factors influencing trust in advanced driver assistance systems (ADAS) may help inform training and education to support appropriate use. We surveyed 369 drivers with experience using both adaptive cruise control (ACC) and lane keeping assist (LKA). The survey included questions to assess trust in ADAS, along with objective knowledge about ADAS limitations, self-reported understanding of ADAS, familiarity with technology, propensity to trust technology, and demographics. Regression results showed that self-reported understanding, but not objective knowledge, predicted trust in ADAS. Self-reported understanding was not correlated with objective knowledge; overall, participants were not aware of many of the system limitations included in the survey. Propensity to trust technology was also a significant predictor of trust. Training/educational materials could be designed to inform drivers of potential gaps in their understanding and adjust expectations of ADAS to support appropriate trust for those with a high propensity to trust technology.

Using Sensitivity and Bias in Signal Detection Theory to Predict Proportion Correctness: Simulation and Case Study on ADAS Mental Model Evaluation

The sensitivity ( d’) and bias ( c) in signal detection theory can reflect respondents’ objective performance in understanding a system and their bias towards saying yes. Thus, d’ and c can be used as alternatives of proportion correctness (PC) when evaluating drivers’ mental models of advanced driving assistance system (ADAS). The adoption of d’ and c as mental model metrics also allows cross-study comparisons as their values are independent of signal (i.e., ADAS function present) and noise (i.e., ADAS function absent) ratio. However, there is no closed-form solution of the relationships among d’, c, and PC. Hence, using numerical simulations, we extracted an empirical equation that quantifies how d’ and c can estimate PC. The equation was then validated based on participants’ responses from a survey study that targeted towards drivers’ ADAS mental model. The results show that the empirical equation reached a satisfying performance (R2 > 0.8) in estimating PC.

Dynamic measurements of speed and risk perception during driving: Evidence of speed misestimation from continuous ratings and video analysis.

Investigating the factors underlying perceived speed and risk is crucial to ensure safe driving. However, existing studies on this topic usually measure speed and risk perception indirectly after a driving session, which makes it difficult to trace dynamic effects and time points of potential misestimates. To address this problem, we developed and validated a novel continuous method for dynamically measuring risk and speed perceptions. To study the factors affecting risk and speed perception, we presented participants with videos captured on the same racing track from the same point of view but with different drivers who varied in their speed and risk profiles. During the experiment, participants used a joystick to continuously rate the subjectively perceived risk of driving in the first block and the perceived speed in the second block. Our analysis of these dynamic ratings indicates that risk and speed estimates were decoupled, with curves resulting in decreased speeds but increased risk ratings. However, a close distance to the car in front increased both speed and risk. Based on actual and estimated speed data, we found that overtaking cars on curves resulted in participants overestimating their own speed, whereas an increase in the distance to the car in front on a straight course led to underestimations of their own speed. Our results showcase the usefulness of dynamic rating profiles for in-depth investigations into situations that could result in drivers misjudging speed or risk and will thus help the development of more intelligent, human-centered driving assistance systems.

Keep your hands on the wheel: The effect of driver supervision strategy on change detection, mind wandering, and gaze behavior

Advanced driver-assist systems (ADAS) enable drivers to relinquish operational control of the vehicle to automation for part of the total drive. While these features are engaged, drivers have an increased risk of losing awareness of their environment. Current ADAS broadly utilizes hands-on-the-wheel or eyes-on-the-road driver supervision strategies to continually monitor steering-wheel torque and drivers’ head and eye positions to ensure driver attention. The current work examines the effect of hands-on-the-wheel and eyes-on-the-road driver supervision strategies on change detection, mind wandering, and gaze behavior in a low-fidelity semi-autonomous driving task.

Can We Replace Real-World With Synthetic Data in Deep Learning-Based ADAS Algorithm Development?

Deep learning models require vast amounts of annotated training data. Gathering and annotating the data from the real world is an expensive and time-consuming process. Thus, synthetically generated data is being researched more and more. This paper tries to answer the question of whether and to what extent synthetically generated data can help in developing the advanced driver-assistance system (ADAS) algorithms for autonomous vehicles, for the object detection task based on deep learning. Two state-of-the-art deep learning object detectors were trained on various combinations of real-world and synthetic data. A total of 12 detectors were tested on real-world test images. Results show that synthetic data can contribute to better detector performance until a certain ratio between real-world and synthetic data is reached.

Metamobility: Connecting Future Mobility With the Metaverse

The metaverse is a perpetual, immersive, and shared digital universe that is linked to but is beyond the physical reality, and this emerging technology is attracting enormous attention from different industries. In this article, we define the first holistic realization of the metaverse in the mobility domain, coined as metamobility. We present our vision of what metamobility will be and describe its basic architecture. We also propose two use cases, tactile live maps and metaempowered advanced driver-assistance systems (ADASs), to demonstrate how metamobility will benefit and reshape future mobility systems. Each use case is discussed from the perspective of the technology evolution, future vision, and critical research challenges. Finally, we identify multiple concrete open research issues.

Evasive actions to prevent pedestrian collisions in varying space/time contexts in diverse urban and non-urban areas

This study aims to identify driver-safe evasive actions associated with pedestrian crash risk in diverse urban and non-urban areas. The research focuses on the integration of quantitative methods and granular naturalistic data to examine the impacts of different driving contexts on transportation system performance, safety, and reliability. The data is derived from real-life driving encounters between pedestrians and drivers in various settings, including urban areas (UAs), suburban areas (SUAs), marked crossing areas (MCAs), and unmarked crossing areas (UMCAs). By determining critical thresholds of spatial/temporal proximity-based safety surrogate techniques, vehicle–pedestrian conflicts are clustered through a K-means algorithm into different risk levels based on drivers' evasive actions in different areas. The results of the data analysis indicate that changing lanes is the key evasive action employed by drivers to avoid pedestrian crashes in SUAs and UMCAs, while in UAs and MCAs, drivers rely on soft evasive actions, such as deceleration. Moreover, critical thresholds for several Safety Surrogate Measures (SSMs) reveal similar conflict patterns between SUAs and UMCAs, as well as between UAs and MCAs. Furthermore, this study develops and delivers a pseudo-code algorithm that utilizes the critical thresholds of SSMs to provide tangible guidance on the appropriate evasive actions for drivers in different space/time contexts, aiming to prevent collisions with pedestrians. The developed research methodology as well as the outputs of this study could be potentially useful for the development of a driver support and assistance system in the future.

Improvement of small objects detection in thermal images

Thermal images are widely used for various applications such as safety, surveillance, and Advanced Driver Assistance Systems (ADAS). However, these images typically have low contrast, blurred aspect, and low resolution, making it difficult to detect distant and small-sized objects. To address these issues, this paper explores various preprocessing algorithms to improve the performance of already trained object detection networks. Specifically, mathematical morphology is used to favor the detection of small bright objects, while deblurring and super-resolution techniques are employed to enhance the image quality. The Logarithmic Image Processing (LIP) framework is chosen to perform mathematical morphology, as it is consistent with the Human Visual System. The efficacy of the proposed algorithms is evaluated on the FLIR dataset, with a sub-base focused on images containing distant objects. The mean Average-Precision (mAP) score is computed to objectively evaluate the results, showing a significant improvement in the detection of small objects in thermal images using CNNs such as YOLOv4 and EfficientDet.

One-Stage Brake Light Status Detection Based on YOLOv8.

Despite the advancement of advanced driver assistance systems (ADAS) and autonomous driving systems, surpassing the threshold of level 3 of driving automation remains a challenging task. Level 3 of driving automation requires assuming full responsibility for the vehicle's actions, necessitating the acquisition of safer and more interpretable cues. To approach level 3, we propose a novel method for detecting driving vehicles and their brake light status, which is a crucial visual cue relied upon by human drivers. Our proposal consists of two main components. First, we introduce a fast and accurate one-stage brake light status detection network based on YOLOv8. Through transfer learning using a custom dataset, we enable YOLOv8 not only to detect the driving vehicle, but also to determine its brake light status. Furthermore, we present the publicly available custom dataset, which includes over 11,000 forward images along with manual annotations. We evaluate the performance of our proposed method in terms of detection accuracy and inference time on an edge device. The experimental results demonstrate high detection performance with an mAP50 (mean average precision at IoU threshold of 0.50) ranging from 0.766 to 0.793 on the test dataset, along with a short inference time of 133.30 ms on the Jetson Nano device. In conclusion, our proposed method achieves high accuracy and fast inference time in detecting brake light status. This contribution effectively improves safety, interpretability, and comfortability by providing valuable input information for ADAS and autonomous driving technologies.

An Objective Evaluation Method for Driver/Passenger Acceptance of an Autonomous Driving System for Lane Changes

Reliable assessment methods of driver acceptance are needed due to increased interest in high levels of autonomous driving systems. Subjective evaluation methods have mostly been utilized to assess the acceptance of newly developed advanced driver assistance systems because acceptance varies depending on the individual. In this paper, an objective evaluation methodology of driver acceptance for an autonomous driving system was proposed based on objective measurable parameters in the case of automatic lane change situations. To this end, a massive driver–vehicle interaction database was utilized, constructed by a specially designed experimental program. The experiment was carried out with 19 selected drivers (9 experts and 10 novices), supposed as an autonomous driving system. The database consisted of not only various measurable parameters on control commands, vehicle behaviors, and relations with other vehicles but also subjective acceptances. To interpret the driver acceptance, objective parameter sets were derived by two different methods: a statistical significance test and an acceptance sensitivity analysis. Then, a modeling method based on stochastic estimation to evaluate driver acceptance was suggested as an objective evaluation method for the driver acceptance of an automatic lane change system. The data set of the expert drivers was only used for the acceptance evaluation modeling; the other data sets of the novice drivers were used for verifications for the suggested model. The estimation accuracies of the two different models using a significance test and sensitivity analysis were 90.2% and 99.5%, respectively. This objective method for acceptance evaluation can not only be expanded to other functions of an autonomous driving system but also to an entirely autonomous driving vehicle.

The Impact of Environmental Features on Drivers’ Situation Awareness Using Real-World Driving Scenarios

Advanced driver assistance systems (ADAS) need to account for the driver’s awareness of the environment to be effectively used. This study examines the impact of environmental features (eg, visual complexity, object density, roadway type, lighting) on drivers’ situation awareness (SA). This is achieved using a controlled study with 40 participants. Using a split-plot design, the participants were shown 30 out of 75 real-world driving scenarios displayed in a driving simulator environment. Participants’ responses to Situational Awareness Global Assessment Technique (SAGAT) queries on the type and coordinates of objects in the scene were used to calculate SA scores. A hurdle model was developed to estimate participants’ SA scores. The key findings highlight visual complexity as a significant predictor of SA scores. This predictor was easy to compute and able to capture the complexity of objects that impact road safety as well as the visual clutter in the background. The model showed that drivers were able to identify at least one object of interest in complex environments with high visual complexity and with many objects. A higher proportion of vulnerable road users was associated with a greater likelihood of a non-zero SA score, but the SA score was lower compared to environments with higher proportions of cars. The findings of this study provide insights into the environmental factors to be considered for SA predictive models.