Driver Activity Recognition Research Articles

RFDANet: an FMCW and TOF radar fusion approach for driver activity recognition using multi-level attention based CNN and LSTM network

Dangerous driving behavior is a major contributing factor to road traffic accidents. Identifying and intervening in drivers’ unsafe driving behaviors is thus crucial for preventing accidents and ensuring road safety. However, many of the existing methods for monitoring drivers’ behaviors rely on computer vision technology, which has the potential to invade privacy. This paper proposes a radar-based deep learning method to analyze driver behavior. The method utilizes FMCW radar along with TOF radar to identify five types of driving behavior: normal driving, head up, head twisting, picking up the phone, and dancing to music. The proposed model, called RFDANet, includes two parallel forward propagation channels that are relatively independent of each other. The range-Doppler information from the FMCW radar and the position information from the TOF radar are used as inputs. After feature extraction by CNN, an attention mechanism is introduced into the deep architecture of the branch layer to adjust the weight of different branches. To further recognize driving behavior, LSTM is used. The effectiveness of the proposed method is verified by actual driving data. The results indicate that the average accuracy of each of the five types of driving behavior is 94.5%, which shows the advantage of using the proposed deep learning method. Overall, the experimental results confirm that the proposed method is highly effective for detecting drivers’ behavior.

A lightweight temporal attention-based convolution neural network for driver's activity recognition in edge

Low inference latency and accurate response to environment changes play a crucial role in the automated driving system, especially in the current Level 3 automated driving. Achieving the rapid and reliable recognition of driver's non-driving related activities (NDRAs) is important for designing an intelligent takeover strategy that ensures a safe and quick control transition. This paper proposes a novel lightweight temporal attention-based convolutional neural network (LTA-CNN) module dedicated to edge computing platforms, specifically for NDRAs recognition. This module effectively learns spatial and temporal representations at a relatively low computational cost. Its superiority has been demonstrated in an NDRA recognition dataset, achieving 81.01% classification accuracy and an 8.37% increase compared to the best result of the efficient network (MobileNet V3) found in the literature. The inference latency has been evaluated to demonstrate its effectiveness in real applications. The latest NVIDIA Jetson AGX Orin could complete one inference in only 63 ms.

Analysis of Privacy-Enhancing Technologies in Open-Source Federated Learning Frameworks for Driver Activity Recognition.

Wearable devices and smartphones that are used to monitor the activity and the state of the driver collect a lot of sensitive data such as audio, video, location and even health data. The analysis and processing of such data require observing the strict legal requirements for personal data security and privacy. The federated learning (FL) computation paradigm has been proposed as a privacy-preserving computational model that allows securing the privacy of the data owner. However, it still has no formal proof of privacy guarantees, and recent research showed that the attacks targeted both the model integrity and privacy of the data owners could be performed at all stages of the FL process. This paper focuses on the analysis of the privacy-preserving techniques adopted for FL and presents a comparative review and analysis of their implementations in the open-source FL frameworks. The authors evaluated their impact on the overall training process in terms of global model accuracy, training time and network traffic generated during the training process in order to assess their applicability to driver’s state and behaviour monitoring. As the usage scenario, the authors considered the case of the driver’s activity monitoring using the data from smartphone sensors. The experiments showed that the current implementation of the privacy-preserving techniques in open-source FL frameworks limits the practical application of FL to cross-silo settings.

Driver activity recognition by learning spatiotemporal features of pose and human object interaction

Detecting hazardous activity during driving can be useful in curbing roadside accidents. Existing techniques utilizing image based features for encoding such activity can sometimes misclassify crucial scenarios. One particular work by Zhao et al. (2013 [1], 2013 [2], 2011 [3]) suggests an image based feature set that encodes the driver’s pose, which is categorized into one of four activities. We bring more clarity in understanding the activity by proposing a richer, video based feature set that adeptly exploits spatiotemporal information of the driver. Our feature set encodes the driver’s pose, crucial variations in pose and interactions with objects within the vehicle. The feature set is tested on our newly created dataset since the ones used in literature are not publicly available. Our proposed feature set captures a larger number of activities and using standard classifiers and benchmarks it has shown significant improvements over the existing ones.

Driver activity recognition using spatial‐temporal graph convolutional LSTM networks with attention mechanism

Driver activity engagement while driving plays a vital role that leads to negative outcomes of driving safety. To reduce traffic accidents and ensure driving safety, real-time driver activity recognition architecture is proposed in this study. Specifically, a total of eight kinds of common driving-related activities are identified, which include the normal driving, left or right checking, texting, answering the phone, using media, drinking, and picking up objects. Raw experiment videos are collected via onboard monocular cameras, which are used for the upper body skeleton information extraction of the driver. Then, the graph convolutional networks (GCN) are constructed for spatial structure feature reasoning in a single frame, which is consecutively followed by long short-term memory (LSTM) networks for temporal motion feature learning within the sequence. Moreover, the attention mechanism is further utilised to emphasise the keyframes to select discriminative sequential information. Finally, a large-scale driver activity dataset, consisting of both naturalistic driving data and simulative driving data, is collected for model training and evaluations. Experimental results show that the general recall ratio of those eight driving-related activities reaches up to 88.8% and the real-time recognition efficiency can reach up to 24 fps, which would satisfy the real-time requirements of engineering applications.

Deep CNN models for Driver Activity Recognition for Intelligent Vehicles

Deep CNN models for Driver Activity Recognition for Intelligent Vehicles

Driver Activity Recognition by Means of Temporal HTN Planning

When delivering a transport service, scheduled driver workplans have to be aligned with world wide complex hours of service (HoS) regulations which constraint the amount of working and driving time without resting. The activities of such workplans are recorded by onboard sensors in large temporal event logs. Transport companies are interested on recognizing what a driver is doing, based on the temporal observations from event logs, considering the terms defined by HoS regulations. This work presents an application of temporal HTN planning to plan and goal recognition that, starting from a real event log extracted from a tachograph, identifies different sub-sequences of a driver's daily and weekly driving activity and labels them according to the terms defined by HoS regulation.

CARIN

Recent studies have proposed to use the Channel State Information (CSI) of WiFi wireless channel for human gesture recognition. As an important application, CSI-based driver activity recognition in passenger vehicles has received increasing research attention. However, a serious limitation of almost all the existing WiFi-based recognition solutions is that they can only recognize the activity of a single person at a time, because the activities of other people (if performed at the same time) can interfere with the WiFi signals. In a sharp contrast, there can often be one or more passengers in any vehicles. In this paper, we propose CARIN, CSI-based driver Activity Recognition under the INterference of passengers. CARIN features a combination-based solution that profiles all the possible activity combinations of driver and (one or more) passengers in offline training and then performs recognition online. To attack possible combination explosion, we first leverage in-car pressure sensors to significantly reduce combinations, because there are only limited seating options in a passenger vehicle. We then formulate a distance minimization problem for fast runtime recognition. In addition, a period analysis methodology is designed based on the kNN classifier to recognize activities that have a sequence of body movements, like continuous head nodding due to driver fatigue. Our results in a real car with 3,000 real-world traces show that CARIN can achieve an overall F1 score of 90.9%, and outperforms the three state-of-the-art solutions by 32.2%.

WiDriver: Driver Activity Recognition System Based on WiFi CSI

Driver is the most active factor in people–vehicle–road system, so the driver activity monitoring has become increasingly important to support the driver assistant system application. The possibility of using device-free sensing technology for driver activity recognition in a simulated driving environment is investigated in this paper. We present WiDriver, among the first efforts to employ channel state information (CSI) amplitude variation data to intelligently estimate driving actions with commodity WiFi devices. The WiDriver proposes the scheme of screening sensitive input data from original CSI matrix of WiFi signals based on BP neural network algorithm; and the continuous driving activities classification algorithm by introducing the posture sequence, driving context finite automate model. Our experimental driving study in carriage with 5 subjects shows that the sensitive input selection scheme can achieve high accuracy of 96.8% in posture recognition and the continuous action classification algorithm can reach 90.76% maneuver operation detection rate.

Online Recognition of Driver-Activity Based on Visual Scanpath Classification

The next step towards the fully automated vehicle is the level of conditional automation, where the automated driving system can take over the control and responsibility for a limited time interval. Nevertheless, take-over situations may occur, forcing the driver to resume the driving task. Despite such situations, the driver is able to perform secondary tasks during conditionally automated driving, hence a low take-over quality must be expected. Methods for Driver-Activity Recognition (DAR) usually extract features for the classification within a moving time window. In this paper, the first DAR architecture based on the driver's scanpath, which is extracted by means of dynamic clustering and symbolic aggregate approximation patterns, is introduced. To demonstrate the potential of this approach, it is compared to a state-of-the-art method based on the data of a driving simulator study with 82 subjects. The classification performance of both DAR approaches was examined for decreasing window sizes with regard to the recognition of different secondary tasks and the separability of drivers using a handheld or hands-free device. Compared to the state-of-the-art approach, the proposed method shows a classification accuracy increase of nearly 20%, a significant improvement of the overall classification performance, and is able to classify the secondary tasks of the driver even for short windows of a duration of 5 s, i.e. with little information.

Multiview fusion for activity recognition using deep neural networks

Convolutional neural networks (ConvNets) coupled with long short term memory (LSTM) networks have been recently shown to be effective for video classification as they combine the automatic feature extraction capabilities of a neural network with additional memory in the temporal domain. This paper shows how multiview fusion can be applied to such a ConvNet LSTM architecture. Two different fusion techniques are presented. The system is first evaluated in the context of a driver activity recognition system using data collected in a multicamera driving simulator. These results show significant improvement in accuracy with multiview fusion and also show that deep learning performs better than a traditional approach using spatiotemporal features even without requiring any background subtraction. The system is also validated on another publicly available multiview action recognition dataset that has 12 action classes and 8 camera views.