Self-driving System Research Articles

Structural Information Guided Multimodal Pre-training for Vehicle-Centric Perception

Understanding vehicles in images is important for various applications such as intelligent transportation and self-driving system. Existing vehicle-centric works typically pre-train models on large-scale classification datasets and then fine-tune them for specific downstream tasks. However, they neglect the specific characteristics of vehicle perception in different tasks and might thus lead to sub-optimal performance. To address this issue, we propose a novel vehicle-centric pre-training framework called VehicleMAE, which incorporates the structural information including the spatial structure from vehicle profile information and the semantic structure from informative high-level natural language descriptions for effective masked vehicle appearance reconstruction. To be specific, we explicitly extract the sketch lines of vehicles as a form of the spatial structure to guide vehicle reconstruction. The more comprehensive knowledge distilled from the CLIP big model based on the similarity between the paired/unpaired vehicle image-text sample is further taken into consideration to help achieve a better understanding of vehicles. A large-scale dataset is built to pre-train our model, termed Autobot1M, which contains about 1M vehicle images and 12693 text information. Extensive experiments on four vehicle-based downstream tasks fully validated the effectiveness of our VehicleMAE. The source code and pre-trained models will be released at https://github.com/Event-AHU/VehicleMAE.

LLM-Powered Synthetic Environments for Self-Driving Scenarios

This paper outlines a proposal exploring the potential use of Large Language Models (LLMs), particularly GPT-4, in crafting realistic synthetic environments for self-driving scenarios. The envisioned approach involves dynamic scene generation within game engines, leveraging LLMs to introduce challenging elements for autonomous vehicles. The proposed evaluation process outlines assessments such as realistic testing, safety metrics, and user interaction, aiming to set the stage for potential improvements in self-driving system performance. The paper aims to contribute to the AI field by discussing how LLMs could be utilized to create valuable testing grounds for autonomous vehicles, potentially fostering the development of more robust self-driving technology. The envisioned impact is the eventual enhancement of road safety and the possible acceleration of the adoption of autonomous vehicles, paving the way for a future with safer and more efficient transportation.

Tri-HGNN: Learning triple policies fused hierarchical graph neural networks for pedestrian trajectory prediction

In complex and dynamic urban traffic scenarios, the accurate trajectory prediction of surrounding pedestrians with interactive behaviors plays a vital role in the self-driving system. Intrinsic factors and extrinsic factors will inevitably influence the pedestrians trajectory. Intrinsic factors such as pedestrians diversified intentions bring rich and diverse multi-modal future possibilities. Besides, extrinsic factors affecting the future trajectory are accompanied by context semantics such as interactions among pedestrians. However, most of the existing methods discuss two problems (interaction and intention) separately. Considering both two factors impact the trajectory of pedestrians, a Triple Policies Fused Hierarchical Graph Neural Networks (Tri-HGNN) is proposed to model spatial and temporal interactions and intentions among the whole scene of pedestrians at each time step and predict the multiple future trajectories. Tri-HGNN contains three different policies: (i) Extrinsic-level policy is used to extract spatial nodes embedding from the interaction graph of pedestrian trajectories by using the Graph Attention Network. (ii) Intrinsic-level policy adopts the Graph Convolutional Network to infer the human intention for more accurate prediction. Moreover, human intention is influenced by the intrinsic interaction generated among pedestrians, so we fuse the interaction features to grasp the influence of the extrinsic interaction. (iii) Basic-level policy then integrates the heuristic information obtained from other two policies and concatenates it with historical trajectories to make multiple predictions through Temporal Convolution Network. Experimental results show that our model improves performance compared with state-of-the-art methods on the ETH/UCY and SDD benchmarks.

Just-in-Time Defect Prediction for Self-driving Software via a Deep Learning Model

Edge computing is applied to various applications and is typically applied to autonomous driving software. As the self-driving system becomes complicated and the proportion of software increases, accidents caused by software defects increase. Just-in-time (JIT) defect prediction is a technique that identifies defects during the software development phase, which helps developers prioritize code inspection. Many researchers have proposed various JIT models, but it is difficult to find a case in which JIT defect prediction was performed on edge computing applications. In particular, due to the characteristic of self-driving software, which is frequently updated, there is a high risk of inducing defects into the update process. In this work, we propose a JIT defect prediction model via deep learning for edge computing applications called JIT4EA. Our research goal is to develop an effective model to predict defects in edge computing applications. To do this, we perform defect prediction on self-driving software, a representative edge computing application. We use pre-trained unified cross-modal pre-training for code representation (UniXCoder) to embed commit messages and code changes. We use bidirectional-LSTM(Bi-LSTM) for context and semantic learning. As a result of the experiment, it was confirmed that the proposed JIT4EA performed better than state-of-the-art methods and could reduce the code inspection effort.

Self-powered triboelectric sensor with N-doped graphene quantum dots decorated polyaniline layer for non-invasive glucose monitoring in human sweat

A novel non-invasive wearable self-powered triboelectric sensor (TES) for simultaneous physiological monitoring had shed the light on its extraordinary performance, which become a promising choice for detecting glucose concentration from human sweat via non-invasive strategy. For realizing the function of wearable TES, the flexible substrate of conducting polymers commonly served as matrix for not only immobilizing the specific enzyme, but also played as the sensing layer by the coupling effect of triboelectrification/enzymatic reaction. To boost up the reliability and sensitivity of conducting polymer-based TES, in this study, a nanocomposite of N-doped graphene quantum dots decorated polyaniline (NGQDs/PANI) is designed as a triboelectric layer for non-invasive glucose detection. The surface electronegativity of PANI was changed by decorating the NGQDs with electron-rich functional groups for increasing the surface charge, which further facilitating the charge transfer between the PANI and intermedia and then enhancing triboelectric outputs. Upon the enhanced electron transfer and surface charge by NGQDs, NGQDs/PANI/GOx based TES offers greater sensitivity (23.52 mM−1) in glucose detection compared to pristine PANI/GOx one (16.44 mM−1). Notably, this study demonstrates the feasibility of wearable TES on human skin and their reliable and sensitive performance with a self-driving system without any external power source.

DSiV: Data Science for Intelligent Vehicles

Data science (DS) devotes to extract useful data from noisy one to form actionable insights. It has broad applications in many domains such as internet search, tourism and social media. However, less systematic studies related to DS have been done in the field of intelligent vehicles (IV). As the size of data in IV becomes more enormous than ever before, it is necessary to give a high-level view on how to utilize DS for achieving better IV. After briefly retrospecting the history of DS, we shed light on the potential and prospects of DS in the IV domain through discussing data management, trusted data, interpretability, high-order relationship of intelligent vehicle data and so on. We also analyze the relationship between DS and several practical intelligent vehicle applications including self-driving system, data visualization, parallel intelligent vehicle system.

Deadline Miss Early Detection Method for Mixed Timer-Driven and Event-Driven DAG Tasks

Self-driving systems have a hard real-time nature, and the control commands of the vehicle must be output before the end-to-end deadline using sufficiently new data. It must also have the functionality to quickly shift to a safe state in the event of a deadline miss. However, the current self-driving system can only detect a deadline miss at the end of the process. To further improve safety, a method for detecting the possibility of a deadline miss in the middle of the process is required. Therefore, we represent such a real-time system as a mixed timer-driven and event-driven directed acyclic graph (DAG) and propose an early detection method for deadline misses by deriving a time constraint for each node. The experimental evaluation shows that the proposed method can detect deadline misses early for various scheduling algorithms. It also shows that the deadline miss ratio can be reduced by prioritizing the scheduling of jobs with a small margin to the time constraint determined using the proposed method for each job.

TrajGen: Generating Realistic and Diverse Trajectories With Reactive and Feasible Agent Behaviors for Autonomous Driving

Realistic and diverse simulation scenarios with reactive and feasible agent behaviors can be used for validation and verification of self-driving system performance without relying on expensive and time-consuming real-world testing. Existing simulators rely on heuristic-based behavior models for background vehicles, which cannot capture the complex interactive behaviors in real-world scenarios. Meanwhile, existing learning-based methods are typical insufficient, yielding behaviors of traffic participants that frequently collide or drive off the road especially for a long horizon. To address this issue, we propose TrajGen, a two-stage trajectory generation framework, which can capture more realistic and diverse behaviors directly from human demonstration. In particular, TrajGen consists of the multi-modal trajectory prediction stage and the reinforcement learning based trajectory modification stage. In the first stage, we propose a novel auxiliary RouteLoss for the trajectory prediction model to generate multi-modal diverse trajectories in the drivable area. In the second stage, reinforcement learning is used to track the predicted trajectories while avoiding collisions, which can improve the reactivity and feasibility of generated trajectories. In addition, we develop a simulator I-Sim that can provide support for data-driven agent behavior simulation and train reinforcement learning models in parallel based on naturalistic driving data. The vehicle model in I-Sim can guarantee that the generated trajectories by TrajGen satisfy vehicle kinematic constraints. Finally, we give comprehensive metrics to evaluate generated trajectories for simulation scenarios, which shows that TrajGen outperforms either trajectory prediction or inverse reinforcement learning in terms of fidelity, reactivity, feasibility, and diversity.

The Relationship between Innovative Technology and Driver’s Resistance and Acceptance Intention for Sustainable Use of Automobile Self-Driving System

The purpose of this study was to empirically analyze the relationship between the characteristics of innovative technology, innovation resistance, and acceptance intention of automobile autonomous driving systems by applying the technology acceptance model (TAM), a representative innovative technology in the era of the 4th Industrial Revolution. A survey was conducted on drivers living in Seoul, Korea, and the following main results were derived based on the survey data for 567 people. First, the perceived usefulness and perceived ease of use of the autonomous driving system were found to have a significant negative effect on innovation resistance, but the perceived risk factors did not have a significant effect on innovation resistance. Second, it was found that the driver’s innovation resistance to the autonomous driving system had a significant negative effect on the acceptance intention. Third, the perceived risk of the autonomous driving system directly had a significant negative effect on the acceptance intention, but the perceived usefulness and perceived ease of use did not directly significantly affect the acceptance intention. Through this study, it was confirmed that resistance to innovative technologies can play an important role in enhancing the acceptance intention for sustainable use of automobile autonomous driving systems by drivers.

Explainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges

This survey reviews explainability methods for vision-based self-driving systems trained with behavior cloning. The concept of explainability has several facets and the need for explainability is strong in driving, a safety-critical application. Gathering contributions from several research fields, namely computer vision, deep learning, autonomous driving, explainable AI (X-AI), this survey tackles several points. First, it discusses definitions, context, and motivation for gaining more interpretability and explainability from self-driving systems, as well as the challenges that are specific to this application. Second, methods providing explanations to a black-box self-driving system in a post-hoc fashion are comprehensively organized and detailed. Third, approaches from the literature that aim at building more interpretable self-driving systems by design are presented and discussed in detail. Finally, remaining open-challenges and potential future research directions are identified and examined.

Ramifications of incorrect image segmentations; emphasizing on the potential effects on deep learning methods failure

IntroductionDetecting failure cases is critical to ensure a secure self-driving system. Any flaw in the system directly results in an accident. In genuine class, the model’s probability reflects better-reflected model confidence. As a result, the confidence distributions of failed predictions were changed to lower values. In contrast, accurate predictions were remained associated with high values, allowing for considerably more excellent separability between such prediction types. The study investigates the association of ramifications with computational color constancy that can negatively influence CNN’s image classification and semantic segmentation.MethodologyImage datasets were used to conduct different scales and complexity experiments. For instance, minimal and straightforward images of digits were comparatively provided through MNIST and SVHN datasets. The dataset’s standard validation set was employed to test and compute additional metrics because ground truth that is not publicly available for some test sets.ResultsThe results depicted that baseline methods were outperformed through the proposed approach with a considerable variant on minimal datasets or models in every context. Therefore, Transmission Control Protocol (TCP) is appropriate in failure prediction, and ConfidNet is competent to be fulfilled as confidence criterion. Further, one of the solutions would be to elevate the validation set size, but this would influence the prediction performance of a failure model. On the contrary, the confidence estimation was based on models with test predictive performance levels, similar to baselines.ConclusionsThe gap between validation accuracy and training accuracy was significant on CIFAR-100, which indicates the modest enhancement for failure detection via the validation set.

Significance of low-level control to string stability under adaptive cruise control: Algorithms, theory and experiments

Commercial adaptive cruise control (ACC) systems are bi-level: an upper-level planner decides the target trajectory and the low-level system executes it. Existing literature on ACCs mostly focus on the planner algorithms or the actuator delay, while the transition process between them, e.g. the low-level control design and its impact are often ignored. This paper tries to fill this gap by digging into the codebase of a recent open-source self-driving system, Openpilot (OP), Comma.ai, from which we extract and formulate the algorithms at both the upper and lower levels. For linear ACCs, the paper extends the transfer function analysis from planners only to full control loops and investigates the impact of slow/fast low-level control on the overall string stability (SS). For MPC ACCs, it studies their planning characteristics based its optimization objectives and approximates the low-level impact using an ODE approach.We find that low-level control has a significant impact on the overall SS of ACCs: (i) slow low-level control undermines SS under small frequencies and improves SS given large frequencies for linear systems, (ii) MPC features a varying gain throughout an oscillation, where the fast low-level control typically results in a ‘fast-slow’ changing process of the MPC gain, which benefits the SS, whereas the slow low-level control leads to a ‘slow-fast’ varying gain which undermines the SS, (iii) slow low-level control are common as they arise from comfort-oriented control gains, from a ”weak” actuator performance or both, and (iv) the SS is very sensitive to the integral gain under slow low-level control for both PI and PIF controllers. Overall, the study recommends fast low-level control for ensuring vehicular SS to reduce traffic congestion, considering that large congestion waves usually feature both small frequencies and large amplitudes, although slow controllers could perform even better provided a short and small leader perturbation. The findings of this paper are verified both numerically and experimentally. For the first time in the literature we implement custom ACC algorithms on market cars, and achieve SS on open roads with a random leader by only tuning the low-level controllers. The source code is shared at https://github.com/HaoZhouGT/openpilot to support on-road experiments of arbitrary car-following models, which may be of interest to other studies.

Intelligent Semantic Segmentation for Self-Driving Vehicles Using Deep Learning.

Understanding the situation is a critical component of any self-driving system. Accurate real-time visual signal processing to create pixelwise classed pictures, also known as semantic segmentation, is critical for scenario comprehension and subsequent acceptance of this new technology. Due to the intricate interaction between pixels in each frame of the received camera data, such efficiency in terms of processing time and accuracy could not be achieved prior to recent advances in deep learning algorithms. We present an effective approach for semantic segmentation for self-driving automobiles in this study. We combine deep learning architectures like convolutional neural networks and autoencoders, as well as cutting-edge approaches like feature pyramid networks and bottleneck residual blocks, to develop our model. The CamVid dataset, which has undergone considerable data augmentation, is utilised to train and test our model. To validate the suggested model, we compare the acquired findings to various baseline models reported in the literature.

A Decision-Making Model for Self-Driving Vehicles Based on Overtaking Frequency

The driving state of a self-driving vehicle represents an important component in the self-driving decision system. To ensure the safe and efficient driving state of a self-driving vehicle, the driving state of the self-driving vehicle needs to be evaluated quantitatively. In this paper, a driving state assessment method for the decision system of self-driving vehicles is proposed. First, a self-driving vehicle and surrounding vehicles are compared in terms of the overtaking frequency (OTF), and an OTF-based driving state evaluation algorithm is proposed considering the future driving efficiency. Next, a decision model based on the deep deterministic policy gradient (DDPG) algorithm and the proposed method is designed, and the driving state assessment method is integrated with the existing time-to-collision (TTC) and minimum safe distance. In addition, the reward function and multiple driving scenarios are designed so that the most efficient driving strategy at the current moment can be determined by optimal search under the condition of ensuring safety. Finally, the proposed decision model is verified by simulations in four three-lane highway scenarios. The simulation results show that the proposed decision model that integrates the self-driving vehicle driving state assessment method can help self-driving vehicles to drive safely and to maintain good maneuverability.

CoPace: Edge Computation Offloading and Caching for Self-Driving With Deep Reinforcement Learning

Currently, self-driving, emerging as a key automatic application, has brought a huge potential for the provision of in-vehicle services (e.g., automatic path planning) to mitigate urban traffic congestion and enhance travel safety. To provide high-quality vehicular services with stringent delay constraints, edge computing (EC) enables resource-hungry self-driving vehicles (SDVs) to offload computation-intensive tasks to the edge servers (ESs). In addition, caching highly reusable contents decreases the redundant transmission time and improves the quality of services (QoS) of SDVs, which is envisioned as a supplement to the computation offloading. However, the high mobility and time-varying requests of SDVs make it challenging to provide reliable offloading decisions while guaranteeing the resource utilization of content caching. To this end, in this paper we propose a <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">co</u> llaborative com <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</u> utation offlo <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a</u> ding and <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</u> ont <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</u> nt caching method, named CoPace, by leveraging deep reinforcement learning (DRL) in EC for self-driving system. Specifically, we first introduce OSTP to predict the future time-varying content popularity, taking into account the temporal-spatial attributes of requests. Moreover, a DRL-based algorithm is developed to jointly optimize the offloading and caching decisions, as well as the resource allocation (i.e., computing and communication resources) strategies. Extensive experiments with real-world datasets in Shanghai, China, are conducted to evaluate the performance, which demonstrates that CoPace is both effective and well-performed.

DBMind

We demonstrate a self-driving system DBMind, which provides three autonomous capabilities in database, including self-monitoring, self-diagnosis and self-optimization. First, self-monitoring judiciously collects database metrics and detects anomalies (e.g., slow queries and IO contention), which can profile database status while only slightly affecting system performance (&lt;5%). Then, self-diagnosis utilizes an LSTM model to analyze the root causes of the anomalies and automatically detect root causes from a pre-defined failure hierarchy. Next, self-optimization automatically optimizes the database performance using learning-based techniques, including deep reinforcement learning based knob tuning, reinforcement learning based index selection, and encoder-decoder based view selection. We have implemented DBMind in an open source database openGauss and demonstrated real scenarios.

How can we know a self-driving car is safe?

Self-driving cars promise solutions to some of the hazards of human driving but there are important questions about the safety of these new technologies. This paper takes a qualitative social science approach to the question ‘how safe is safe enough?’ Drawing on 50 interviews with people developing and researching self-driving cars, I describe two dominant narratives of safety. The first, safety-in-numbers, sees safety as a self-evident property of the technology and offers metrics in an attempt to reassure the public. The second approach, safety-by-design, starts with the challenge of safety assurance and sees the technology as intrinsically problematic. The first approach is concerned only with performance—what a self-driving system does. The second is also concerned with why systems do what they do and how they should be tested. Using insights from workshops with members of the public, I introduce a further concern that will define trustworthy self-driving cars: the intended and perceived purposes of a system. Engineers’ safety assurances will have their credibility tested in public. ‘How safe is safe enough?’ prompts further questions: ‘safe enough for what?’ and ‘safe enough for whom?’

Deep Learning Enhanced Driving Behavior Evaluation Based on Vehicle-Edge-Cloud Architecture

With the rapid development of 5 G, artificial intelligence and other technologies, the intelligent transportation system (ITS) bursts flourish fireworks. It is acknowledged that driving security problem still runs through the ITS development history.It is the driver who plays the decisive role in a vehicle accident, and the performance of autopilot system is also the kernel in guaranteeing the security of autonomous vehicle. Therefore, many researchers devote to self-driving system optimization and human abnormal driving behavior detection. Note that they either relied on simulators, or confined to several specific driving patterns, which undoubtedly limited their application value. In addition, some works require the vehicles have high computing power and abundant storage memory, which aggravated their burden. Different from previous works, we propose a driving behavior evaluation scheme based on vehicle-edge-cloud architecture. When vehicles running on the road, they transmit the data reflecting the autopilots/driver behaviors to the edge networks via the telematics box. The edge networks use the driving behavior evaluation model trained by cloud server, and send the behavior rankings back to vehicles. The cloud server continuously trains and optimizes the driving behavior evaluation model using vehicle data, and regularly transmits the model to the edge networks for upgrading. The experimental results show robustness and feasibility of the scheme.

Driving Assistance System with Lane Change Detection

In this study, a simple technology for a self-driving system called “driver assistance system” is developed based on embedded image identification. The system consists of a camera, a Raspberry Pi board, and OpenCV. The camera is used to capture lane images, and the image noise is overcome through color space conversion, grayscale, Otsu thresholding, binarization, erosion, and dilation. Subsequently, two horizontal lines parallel to the X-axis with a fixed range and interval are used to detect left and right lane lines. The intersection points between the left and right lane lines and the two horizontal lines can be obtained, and can be used to calculate the slopes of the left and right lanes. Finally, the slope change of the left and right lanes and the offset of the lane intersection are determined to detect the deviation. When the angle of lanes changes drastically, the driver receives a deviation warning. The results of this study suggest that the proposed algorithm is 1.96 times faster than the conventional algorithm.

Multi-task Learning of Semantics, Geometry and Motion for Vision-based End-to-End Self-Driving

&lt;div class="section abstract"&gt;&lt;div class="htmlview paragraph"&gt;It’s hard to achieve complete self-driving using hand-crafting generalized decision-making rules, while the end-to-end self-driving system is low in complexity, does not require hand-crafting rules, and can deal with complex situations. Modular-based self-driving systems require multi-task fusion and high-precision maps, resulting in high system complexity and increased costs. In end-to-end self-driving, we usually only use camera to obtain scene status information, so image processing is very important. Numerous deep learning applications benefit from multi-task learning, as the multi-task learning can accelerate model training and improve accuracy with combine all tasks into one model, which reduces the amount of calculation and allows these systems to run in real-time. Therefore, the approach of obtaining rich scene state information based on multi-task learning is very attractive. In this paper, we propose an approach to multi-task learning for semantics, geometry and motion. The multi-task learning includes four tasks: semantics segmentation, instance segmentation, depth regression, optical flow estimation. Optical flow method is currently an important method of moving image analysis and it not only contains the motion information of the observed object, but also contains rich information about the three-dimensional structure of the scene. Through the above work we can get compressed information about semantics, distance estimation and action recognition. As we all know, self-driving based on deep learning requires a lot of data to train neural network models. However, the basic attributes of the end-to-end system determine that it cannot be trained and learning in real world, such as collision testing and traffic accidents during reinforcement learning. Therefore, integrating existing autonomous driving data sets for model training and generalizing them to virtual environments for application is very critical. Finally, we use the virtual scene constructed by CARLA to train and evaluate the end-to-end self-driving system.&lt;/div&gt;&lt;/div&gt;