Testing Of Autonomous Vehicles Research Articles

CornerSim: A Virtualization Framework to Generate Realistic Corner-Case Scenarios for Autonomous Driving Perception Testing

Autonomous driving development requires rigorous testing in real-world scenarios, including adverse weather, unpredictable events, object variations, and sensor limitations. However, these challenging “corner cases” are elusive in conventional datasets due to their unpredictability, high costs, and inherent risks. Recognizing the critical role of ground truth data in autonomous driving, the demand for synthetic data becomes evident. Contemporary machine learning-based algorithms essential to autonomous vehicles heavily depend on labeled data for training and validation. Simulation of scenarios not only mitigates the scarcity of real-world data but also facilitates controlled experimentation in situations that are challenging to replicate physically. The challenge extends beyond data scarcity, encompassing the impediment posed by the inability to systematically control and manipulate specific scenarios, hindering progress.To overcome these challenges, we present CornerSim, a dynamic virtualization framework simplifying the creation and modification of diverse driving scenarios. Leveraging simulation, CornerSim generates synthetic environments for comprehensive testing, providing essential outputs like raw sensor data (cameras, LiDAR, etc.) and labeled data (object detection bounding boxes, classes, semantic segmentation). The unpredictable nature of real-world corner cases complicates obtaining a sufficiently large and diverse annotated dataset. CornerSim addresses this challenge by not only generating synthetic data but also supplying necessary ground truth for training and evaluating machine learning models.This paper emphasizes the introduction of CornerSim and its ability to challenges related to testing autonomous vehicles in realistic scenarios. It focuses on the framework’s capabilities, design principles, and integration, with the goal of enhancing thorough testing and validation of autonomous driving systems in a simulated environment, improving their robustness and safety. Our approach involves running simulations to generate datasets, which are statistically studied and compared with real data. Furthermore, we apply state-of-the-art detection algorithms to assess if data generated by CornerSim is suitable for both training and validation stages.

Study on the Safety Management of Connected and Autonomous Vehicle Test Roads Based on the Evaluation of Traffic Safety Facilities

More and more connected and autonomous vehicle (CAV) open test roads reconstructed on the basis of traditional roads have appeared in China. However, the management policies vary, which makes the traffic environment complicated. This paper takes CAV test road safety management as the research aim and investigates the open test condition through the evaluation of the traffic safety facilities. Indicators were rigorously screened, then the game theory model was used to determine the combination weight of the indicators, and the set pair analysis was applied to solve the uncertain problems. A case study for the CAV test road network of a city in central China was implemented and the results show, regarding the traffic safety facilities’ condition, among the 20 sections of the city’s CAV test road network, 15% of which are at an excellent level, 75% of which are at a good level and 10% of which are at a moderate level; road signs, guardrail facilities, isolation facilities and road features are the main limiting factors affecting the level of traffic safety facilities. Based on the results, recommendations have been made for the transport management authorities in the aspects of safety management policy-making and facilities maintenance.

Risk analysis of autonomous vehicle test scenarios using a novel analytic hierarchy process method

AbstractScenario‐based test methods are employed to assess the safety and performance of autonomous vehicles. The analytic hierarchy process (AHP) method is a common assessment method for determining the criticality of test scenarios. However, the AHP method is subjective and less reproducible when performed by different persons, as the elements of pairwise comparison values that are directly linked to the outcome must be assigned by the person involved. This paper proposes a novel AHP method that automatically generates pairwise comparison values by optimizing the correlation between performance metrics and risk of test scenarios by simulation. Performance metrics are defined as the minimum relative distances and corresponding relative velocities between vehicles, and the risk of the test scenario is determined by the pairwise comparison values of AHP. The novel AHP method was evaluated using a cut‐in scenario. The results showed that the minimum relative distance and the risk determined by the novel AHP method achieved a better correlation coefficient of −0.96, which is better than the conventional AHP of −0.828 and Fuzzy AHP of −0.824. These results suggest that the criticality of the test scenarios determined by the novel AHP method can more accurately reflect real‐world driving environments.

Qualification of the PAVIN Fog and Rain Platform and Its Digital Twin for the Evaluation of a Pedestrian Detector in Fog.

Vehicles featuring partially automated driving can now be certified within a guaranteed operational design domain. The verification in all kinds of scenarios, including fog, cannot be carried out in real conditions (risks or low occurrence). Simulation tools for adverse weather conditions (e.g., physical, numerical) must be implemented and validated. The aim of this study is, therefore, to verify what criteria need to be met to obtain sufficient data to test AI-based pedestrian detection algorithms. It presents both analyses on real and numerically simulated data. A novel method for the test environment evaluation, based on a reference detection algorithm, was set up. The following parameters are taken into account in this study: weather conditions, pedestrian variety, the distance of pedestrians to the camera, fog uncertainty, the number of frames, and artificial fog vs. numerically simulated fog. Across all examined elements, the disparity between results derived from real and simulated data is less than 10%. The results obtained provide a basis for validating and improving standards dedicated to the testing and approval of autonomous vehicles.

Autonomous Vehicles Testing Considering Utility-Based Operable Tasks

Virtual simulation testing of Autonomous Vehicles (AVs) is gradually being accepted as a mandatory way to test the feasibility of driving strategies for AVs. Mainstream methods focus on improving testing efficiency by extracting critical scenarios from naturalistic driving datasets. However, the criticalities defined in their testing tasks are based on fixed assumptions, the obtained scenarios cannot pose a challenge to AVs with different strategies. To fill this gap, we propose an intelligent testing method based on operable testing tasks. We found that the driving behavior of Surrounding Vehicles (SVs) has a critical impact on AV, which can be used to adjust the testing task difficulty to find more challenging scenarios. To model different driving behaviors, we utilize behavioral utility functions with binary driving strategies. Further, we construct a vehicle interaction model, based on which we theoretically analyze the impact of changing the driving behaviors on the testing task difficulty. Finally, by adjusting SV's strategies, we can generate more corner cases when testing different AVs in a finite number of simulations.

시나리오 분석 및 위험도 평가에 의한 자율주행자동차 검증방법

In this study, we propose an autonomous vehicle safety validation method to address the limitations of existing vehicle safety validation methods. Using scenario analysis and risk assessment of the autonomous vehicle safety validation scenario, the proposed method determines the significance of scenario validation, difficulty of scenario implementation, and critical nature of accidents that occur during validation testing. Then, it calculates the autonomous vehicle test level based on these results. Various safety validation tests, such as audits, simulation tests, proving ground tests, and real road tests, were performed according to the calculated level. Based on the validation results, the proposed method enables authorities to request improvement in the original equipment manufacturer address areas and is expected to contribute to providing safe autonomous vehicles to consumers.

Rethinking engagement in urban design: reimagining the value of co-design and participation at every stage of planning for autonomous vehicles

PurposeThe practical demonstrations and research which led to the preparation of this paper involved a combination of stakeholder engagement, policy debate and the practical demonstration and testing of autonomous vehicles. By adhering to a design approach which in centred on participation and human-centred engagement, the advent of autonomous vehicles might avoid many of the problems encountered in relation to conventional transport.Design/methodology/approachThe research explored how a new and potentially disruptive technology might be incorporated in urban settings, through the lens of participation and problem-based design. The research critically reviews key strands in the literature (autonomous vehicles, social research and participatory design), with allusion to current case study experiments.FindingsAlthough there are numerous examples of autonomous vehicles (AV) research concentrating on technical aspects alone, this paper finds that such an approach appears to be an unusual starting point for the design of innovative technology. That is, AVs would appear to hold the potential to be genuinely disruptive in terms of innovation, yet the way that disruption takes place should surely be guided by design principles and by issues and problems encountered by potential users.Practical implicationsThe research carries significant implications for practice in that it advocates locating those socio-contextual issues at the heart of the problem definition and design process and ahead of technical solutions.Originality/valueWhat sets this research apart from other studies concerning AVs was that the starting point for investigation was the framing of AVs within contexts and scenarios leading to the emergence of wicked problems. This begins with a research position where the potential uses for AVs are considered in a social context, within which the problems and issues to be solved become the starting point for design at a fundamental level.

Decision-Making Model for Dynamic Scenario Vehicles in Autonomous Driving Simulations

A scenario vehicle in autonomous driving simulations is a dynamic entity that is expected to perform trustworthy bidirectional interaction tasks with the autonomous vehicle under test. Modeling interactive behavior can not only facilitate better prediction of human drivers’ intentions and motions but also be valuable in generating more human-like decisions and trajectories for autonomous vehicle testing. However, simulations of most of the available scenario vehicles on existing platforms behave conservatively. This study summarizes five driving motivations based on human-need theories of multiple psychologists, namely safety, dominance, achievement, order, and relatedness, and organizes the framework using a behavior tree. The proposed model generates different driving behaviors by simulating the changing psychological needs of human drivers during vehicle operation. Using a self-developed two-dimensional simulator, experiments were conducted by considering multiple scenarios in urban, rural, and highway road sections. The obtained results indicate that the scenario vehicles controlled by the proposed model exhibit a significant interactive nature, facilitating proactive communication rather than providing simple responses.

Overweight and overload implementation in Apollo systems

Apollo is a high-performance, flexible architecture developed by Baidu, Kinglong, and a consortium of more than 40 companies for the purpose of accelerating the development, testing, and deployment of Autonomous Vehicles. This article first describes Apollo in terms of the functionality of the system and the interaction between its components, and then analyzes the conceptual architecture through system evolution, control and data flow, and concurrency, while illustrating the impact of the division of developer responsibilities on this. Then will go through a detailed overview of the concrete architecture of Baidu Apollo with working on mapping the source code from the Apollo Github Website to our architecture using the diagram drawing tool Scitools Understand. And divided into five parts, the report concludes with five major parts: the process from mapping code to diagram; concrete architecture explanation; unexpected dependencies discovery; subsystem analysis, and sequence diagrams presented in a more accurate and specific way. Finally, this report will propose a specific feature or enhancement to the current concrete architecture. The impact that this feature or enhancement can have on the non-functional requirements and stakeholders of the system will be explored based on the existing architecture and component interactions. In this regard, this report discusses two specific approaches that can be implemented and compares the two approaches through SAAM analysis to determine the best approach.

An Evaluation Method for Automated Vehicles Combining Subjective and Objective Factors

The rapid development of automated vehicle technology requires reasonable test scenarios and comprehensive evaluation methods. This paper proposes an evaluation method for automated vehicles combining subjective and objective factors. First, we propose a method for automatically generating test scenarios and for batch testing autonomous vehicles. Then, the use of the target layer, total index layer, and index layer of automated vehicles is proposed to establish a more comprehensive evaluation system for automated vehicles. Specifically, the analytic hierarchy process (AHP, subjective) and improved criteria importance though intercriteria correlation (CRITIC, objective) methods are used to determine the weight of the indicators, and a two-level fuzzy comprehensive (subjective and objective) evaluation method is adopted to comprehensively evaluate the performance of the automated vehicles. Finally, the effectiveness of the proposed evaluation method combining subjective and objective factors is verified through virtual simulations and real-world experiments. Through a combination of subjective and objective methods, improved results can be obtained for safety, efficiency, economy, intelligence, and comfort tests.

Model vs system level testing of autonomous driving systems: a replication and extension study

Offline model-level testing of autonomous driving software is much cheaper, faster, and diversified than in-field, online system-level testing. Hence, researchers have compared empirically model-level vs system-level testing using driving simulators. They reported the general usefulness of simulators at reproducing the same conditions experienced in-field, but also some inadequacy of model-level testing at exposing failures that are observable only in online mode. In this work, we replicate the reference study on model vs system-level testing of autonomous vehicles while acknowledging several assumptions that we had reconsidered. These assumptions are related to several threats to validity affecting the original study that motivated additional analysis and the development of techniques to mitigate them. Moreover, we also extend the replicated study by evaluating the original findings when considering a physical, radio-controlled autonomous vehicle. Our results show that simulator-based testing of autonomous driving systems yields predictions that are close to the ones of real-world datasets when using neural-based translation to mitigate the reality gap induced by the simulation platform. On the other hand, model-level testing failures are in line with those experienced at the system level, both in simulated and physical environments, when considering the pre-failure site, similar-looking images, and accurate labels.

Learning naturalistic driving environment with statistical realism

For simulation to be an effective tool for the development and testing of autonomous vehicles, the simulator must be able to produce realistic safety-critical scenarios with distribution-level accuracy. However, due to the high dimensionality of real-world driving environments and the rarity of long-tail safety-critical events, how to achieve statistical realism in simulation is a long-standing problem. In this paper, we develop NeuralNDE, a deep learning-based framework to learn multi-agent interaction behavior from vehicle trajectory data, and propose a conflict critic model and a safety mapping network to refine the generation process of safety-critical events, following real-world occurring frequencies and patterns. The results show that NeuralNDE can achieve both accurate safety-critical driving statistics (e.g., crash rate/type/severity and near-miss statistics, etc.) and normal driving statistics (e.g., vehicle speed/distance/yielding behavior distributions, etc.), as demonstrated in the simulation of urban driving environments. To the best of our knowledge, this is the first time that a simulation model can reproduce the real-world driving environment with statistical realism, particularly for safety-critical situations.

Dense reinforcement learning for safety validation of autonomous vehicles.

One critical bottleneck that impedes the development and deployment of autonomous vehicles is the prohibitively high economic and time costs required to validate their safety in a naturalistic driving environment, owing to the rarity of safety-critical events1. Here we report the development of an intelligent testing environment, where artificial-intelligence-based background agents are trained to validate the safety performances of autonomous vehicles in an accelerated mode, without loss of unbiasedness. From naturalistic driving data, the background agents learn what adversarial manoeuvre to executethrough a dense deep-reinforcement-learning (D2RL) approach, in which Markov decision processes are edited by removing non-safety-critical states and reconnecting critical ones so that the information in the training data is densified. D2RL enables neural networks to learn from densified information with safety-critical events and achieves tasks that are intractable for traditional deep-reinforcement-learning approaches. We demonstrate the effectiveness of our approach by testing a highly automated vehicle in both highway and urban test tracks with an augmented-reality environment, combining simulated background vehicles with physical road infrastructure anda real autonomous test vehicle. Our results show that the D2RL-trained agents can accelerate the evaluation process by multiple orders of magnitude (103 to 105 times faster). In addition, D2RL will enable accelerated testing and training with other safety-critical autonomous systems.

Impact of Autonomous Vehicles on the Car-Following Behavior of Human Drivers

The past few years have been witness to an increase in autonomous vehicle (AV) development and testing. However, even with a fully developed and comprehensively tested AV technology, AVs are anticipated to share the roadway network with human drivers for the unforeseeable future. In such a mixed environment, we use naturalistic driving data from the next generation simulation (NGSIM) and Lyft Level 5 (Lyft L5) prediction data sets to investigate whether the existence of AVs influences the car-following behavior of human drivers. We use time headway time series as a proxy to capture the car-following behavior of human drivers. We then develop a nested fixed model to find possible changes in behavior when human drivers are following different types of vehicles (i.e., human-driven vehicles or AVs). The factors included in this model are the platoon structure (a legacy vehicle following a legacy vehicle, and a legacy vehicle following an autonomous vehicle), road type (freeway and urban), time period (morning and afternoon), lane (right, middle, and left), and the source of the data (NGSIM and Lyft L5). Results indicate a statistically significant difference between the car-following behavior of drivers when they follow a human-driven vehicle compared to an AV. This change in the car-following behavior of drivers is manifested in the form of a reduction in the mean and variance of time headways when human drivers follow an AV. These findings can bridge the gap between anticipated and real-world impacts of AVs on traffic streams and roadway capacity, providing meaningful insights on the influence of AVs on the driving behavior of humans in a naturalistic driving environment.

Dynamic Vehicle Modelling and Controlling Techniques for Autonomous Vehicle Systems

The driving scenario of an automated vehicle is the crucial technology in the design of autonomous cars. This suggested approach aims to address the shortcomings of autonomous cars, such as their poor real- time performance and low control precision. The process for building a virtual simulation environment for autonomous vehicle testing and validation is described in this study. Model Predictive Control and Proportional Integral and Derivative Control are used in MATLAB simulation to build three car models. These are related to the 2D and 3D animation used in collision detection and visualization. The virtual engine visualization is included throughout the model. A variety of test circumstances are used to validate the simulation model, and the model’s performance is assessed in the presence of various barriers. The simulation's findings demonstrate that the autonomous vehicle has a strong potential for self-adaptation even in challenging and complex working environments. No instances of car sideslip or track departure have been noted. It is discovered that this autonomous car performs remarkably well overall when compared to other autonomous vehicles. The suggested approach is essential for enhancing autonomous vehicle driving safety, maintaining vehicle control in challenging situations, and improving the advancement of intelligent vehicle driving assistance.

Multiple Autonomous Surface Vehicles for Autonomous Cooperative Navigation Tasks in a Marine Environment: Development and Preliminary Field Tests

This paper presents the development and field testing of multiple autonomous surface vehicles (ASVs) for autonomous cooperative navigation tasks in marine environments. The developed ASVs were designed in a catamaran hull form, and various systems related to autonomous functionalities, including the electrical propulsion, wireless communication, and guidance, navigation, and control (GNC) systems, were integrated with the situational awareness sensor system. Because not all of the navigation and situational awareness sensor data can be shared owing to the large network load, the network communication and data protocol structure was designed to enhance the data management efficiency. Furthermore, for cooperative navigation between multiple ASVs, the network communication system was synchronously shared with motion information through a robot operating system (ROS) multi-master system. In particular, an autonomy framework with cooperative navigation and control was designed to enable autonomous cooperative maneuvering with multiple ASVs in a global navigation satellite system (GNSS)-denied environment. Furthermore, a cooperative navigation and control approach based on relative geometric information between vehicles was implemented in the framework, which incorporated the capabilities of following a predefined path and maintaining a specific formation based on the relative distance between ASVs. Moreover, to demonstrate the essential maneuvering and functional capabilities of the cooperative navigation and control approach of the developed ASVs, preliminary field tests were conducted in an inland water environment, and the test results are discussed in this paper.

OVERVIEW OF TEST INFRASTRUCTURE INVESTMENTS ON OPEN ROADS TO TEST CONNECTED AND AUTOMATED VEHICLES

Open road testing is an essential stage of the process of approving driver assistance and other self-driving functions. However several countries have opened their network with or without limitations for autonomous vehicle testing, there are some sections where the infrastructure was designed to support testing. The aim of this article is to give an overview of such open road testing facilities around the world, introducing the main purposes, the used infrastructure (eg. sensor network, communication) and features (eg. data processing, and control functions), and compare them to the planned investments in Hungary.

Weight Adaptive Path Tracking Control for Autonomous Vehicles Based on PSO-BP Neural Network.

In order to improve the tracking adaptability of autonomous vehicles under different vehicle speeds and road curvature, this paper develops a weight adaptive model prediction control system (AMPC) based on PSO-BP neural network, which consists of a dynamics-based model prediction controller (MPC) and an optimal weight adaptive regulator. Based on the application of MPC to achieve high-precision tracking control, the optimal weight under different operating conditions obtained by automated simulation is used to train the PSO-BP neural network offline to achieve online adjustment of MPC weight. The validation results of the Prescan-Carsim-Simulink joint simulation platform show that the adaptive control system has better tracking adaptation capability compared with the original classical MPC control. The control strategy was also verified on an autonomous vehicle test platform, and the test results showed that the adaptive control strategy improved tracking accuracy while meeting the vehicle's requirements for real-time control and lateral stability.

AdaSplats: Adaptive Splatting of Point Clouds for Accurate 3D Modeling and Real-Time High-Fidelity LiDAR Simulation

LiDAR sensors provide rich 3D information about their surroundings and are becoming increasingly important for autonomous vehicles tasks such as localization, semantic segmentation, object detection, and tracking. Simulation accelerates the testing, validation, and deployment of autonomous vehicles while also reducing cost and eliminating the risks of testing in real-world scenarios. We address the problem of high-fidelity LiDAR simulation and present a pipeline that leverages real-world point clouds acquired by mobile mapping systems. Point-based geometry representations, more specifically splats (2D oriented disks with normals), have proven their ability to accurately model the underlying surface in large point clouds, mainly with uniform density. We introduce an adaptive splat generation method that accurately models the underlying 3D geometry to handle real-world point clouds with variable densities, especially for thin structures. Moreover, we introduce a fast LiDAR sensor simulator, working in the splatted model, that leverages the GPU parallel architecture with an acceleration structure while focusing on efficiently handling large point clouds. We test our LiDAR simulation in real-world conditions, showing qualitative and quantitative results compared to basic splatting and meshing techniques, demonstrating the interest of our modeling technique.

Disengagement Cause-and-Effect Relationships Extraction Using an NLP Pipeline

The advancement in machine learning and artificial intelligence promotes the testing and deployment of autonomous vehicles (AVs) on public roads. The California Department of Motor Vehicles (CA DMV) has launched the Autonomous Vehicle Tester Program, which collects and releases reports related to Autonomous Vehicle Disengagement (AVD) from autonomous driving. Understanding the causes of AVD is critical to improving the AV system’s safety and stability and providing guidance for AV testing and deployment. In this work, we built a scalable end-to-end pipeline to collect, process, model, and analyze the disengagement reports released from 2014 to 2020 using natural language processing and deep transfer learning. The analysis of disengagement data using taxonomy, visualization, and statistical tests revealed the trends of AV testing, cause frequency, and significant relationships between causes and effects of AVD. We found that (1) manufacturers tested AVs intensively during the Spring and/or Winter, (2) test drivers initiated more than 80% of the disengagement while more than 75% of the disengagement were because of errors in perception, localization & mapping, planning and control of the AV system, and (3) there was a significant relationship between the initiator of AVD and the cause category. This study serves as a successful practice of deep transfer learning using pre-trained models and generates a consolidated disengagement database allowing further investigation for other researchers. The related code and data are available on github. <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</xref>