Testing Of Autonomous Vehicles Research Articles

Comprehensive Survey of Autonomous Driving Systems and Advanced Technologies: Literature Review, Gap Analysis and Future Directions

Abstract: This article presents a literature review of numerous research papers focusing around the topic ‘Development of Autonomous Driving Vehicle System’, and related works. The advent of autonomous driving technology has revolutionized the automotive industry, promising enhanced safety, efficiency, and convenience on the road. A critical aspect of autonomous vehicle development is the assessment of their performance ina controlled and risk-free environment. Virtual environments, enabled by cutting-edge technologies, offer a dynamic plat- form for rigorous testing and training of autonomous vehicles. This paper aims to study and analyze different approaches and methodologies that are implemented for the development of Autonomous Driving Systems(ADS). Some of the primary aspects that are found common in most papers are Virtual Environment, pedestrians’ safety, privacy, security, real-time datapresentation, precise torque control, Reinforcement Learning, Deep Learning, Hardware-in-the-loop (HIL)- simulation, Model- in-the-loop (MIL)-simulation, etc. The sole idea of extracting valuable information from a virtual environment ensures a sense of safety as there are no humans involuntarily involved to take part in the development. However, the data extracted from the virtual environment must be highly accurate and reliable, as it will be trained and tested in real environments post deployment. Collision scenarios need to be carefully studied, for which relative positioning of vehicle and pedestrians should be taken into account, so as to examine their velocities, time to collision, and appropriately taking actions. Above all, it should be noted that the safety of human lives holds the highest priority, if a method suggests a high risk factor for a human life, then it should be either discarded, or improved

Pre-crash scenarios for safety testing of autonomous vehicles: A clustering method for in-depth crash data

Autonomous vehicles (AVs) provide an opportunity to enhance traffic safety. However, AVs market penetration is still restricted due to their safety concerns and dependability. For widespread adoption, it is crucial to thoroughly assess the safety response of AVs in various high-risk scenarios. To achieve this objective, a clustering method was used to construct typical testing scenarios based on the China In-depth Mobility Safety Study-Traffic Accident (CIMSS-TA) database. Initially, 222 car-to-powered two-wheelers (PTWs) crashes and 180 car-to-car crashes were reconstructed from CIMSS-TA database. Second, six variables were extracted and analyzed, including the motion of the two vehicles involved, relative movement, lighting condition, road condition, and visual obstruction. Third, these variables were clustered using the k-medoids algorithm, identifying five typical pre-crash scenarios for car-to-PTWs and seven for car-to-car. Additionally, we extracted the velocities and surrounding environmental information of the crash-involved parties to enrich the scenario description. The approach used in this study used in-depth case review and thus provided more insightful information for identifying and quantifying representative high-risk scenarios than prior studies that analyzed overall descriptive variables from Chinese crash databases. Furthermore, it is crucial to separately test car-to-car scenarios and car-to-PTWs scenarios due to their distinct motion characteristics, which significantly affect the resulting typical scenarios.

An entropy‐based model for quantifying multi‐dimensional traffic scenario complexity

AbstractQuantifying the complexity of traffic scenarios not only provides an essential foundation for constructing the scenarios used in autonomous vehicle training and testing, but also enhances the robustness of the resulting driving decisions and planning operations. However, currently available quantification methods suffer from inaccuracies and coarse‐granularity in complexity measurements due to issues such as insufficient specificity or indirect quantification. The present work addresses these challenges by proposing a comprehensive entropy‐based model for quantifying traffic scenario complexity across multiple dimensions based on a consideration of the essential components of the traffic environment, including traffic participants, static elements, and dynamic elements. In addition, the limitations of the classical information entropy models applied for assessing traffic scenarios are addressed by calculating magnitude entropy. The proposed entropy‐based model is analyzed in detail according to its application to simulated traffic scenarios. Moreover, the model is applied to real world data within a naturalistic driving dataset. Finally, the effectiveness of the proposed quantification model is illustrated by comparing the complexity results obtained for three typical traffic scenarios with those obtained using an existing multi‐factor complexity quantification method.

PEM: Perception Error Model for Virtual Testing of Autonomous Vehicles

PEM: Perception Error Model for Virtual Testing of Autonomous Vehicles

Beyond the Value for AI Adopters: Analyzing the Impacts of Autonomous Vehicle Testing on Traffic Conditions in California

Beyond the Value for AI Adopters: Analyzing the Impacts of Autonomous Vehicle Testing on Traffic Conditions in California

Study on the Safety Management of Connected and Autonomous Vehicle Test Roads Based on the Evaluation of Traffic Safety Facilities

More and more connected and autonomous vehicle (CAV) open test roads reconstructed on the basis of traditional roads have appeared in China. However, the management policies vary, which makes the traffic environment complicated. This paper takes CAV test road safety management as the research aim and investigates the open test condition through the evaluation of the traffic safety facilities. Indicators were rigorously screened, then the game theory model was used to determine the combination weight of the indicators, and the set pair analysis was applied to solve the uncertain problems. A case study for the CAV test road network of a city in central China was implemented and the results show, regarding the traffic safety facilities’ condition, among the 20 sections of the city’s CAV test road network, 15% of which are at an excellent level, 75% of which are at a good level and 10% of which are at a moderate level; road signs, guardrail facilities, isolation facilities and road features are the main limiting factors affecting the level of traffic safety facilities. Based on the results, recommendations have been made for the transport management authorities in the aspects of safety management policy-making and facilities maintenance.

Risk analysis of autonomous vehicle test scenarios using a novel analytic hierarchy process method

AbstractScenario‐based test methods are employed to assess the safety and performance of autonomous vehicles. The analytic hierarchy process (AHP) method is a common assessment method for determining the criticality of test scenarios. However, the AHP method is subjective and less reproducible when performed by different persons, as the elements of pairwise comparison values that are directly linked to the outcome must be assigned by the person involved. This paper proposes a novel AHP method that automatically generates pairwise comparison values by optimizing the correlation between performance metrics and risk of test scenarios by simulation. Performance metrics are defined as the minimum relative distances and corresponding relative velocities between vehicles, and the risk of the test scenario is determined by the pairwise comparison values of AHP. The novel AHP method was evaluated using a cut‐in scenario. The results showed that the minimum relative distance and the risk determined by the novel AHP method achieved a better correlation coefficient of −0.96, which is better than the conventional AHP of −0.828 and Fuzzy AHP of −0.824. These results suggest that the criticality of the test scenarios determined by the novel AHP method can more accurately reflect real‐world driving environments.

Qualification of the PAVIN Fog and Rain Platform and Its Digital Twin for the Evaluation of a Pedestrian Detector in Fog.

Vehicles featuring partially automated driving can now be certified within a guaranteed operational design domain. The verification in all kinds of scenarios, including fog, cannot be carried out in real conditions (risks or low occurrence). Simulation tools for adverse weather conditions (e.g., physical, numerical) must be implemented and validated. The aim of this study is, therefore, to verify what criteria need to be met to obtain sufficient data to test AI-based pedestrian detection algorithms. It presents both analyses on real and numerically simulated data. A novel method for the test environment evaluation, based on a reference detection algorithm, was set up. The following parameters are taken into account in this study: weather conditions, pedestrian variety, the distance of pedestrians to the camera, fog uncertainty, the number of frames, and artificial fog vs. numerically simulated fog. Across all examined elements, the disparity between results derived from real and simulated data is less than 10%. The results obtained provide a basis for validating and improving standards dedicated to the testing and approval of autonomous vehicles.

Autonomous Vehicles Testing Considering Utility-Based Operable Tasks

Virtual simulation testing of Autonomous Vehicles (AVs) is gradually being accepted as a mandatory way to test the feasibility of driving strategies for AVs. Mainstream methods focus on improving testing efficiency by extracting critical scenarios from naturalistic driving datasets. However, the criticalities defined in their testing tasks are based on fixed assumptions, the obtained scenarios cannot pose a challenge to AVs with different strategies. To fill this gap, we propose an intelligent testing method based on operable testing tasks. We found that the driving behavior of Surrounding Vehicles (SVs) has a critical impact on AV, which can be used to adjust the testing task difficulty to find more challenging scenarios. To model different driving behaviors, we utilize behavioral utility functions with binary driving strategies. Further, we construct a vehicle interaction model, based on which we theoretically analyze the impact of changing the driving behaviors on the testing task difficulty. Finally, by adjusting SV's strategies, we can generate more corner cases when testing different AVs in a finite number of simulations.

시나리오 분석 및 위험도 평가에 의한 자율주행자동차 검증방법

In this study, we propose an autonomous vehicle safety validation method to address the limitations of existing vehicle safety validation methods. Using scenario analysis and risk assessment of the autonomous vehicle safety validation scenario, the proposed method determines the significance of scenario validation, difficulty of scenario implementation, and critical nature of accidents that occur during validation testing. Then, it calculates the autonomous vehicle test level based on these results. Various safety validation tests, such as audits, simulation tests, proving ground tests, and real road tests, were performed according to the calculated level. Based on the validation results, the proposed method enables authorities to request improvement in the original equipment manufacturer address areas and is expected to contribute to providing safe autonomous vehicles to consumers.

Rethinking engagement in urban design: reimagining the value of co-design and participation at every stage of planning for autonomous vehicles

PurposeThe practical demonstrations and research which led to the preparation of this paper involved a combination of stakeholder engagement, policy debate and the practical demonstration and testing of autonomous vehicles. By adhering to a design approach which in centred on participation and human-centred engagement, the advent of autonomous vehicles might avoid many of the problems encountered in relation to conventional transport.Design/methodology/approachThe research explored how a new and potentially disruptive technology might be incorporated in urban settings, through the lens of participation and problem-based design. The research critically reviews key strands in the literature (autonomous vehicles, social research and participatory design), with allusion to current case study experiments.FindingsAlthough there are numerous examples of autonomous vehicles (AV) research concentrating on technical aspects alone, this paper finds that such an approach appears to be an unusual starting point for the design of innovative technology. That is, AVs would appear to hold the potential to be genuinely disruptive in terms of innovation, yet the way that disruption takes place should surely be guided by design principles and by issues and problems encountered by potential users.Practical implicationsThe research carries significant implications for practice in that it advocates locating those socio-contextual issues at the heart of the problem definition and design process and ahead of technical solutions.Originality/valueWhat sets this research apart from other studies concerning AVs was that the starting point for investigation was the framing of AVs within contexts and scenarios leading to the emergence of wicked problems. This begins with a research position where the potential uses for AVs are considered in a social context, within which the problems and issues to be solved become the starting point for design at a fundamental level.

Decision-Making Model for Dynamic Scenario Vehicles in Autonomous Driving Simulations

A scenario vehicle in autonomous driving simulations is a dynamic entity that is expected to perform trustworthy bidirectional interaction tasks with the autonomous vehicle under test. Modeling interactive behavior can not only facilitate better prediction of human drivers’ intentions and motions but also be valuable in generating more human-like decisions and trajectories for autonomous vehicle testing. However, simulations of most of the available scenario vehicles on existing platforms behave conservatively. This study summarizes five driving motivations based on human-need theories of multiple psychologists, namely safety, dominance, achievement, order, and relatedness, and organizes the framework using a behavior tree. The proposed model generates different driving behaviors by simulating the changing psychological needs of human drivers during vehicle operation. Using a self-developed two-dimensional simulator, experiments were conducted by considering multiple scenarios in urban, rural, and highway road sections. The obtained results indicate that the scenario vehicles controlled by the proposed model exhibit a significant interactive nature, facilitating proactive communication rather than providing simple responses.

Overweight and overload implementation in Apollo systems

Apollo is a high-performance, flexible architecture developed by Baidu, Kinglong, and a consortium of more than 40 companies for the purpose of accelerating the development, testing, and deployment of Autonomous Vehicles. This article first describes Apollo in terms of the functionality of the system and the interaction between its components, and then analyzes the conceptual architecture through system evolution, control and data flow, and concurrency, while illustrating the impact of the division of developer responsibilities on this. Then will go through a detailed overview of the concrete architecture of Baidu Apollo with working on mapping the source code from the Apollo Github Website to our architecture using the diagram drawing tool Scitools Understand. And divided into five parts, the report concludes with five major parts: the process from mapping code to diagram; concrete architecture explanation; unexpected dependencies discovery; subsystem analysis, and sequence diagrams presented in a more accurate and specific way. Finally, this report will propose a specific feature or enhancement to the current concrete architecture. The impact that this feature or enhancement can have on the non-functional requirements and stakeholders of the system will be explored based on the existing architecture and component interactions. In this regard, this report discusses two specific approaches that can be implemented and compares the two approaches through SAAM analysis to determine the best approach.

An Evaluation Method for Automated Vehicles Combining Subjective and Objective Factors

The rapid development of automated vehicle technology requires reasonable test scenarios and comprehensive evaluation methods. This paper proposes an evaluation method for automated vehicles combining subjective and objective factors. First, we propose a method for automatically generating test scenarios and for batch testing autonomous vehicles. Then, the use of the target layer, total index layer, and index layer of automated vehicles is proposed to establish a more comprehensive evaluation system for automated vehicles. Specifically, the analytic hierarchy process (AHP, subjective) and improved criteria importance though intercriteria correlation (CRITIC, objective) methods are used to determine the weight of the indicators, and a two-level fuzzy comprehensive (subjective and objective) evaluation method is adopted to comprehensively evaluate the performance of the automated vehicles. Finally, the effectiveness of the proposed evaluation method combining subjective and objective factors is verified through virtual simulations and real-world experiments. Through a combination of subjective and objective methods, improved results can be obtained for safety, efficiency, economy, intelligence, and comfort tests.

Model vs system level testing of autonomous driving systems: a replication and extension study

Offline model-level testing of autonomous driving software is much cheaper, faster, and diversified than in-field, online system-level testing. Hence, researchers have compared empirically model-level vs system-level testing using driving simulators. They reported the general usefulness of simulators at reproducing the same conditions experienced in-field, but also some inadequacy of model-level testing at exposing failures that are observable only in online mode. In this work, we replicate the reference study on model vs system-level testing of autonomous vehicles while acknowledging several assumptions that we had reconsidered. These assumptions are related to several threats to validity affecting the original study that motivated additional analysis and the development of techniques to mitigate them. Moreover, we also extend the replicated study by evaluating the original findings when considering a physical, radio-controlled autonomous vehicle. Our results show that simulator-based testing of autonomous driving systems yields predictions that are close to the ones of real-world datasets when using neural-based translation to mitigate the reality gap induced by the simulation platform. On the other hand, model-level testing failures are in line with those experienced at the system level, both in simulated and physical environments, when considering the pre-failure site, similar-looking images, and accurate labels.

Learning naturalistic driving environment with statistical realism

For simulation to be an effective tool for the development and testing of autonomous vehicles, the simulator must be able to produce realistic safety-critical scenarios with distribution-level accuracy. However, due to the high dimensionality of real-world driving environments and the rarity of long-tail safety-critical events, how to achieve statistical realism in simulation is a long-standing problem. In this paper, we develop NeuralNDE, a deep learning-based framework to learn multi-agent interaction behavior from vehicle trajectory data, and propose a conflict critic model and a safety mapping network to refine the generation process of safety-critical events, following real-world occurring frequencies and patterns. The results show that NeuralNDE can achieve both accurate safety-critical driving statistics (e.g., crash rate/type/severity and near-miss statistics, etc.) and normal driving statistics (e.g., vehicle speed/distance/yielding behavior distributions, etc.), as demonstrated in the simulation of urban driving environments. To the best of our knowledge, this is the first time that a simulation model can reproduce the real-world driving environment with statistical realism, particularly for safety-critical situations.

Dense reinforcement learning for safety validation of autonomous vehicles.

One critical bottleneck that impedes the development and deployment of autonomous vehicles is the prohibitively high economic and time costs required to validate their safety in a naturalistic driving environment, owing to the rarity of safety-critical events1. Here we report the development of an intelligent testing environment, where artificial-intelligence-based background agents are trained to validate the safety performances of autonomous vehicles in an accelerated mode, without loss of unbiasedness. From naturalistic driving data, the background agents learn what adversarial manoeuvre to executethrough a dense deep-reinforcement-learning (D2RL) approach, in which Markov decision processes are edited by removing non-safety-critical states and reconnecting critical ones so that the information in the training data is densified. D2RL enables neural networks to learn from densified information with safety-critical events and achieves tasks that are intractable for traditional deep-reinforcement-learning approaches. We demonstrate the effectiveness of our approach by testing a highly automated vehicle in both highway and urban test tracks with an augmented-reality environment, combining simulated background vehicles with physical road infrastructure anda real autonomous test vehicle. Our results show that the D2RL-trained agents can accelerate the evaluation process by multiple orders of magnitude (103 to 105 times faster). In addition, D2RL will enable accelerated testing and training with other safety-critical autonomous systems.

Impact of Autonomous Vehicles on the Car-Following Behavior of Human Drivers

The past few years have been witness to an increase in autonomous vehicle (AV) development and testing. However, even with a fully developed and comprehensively tested AV technology, AVs are anticipated to share the roadway network with human drivers for the unforeseeable future. In such a mixed environment, we use naturalistic driving data from the next generation simulation (NGSIM) and Lyft Level 5 (Lyft L5) prediction data sets to investigate whether the existence of AVs influences the car-following behavior of human drivers. We use time headway time series as a proxy to capture the car-following behavior of human drivers. We then develop a nested fixed model to find possible changes in behavior when human drivers are following different types of vehicles (i.e., human-driven vehicles or AVs). The factors included in this model are the platoon structure (a legacy vehicle following a legacy vehicle, and a legacy vehicle following an autonomous vehicle), road type (freeway and urban), time period (morning and afternoon), lane (right, middle, and left), and the source of the data (NGSIM and Lyft L5). Results indicate a statistically significant difference between the car-following behavior of drivers when they follow a human-driven vehicle compared to an AV. This change in the car-following behavior of drivers is manifested in the form of a reduction in the mean and variance of time headways when human drivers follow an AV. These findings can bridge the gap between anticipated and real-world impacts of AVs on traffic streams and roadway capacity, providing meaningful insights on the influence of AVs on the driving behavior of humans in a naturalistic driving environment.

Dynamic Vehicle Modelling and Controlling Techniques for Autonomous Vehicle Systems

The driving scenario of an automated vehicle is the crucial technology in the design of autonomous cars. This suggested approach aims to address the shortcomings of autonomous cars, such as their poor real- time performance and low control precision. The process for building a virtual simulation environment for autonomous vehicle testing and validation is described in this study. Model Predictive Control and Proportional Integral and Derivative Control are used in MATLAB simulation to build three car models. These are related to the 2D and 3D animation used in collision detection and visualization. The virtual engine visualization is included throughout the model. A variety of test circumstances are used to validate the simulation model, and the model’s performance is assessed in the presence of various barriers. The simulation's findings demonstrate that the autonomous vehicle has a strong potential for self-adaptation even in challenging and complex working environments. No instances of car sideslip or track departure have been noted. It is discovered that this autonomous car performs remarkably well overall when compared to other autonomous vehicles. The suggested approach is essential for enhancing autonomous vehicle driving safety, maintaining vehicle control in challenging situations, and improving the advancement of intelligent vehicle driving assistance.

Multiple Autonomous Surface Vehicles for Autonomous Cooperative Navigation Tasks in a Marine Environment: Development and Preliminary Field Tests

This paper presents the development and field testing of multiple autonomous surface vehicles (ASVs) for autonomous cooperative navigation tasks in marine environments. The developed ASVs were designed in a catamaran hull form, and various systems related to autonomous functionalities, including the electrical propulsion, wireless communication, and guidance, navigation, and control (GNC) systems, were integrated with the situational awareness sensor system. Because not all of the navigation and situational awareness sensor data can be shared owing to the large network load, the network communication and data protocol structure was designed to enhance the data management efficiency. Furthermore, for cooperative navigation between multiple ASVs, the network communication system was synchronously shared with motion information through a robot operating system (ROS) multi-master system. In particular, an autonomy framework with cooperative navigation and control was designed to enable autonomous cooperative maneuvering with multiple ASVs in a global navigation satellite system (GNSS)-denied environment. Furthermore, a cooperative navigation and control approach based on relative geometric information between vehicles was implemented in the framework, which incorporated the capabilities of following a predefined path and maintaining a specific formation based on the relative distance between ASVs. Moreover, to demonstrate the essential maneuvering and functional capabilities of the cooperative navigation and control approach of the developed ASVs, preliminary field tests were conducted in an inland water environment, and the test results are discussed in this paper.