Scenario-based Testing Research Articles

Structured validation of AI-based systems by virtual testing in simulated test scenarios

The growing relevance of artificial intelligence (AI) for technical systems offers significant potential for the realization and operation of autonomous systems in complex and potentially unknown environments. However, unlike classical solution approaches, the functionality of an AI system cannot be verified analytically, which is why data-driven approaches such as scenario-based testing are used. With the increasing complexity of the required functionality of the AI-based system, the quantity, and quality of the data needed for development and validation also increase. To meet this demand, data generated synthetically using simulation is increasingly being used. Compared to the acquisition of real-world reference data, simulation offers the major advantage that it can be configured to test specific scenarios of interest. This paper presents an architecture for the systematic generation of virtual test scenarios to establish synthetically generated test data as an integral part of the development and validation process for AI systems. Key aspects of this architecture are the consistent use of digital twins as virtual 1-to-1 replicas and a simulation infrastructure that enables the generation of training and validation data for AI-based systems in appropriate quantity, quality, and time. In particular, this paper focuses on the application of the architecture in the context of two use cases from different application domains.

Ad hoc HLA simulation model derived from a model-based traffic scenario

Modern highly automated and autonomous traffic systems and sub-systems require new approaches to test their functional safety in the context of validation and verification. One approach that has taken a leading role in current research is scenario-based testing. For various reasons, simulation is considered to be the most practicable solution for a wide range of test scenarios. However, this is where many existing simulation systems in research reach their limits. In order to be able to integrate the widest possible range of systems to be tested into the simulation, the use of co-simulation has proven to be particularly useful. In this work, the High-Level Architecture defined in the IEEE 1516-2010 standard is specifically addressed, and a concept is developed that establishes the foundation for the feasible use of scenario-based distributed co-simulation on its basis. The main challenge identified and addressed is the resolution of the double-sided dependency between scenario and simulation models. The solution was to fully automate the generation and instantiation of the simulation environment on the basis of a scenario instance. Finally, the developed concept was implemented as a prototype, and the resulting process for its use is presented here using an example scenario. Based on the experience gained during the creation of the concept and the prototype, the next steps for future work are outlined in conclusion.

Toward Unsupervised Test Scenario Extraction for Automated Driving Systems from Urban Naturalistic Road Traffic Data

Scenario-based testing is a promising approach to solve the challenge of proving the safe behavior of vehicles equipped with automated driving systems. Since an infinite number of concrete scenarios can theoretically occur in real-world road traffic, the extraction of scenarios relevant in terms of the safety-related behavior of these systems is a key aspect for their successful verification and validation. Therefore, a method for extracting multimodal urban traffic scenarios from naturalistic road traffic data in an unsupervised manner, minimizing the amount of (potentially biased) prior expert knowledge, is proposed. Rather than an (elaborate) rule-based assignment by extracting concrete scenarios into predefined functional scenarios, the presented method deploys an unsupervised machine learning pipeline. The approach allows exploring the unknown nature of the data and their interpretation as test scenarios that experts could not have anticipated. The method is evaluated for naturalistic road traffic data at urban intersections from the inD and the Silicon Valley Intersections datasets. For this purpose, it is analyzed with which clustering approach (K-Means, hierarchical clustering, and DBSCAN) the scenario extraction method performs best (referring to an elaborate rule-based implementation). Subsequently, using hierarchical clustering the results show both a jump in overall accuracy of around 20% when moving from 4 to 5 clusters and a saturation effect starting at 41 clusters with an overall accuracy of 84%. These observations can be a valuable contribution in the context of the trade-off between the number of functional scenarios (i.e., clustering accuracy) and testing effort. Possible reasons for the observed accuracy variations of different clusters, each with a fixed total number of given clusters, are discussed.

Falsifying Motion Plans of Autonomous Vehicles With Abstractly Specified Traffic Scenarios

Verifying the safety of autonomous vehicles is one of the major challenges towards their deployment on public roads due to the vast number of possible situations that can occur in traffic. Scenario-based testing has been proposed to reduce the number of required tests using catalogs of abstractly defined scenarios. However, when specifying test scenarios abstractly, there is still an infinite number of possible concrete scenarios that can be derived from a specification. Available computational resources can thus be easily exceeded when using inefficient strategies for deriving concrete scenarios. In this work, we propose a novel method that synthesizes concrete scenarios complying with abstract scenario specifications. We compute the set of compliant trajectories for surrounding traffic participants and only sample trajectories within those sets. Our synthesis integrates a falsification algorithm that searches for specified failures of the vehicle under test. Compared to existing work, we can efficiently find scenario concretizations, especially for complex maneuver specifications. By directly considering failure specifications during the scenario synthesis, we avoid executing irrelevant simulations that cannot possibly result in failures. Our approach is demonstrated in several scenarios using Monte Carlo tree search as a search algorithm.

Validation of personal protective equipment ensembles, incorporating powered air-purifying respirators protected from contamination, for the care of patients with high-consequence infectious diseases

The UK High-Consequence Infectious Diseases (HCID) Network of high-level isolation units provides care for patients with contact- or airborne-transmissible highly infectious and highly dangerous diseases. In most HCID units, the healthcare workers (HCWs) wear personal protective equipment (PPE) ensembles incorporating a powered air-purifying respirator (PAPR) for head and respiratory protection. Some PAPRs have components worn outside/over other PPE, necessitating decontamination of re-usable elements. Two alternative PAPRs, with all re-usable elements worn under PPE, were trialled in this study. To undertake scenario-based testing of PAPRs and PPE to determine usability, comfort and ability to remove contaminated PPE without personal cross-contamination. Trained healthcare volunteers (N=20) wearing PAPR/PPE ensembles were sprayed with ultraviolet fluorescent markers. They undertook exercises to mimic patient care, and subsequently, after doffing the contaminated PPE following an established protocol, any personal cross-contamination was visualized under ultraviolet light. Participants also completed a questionnaire to gauge how comfortable they found the PPE. The ensembles were tested under extreme 'worst case scenario' conditions, augmented by physical and manual dexterity tests. Participating volunteers considered the exercise to be beneficial in terms of training and PPE evaluation. Data obtained, including feedback from questionnaires and doffing buddy observations, supported evidence-based decisions on the PAPR/PPE ensemble to be adopted by the HCID Network. One cross-contamination event was recorded in the ensemble chosen; this could be attributed to doffing error, and could therefore be eliminated with further practice.

Performance Comparison of Three Rival AI-Powered Intelligent Trajectory Tracking Controllers for an Autonomous Delivery Van

Due to the advent of E-commerce businesses, delivery vehicle traffic has grown by 71% over the last two decades. Hence, the automation of vans has received special attention to improve road safety. This study compares the performance of a trajectory tracking controller powered by three rival AI-based techniques namely, Particle Swarm, Genetic Algorithm and Ant Colony. Through scenario-based testing, all three techniques are compared from the tracking performance, algorithm complexity, accuracy and precision, and simplicity in finding the optimal weighting factors of the lateral position error and heading angle error which define the desired lateral controller output. Taken individual performance, ant colony powered fuzzy tracking controller was superior in terms of computational time, accuracy, and precision whereas, poor in ease of programming and explainability.

Operational Design Domain-Driven Coverage for the Safety Argumentation of Automated Vehicles

Automated Vehicles aim to increase road safety as automated driving systems (ADS) take over the human driving task in the operational design domain (ODD), introducing severe challenges for safety validation. Pure driving over many kilometers to gather enough evidence for a safety argument is not feasible. Scenario-based testing is an approach to overcome this, but challenges like parameter discretization still prevail, hindering safety assurance. This work proposes contributions towards a traceable and efficient safety argumentation for ADS built upon ODD coverage. First, ODD coverage is thoroughly quantified across all scenario levels, assuming distribution functions’ availability for the scenario parameters. Secondly, a sampling method for n-dimensional scenario parameter distributions is proposed. The provided algorithms adapt an initial k-means clustering using pre-defined boundary conditions requiring significantly fewer scenarios. Furthermore, a risk metric for urban intersections is presented for scenario evaluation. The risk metric consists of two parts, scene prediction of traffic participants (TPs) and risk assessment. The scene prediction uses a manoeuvre-based motion model with a data-driven approach towards trajectory prediction, increasing the validity. For the risk assessment, a probabilistic risk prediction for the TPs is performed for each scenario scene. The risk metric shows a reasonable tradeoff between sensitivity and specificity, outperforming time-to-collision. These contributions are exemplarily applied at an intersection using a simplified setup for generating TPs and ego vehicle trajectories. The results indicate that an increased safety argumentation is enabled using the proposed methods alongside a coverage process, facilitating further research.

Study of Cross-Site Request Forgery on Web-Based Application: Exploitations and Preventions

Cross-Site Request Forgery (CSRF) is a prominent web exploit that continues to pose significant security risks, even on highly ranked websites. This research focuses on identifying the underlying vulnerability, understanding the techniques employed, and proposing effective preventive measures. The hybrid method which is systematic report review method and lab-based scenarios method is conducted to identify CSRF exploitation and prevention mechanisms. Using a scenario-based testing approach, this study investigated the cause and effect of CSRF attacks, uncovering numerous techniques for exploiting this vulnerability. Based on the report review, 35 CSRF vulnerability reports from the Hacker one Bug Bounty Platform are analyzed. The result shows there is a lack of awareness and implementation of primary defenses, resulting in a high violation of end-user data integrity. The result also extensively describes some of the profound causes and effects of CSRF attacks and uncovers a broad range of techniques for exploiting this vulnerability. Furthermore, this study provides actionable recommendations to enhance system security and raise awareness among developers and users. The findings from this research serve as a valuable resource for improving security practices and mitigating CSRF attacks.

Teaching Social Determinants of Health Through an Unfolding Case Study.

The impact of social determinants of health (SDOH) was developed to educate nursing students through the use of an unfolding case study. SDOH and population health are critical components of prelicensure nursing education. Unfolding case studies are a strategy to develop critical thinking and teach SDOH to nursing students. A model was used to develop the case study including a community assessment, which follows a male veteran and family through life events. Implementation of the unfolding case study took place over 3 consecutive semesters in a community health nursing course. This educational activity achieved standardized examination scores, which are intended to assess student preparedness for the National Council Licensure Examination (NCLEX), above the national averages. Student participation was above 90%. Unfolding case studies can present realistic scenarios that are useful to teach critical thinking. As the Next Generation NCLEX moves to scenario-based testing, unfolding case studies are a teaching strategy to prepare students.

Probabilistic Metamodels for an Efficient Characterization of Complex Driving Scenarios

To validate the safety of automated vehicles (AV), scenario-based testing aims to systematically describe driving scenarios an AV might encounter. In this process, continuous inputs such as velocities result in an infinite number of possible variations of a scenario. Thus, metamodels are used to perform analyses or to select specific variations for examination. However, despite the safety criticality of AV testing, metamodels are usually seen as a part of an overall approach, and their predictions are not questioned. This paper analyzes the predictive performance of Gaussian processes (GP), deep Gaussian processes, extra-trees, and Bayesian neural networks (BNN), considering four scenarios with 5 to 20 inputs. Building on this, an iterative approach is introduced and evaluated, which allows to efficiently select test cases for common analysis tasks. The results show that regarding predictive performance, the appropriate selection of test cases is more important than the choice of metamodels. However, the choice of metamodels remains crucial: Their great flexibility allows BNNs to benefit from large amounts of data and to model even the most complex scenarios. In contrast, less flexible models like GPs convince with higher reliability. Hence, relevant test cases are best explored using scalable virtual test setups and flexible models. Subsequently, more realistic test setups and more reliable models can be used for targeted testing and validation.

Safety Performance Boundary Identification of Highly Automated Vehicles: A Surrogate Model-Based Gradient Descent Searching Approach

Highly automated vehicles (HAVs) have been introduced to the transportation system for the purpose of providing safer mobility. Considering the expected long co-existence period of HAVs and human-driven vehicles (HDVs), the safety operation of HAVs interacting with HDVs needs to be verified. To achieve this, HAVs’ Operational Design Domain (ODD) needs to be identified under the scenario-based testing framework. In this study, a novel testing framework aiming at identifying the Safety performance boundary (SPB) is proposed, which assures the coverage of safety-critical scenarios and compatible with the black-box feature of HAV control algorithm. A surrogate model was utilized to approximate the safety performance of HAV, and a gradient descent searching algorithm was employed to accelerate the search for SPB. For empirical analyses, a three-vehicle following scenario was adopted and the Intelligent Driver Model (IDM) was tested as a case study. The results show that only 4% of the total scenarios are required to establish a reliable surrogate model. And the gradient descent algorithm was able to establish the SPB by identifying 97.42% of collision scenarios and only false alarming 0.29% of non-collision scenarios. Furthermore, the concept of safety tolerance was proposed to measure the possibilities of boundary scenarios dropping in safety performance. The applications of helping to construct ODD and compare different control algorithms were discussed. It shows that the IDM performs better than the Wiedemann 99 (W99) model with larger ODD.

A Survey on Data-Driven Scenario Generation for Automated Vehicle Testing

Automated driving is a promising tool for reducing traffic accidents. While some companies claim that many cutting-edge automated driving functions have been developed, how to evaluate the safety of automated vehicles remains an open question, which has become a crucial bottleneck. Scenario-based testing has been introduced to test automated vehicles, and much progress has been achieved. While data-driven and knowledge-based approaches are hot research topics, this survey is mainly about Data-Driven Scenario Generation (DDSG) for automated vehicle testing. Rather than describe the contributions of every study respectively, in this survey, methodologies from various studies are anatomized as solutions for several significant problems and compared with each other. This way, scholars and engineers can quickly find state-of-the-art approaches to the issues they might encounter. Furthermore, several critical challenges that might hinder DDSG are described, and responding solutions are presented at the end of this survey.

Effectiveness Analysis of Unwanted Fire Alarm Using Carbon Monoxide Sensors

A carbon monoxide sensor was applied with smoke detectors, which trigger unwanted fire alarms, to improve detection reliability. Standardized fire and unwanted fire tests according to UL 268 and scenario-based tests for unwanted fire alarms were performed to analyze the detection effectiveness in fire and unwanted fire situations. Studies on improving unwanted fire alarms using carbon monoxide sensors have been conducted in various fields. However, additional concentration measurements using carbon monoxide sensors in various fire and unwanted fire conditions should be performed for institutional approval of detection effectiveness. Further research is required to determine the carbon monoxide concentration that can clearly distinguish unwanted fires from actual fires.

Verification and Validation Methods for Decision-Making and Planning of Automated Vehicles: A Review

Verification and validation (V&V) hold a significant position in the research and development of automated vehicles (AVs). Current literature indicates that different V&V techniques have been implemented in the decision-making and planning (DMP) system to improve AVs' safety, comfort, and energy optimization. This paper aims to review a range of different V&V approaches for the DMP system of AVs and divides these approaches into three distinct categories: scenario-based testing, fault injection testing, and formal verification. Further, scenario-based testing is categorized into fundamental and advanced approaches based on the interaction between road users in generated scenarios. In this paper, six criteria are proposed to compare and evaluate the characteristics of V&V approaches, which could help researchers gain insight into the benefits and limitations of the reviewed approaches and assist with approach choices. Next, the DMP system is broken down into a hierarchy of modules, and the functional requirements of each module are deduced. The suitable approaches are matched to verify and validate each module aiming at their different functional requirements. Finally, the current challenges and future research directions are concluded.

Scenario-Based Test Automation for Highly Automated Vehicles: A Review and Paving the Way for Systematic Safety Assurance

Highly Automated Vehicles (HAVs) must undergo strict safety testing before being released to the public. Mileage-based on-road testing suffers from unaffordable time costs and high safety risks. Simulated scenario-based testing has been found to be a trustworthy alternative for testing HAVs’ built-in algorithms and functionalities. Test automation is typically used to generate target scenarios. This approach facilitates customized testing and avoids wasting time on simple and redundant scenarios. This study aims to review test automation methods and discuss how to accentuate their strengths rather than be trapped in their weaknesses under certain applicable conditions. According to their main purposes, we classify test automation methods into coverage-oriented, unsafe-scenario-oriented, and naturalistic-assessment-oriented categories. To further demonstrate the differences of these methods, we then design numerical experiment to compare the capabilities of seven test automation methods. Finally, we compile our observations to form a comprehensive guide for selecting test automation methods with different test requirements in mind.

Performance Evaluation Method for Automated Driving System in Logical Scenario

With the continuous improvement of automated driving technology, how to evaluate the performance of an automated driving system is attracting more and more attention. Meanwhile, with the creation of scenario-based test methods, the traditional evaluation index based on a single test can no longer meet the requirements of high-level safety verification for automated driving system, and the performance evaluation of such a system in logical scenarios will be the mainstream. Based on the scenario-based test method and Turing test theory, a performance evaluation method for an automated driving system in the whole parameter space of a logical scenario is proposed. The logical scenario parameter space is partitioned according to the risk degree of concrete scenario, and the evaluation process in different zones are determined. Subsequently, the anthropomorphic index in the safe zone and the collision-avoidance index in the danger zone are defined by comparing test results of human driving and ideal vehicle motion. Taking front vehicle low-speed and cut-out scenarios as examples, two automated driving algorithms are tested in the virtual environment, and the test results are evaluated both by the proposed method and by human observation. The results show that the results of the proposed method are consistent with the subjective feelings of humans; additionally, it can be applied to scenario-based tests and the verification process of an automated driving system.

AR Head-up Display Validation with Scenario-based and Keyword-driven Testing

AR Head-up Display Validation with Scenario-based and Keyword-driven Testing

Uses of Personal Health Records for Communication Among Colorectal Cancer Survivors, Caregivers, and Providers: Interview and Observational Study in a Human-Computer Interaction Laboratory.

BackgroundPersonal health records (PHRs) may be useful for patient self-management and participation in communication with their caregivers and health care providers. As each potential participant’s role is different, their perception of the best uses of a PHR may vary.ObjectiveThe perspectives of patients, caregivers, and providers were all evaluated concurrently in relation to a PHR developed for colorectal cancer (CRC) survivors.MethodsWe explored group perceptions of a CRC PHR prototype. Scenario-based testing across eight use cases, with semistructured follow-up interviews, was videotaped in a human-computer interaction laboratory with patients, caregivers, and health care providers. Providers included oncologists, gastroenterologists, and primary care physicians. Discrete observations underwent grounded theory visual affinity analysis to identify emergent themes.ResultsObservations fell into three major themes: the network (who should be granted access to the PHR by the patient), functions (helpful activities the PHR enabled), and implementation (how to adopt the PHR into workflow). Patients wanted physician access to their PHR, as well as family member access, especially when they lived at a distance. All groups noted the added value of linking the PHR to an electronic health record, self-tracking, self-management, and secure messaging. Patients and caregivers also saw information in the PHR as a useful memory tool given their visits to multiple doctors. Providers had reservations about patients viewing raw data, which they were not prepared to interpret or might be inaccurate; patients and caregivers did not express any reservations about having access to more information. Patients saw PHR communication functions as a potential tool for relationship building. Patients and caregivers valued the journal as a tool for reflection and delivery of emotional support. Providers felt the PHR would facilitate patient-physician communication but worried that sharing journal access would make the doctor-patient relationship less professional and had reservations about the time burden of reviewing. Strategies suggested for efficient adoption into workflow included team delegation. Establishment of parameters for patient uses and provider responses was perceived as good standard practice.ConclusionsPHR perceptions differed by role, with providers seeing the PHR as informational, while patients and caregivers viewed the tool as more relational. Personal health records should be linked to electronic health records for ease of use. Tailoring access, content, and implementation of the PHR is essential. Technology changes have the potential to change the nature of the patient-physician relationship. Patients and providers should establish shared expectations about the optimal use of the PHR and explore how emerging patient-centered technologies can be successfully implemented in modern medical practice to improve the relational quality of care.

The Inadequacy of Discrete Scenarios in Assessing Deep Neural Networks

Many recent approaches for automated driving (AD) functions currently include components relying on deep neural networks (DNNs). One approach in order to test AD functions is the scenario-based approach. This work formalizes and evaluates the parameter discretization process required in order to yield concrete scenarios for which an AD function can be tested. Using a common perception algorithm for camera images, a simulation case study is conducted for a simple static scenario containing one other vehicle. The results are analyzed with methods akin to those applied in the domain of computational fluid dynamics (CFD). The performance of the perception algorithm shows strong fluctuations even for small input changes and displays unpredictable outliers even at very small discretization samples. The convergence criteria as known from CFD fail, meaning that no parametrization is found which is sufficient for the validation of the perception component. Indeed, the results do not indicate consistent improvement with a finer discretization. These results agree well with theoretical attributes known for existing neural networks. However, the impact appears to be large even for the most basic scenario without malicious input. This indicates the necessity of directing more attention towards the parameter discretization process of the scenario-based testing approach to enable the safety argumentation of AD functions.

Traffic Scenarios for Automated Vehicle Testing: A Review of Description Languages and Systems

Testing and validation of the functionalities and safety of automated vehicles shifted from a distance-based to a scenario-based method in the past decade. A number of domain-specific languages and systems were developed to support scenario-based testing. The aim of this paper is to review and compare the features and characteristics of the major scenario description languages and systems (SDLS). Each of them is designed for different purposes and with different goals; therefore, they have their strengths and weaknesses. Their characteristics are highlighted with an example nontrivial traffic scenario that we designed. We also discuss some directions for further development and research of these SDLS.