Scenario-based Testing Research Articles

FutureWare: Designing a Middleware for Anticipatory Mobile Computing

Ubiquitous computing is moving from context-awareness to context-prediction. In order to build truly anticipatory systems developers have to deal with many challenges, from multimodal sensing to modeling context from sensed data, and, when necessary, coordinating multiple predictive models across devices. Novel expressive programming interfaces and paradigms are needed for this new class of mobile and ubiquitous applications. In this paper we present FutureWare, a middleware for seamless development of mobile applications that rely on context prediction. FutureWare exposes an expressive API to lift the burden of mobile sensing, individual and group behavior modeling, and future context querying, from an application developer. We implement FutureWare as an Android library, and through a scenario-based testing and a demo app we show that it represents an efficient way of supporting anticipatory applications, reducing the necessary coding effort by two orders of magnitude.

Design and Implementation of an Ontology for Semantic Labeling and Testing: Automotive Global Ontology (AGO)

Modern Artificial Intelligence (AI) methods can produce a large quantity of accurate and richly described data, in domains such as surveillance or automation. As a result, the need to organize data at a large scale in a semantic structure has arisen for long-term data maintenance and consumption. Ontologies and graph databases have gained popularity as mechanisms to satisfy this need. Ontologies provide the means to formally structure descriptive and semantic relations of a domain. Graph databases allow efficient and well-adapted storage, manipulation, and consumption of these linked data resources. However, at present, there is no a universally defined strategy for building AI-oriented ontologies for the automotive sector. One of the key challenges is the lack of a global standardized vocabulary. Most private initiatives and large open datasets for Advanced Driver Assistance Systems (ADASs) and Autonomous Driving (AD) development include their own definitions of terms, with incompatible taxonomies and structures, thus resulting in a well-known lack of interoperability. This paper presents the Automotive Global Ontology (AGO) as a Knowledge Organization System (KOS) using a graph database (Neo4j). Two different use cases for the AGO domain ontology are presented to showcase its capabilities in terms of semantic labeling and scenario-based testing. The ontology and related material have been made public for their subsequent use by the industry and academic communities.

Supporting deprescribing in hospitalised patients: formative usability testing of a computerised decision support tool

BackgroundDespite growing evidence that deprescribing can improve clinical outcomes, quality of life and reduce the likelihood of adverse drug events, the practice is not widespread, particularly in hospital settings. Clinical risk assessment tools, like the Drug Burden Index (DBI), can help prioritise patients for medication review and prioritise medications to deprescribe, but are not integrated within routine care. The aim of this study was to conduct formative usability testing of a computerised decision support (CDS) tool, based on DBI, to identify modifications required to the tool prior to trialling in practice.MethodsOur CDS tool comprised a DBI MPage in the electronic medical record (clinical workspace) that facilitated review of a patient’s DBI and medication list, access to deprescribing resources, and the ability to deprescribe. Two rounds of scenario-based formative usability testing with think-aloud protocol were used. Seventeen end-users participated in the testing, including junior and senior doctors, and pharmacists.ResultsParticipants expressed positive views about the DBI CDS tool but testing revealed a number of clear areas for improvement. These primarily related to terminology used (i.e. what is a DBI and how is it calculated?), and consistency of functionality and display. A key finding was that users wanted the CDS tool to look and function in a similar way to other decision support tools in the electronic medical record. Modifications were made to the CDS tool in response to user feedback.ConclusionUsability testing proved extremely useful for identifying components of our CDS tool that were confusing, difficult to locate or to understand. We recommend usability testing be adopted prior to implementation of any digital health intervention. We hope our revised CDS tool equips clinicians with the knowledge and confidence to consider discontinuation of inappropriate medications in routine care of hospitalised patients. In the next phase of our project, we plan to pilot test the tool in practice to evaluate its uptake and effectiveness in supporting deprescribing in routine hospital care.

Dynamic driving environment complexity quantification method and its verification

To meet the requirements of scenario-based testing for Autonomous Vehicles (AVs), driving scenario characterization has become a critical issue. Existing studies have concluded that complexity is a necessary criticality measure for supporting critical AV testing scenario identification. However, the existing scenario complexity quantification studies mainly have two limitations, namely, subjective quantification methods highly rely on human participations and are difficult to apply to big data, and existing objective methods lack consideration of human driver characteristics and hence the performance cannot be guaranteed. To bridge these research gaps, a general objective quantification framework is proposed to quantify human drivers’ judgement on driving environment complexity, by describing the vehicle-vehicle spatial-temporal interactions from the perspectives of quantity, variety, and relations. The model mainly contains three parts. First, to describe quantity information, a dynamic influencing area was set to identify the surrounding vehicles that contribute to driving environment complexity based on Responsibility-Sensitive Safety (RSS) theory. Second, considering the various surrounding vehicles’ driving statuses, behaviors, and intentions, a basic vehicle-pair complexity quantification model was constructed based on encounter angles. Then, nonlinear relationships based upon information entropy theory were introduced to capture the heterogeneous longitudinal and lateral complexities. Third, a vehicle-pair complexity aggregation and smoothing step was conducted to reflect the characteristics of human driver's cognition. To demonstrate the abovementioned model, empirical Field Operational Test (FOT) data from Shanghai urban roadways were used to conduct case studies, and it can be concluded that this model can accurately describe the timing and extent of the complexity change, and reveal the complexity differences due to scenario type and spatial-temporal heterogeneity. Besides, Inter-Rater Reliability (IRR) index was calculated to validate the consistency of scenario complexity judgement between the proposed model and human drivers, and for performance comparison with the existing models. Finally, the applications of the proposed model and its further investigations have been discussed.

Non-deterministic model validation methodology for simulation-based safety assessment of automated vehicles

Safeguarding and type approval of automated vehicles is a key enabler for their market launch in our complex traffic environment. Scenario-based testing by means of computer simulation is becoming increasingly important to cope with the enormous complexity and effort. However, there is a huge gap when assessing the safety of the virtual vehicle while the real vehicle will drive on the road. Simulation must be accompanied by model validation to ensure its credibility since errors and uncertainties are inherent in every model. Unfortunately, this is rarely addressed in the current literature. In this paper, a modular process is presented covering both model validation and safeguarding. It is characterized by the fact that it quantifies a large number of errors and uncertainties, represents them in the form of an error model, and ultimately integrates them into the safeguarding results. It is applied to a type-approval regulation for the lane-keeping behavior of a vehicle under various scenario conditions. The paper contains a thorough validation of the methodology itself by comparing its results with actual ground truth values. For this comparison, a binary classifier and confusion matrices are used that relate the binary type-approval decisions. The classifier demonstrates that the methodology of this paper identifies a systematic error of the simulation model across several safeguarding scenarios. Finally, the paper provides recommendations for alternative configurations of the modular methodology depending on different requirements.

Scenario Optimisation and Sensitivity Analysis for Safe Automated Driving Using Gaussian Processes

Assuring the safety of automated vehicles is essential for their timely introduction and acceptance by policymakers and the public. To assess their safe design and robust decision making in response to all possible scenarios, new methods that use a scenario-based testing approach are needed, as testing on public roads in normal traffic would require driving millions of kilometres. We make use of the scenario-based testing approach and propose a method to model simulated scenarios using Gaussian Process based models to predict untested scenario outcomes. This enables us to efficiently determine the performance boundary, where the safe and unsafe scenarios can be evidently distinguished from each other. We present an iterative method that optimises the parameter space of a logical scenario towards the most critical scenarios on this performance boundary. Additionally, we conduct a novel probabilistic sensitivity analysis by efficiently computing several variance-based sensitivity indices using the Gaussian Process models and evaluate the relative importance of the scenario input parameters on the scenario outcome. We critically evaluate and investigate the usefulness of the proposed Gaussian Process based approach as a very efficient surrogate model, which can model the logical scenarios effectively in the presence of uncertainty. The proposed approach is applied on an exemplary logical scenario and shows viability in finding concrete critical scenarios. The reported results, derived from the proposed approach, could pave the way to more efficient testing of automated vehicles and instruct further physical tests on the determined critical scenarios.

Fault Injection in Model-Based System Failure Analysis of Highly Automated Vehicles

The active safety control systems of highly automated vehicles for SAE level 3 and higher are still not fully developed and facing some unresolved issues. The deployment of automated driving systems and the functional safety development present challenges in driver – machine control relationship when there is a system failure or malfunction. The current definition of the product development and controllability classes of the road vehicles functional safety (ISO26262) are not feasible in highly automated vehicles (HAV). This research developed an overview of fault or disturbance injection on the steering system of highly automated model to study the impact of steering system sensors malfunction. The approach was to study the fault propagation using a model-based engineering development in a virtual environment of MATLAB. Subsequently, the steering control system of automated vehicle was developed using an adaptive model predictive control structure to study the control system sensors failures on a system-feature level of the vehicle. It was concluded that the steering wheel angle sensor failure has a significant impact on the planned trajectory of the vehicle and thus it was classified as ASIL D, which represents the highest critical safety component and requires comprehensive safety mechanisms to meet the safety goals of the system. The study also introduced a new criterion for controllability classes suitable for highly automated systems based on the global vehicle position relative of the lane marker lines, to deal with the active safety systems and risk handling strategies. The drivers – vehicle control systems are changing significantly in SAE level 3 automated vehicle and above that driving functions are controlled by the vehicle control systems. This presents human factors challenge in this interactive system with moving to SAE levels 4 and 5. Hence, several human machine interfaces and scenario-based testing are introduced to mitigate any risk or safety uncertainty resulting from control handing-over between the driver and the vehicle control system.

Test Scenarios Construction Based on Combinatorial Testing Strategy for Automated Vehicles

Scenario-based testing is an important verification and certification measure to evaluate the safety of automated vehicles. In view of the existing test scenario composition methods, which may miss some critical scenario problems that have low occurrence probability, we fully combined the ego-vehicle with the possible relative positions and movement directions of surrounding traffic participants based on a complex scenario group. We applied scenario-screening rules to obtain the functional test scenarios with different traffic environments and driving task complexities, which ensured the coverage of the test scenarios and reduced the number of test scenarios. The problem arose that the amount of test cases was too large after the discretized combination of test scenario parameters, so we adopted a three-way combinatorial testing strategy to greatly reduce the number of test cases. Taking the complicated lane changing scenario of the ego-vehicle as an example, the simulation method was adopted, and the critical test cases were obtained by screening through safety indicators. Finally, the K-medoids clustering method was used to further reduce the number of critical test cases, and a pairwise combinatorial test strategy was used to combine dynamic scenario and static scenario elements to obtain critical test cases for closed-road testing.

Online Verification Enabling Approval of Driving Functions—Implementation for a Planner of an Autonomous Race Vehicle

Safety guarantees and regulatory approval for autonomous vehicles remain an ongoing challenge. In particular, software that is frequently adapted or contains complex, non-transparent components, such as artificial intelligence, is exceeding the limits of safety standards. This paper presents a detailed implementation of an online verification module – the Supervisor – that copes with these challenges. The presented implementation focuses on autonomous race vehicles without loss of generality. Following an identified holistic list of safety-relevant requirements for a trajectory, metrics are developed to monitor whether the trajectory can safely be executed. To evaluate safety with respect to dynamic objects in a semi-structured and highly dynamic racing environment, rule-based reachable sets are presented. As a result, the pure reachable set is further constrained by applicable regulations. Real-time capability and effectiveness are demonstrated in fault-injected scenario-based tests and on real-world run data. The implemented Supervisor will be publicly available on GitHub.

Model Validation and Scenario Selection for Virtual-Based Homologation of Automated Vehicles

Due to the rapid progress in the development of automated vehicles over the last decade, their market entry is getting closer. One of the remaining challenges is the safety assessment and type approval of automated vehicles, as conventional testing in the real world would involve an unmanageable mileage. Scenario-based testing using simulation is a promising candidate for overcoming this approval trap. Although the research community has recognized the importance of safeguarding in recent years, the quality of simulation models is rarely taken into account. Without investigating the errors and uncertainties of models, virtual statements about vehicle safety are meaningless. This paper describes a whole process combining model validation and safety assessment. It is demonstrated by means of an actual type-approval regulation that deals with the safety assessment of lane-keeping systems. Based on a thorough analysis of the current state-of-the-art, this paper introduces two approaches for selecting test scenarios. While the model validation scenarios are planned from scratch and focus on scenario coverage, the type-approval scenarios are extracted from measurement data based on a data-driven pipeline. The deviations between lane-keeping behavior in the real and virtual world are quantified using a statistical validation metric. They are then modeled using a regression technique and inferred from the validation experiments to the unseen virtual type-approval scenarios. Finally, this paper examines safety-critical lane crossings, taking into account the modeling errors. It demonstrates the potential of the virtual-based safeguarding process using exemplary simulations and real driving tests.

User-centered design of a longitudinal care plan for children with medical complexity

To determine the content priorities and design preferences for a longitudinal care plan (LCP) among caregivers and healthcare providers who care for children with medical complexity (CMC) in acute care settings. We conducted iterative one-on-one design sessions with CMC caregivers (ie, parents/legal guardians) and providers from 5 groups: complex care, primary care, subspecialists, emergency care, and care coordinators. Audio-recorded sessions included content categorization activities, drawing exercises, and scenario-based testing of an electronic LCP prototype. We applied inductive content analysis of session materials to elicit content priorities and design preferences between sessions. Analysis informed iterative prototype revisions. We conducted 30 design sessions (10 with caregivers, 20 with providers). Caregivers expressed high within-group variability in their content priorities compared to provider groups. Emergency providers had the most unique content priorities among clinicians. We identified 6 key design preferences: a familiar yet customizable layout, a problem-based organization schema, linked content between sections, a table layout for most sections, a balance between unstructured and structured data fields, and use of family-centered terminology. Findings from this study will inform enhancements of electronic health record-embedded LCPs and the development of new LCP tools and applications. The design preferences we identified provide a framework for optimizing integration of family and provider content priorities while maintaining a user-tailored experience. Health information platforms that incorporate these design preferences into electronic LCPs will help meet the information needs of caregivers and providers caring for CMC in acute care settings.

Mining and comparative analysis of typical pre-crash scenarios from IGLAD

Scenario-based testing is crucial for considering the intended functional safety of automated driving vehicles. For the first time, pre-crash scenario mining research was conducted using worldwide accident data obtained from the Initiative for the Global Harmonization of Accident Data (IGLAD). First, data from the IGLAD database were analyzed and divided into four categories based on differences in traffic environments among countries and regions. Second, according to actual accident characteristics, fields and methods of clustering were selected, and 21 typical pre-crash scenarios were obtained using clustering and analysis. Finally, the typical scenarios were analyzed and compared in detail.Four conclusions were drawn as follows: 1. Considerable differences exist in traffic participant types, accident forms, and typical scenarios across countries and regions. 2. The third group of countries (3-G, represented by China and Brazil) in which accidents and pre-crash scenarios are the most representative and diverse is an ideal data source for the international scenario research. 3. The typical scenarios mined through clustering were highly consistent with the new test scenarios added in the Euro-NCAP 2025 Roadmap, but a few typical scenario elements which are critical for safety evaluations were still not covered in Roadmap. 4. Data from the IGLAD database still lacks a few important pieces of information for scenario research, such as obstruction of visual field due to obstacles, and the data representativeness need to be improved, therefore we recommend that IGLAD database adds some new data parameters to fit the further scenario research, and propose distribution requirements of accident data considering scenario elements. The analysis methods and conclusions presented used in this study could serve as guidelines or references for automated vehicle safety evaluations.

Low-Cost Urban Test Scenario Generation Using Microscopic Traffic Simulation

Scenario-based testing is already a well-known test approach to the automotive industry for the Validation, Verification and Testing of Connected and Automated Vehicles. How to construct the scenarios of complex traffic environment is a major challenge to this method. A common approach for generating scenarios is collecting and post-processing natural traffic data with a lot of time and money costs. In order to reduce the cost of scenario collection, we propose a low-cost method based on Microscopic Traffic Simulation to obtain a large number of urban traffic scenarios. Based on public data, we establish a microscopic traffic model for a specific area within the Shenzhen urban. Through a simulation, the 24-hour traffic behavior of vehicles for this area is simulated. About 189,752 scenarios covering the entire travel process are generated, which is equivalent to the data collected by a well-equipped car traveling 254,480 kms or 8,288 h. Our scenarios include not only typical urban scenarios such as U-turn and Parallel-driving (two vehicles driving in parallel on a single lane), but also collision accident scenarios of various forms. In addition, in order to evaluate the risks faced by Connected and Automated Vehicles in different scenarios, we design a new criticality metric, Scenario Risk Index, based on the risk assessment principle. The Scenario Risk Index has nothing to do with a certain function, and can quantitatively comprehensively evaluate the criticality and loss of potential accidents.

Stress Testing Method for Scenario-Based Testing of Automated Driving Systems

Classical approaches and procedures for testing of automated vehicles of SAE levels 1 and 2 were based on defined scenarios with specific maneuvers, depending on the function under test. For automated driving systems (ADS) of SAE level 3+, the scenario space is infinite and calling for virtual testing and verification. However, even in simulation, the generation of safety-relevant scenarios for ADS is expensive and time-consuming. This leads to a demand for stochastic and realistic traffic simulation. Therefore, microscopic traffic flow simulation models (TFSM) are becoming a crucial part of scenario-based testing of ADS. In this paper, a co-simulation between the multi-body simulation software IPG CarMaker and the microscopic traffic flow simulation software (TFSS) PTV Vissim is used. Although the TFSS could provide realistic and stochastic behavior of the traffic participants, safety-critical scenarios (SCS) occur rarely. In order to avoid this, a novel Stress Testing Method (STM) is introduced. With this method, traffic participants are manipulated via external driver DLL interface from PTV Vissim in the vicinity of the vehicle under test in order to provoke defined critical maneuvers derived from statistical accident data on highways in Austria. These external driver models imitate human driving errors, resulting in an increase of safety-critical scenarios. As a result, the presented STM method contributes to an increase of safety-relevant scenarios for verification, testing and assessment of ADS.

Functional decomposition—A contribution to overcome the parameter space explosion during validation of highly automated driving

Objective: Particular testing by functional decomposition of the automated driving function can potentially contribute to reducing the effort of validating highly automated driving functions. In this study, the required size of test suites for scenario-based testing and the potential to reduce it by functional decomposition are quantified for the first time.Methods: The required size of test suites for scenario-based approval of a so-called Autobahn-Chauffeur (SAE Level 3) is analyzed for an exemplary set of scenarios. Based on studies of data from failure analyses in other domains, the possible range for the required test coverage is narrowed down and suitable discretization steps, as well as ranges for the influence parameters, are assumed. Based on those assumptions, the size of the test suites for testing the complete system is quantified. The effects that lead to a reduction in the parameter space for particular testing of the decomposed driving function are analyzed and the potential to reduce the validation effort is estimated by comparing the resulting test suite sizes for both methods.Results: The combination of all effects leads to a reduction in the test suites’ size by a factor between 20 and 130, depending on the required test coverage. This means that the size of the required test suite can be reduced by 95–99% by particular testing compared to scenario-based testing of the complete system.Conclusions: The reduction potential is a valuable contribution to overcome the parameter space explosion during the validation of highly automated driving. However, this study is based on assumptions and only a small set of exemplary scenarios. Thus, the findings have to be validated in further studies.

Solving interpersonal problems when fatigued: The beneficial effect of positive mood

I examined factors that inhibit and augment effective problem solving in social contexts. I tested the joint effect of mental fatigue and mood on interpersonal problem solving. Utilizing scenario-based tests, I found that individuals experiencing high (vs. low) mental fatigue generated less effective solutions to social problems, and identified fewer relevant steps to solve these problems. I also found that the experience of positive emotion served to counteract these fatigue-induced deficits, bolstering mental resources needed for productive social problem solving. My results also have practical relevance, as the extent to which individuals can generate effective strategies to overcome interpersonal problems has far-reaching consequences with respect to personal adjustment and the ability to enact socially skilled behavior.

American Board of Orthodontics: Time for change.

The American Board of Orthodontics (ABO) works to certify orthodontists in a fair, reliable, and valid manner. The process must examine an orthodontist's knowledge, abilities, and critical thinking skills to ensure that each certified orthodontist has the expertise to provide the highest level of patient care. Many medical specialty boards and 4 American Dental Association specialty boards use scenario-based testing for board certification. Changing to a scenario-based clinical examination will allow the ABO to test more orthodontists. The new process will not result in an easier examination; standards will not be lowered. It will offer an improved testing method that will be fair, valid, and reliable for the specialty of orthodontics while increasing accessibility and complementing residency curricula. The ABO's written examination will remain as it is.

User-Centered Evaluations with Older Adults: Testing the Usability of a Mobile Health System for Heart Failure Self-Management.

Many older adults living with heart failure struggle to follow recommended self-management routines. To help older adults with heart failure more effectively and efficiently self-manage their disease, we developed Engage, a mobile health application promoting the performance, logging, and sharing of routine self-management behaviors. This paper reports on the usability evaluation of the Engage system with 15 older adults with heart failure and informal caregivers. In two phases, participants used Engage during a task-based usability test (n=5) and a scenario-based usability test (n=10). Usability and performance data were assessed through video-recorded observation and the administration of the system usability scale (SUS) and NASA Task Load Index (TLX). We found that task-based testing was useful in quickly identifying problems within our application, but scenario-based testing elicited more valuable feedback from older adults. A comparison of the different evaluation methods used and the discussion of the challenges encountered provide multiple implications for the practice of usability testing of mobile health products with older adults.

Geriatric education utilizing a palliative care framework

ABSTRACTThe dramatic growth of persons older than age 65 and the increased incidence of multiple, chronic illness has resulted in the need for more comprehensive health care. Geriatrics and palliative care are medical specialties pertinent to individuals who are elderly, yet neither completely addresses the needs of older adults with chronic illness. Interprofessional faculty developed Geriatric Education Utilizing a Palliative Care Framework (GEPaC) to teach an integrated approach to care. Interactive online modules use a variety of instructional methods, including case-based interactive questions, audio-visual presentations, reflective questions, and scenario-based tests. Modules are designed for online education and/or traditional classroom and have been approved for Continuing Medical Education. Pre- and posttest scores showed significant improvements in knowledge, attitudes, and skills. Participants were highly satisfied with the coursework’s relevance and usefulness for their practice and believed that GEPaC prepared them to address the needs of older adults for disease and symptom management, communicating goals of care, and supportive/compassionate care.

A System Dynamics Approach of Users’ Dynamic Behavior for Mobile Services in Academic Library

AbstractThe application and innovation of mobile library services are changing the way users communicate and access information. As academic libraries provide users with ubiquitous services, the need to holistically explore the transformation mechanism of different levels of users driven by their dynamic behavior of mobile services assumes greater importance. The systems thinking perspective is used as a tool to relate to key factors, interactional processes and feedback outcomes. The system dynamics model is used to present two scenario-based tests so as to evaluate influences of managing the value of relationship of different levels of users, with an emphasis on participation and collaboration with users, while three different purposive types of users are also identified, need, exploration and interest orientation, to reflect the effectiveness of adopting. Mobile service providers can meet individuation and diversification of users’ demands in the process of providing dynamic services by discerning and clustering user types timely. The developed model can serve as a systematic explanation or effective reference tool for the development of mobile libraries under a variety of circumstances.