Automatic Scenario Generation Research Articles

Automated scenario generation from Operational Design Domain model for testing AI-based systems in aviation

AbstractApplications based on artificial intelligence (AI) promise benefits, ranging from improved performance to increased capabilities in many industries. In the aviation domain, one example is the new Airborne Collision Avoidance System (ACAS X). The current investigation aims at combining ACAS X and AI to maintain its performance while decreasing the memory footprint. However, the anticipation of AI being increasingly used confronts regulators with challenges in terms of safety assurance and certification. Consequently, the European Union Aviation Safety Agency (EASA) published a concept paper for machine learning applications in aviation. Both, the Concept of Operation (ConOps) in combination with an Operational Design Domain (ODD), are listed as objectives to be met for the safety analysis. From a developer’s perspective, this raises questions on how to effectively derive the ODD from ConOps and test the given system based on the ODD description. Based on an exemplary use case of a Near Mid-Air Collision avoidance between two aircraft through the advisories of ACAS X, a highly automated framework for generating and testing synthetic data is proposed. Using this framework, 1800 Near Mid-Air Collision scenario files are created and automatically executed in the simulation environment FlightGear. Scenario-based testing is used for the logging of ACAS X advisory data and evaluating it against predefined requirements. By this approach, an efficient way of verifying system requirements and conducting automated testing based on the ODD definition is demonstrated. Throughout this process, Model-Based Systems Engineering (MBSE) is used to reduce and manage complexity. The framework in this paper enables a systematic and highly automated approach for scenario generation based on the ODD.

Multi-Robot Coordination and Layout Design for Automated Warehousing (Extended Abstract)

With the rapid progress in Multi-Agent Path Finding (MAPF), researchers have studied how MAPF algorithms can be deployed to coordinate hundreds of robots in large automated warehouses. While most works try to improve the throughput of such warehouses by developing better MAPF algorithms, we focus on improving the throughput by optimizing the warehouse layout. We show that, even with state-of-the-art MAPF algorithms, commonly used human-designed layouts can lead to congestion for warehouses with large numbers of robots and thus have limited scalability. We extend existing automatic scenario generation methods to optimize warehouse layouts. Results show that our optimized warehouse layouts (1) reduce traffic congestion and thus improve throughput, (2) improve the scalability of the automated warehouses by doubling the number of robots in some cases, and (3) are capable of generating layouts with user-specified diversity measures. We include the source code at: https://github.com/lunjohnzhang/warehouse_env_gen_public

Automatic scenario generation for efficient solution of robust optimal control problems

AbstractExisting methods for nonlinear robust control often use scenario‐based approaches to formulate the control problem as large nonlinear optimization problems. The optimization problems are challenging to solve due to their size, especially if the control problems include time‐varying uncertainty. This paper draws from local reduction methods used in semi‐infinite optimization to solve robust optimal control problems with parametric and time‐varying uncertainty. By iteratively adding interim worst‐case scenarios to the problem, methods based on local reduction provide a way to manage the total number of scenarios. We show that the local reduction method for optimal control problems consists of solving a series of simplified optimal control problems to find worst‐case constraint violations. In particular, we present examples where local reduction methods find worst‐case scenarios that are not on the boundary of the uncertainty set. We also provide bounds on the error if local solvers are used. The proposed approach is illustrated with two case studies with parametric and additive time‐varying uncertainty. In the first case study, the number of scenarios obtained from local reduction is 101, smaller than in the case when all extreme scenarios are considered. In the second case study, the number of scenarios obtained from the local reduction is two compared to 512 extreme scenarios. Our approach was able to satisfy the constraints both for parametric uncertainty and time‐varying disturbances, whereas approaches from literature either violated the constraints or became computationally expensive.

A VR training scenario editor for operation in construction based on BIM 4D and domain expert authoring

PurposeVirtual Reality (VR) based training is an efficient and promising digital tool. In the construction industry, studies have been conducted on the realisation of VR scenarios for construction operations, but the challenges lie in procedural scenario creation based on the knowledge and digital data of the construction project. The purpose of this paper is to address this issue.Design/methodology/approachA new system and interface allowing an automatic generation of a VR scenario for operating procedure (OP) based on 4D Building Information Modelling (BIM) and tasks authoring by the domain experts are presented. The first process, Automatic Scenario Generation, generates assembly tasks based on the BIM metadata. The second process, Expert Scene and Scenario Authoring, allows adding tasks that cannot be modeled in the BIM OP.FindingsWith the 4D BIM methodology, the 3D model of building as well as the 3D OPs contain a time dimension representing the construction phases or planning. The evolution in time of the construction site and the tools and procedures adapted to each construction project must be integrated in the VR simulations. Moreover, domain experts should have control over the scenario and should be able to author tasks assigned to the operator in training.Originality/valueA Unified Modeling Language based framework allows the reuse of BIM metadata to create the scene, VR interactions and the assembly procedure; the assembly procedure is embedded in the virtual environment considering the time (4D) in the construction of the building and is used as a training scenario for the operator; and the training scenario can be refined by adding operational knowledge, such as quality.

Mission-Based Scenario Modeling and Generation for Virtual Training

Automated scenario generation for virtual training has become an emerging research problem, as manual authoring is often time consuming and costly. In this paper, we present a mission-based scenario modeling and generation framework for virtual training. In particular, we consider the issue of how the timing of the events in a scenario can impact the training process and how to incorporate such impact into the scenario generation. To this end, our framework is designed to explicitly capture the propagated effect of an event and its influence to other events. For representing mission-based scenarios, the concepts of mission objectives, events, and scenario beats are introduced. The generation process is designed to generate scenarios that can tailor to trainer's preference and adapt to different trainees' skill levels. The efficacy of the proposed framework is demonstrated through an empirical study of human players in a food distribution training game.

Simulation-Based Evaluation of Cooperative Maneuver Coordination and Its Impact on Traffic Quality

<div class="section abstract"><div class="htmlview paragraph">Cooperative Maneuver Coordination (CMC) is one of the cornerstone technologies on the way to automated and connected driving of the future. The goal of this technology is to increase traffic safety and efficiency, by solving potential conflict situations on the road, based on cooperative maneuver planning, negotiation and decision-making, with the aid of inter-vehicular communication. This innovative topic represents a wide area for research projects, e.g., such as German funded project IMAGinE. A variety of different approaches for CMC has already been implemented and evaluated in the related work of the recent past. However, due to the high system complexity in general and vast testing effort in particular of these solutions, their actual impact on the traffic quality has not yet been extensively addressed, and therefore must be further investigated. In order to fulfill this task, one needs a methodology with appropriate evaluation metrics alongside with a suitable testing environment, in order to obtain comprehensive results. The scientific contribution of the work at hand involves a simulation-based evaluation methodology for CMC. For this, we will propose a novel CMC algorithm, which is built upon a direct trajectory exchange via inter-vehicular communication and a decentralized decision-making process, suitable for both manually operated as well as autonomous vehicles. In order to examine this algorithm, we will introduce a co-simulation environment with automatic scenario generation and multi-instances capability, which consists of a coupled simulation of traffic flow and vehicle dynamics. Eventually, we will present a set of metrics, which we determined in order to evaluate effectiveness and efficiency of our CMC algorithm, considering its impact on various aspects of the traffic quality.</div></div>

Mathematical Definitions of Scene and Scenario for Analysis of Automated Driving Systems in Mixed-Traffic Simulations

This paper introduces a unified mathematical definition for describing commonly used terms encountered in systematical analysis of automated driving systems in mixed-traffic simulations. The most significant contribution of this work is in translating the terms that are clarified previously in literature into a mathematical set and function based format. Our work can be seen as an incremental step towards further formalisation of Domain-Specific-Language (DSL) for scenario representation. We also extended the previous work in the literature to allow more complex scenarios by expanding the model-incompliant information using set-theory to represent the perception capacity of the road-user agents. With this dynamic perception definition, we also support interactive scenarios and are not limited to reactive and pre-defined agent behavior. Our main focus is to give a framework to represent realistic road-user behavior to be used in simulation or computational tool to examine interaction patterns in mixed-traffic conditions. We believe that, by formalising the verbose definitions and extending theprevious work in DSL, we can support automatic scenario generation and dynamic/evolving agent behavior models for simulating mixed traffic situations and scenarios. In addition, we can obtain scenarios that are realistic but also can represent rare-conditions that are difficult to extract from field-tests and real driving data repositories.

Towards simulation-based verification of autonomous navigation systems

Abstract Autonomous ships are expected to change water-based transport of both cargo and people, and large investments are being made internationally. There are many reasons for such transformation and interest, including shifting transport of goods from road to sea, reducing ship manning costs, reduced dangerous exposure for crew, and reduced environmental impact. Situational awareness (SA) systems and Autonomous navigation systems (ANS) are key elements of autonomous ships. Safe deployment of ANS will not be feasible based on real-life testing only, but will require large-scale, systematic simulation-based testing in addition to assurance of the development process. DNV GL proposes to use a digital twin, meaning a digital representation of key elements of the autonomous ship as a key tool for simulation-based testing. The digital twin contains comprehensive mathematical models of the ship and its equipment, including all sensors and actuators. The complete simulation-based test system complementing the digital twin should consist of a virtual world to simulate environmental conditions, geographical information and interaction with other maritime traffic. Finally, the test system must include a test management system that controls simulation of the digital twin and the virtual world, generates test scenarios as well as evaluates the test scenario results. An automatic scenario generation tool should search for low ANS performance, and ultimately establish sufficient coverage of the possible scenario space. The test scenario evaluation should automatically consider safety, conformance to collision regulations at sea (COLREG), and possibly also the efficiency of the ship navigation. This paper presents a comprehensive prototype of a test system for ANS. Key topics are simulation-based testing, interfacing the simulator and ANS, cooperation with ANS manufacturers, dynamic test scenario generation, automatic assessment towards COLREG and experiences from the cooperation with The Norwegian Defense Establishment (FFI).

Automated Scenario Generation for Training of Humanitarian Responders in High-Risk Settings

Automated Scenario Generation for Training of Humanitarian Responders in High-Risk Settings

A Combinatorial Procedure to Determine the Full Range of Potential Operating Scenarios for a Dam System

Traditional dam safety assessment tends to place the focus on estimating the probabilities of failure for the system based on a few subjectively-chosen operating scenarios. The techniques used to assess these systems rely on linear chains of events and are incapable of considering component interactions, feedbacks and non-linear behaviour. This paper uses a systems approach to develop a new technique that puts possibilities first and is capable of generating an exhaustive list of potential component operating state scenarios for the system. The scenarios can then be simulated using a Monte-Carlo approach to determine a wide range of system behaviour (outcomes) for each scenario, taking into account component interactions and feedbacks. A component operating states database is presented where the system can be broken down into various levels of detail and the operating states and causal factors for each component can be defined. The Cartesian product of each operating state set is used to derive the full range of potential operating scenarios, with each scenario consisting of a single operating state for every component in the system. The list of scenarios can be used as an input to a simulation model which can then be run many times for each scenario, using Monte-Carlo inputs which vary the timing and severity of events as well as the inflows. The approach for automated scenario generation is demonstrated on a simple and complex representation of a hydropower system and the scenarios are generated and counted. Results show that increasing system complexity results in exponentially increasing numbers of potential operating scenarios, with the simple system having a total of 1.1 × 106 operating states in comparison to the complex system which has 1.83 × 1027. The approach presented in this paper (a) reduces the subjectivity associated with traditional dam safety assessments through automated scenario generation, and (b) improves the ability to understand component interaction and feedbacks by describing a Monte Carlo simulation approach which can be used for scenario simulation. By understanding how the system responds to the full range of potential operating conditions, dam owners and asset managers can make informed decisions relating to the improvement of operating strategies and implementation of system upgrades.

Implicit communication robots based on automatic scenario generation using web intelligence

Purpose This paper aims to propose a system that generates dialogue scenarios automatically in real time from Web news articles. Then, the authors used the Manzai metaphor, a form of Japanese traditional humorous comedy, in the system. The generated Manzai scenario consists of snappy patter and a humorous misunderstanding of dialogue based on the gap of our structure of funny points. The authors create communication robots to amuse people with the generated humorous robot dialogue scenarios. Design/methodology/approach The authors propose the following: how to generate funny dialogue-based scenario from Web news and Web intelligence, automatically? How to create direction of robots based on the pre-experiment? The authors conducted experiments from three viewpoints, namely, effectiveness of Manzai scenarios as content, effectiveness of Manzai-Robots as a medium and familiarity of Manzai-Robots. Findings In this paper, the authors find two points, namely, the new communication style called “human–robots implicit communication-and bridging the knowledge gap using Web intelligence, to communicate smoothly between humans and robots. Originality/value Numerous studies have examined communication robots that mutually communicate with people. However, for several reasons, communicating smoothly with people is difficult for robots. One reason is the problem of communication style. Another is knowledge gaps separating humans and robots. The authors propose a new communication style to solve the problems and designate the communication style based on dialogue between robots as “human-robot implicit communication”. The authors then create communication robots to communicate with people naturally, smoothly and with familiarity according to their dialogue.

Tourism travel under climate change mitigation constraints

The paper first describes an inventory for 2005 giving the tourism related CO2 emission caused by global tourism, and presents a 30-year projection and a 45-year simulation. The study found that tourists cause 4.4% of global CO2 emissions. Also these emissions are projected to grow at an average rate of 3.2% per year up to 2035. This increase is problematic as globally a reduction of emissions by 3–6% is required to avoid ‘dangerous’ climate change. Using contemporary scenario techniques it appeared difficult to find a future tourist travel system consistent with CO2 emission reductions of up to 70% by 2050 with respect to 2005. Based on the model underlying the 30-year projection, 70 scenarios are presented in a ‘landscape’ graph exploring the effect of opportunities to reduce the emissions, but this attempt did not reach the large reductions envisaged. We therefore explored automated scenario generation as a way to define backcasting scenarios that both reach the emission reduction target and retain the highest possible economic value for the sector. The main contributions made by this study are (1) in comparing the value of different ways to approach a (desired) future and (2) giving insight into the kind of structural changes required within tourism and tourism transport in case very strong emission reductions are required. Finally the model showed signs of ‘complex’ behaviour.

Automatic Scenario Generation through Procedural Modeling for Scenario-Based Training

Automatic Scenario Generation through Procedural Modeling for Scenario-Based Training

Automatic Scenario Generation through Procedural Modeling for Scenario-Based Training

We discuss our current efforts at developing automatic scenario generation software. We begin by explaining the rationale, and then review successful previous efforts. We discuss the lessons-learned from the past work, and the conceptual pieces that are required to generate operationally-valid scenarios that support effective training. We then present the conceptual design of our scenario generation approach, which uses novel procedural modeling approaches to ensure operational and training requirements are adequately met.

Environment behavior models for automation of testing and assessment of system safety

This paper presents an approach to automatic scenario generation from environment behavior models for testing of real-time reactive systems. The model of behavior is defined as a set of events (event trace) with two basic relations: precedence and inclusion. The attributed event grammar (AEG) specifies possible event traces and provides a uniform approach for automatically generating and executing test cases. The environment model includes a description of hazardous states in which the system may arrive and makes it possible to gather statistics for system safety assessment. The approach is supported by a generator that creates test cases from the AEG models. We demonstrate the approach with a case study of a software prototype of the computer-assisted resuscitation algorithm for a safety-critical casualty intravenous fluid infusion pump.

Environment behavior models for scenario generation and testing automation

This paper suggests an approach to automatic scenario generation from environment models for testing of real-time reactive systems. The behavior of the system is defined as a set of events (event trace) with two basic relations: precedence and inclusion. The attributed event grammar (AEG) specifies possible event traces and provides a uniform approach for automatically generating, executing, and analyzing test cases. The environment model includes a description of hazardous states in which the system may arrive and makes it possible to gather statistics for system safety assessment. The approach is supported by a generator that creates test cases from the AEG models. We demonstrate the approach with case studies of prototypes for the safety-critical computer-assisted resuscitation algorithm (CARA) software for a casualty intravenous fluid infusion pump and the Paderborn Shuttle System.