Structured validation of AI-based systems by virtual testing in simulated test scenarios

Abstract

The growing relevance of artificial intelligence (AI) for technical systems offers significant potential for the realization and operation of autonomous systems in complex and potentially unknown environments. However, unlike classical solution approaches, the functionality of an AI system cannot be verified analytically, which is why data-driven approaches such as scenario-based testing are used. With the increasing complexity of the required functionality of the AI-based system, the quantity, and quality of the data needed for development and validation also increase. To meet this demand, data generated synthetically using simulation is increasingly being used. Compared to the acquisition of real-world reference data, simulation offers the major advantage that it can be configured to test specific scenarios of interest. This paper presents an architecture for the systematic generation of virtual test scenarios to establish synthetically generated test data as an integral part of the development and validation process for AI systems. Key aspects of this architecture are the consistent use of digital twins as virtual 1-to-1 replicas and a simulation infrastructure that enables the generation of training and validation data for AI-based systems in appropriate quantity, quality, and time. In particular, this paper focuses on the application of the architecture in the context of two use cases from different application domains.