Evaluation of Autonomous Approaches Using Virtual Environments

Abstract

In this paper, we address the challenging problem of evaluating autonomous research approaches by the example of an online anomaly detection framework for dynamical real-time systems. We propose to use a virtual test environment that was conceptualized based on the specific evaluation requirements. The architecture is composed of all system parts required for evaluation: the operating system implementing the anomaly detection framework, reconfigurable autonomous applications, an execution platform device for the operating system and its applications, and the device’s environment. We demonstrate our concepts by the example of our miniature robot BeBot that acts as our virtual prototype (VP) to execute autonomous applications. With an interactive module, the virtual environment (VE) offers full control over the environment and the VP so that using different levels of hardware implementation for evaluation, but also failure injection at runtime becomes possible. Our architecture allows to determine clear system boundaries of the particular parts composed of perception function, decision making function and execution function which is essential for evaluating autonomous approaches. We define evaluation scenarios to show the effectiveness of each part of our approach and illustrate the powerfulness of applying virtual test environments to evaluate such approaches as the here referred one.