OGATE: A framework for autonomous controllers assessment

Abstract

Autonomous robots can face a variety of applications integrating Artificial Intelligence (AI) techniques, for instance Planning & Scheduling (P&S). While robotics and planning systems are commonly well assessed through test benchs and metrics, in autonomous robots literature it is usual to present isolated case studies for evaluating such works. For instance, experiments are presented in very specific circumstances and often the data provided is not enough to enable a characterization of the autonomous features performance, providing only a demonstration of effectiveness. The main issue is the absence of a framework to assess autonomous controllers from a general perspective. We propose a research focused on a set of general applicable metrics to enable assessment of autonomous controllers. In this way, our objective is to analyse the deliberation and reaction capabilities of an autonomous robot in real operative scenarios. Such metrics are implemented in OGATE, a domain independent tool that automatically carries on with controllers’ testing, generating objective and reproducible performance assessments. To test this framework we have used two autonomous controllers that rely on different technologies for P&S. Results show that we are able to obtain relevant data, enabling the characterization of different P&S integration in robotics.