Verification, Trustworthiness and Accountability of Human-Driven Autonomous Systems

Abstract

Listen

Translate article

Despite the fact that autonomous systems’ science and control theory have almost 50 years of history, the community is facing major challenges to ensure the safety of fully autonomous consumer systems. It mostly concerns the verification and high fidelity operation of safety-critical systems, may that be a self-driving car, a homecare robot or a surgical manipulator. The community still struggles to establish objective criteria for trustworthiness of AI driven / machine learning based control systems. On one hand, we celebrate the rise of cognitive capabilities in robotic systems, leading independent decision making; on the other hand, decisions made in complex environments, based on multi-sensory data will surly lead to some wrong conclusions and hazardous outcome, jeopardizing the public trust in entire application domains. This ambiguity led to the currently ruling safety principle to offer the possibility for a human-driven override, translating to Level of Autonomy 3 and 4 with autonomous vehicles.The aim of the development community is to establish processes and metrics to ensure the reliability of the takeover process, when the human driver or operator takes back the partial or full control from the autonomous system. We have been building complex simulators and data collection systems to benchmark human decision making against the computer. Situation Awareness (SA) has been identified as a key, as it defines the level of cognitive understanding and capability of a human operator in a given environment. Assessing, maintaining and regaining efficiently SA are core elements of the relevant research projects, reviewed and compared in this talk. Based on the research at the Antal Bejczy Center for Intelligent Robotics at Óbuda University, we created an assessment method for critical handover performance, to quantitatively define the required level and components of SA with respect to the autonomous functionalities present. To improve system safety, driver assistance systems and automated driving functionalities shall be collected and organized in a hierarchical way, along with the two criteria of SA presented, as a standardized risk assessment protocol: 1) the Level of SA, based on state of the environment; 2) the components of SA, based on knowledge.The outcome of our experiments may find its way to new verification standards through ongoing IEEE initiatives, such as the P1872.1, P2817, P7000 and P7007, moreover this systematic approach has already proved to bring benefit to other domains, such as medical robotics.