A Latent State-Based Multimodal Execution Monitor with Anomaly Detection and Classification for Robot Introspection

Hongmin Wu,Yisheng Guan,Juan Rojas

doi:10.3390/app9061072

Abstract

Robot introspection is expected to greatly aid longer-term autonomy of autonomous manipulation systems. By equipping robots with abilities that allow them to assess the quality of their sensory data, robots can detect and classify anomalies and recover appropriately from common anomalies. This work builds on our previous Sense-Plan-Act-Introspect-Recover (SPAIR) system. We introduce an improved anomaly detector that exploits latent states to monitor anomaly occurrence when robots collaborate with humans in shared workspaces, but also present a multiclass classifier that is activated with anomaly detection. Both implementations are derived from Bayesian non-parametric methods with strong modeling capabilities for learning and inference of multivariate time series with complex and uncertain behavior patterns. In particular, we explore the use of a hierarchical Dirichlet stochastic process prior to learning a Hidden Markov Model (HMM) with a switching vector auto-regressive observation model (sHDP-VAR-HMM). The detector uses a dynamic log-likelihood threshold that varies by latent state for anomaly detection and the anomaly classifier is implemented by calculating the cumulative log-likelihood of testing observation based on trained models. The purpose of our work is to equip the robot with anomaly detection and anomaly classification for the full set of skills associated with a given manipulation task. We consider a human–robot cooperation task to verify our work and measure the robustness and accuracy of each skill. Our improved detector succeeded in detecting 136 common anomalies and 368 nominal executions with a total accuracy of 91.0%. An overall anomaly classification accuracy of 97.1% is derived by performing the anomaly classification on an anomaly dataset that consists of 7 kinds of detected anomalies from a total of 136 anomalies samples.

Highlights

Human interactions with the environment are directed by their ability to understand the consequences of their actions, including failures, and their ability to learn from such failures
We propose that the anomaly detector is a thresholding marginal log-likelihood with Bayesian non-parametric Hidden Markov Model (HMM), which leads to tackling the anomaly detection problem in a more natural way to meet the need for real-world applications
We visualized the performance of the proposed method as well as a baseline by assessing the anomaly time reaction of anomaly flags; that is, which algorithm can make a correct detection at a time closer to the ground truth

Summary

Introduction

Human interactions with the environment are directed by their ability to understand the consequences of their actions, including failures, and their ability to learn from such failures. We consider introspection to be the ability to assess streaming sensory data quality, the actions that data underpin, and how actions are performed. When robots learn models from demonstrated data, their long-term autonomy capability depends on their ability to have good insights into the data—allowing them to differentiate nominal conditions from anomalous ones. We improve the introspection ability initially presented in our Sense-Plan-Act-Introspect-Recover (SPAIR) system (http://www.juanrojas.net/re_enact_adapt/).

Objectives

Results

Discussion

Conclusion