Towards Self-Confidence in Autonomous Systems

Abstract

Modern civilian and military systems have created a demand for sophisticated machine autonomy with human supervision and coordination in uncertain environments. These ‘on-the-loop’ roles for humans interacting with intelligent autonomy shift emphasis away from traditional human ‘in-the-loop’ control capabilities in order to extend/complement existing human capabilities. As such, system designed around ‘levels of autonomy’ (in which the responsibilities, and capabilities of autonomous agents and their human counterparts can be predefined and separately delegated) are making way for tightly integrated systems that emphasize dynamic cooperation and team-based interaction. This shift in human roles and system design philosophy naturally raises questions about when and how mutual communication of operational intent and the perceived capabilities of autonomous agents can impact human-autonomy coordination. Related questions have attracted much interest in the human factors, artificial intelligence, robotics, and control engineering literature. Many studies focus on challenges in human-autonomy teaming that arise from the fact that the decision-making and communication algorithms which drive many unmanned systems are non-transparent to human collaborators, and are often designed without consideration of how human problem-solving abilities can be opportunistically exploited at various operational levels to assist (rather than undermine) the autonomy. Planning and information gathering algorithms are typically based on normative models for reasoning under uncertainty: an autonomous agent seeks actions that maximize some expected utility, given models of uncertainty and specifications of costs for some set of tasks/subtasks. This approach can lead to extremely sophisticated non-deterministic behaviors through hierarchical reasoning, and provides a flexible means for coping with imperfect information. However, autonomous reasoning ultimately depends on several key pieces of knowledge that are subject to their own uncertainties, which could potentially be mitigated by interactions with human collaborators. Of particular interest are uncertainties in: (i) world models (i.e. imperfect knowledge of possible outcomes that may develop in a particular operating environment); (ii) capabilities of an autonomous agent; (iii) information sources (e.g. sensor data for own state or task/world state, intelligence reports, etc.). The brittleness of autonomy to these uncertainties has steered the development of sophisticated formal verification techniques to ensure that autonomous agents fulfill desired mission objectives without the need for human intervention. Sophisticated planning and task allocation algorithms have also been developed to ‘repair’ plans in the event of uncertain or worst case outcomes. Yet, these approaches to analyzing and certifying autonomy require considerable offline computational effort to exhaustively root out failure modes or exceptional scenarios that are not anticipated or easily understood by system designers and end users. They are also very sensitive to changes in system architecture, mission requirements, or uncertainty specifications. Furthermore, they typically require sophisticated understanding of the capabilities and inner workings of the autonomy as well as the precise requirements for the mission at hand, which can severely restrict the space of potential end users of autonomy. This in turn can significantly increase the overhead cost of autonomous system development and deployment. It is therefore also worth considering the extent to which the difficult problems of mission uncertainty assessment and plan adaptation can be left as tasks to be jointly tackled by human-autonomy teams. This paper discusses the idea of augmenting human-machine dialog by communicating an autonomous agent’s sense of confidence, i.e. the autonomy’s perceived ability to effective execute assigned tasks. Such