Designing an Automated Agent to Encourage Human Reliance

Abstract

Human reliance on automated agents can be critically important, as exemplified by a pilot relying on an automated ground collision avoidance system. While it is important that the automated agent perform a task well, thus promoting reliance on the automation, it is difficult to test human reliance on automated agents in safety-critical systems. This paper presents an automated agent designed to enable testing of human reliance on automation in the Space Navigator environment. The automated agent performs collision detection and avoidance tasks in the environment, aiding the human participant in real-time. We present a collision detection and avoidance model, comparing three potential methods for collision avoidance. Analysis shows that the new agent’s performance when teamed with another simulated agent improves upon previous individual human and human-agent team performances in the same environment, thus making it logical for humans to rely upon it. A human-subjects study confirms that the resulting automated agent/environment pairing enables human reliance studies in a low-states automation environment.