Measurement and Manipulation in Human-Agent Teams: A Review

Abstract

In this era of the Fourth Industrial Revolution, increasingly autonomous and intelligent artificial agents become more integrated into our daily lives. As such, these agents are capable of conducting independent tasks within a teaming setting, while also becoming more socially invested in the team space. While ample human-teaming theories help understand, explain, and predict the outcome of team endeavors, such theories are not yet existent for human-agent teaming. Furthermore, the development and evaluations of agents are constantly evolving. As a result, many developers utilize their own test plans and their own measures making it difficult to compare findings across agent developers. Many agent developers looking to capture human-team behaviors may not sufficiently understand the benefits of specific team processes and the challenges of measuring these constructs. Ineffective team scenarios and measures could lead to unrepresentative training datasets, prolonged agent development timelines, and less effective agent predictions. With the appropriate measures and conditions, an agent would be able to determine deficits in team processes early enough to intervene during performance. This paper is a step in the direction toward the formulation of a theory of human-agent teaming, wherein we conducted a literature review of team processes that are measurable in order to predict team performance and outcomes. The frameworks presented leverage multiple teaming frameworks such as Marks et al.’s (2001) team process model, the IMOI model (Ilgen, 20005), Salas et al.’s big five model (2005) as well as more modern frameworks on human agent teaming such as Carter-Browne et al. (2021). Specific constructs and measures within the “input” and “process” stages of these models were pulled and then searched within the team’s literature to find specific measurements of team processes. However, the measures are only half of the requirement for an effective team-testing scenario. Teams that are given unlimited amount of time should all complete a task, but only the most effective coordinative and communicative teams can do so in a time efficient manner. As a result, we also identified experimental manipulations that have shown to cause effects in team processes. This paper aims to present the measurement and manipulation frameworks developed under a DARPA effort along with the benefits and costs associated with each measurement and manipulation category.