Using Multi-Modal Physiological Markers and Latent States to Understand Team Performance and Collaboration

Abstract

Squads of the future battlefield will include a mixture of technically savvy humans and artificially intelligent teammates. Contextually aware AI teammates will be essential for war fighter overmatch. To understand how multimodal physiology can impact mixed team performance, we looked at how physiological team properties emerge in a naturalistic and collaborative environment. Here, we examined internal states and team outcomes based on these states within the context of a complex bomb defusal task in a simulated and naturalistic environment. This overarching research integrates eye gaze behavior, neural activity, speech, heart rate variability, and facial expressions to unravel the intricate relationship between individual and team performance. Here we focus on the facial expression data. Using a novel testbed, we aimed to uncover how these physiological processes evolve and interact with human interactions to influence team dynamics and task performance. Compared to traditional highly controlled lab tasks, this novel testbed enables peripheral measurement of multimodal physiology during naturalistic team formation and collaboration. We report differences between an individual task and teaming task in global facial expressivity results and correlations between facial expression synchrony scores and team task performance.