Human-robot Teams Research Articles

Exploring the synergy of human-robot teaming, digital twins, and machine learning in Industry 5.0: a step towards sustainable manufacturing

Abstract Sustainable manufacturing remains a central objective of Industry 5.0. By successfully implementing harmonic human-robot teams in intelligent industrial systems, the efficiency and well-being of human workers can be increased. Achieving this requires a gradual approach from caged robots to advanced, seamless collaboration between humans and robots. Initially, that means transitioning to human-robot interaction (HRI) where there is an exchange of commands between the human and the robot. Further advancements within safety considerations, including collision avoidance through advanced machine vision, enable the exchange of workspace that defines human-robot collaboration (HRC). The next stage is physical HRC (pHRC) which requires safe and controlled exchange of forces through impedance and admittance control. Finally, this paper describes human-robot teaming (HRT), which is defined by the exchange of solutions as teammates. This is enabled by combining cutting-edge technologies such as digital twin (DT), advanced vision sensors, machine learning (ML) algorithms and mixed reality (MR) human–machine interfaces for operators. A key contribution of this work is reviewing the integration of HRT with DT and ML, highlighting how these technologies enable seamless perception, prediction, and decision-making in human-centric industrial systems. By reviewing these technologies, the paper highlights current challenges, limitations and research gaps within the field of HRT and suggests potential future possibilities for HRT, such as advanced disassembly of used goods for a more sustainable manufacturing industry.

Hybrid human–robot teams in the frontline: automated social presence and the role of corrective interrogation

PurposeCustomer perceptions toward hybrid human–robot teams remain largely unexplored. We focus on the impact of frontline robots’ (FLRs) automated social presence (ASP) on customers’ perceived teamwork quality, and ultimately frontline employees’ (FLEs) competence and warmth. We explore the role of interrogation as a relevant contingency. We complement the customer view with insights into the FLEs’ viewpoint.Design/methodology/approachWe manipulate FLR’s ASP cues (speech and identity) in a hybrid team in four business-to-consumer (B2C) video-based experiments and collect data from online participants. We combine these with one business-to-business (B2B) field survey which collected data from FLEs working in hybrid teams.FindingsWhen FLR’s ASP increases, customers more positively evaluate teamwork quality, ultimately affecting FLEs’ competence and warmth. FLEs who correct (interrogate) robotic mistakes strengthen the positive effect of FLRs’ ASP on teamwork quality. When FLRs correct FLEs, ASP’s effect on teamwork quality is also strengthened, while FLEs are not “punished” for erring. In contrast, FLEs themselves do perceive corrections as detrimental to teamwork quality. We term this the hybrid team evaluation paradox.Practical implicationsWe recommend that firms deploy hybrid teams equipped with high-ASP FLRs (name and speech suffice). FLEs should be trained, and FLRs programmed, to appropriately use interrogation. Managers should pay attention to the paradox, given the conflicting perceptions toward interrogative behaviors.Originality/valueWe advance the hybrid teams literature by drawing on ASP, social cognition and collective mindfulness theories and behaviors that ameliorate customer perceptions. Our results support using FLRs to enhance FLEs’ capabilities.

A Game-Theoretic Model of Trust in Human–Robot Teaming: Guiding Human Observation Strategy for Monitoring Robot Behavior

A Game-Theoretic Model of Trust in Human–Robot Teaming: Guiding Human Observation Strategy for Monitoring Robot Behavior

Human Prior Knowledge Estimation from Movement Cues for Information-Based Control of Mobile Robots during Search

Robotic search often involves teleoperating vehicles into unknown environments. In such scenarios, prior knowledge of target location or environmental map may be a viable resource to tap into and control other autonomous robots in the vicinity towards an improved search performance. In this article, we test the hypothesis that despite having the same skill, prior knowledge of target or environment affects teleoperator actions, and such knowledge can therefore be inferred through robot movement. To investigate whether prior knowledge can improve human-robot team performance, we next evaluate an adaptive mutual-information blending strategy that admits a time-dependent weighting for steering autonomous robots. Human-subject experiments show that several features including distance travelled by the teleoperated robot, time spent staying still, speed, and turn rate, all depend on the level of prior knowledge and that absence of prior knowledge increased workload. Building on these results, we identified distance travelled and time spent staying still as movement cues that can be used to robustly infer prior knowledge. Simulations where an autonomous robot accompanied a human teleoperated robot revealed that whereas time to find the target was similar across all information-based search strategies, adaptive strategies that acted on movement cues found the target sooner more often than a single human teleoperator compared to non-adaptive strategies. This gain is diluted with number of robots, likely due to the limited size of the search environment. Results from this work set the stage for developing knowledge-aware control algorithms for autonomous robots in collaborative human-robot teams.

What If I’m Wrong? Team Performance and Trustworthiness When Modeling Risk-Sensitivity in Human–Robot Collaboration

To achieve effective coordination in human–robot teams, robots must have an accurate model of human decision-making (i.e., theory of mind) to predict human actions and plan appropriate supplemental strategies. However, humans are often assumed to be approximately rational decision-makers which is not always the case, especially when faced with risky or uncertain decisions. Recent works in human–robot interaction have begun to address this by implementing risk-sensitive models of human behavior to characterize an individual’s risk-sensitivity. However, little attention is given to the following question: what happens when the robot makes the incorrect inference about human risk-sensitivity? Failure to consider this may lead to ineffective coordination and degradation of perceived trustworthiness in the robot. In this article, we adopt a popular risk-sensitive model based on Cumulative Prospect Theory, where model accuracy is varied in the robot’s theory of mind when interacting with either a risk-averse (pessimistic) or risk-seeking (optimistic) human. We designed a joint-pursuit game where the human and robot are conditioned with different (assumptions of) human risk-sensitivity in a 2 \( \times \) 2, between-subject study. Results from both simulated and human experiments showed that team performance was decreased and perceived trustworthiness in the robot was negatively impacted when the robot made the incorrect assumption of human risk-sensitivity. Overall, this work shows that risk-sensitive models can be used to great effect but only if we remain diligent to the fact that misspecification can lead to negative outcomes.

One Size Does Not Fit All: Mechanisms of Employees’ Acceptance of Robotic Lower-Level Managers

Organizations are facing fundamental changes in digitized work environments. These changes include the introduction of android and humanoid social robots designed to assist employees. The current research addresses whether and under what conditions employees accept such robots as lower-level managers, not just as assistants. The authors extend research on social robots with the expectation-disconfirmation paradigm. They examine how the interplay of expected and experienced performance-related and relational characteristics of robot-supervisors affect employees’ willingness to work with the robotic manager. Results from an online experiment with 7061 U.S. office workers reveal different effects for perceived performance-related and relational characteristics of a robot manager. For performance-related characteristics, employees’ willingness to work with robot managers follows an S-shaped pattern. A slight overfulfillment of expectations is associated with the highest levels of readiness. For relational characteristics, the results instead follow a degressive curve. Increasing positive experiences are associated with decreasing positive evaluations of willingness. Overall, employees indicate a preference for android robots over humanoid robots as managers. This research thus suggests tactics for the successful implementation of mixed human-robot teams and the effective introduction of social robots into lower-level management roles.

A mini-review on mobile manipulators with Variable Autonomy.

This paper presents a mini-review of the current state of research in mobile manipulators with variable levels of autonomy, emphasizing their associated challenges and application environments. The need for mobile manipulators in different environments, especially hazardous ones such as decommissioning and search and rescue, is evident due to the unique challenges and risks each presents. Many systems deployed in these environments are not fully autonomous, requiring human-robot teaming to ensure safe and reliable operations under uncertainties. Through this analysis, we identify gaps and challenges in the literature on Variable Autonomy, including cognitive workload and communication delays, and propose future directions, including whole-body Variable Autonomy for mobile manipulators, virtual reality frameworks, and large language models to reduce operators' complexity and cognitive load in some challenging and uncertain scenarios.

The Journey or the Destination: The Impact of Transparency and Goal Attainment on Trust in Human-Robot Teams

As robots gain autonomy, human-robot task delegation can become more goal-oriented; specifying what to do rather than how. This can lead to unexpected robot behaviour. We investigated the effect of transparency and outcome on the perceived trustworthiness of a robot that deviates from the expected manner to reach a delegated goal. Participants ( N \(=\) 82) engaged in a virtual military mission as a Human-Robot Team using a 2 \(\times\) 2 between-subjects design (low vs. high transparency, positive vs. negative outcome). Participants received training on the expected manner to reach the mission’s goal. In the actual mission, the robot deviated from the planned path. We manipulated whether the robot explained its deviation and whether the outcome was better or worse than the original plan. Results showed that transparency contributed to higher and more stable levels of trust, without increasing subjective workload. While the robot’s deviation led to a violation of trust in the low transparency condition, trust remained stable in the high transparency condition, indicating a buffering effect of transparency on trust in case of unexpected behaviour. The impact of outcome on trust was consistent across transparency conditions. Our findings underscore the role of transparency as a tool for fostering human-robot trust.

Human-Robot Team Performance Compared to Full Robot Autonomy in 16 Real-World Search and Rescue Missions: Adaptation of the DARPA Subterranean Challenge

Human operators in human-robot teams are commonly perceived to be critical for mission success. To explore the direct and perceived impact of operator input on task success and team performance, 16 real-world missions (10 h) were conducted based on the DARPA Subterranean Challenge. Missions involved deploying a heterogeneous team of robots to locate and identify artefacts such as climbing rope, drills and a mannequin representing a human survivor. Two conditions were evaluated: human operators that could control the robot team with state-of-the-art autonomy (Human-Robot Team) compared to autonomous missions without human operator input (Robot-Autonomy). Human interventions included creating waypoints to prioritise high-yield areas, and to navigate through error-prone spaces. Human-Robot Teams were often in directed autonomy mode (70% of mission time), found more items ( \(+\) 10.52%), traversed more distance ( \(+\) 12.71%), covered more unique ground ( \(+\) 10.56%), and longer time between safety-related events (34%). In routine conditions, both condition scores were comparable for artefacts, distance and coverage. Human-Robot Teams were faster at finding the first artefact but slower to respond to information from the robot team. Overall, operators contribute to mission-based outcomes, help to overcome environmental situations that can impede progress, and can assist robots to recover faster from difficult events.

Towards human capability estimation to enhance human-robot team performance

Abstract Skilled labor shortage is a prominent challenge in the world of work. Meanwhile, age-related disabilities or injury lead to at least temporary performance limitations, which make people unfit to work. Consequently, even less workers are available. By employing human-robot teams, the performance of these people may be restored. This requires a good artificial understanding of the human’s capabilities, as generic robot behavior is not feasible with the highly individualized manifestations of disability. We present an approach that allows the robot to autonomously assess human capabilities based on standards from occupational medicine. The method does not only indicate the presence/absence of capabilities, but gives them a discrete rating. This allows the robot to better define its own behavior as a mixture of supportive actions based on gaps in the detailed capabilities.

Embracing Levin’s Legacy: Advancing Socio-Technical Learning and Development in Human-Robot Team Design Through STS Approaches

This paper investigates the synergy between Levin’s theories on technology transfer as a socio-technical learning and developmental process (TLD process), and what we learnt about socio-technical systems (STS) theories in a case study developing human robot solutions for the construction sector. Levin’s extensive work highlights the significance of technology transfer as a means for organizational development. His TLD process emphasizes the intricate interplay between technology, organizational change, and learning and highlights the importance of incorporating cultural knowledge and skills into the technological transfer process. Contemporary STS views developed through our own work are introduced to complement and extend Levin’s theories by providing a systemic lens to understand the broader socio-technical context in which technology transfer occurs. To illustrate the synergies and potential challenges from Levin’s theories of technology transfer with contemporary STS concepts, we use a qualitative study of a unique case about the design and development of human-robot teams (HRTs) for construction tasks. Our findings reveal that while Levin’s theories provide a valuable foundation for understanding technology transfer and organizational change, contemporary socio-technical systems face unique challenges in the context of AI-driven human-robot teams, where intelligent robots also contribute to the socio-technical learning process. Moreover, the rapidly evolving nature of technology and innovations could exponentially impact on multidisciplinary design teams, stakeholder participation and inter-organizational dynamics. The discussions suggest an extension of co-generative learning to incorporate ‘collaborative intelligence’ between human-robot teams enabled by artificial intelligence (AI). Consequently, we suggest that Levin’s theories of technology transfer, developed before the rapid application of AI, may not have fully considered further social challenges caused by the introduction of autonomous systems such as AI-driven HRT systems. We extend Levin’s important work by suggesting that addressing such challenges requires ongoing dialogue and collaboration among researchers, practitioners, and policymakers with different disciplinary backgrounds to develop robust and reliable socio-technical systems frameworks to navigate the complexities of robotics and AI in today’s rapidly evolving technological landscape.

Robot learning on the job: Human-in-the-loop autonomy and learning during deployment

With the rapid growth of computing powers and recent advances in deep learning, we have witnessed impressive demonstrations of novel robot capabilities in research settings. Nonetheless, these learning systems exhibit brittle generalization and require excessive training data for practical tasks. To harness the capabilities of state-of-the-art robot learning models while embracing their imperfections, we present Sirius, a principled framework for humans and robots to collaborate through a division of work. In this framework, partially autonomous robots are tasked with handling a major portion of decision-making where they work reliably; meanwhile, human operators monitor the process and intervene in challenging situations. Such a human–robot team ensures safe deployments in complex tasks. Further, we introduce a new learning algorithm to improve the policy’s performance on the data collected from the task executions. The core idea is re-weighing training samples with approximated human trust and optimizing the policies with weighted behavioral cloning. We evaluate Sirius in simulation and on real hardware, showing that Sirius consistently outperforms baselines over a collection of contact-rich manipulation tasks, achieving an 8% boost in simulation and 27% on real hardware than the state-of-the-art methods in policy success rate, with twice faster convergence and 85% memory size reduction. Videos and more details are available at https://ut-austin-rpl.github.io/sirius/ .

TIP: A trust inference and propagation model in multi-human multi-robot teams

Trust is a crucial factor for effective human–robot teaming. Existing literature on trust modeling predominantly focuses on dyadic human-autonomy teams where one human agent interacts with one robot. There is little, if not no, research on trust modeling in teams consisting of multiple human and robotic agents. To fill this important research gap, we present the Trust Inference and Propagation (TIP) model to model and estimate human trust in multi-human multi-robot teams. In a multi-human multi-robot team, we postulate that there exist two types of experiences that a human agent has with a robot: direct and indirect experiences. The TIP model presents a novel mathematical framework that explicitly accounts for both types of experiences. To evaluate the model, we conducted a human-subject experiment with 15 pairs of participants (N=30\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$N=30$$\\end{document}). Each pair performed a search and detection task with two drones. Results show that our TIP model successfully captured the underlying trust dynamics and significantly outperformed a baseline model. To the best of our knowledge, the TIP model is the first mathematical framework for computational trust modeling in multi-human multi-robot teams.

Sociotechnical considerations on developing human robot teaming solutions for construction: a case study

This research advocates for a paradigm shift in the exploration of human–robot teaming solutions for construction automation, by focusing on an integrated view of sociotechnical systems (STS) that recognize the inter-dependencies among actors at various levels when tracing how innovative ideas about intelligent robotic technologies translate into practice in the construction sector. Through a qualitative case study, the paper examines industry and organizational considerations for developing and adopting robotic technologies, leadership vision, mediation, and change management to propose integrative strategies to enhance expectations, acceptance, and deployment of intelligent technologies in human–robot teams (HRTs). This study contributes to research in construction robotics at three organizational levels—macro, meso, and micro. The Integrated Human–Robot Teaming Framework and associated workplan schema offer guidance for navigating human–robot teaming complexities. The study recommends adopting STS principles in planning and deploying robotics applications for construction, emphasizing the integration of multiple elements across the lifecycle. Active leadership and mediation emerge as critical elements in navigating complex networks, ensuring successful outcomes in the dynamic construction environment.

The paradoxical leader, crafting human worker, and robot teammate: A commentary on the future of leader behaviors

The paradoxical leader, crafting human worker, and robot teammate: A commentary on the future of leader behaviors

Neurocognitive assessment under various human-robot teaming environments.

Human-robot teaming has become increasingly important with the advent of intelligent machines. Prior efforts suggest that performance, mental workload, and trust are critical elements of human-robot dynamics that can be altered by the robot's behavior. Most prior human-robot teaming studies used behavioral analyses, but a limited number used neural markers, without the use of physical robots and complex tasks. Here we combine behavioral and EEG cortical dynamics to examine cognitive-motor processes when individuals complete a complex task under various team environments with a robot. The results revealed that altering the robot quality affected both behavioral and EEG dynamics. Task completion with an experienced robot led to greater team performance and human trust along with lower mental workload compared to an inexperienced teammate or when individuals performed alone. EEG changes suggest that different attentional processes were engaged when humans worked with the robot and performed alone, and that visual processing was more prominent when teaming with an inexperienced teammate. This work can inform human cognitive-motor processes and the design of robotic controllers in human-robot teams.

Effect of human emotional responses on human-robot team pty in construction

Effect of human emotional responses on human-robot team pty in construction

Unlocking underrepresented use-cases for large language model-driven human-robot task planning

Large language models (LLM) are now the de facto task planners for Embodied AI (EAI) systems. This shift can be attributed to LLMs' powerful, emergent properties which enable their adaptation to downstream tasks with minimal to no fine tuning via prompting. However, we find that LLM-driven task planning is not a solved problem. In this work we measure the extent to which these models can be adapted to complex and domain-specific task planning via few-shot prompting. Additionally, we contribute quantitative and qualitative analysis on prompt robustness. Lastly, to meet the challenges of adapting EAI systems to real-world, industrial domains, we adopt a human-in-the-loop approach to guarantee safe and interpretable task planning and execution. We successfully demonstrate co-located, human-robot teaming where an Augmented Reality (AR) headset mediates information exchanged between an EAI agent and human operator for a variety of inspection tasks. To our knowledge the use of an AR headset for multimodal grounding and the application of EAI to industrial tasks are novel contributions within Embodied AI research.

Generating Pattern-Based Conventions for Predictable Planning in Human-Robot Collaboration

For humans to effectively work with robots, they must be able to predict the actions and behaviors of their robot teammates rather than merely react to them. While there are existing techniques enabling robots to adapt to human behavior, there is a demonstrated need for methods that explicitly improve humans’ ability to understand and predict robot behavior at multi-task timescales. In this work, we propose a method leveraging the innate human propensity for pattern recognition in order to improve team dynamics in human-robot teams and to make robots more predictable to the humans that work with them. Patterns are a cognitive tool that humans use and rely on often, and the human brain is in many ways primed for pattern recognition and usage. We propose Pattern-Aware Convention-setting for Teaming (PACT), an entropy-based algorithm that identifies and imposes appropriate patterns over a robot’s planner or policy over long time horizons. These patterns are autonomously generated and chosen via an algorithmic process that considers human-perceptible features and characteristics derived from the tasks to be completed, and as such, produces behavior that is easier for humans to identify and predict. Our evaluation shows that PACT contributes to significant improvements in team dynamics and teammate perceptions of the robot, as compared to robots that utilize traditionally ‘optimal’ plans and robots utilizing unoptimized patterns.

HOTSPOT: An ad hoc teamwork platform for mixed human-robot teams

HOTSPOT: An ad hoc teamwork platform for mixed human-robot teams