Human-robot Teams Research Articles

Deep Learning-based human activity recognition using RGB images in Human-robot collaboration

In human-robot interaction, to ensure safety and effectiveness, robots need to be able to accurately predict human intentions. Hidden Markov Model, Bayesian Filtering, and deep learning methods have been used to predict human intentions. However, few studies have explored deep learning methods to predict variant human intention. Our study aims to evaluate the performance of the human intent recognition inference algorithm, and its impact on the human-robot team for collaborative tasks. Two deep learning algorithms ConvLSTM and LRCN were used to predict human intention. A dataset of 10 participants performing Pick, Throw, Wave, and Carry actions was used. The ConvLSTM method had a prediction accuracy of 74%. The LRCN method had a lower prediction accuracy of 25% compared to ConvLSTM. This result shows that deep learning methods using RGB images can predict human intent with high accuracy. The proposed method is successful in predicting human intents underlying human behavior.

Construction Robotics and Human–Robot Teams Research Methods

Though studies in construction robotics and human–robot teams research have explored varying themes, an in-depth study of the state-of-the-art of methodological paradigms appropriate for construction robotics research is hitherto lacking, taking into consideration that several studies have highlighted conflicting methodological components such as research design, methods, data sources, data types and analytical techniques. To better understand this underexplored area, this study uses a four-stage review approach utilising a scientometric and systematic analysis method based on 112 articles. Using statistical analysis to evaluate the relationship between research components, the study reveals strong associations between components of research methods, data sources and analytical techniques. Researchers are also increasingly looking to mixed paradigms in data sources and designs, highlighting a methodological plurality in construction robotics research. Implications on what this means for the future of construction robotics, policy and stakeholders are discussed in the study.

Influence of Mobile Robots on Human Safety Perception and System Productivity in Wholesale and Retail Trade Environments: A Pilot Study

Ensuring human safety has been one of the most critical considerations within the field of human–robot interaction. To explore the effects of working with autonomous robots on human coworkers’ perceived workload and job performance, two experiments were conducted in this study. Eight participants were recruited in the first experiment. Results revealed an increase in both subjective and objective workload measurements: compared to the baseline “no robot” to “empty payload” condition, the sum of NASA Task Load Index (NASA-TLX) scores increased from 129.3 (58.8) to 147.6 (53.7) ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$p$</tex-math></inline-formula> -value = 0.041) and the pupil diameter increased from 4.07 (0.64) mm to 4.11 (0.67) mm; while working with a full payload robot, the sum of NASA-TLX scores increased to 151.7 (55.9) ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$p$</tex-math></inline-formula> -value = 0.010) and the pupil diameter increased to 4.12 (0.66) mm. Increased task completion time (3.1% for empty payload condition and 6.6% for full payload condition) showed a decrease in human productivity. However, this slight human output reduction was compensated by the substantial gain from the robot. Similar effects of autonomous mobile robots on participants’ NASA-TLX scores and task completion time were observed in the second experiment, where eight participants performed the order picking and sorting tasks in a high-fidelity grocery store setting. Results from the study suggested the feasibility of applying fully autonomous mobile robots in Wholesale and Retail Trade settings to improve human–robot team productivity while prioritizing physical safety and reasonable increases in mental workload.

Discovering meaningful engagement through interaction between customers and service robots

ABSTRACT Although service robots are causing a paradigm shift in the service industry by creating new interactions with customers, there is limited knowledge regarding which elements of service robots create desirable interactions with customers. To address this gap, this study invited situated action theory to investigate elements of service robots shaping customers’ meaningful engagement. 252 responses from the survey were analyzed. These revealed that some socio-functional elements are primary for meaningful engagement by affecting both experiential and instrumental outcomes, whereas others are salient for either one of the outcomes. Our analysis also examined personal innovativeness and viability of human–robot team pertaining to meaningful engagement. This study contributes to the literature by identifying socio-functional elements of service robots and exploring how they create meaningful engagement. It also provides practical implications to service companies, demonstrating critical elements of service robots for enhancing their service quality and for developing robot-driven service environments.

How and When Can Robots Be Team Members? Three Decades of Research on Human–Robot Teams

Artificial intelligence and robotic technologies have grown in sophistication and reach. Accordingly, research into mixed human–robot teams that comprise both robots and humans has expanded as well, attracting the attention of researchers from different disciplines, such as organizational behavior, human–robot interaction, cognitive science, and robotics. With this systematic literature review, the authors seek to establish deeper insights into existing research and sharpen the definitions of relevant terms. With a close consideration of 150 studies published between 1990 and 2020 that investigate mixed human–robot teams, conceptually or empirically, this article provides both a systematic evaluation of extant research and propositions for further research.

Design and Evaluation of an Augmented Reality Head-mounted Display Interface for Human Robot Teams Collaborating in Physically Shared Manufacturing Tasks

We provide an experimental evaluation of a wearable augmented reality (AR) system we have developed for human-robot teams working on tasks requiring collaboration in shared physical workspace. Recent advances in AR technology have facilitated the development of more intuitive user interfaces for many human-robot interaction applications. While it has been anticipated that AR can provide a more intuitive interface to robot assistants helping human workers in various manufacturing scenarios, existing studies in robotics have been largely limited to teleoperation and programming. Industry 5.0 envisions cooperation between human and robot working in teams. Indeed, there exist many industrial tasks that can benefit from human-robot collaboration. A prime example is high-value composite manufacturing. Working with our industry partner towards this example application, we evaluated our AR interface design for shared physical workspace collaboration in human-robot teams. We conducted a multi-dimensional analysis of our interface using established metrics. Results from our user study (n = 26) show that, subjectively, the AR interface feels more novel and a standard joystick interface feels more dependable to users. However, the AR interface was found to reduce physical demand and task completion time, while increasing robot utilization. Furthermore, user’s freedom of choice to collaborate with the robot may also affect the perceived usability of the system.

Metrics for Robot Proficiency Self-assessment and Communication of Proficiency in Human-robot Teams

As development of robots with the ability to self-assess their proficiency for accomplishing tasks continues to grow, metrics are needed to evaluate the characteristics and performance of these robot systems and their interactions with humans. This proficiency-based human-robot interaction (HRI) use case can occur before, during, or after the performance of a task. This article presents a set of metrics for this use case, driven by a four-stage cyclical interaction flow: (1) robot self-assessment of proficiency (RSA), (2) robot communication of proficiency to the human (RCP), (3) human understanding of proficiency (HUP), and (4) robot perception of the human’s intentions, values, and assessments (RPH). This effort leverages work from related fields including explainability, transparency, and introspection, by repurposing metrics under the context of proficiency self-assessment. Considerations for temporal level (a priori, in situ, and post hoc) on the metrics are reviewed, as are the connections between metrics within or across stages in the proficiency-based interaction flow. This article provides a common framework and language for metrics to enhance the development and measurement of HRI in the field of proficiency self-assessment.

A Framework to Explore Proximate Human-Robot Coordination

Proximate human-robot teaming (pxHRT) is a complex subspace within human-robot interaction. Studies in this space involve a range of equipment and methods, including the ability to sense people and robots precisely. Research in this area draws from a wide variety of other fields, from human-human interaction to control theory, making the study design complex, particularly for those outside the field of HRI. In this paper, we introduce a framework that helps researchers consider tradeoffs across various task contexts, platforms, sensors, and analysis methods; metrics frequently used in the field; and common challenges researchers may face. We demonstrate the use of the framework via a case study which employs an autonomous mobile manipulator continuously engaging in shared workspace, handover, and co-manipulation tasks with people, and explores the effect of cognitive workload on pxHRT dynamics. We also demonstrate the utility of the framework in a case study with two groups of researchers new to pxHRT. With this framework, we hope to enable researchers, especially those outside HRI, to more thoroughly consider these complex components within their studies, more easily design experiments, and more fully explore research questions within the space of pxHRT.

Team robot identification theory (TRIT): robot attractiveness and team identification on performance and viability in human–robot teams

Prior literature suggests that shared identity and social attraction between team members and their robots can be vital for the human–robot interaction. However, more attention is needed to understand the potential performance benefits associated with team identification (TI) and robot attractiveness in human–robot teams. We proposed a theoretical framework of team robot identification theory. We conducted a laboratory experiment to examine the impacts of TI and social attraction toward robots on team performance and viability in 30 human–robot teams comprising two humans and two physical robots. Results showed that TI in human–robot teams led to better performance and team viability. Both effects were mediated by the social attraction between team members and their robots. These results evidenced the direct links between TI and objective and subjective team outcomes, explained through social attraction toward robots. We discuss the results and their theoretical and practical implications.

Performance and interaction quality variations of a collaborative Cable-Driven Parallel Robot

In the field of large scale robotic, which is often remotely operated, having direct interaction can be disruptive for applications such as moving heavy loads or 3D printing. A Cable-Driven Parallel Robot (CDPR) is used here in physical Human–Robot Interactions (pHRI) with an admittance-based control strategy to physically interact with a user in tele-operation or in co-manipulation mode. A user experiment involving participants is designed to assess the performance of the human–robot team in a given task completion. Task performance and interaction quality metrics are defined and recorded during experiments with different robot configurations. The novelty is to provide a methodology to compare the configurations based on the performance metrics. The methodology accounts for variations of the metrics along time of use and assert a training effect leading to a progression or a regression of the performances. The experiment apparatus includes a CDPR, a user task composed of targets to reach with the robot and a handle equipped with a force sensor acting as a control input of a fixed admittance control strategy of the robot. Collected data show that the task performances and the interaction quality vary during the experiments and denote different variation profiles among the user population. Distribution of these profiles among configurations are analysed to determine the configuration that has the best training effect on users.

Team Robot Identification Theory (TRIT): Robot Attractiveness and Team Identification on Performance and Viability in Human–Robot Teams

Team Robot Identification Theory (TRIT): Robot Attractiveness and Team Identification on Performance and Viability in Human–Robot Teams

Bidirectional Communications in Human-Agent Teaming: The Effects of Communication Style and Feedback

We examined the effects of communication style on human performance, workload, situation awareness, and trust in a robot in a human-robot team. In a 2 × 2 mixed factor study, participants were teamed with a simulated robot to conduct a cordon-and-search style task. Participants were assigned to a communication style (Directive vs. Non-directive; between subjects), and both groups experienced varied periods of task load (high vs. low task load; within-subjects). Results indicate that task load influenced the participants' task performance more than communication style. However, there were some differential effects on response time and workload due to communication style. Participants in the Non-directive group did not report a higher workload than those in the Directive condition, even though objective measures of workload (i.e., eye-tracking measures) indicated they experienced a higher workload. These results may be due to the presence of feedback inherent in the differing communication styles.

Research focus for construction robotics and human-robot teams towards resilience in construction: scientometric review

PurposeRapid urbanisation and recent shock events have reiterated the need for resilient infrastructure, as seen in the pandemic. Yet, knowledge gaps in construction robotics and human–robot teams (HRTs) research limit maximising these emerging technologies’ potentials. This paper aims to review the state of the art of research in this area to identify future research directions in HRTs able to aid the resilience and responsiveness of the architecture, engineering and construction (AEC) sector.Design/methodology/approachA total of 71 peer-reviewed journal articles centred on robotics and HRTs were reviewed through a quantitative approach using scientometric techniques using Gephi and VOSviewer. Research focus deductions were made through bibliometric analysis and co-occurrence analysis of reviewed publications.FindingsThis study revealed sparse and small research output in this area, indicating immense research potential. Existing clusters signifying the need for further studies are on automation in construction, human–robot teaming, safety in robotics and robotic designs. Key publication outlets and construction robotics contribution towards the built environment’s resilience are discussed.Practical implicationsThe identified gaps in the thematic areas illustrate priorities for future research focus. It raises awareness on human factors in collaborative robots and potential design needs for construction resilience.Originality/valueRapid urbanisation and recent shock events have reiterated the need for resilient infrastructure, as seen in the pandemic. Yet, knowledge gaps in construction robotics and HRTs research limit maximising these emerging technologies’ potentials. This paper aims to review the state of the art of research in this area to identify future research directions in HRTs able to aid the resilience and responsiveness of the AEC sector.

Anytime 3D Object Reconstruction Using Multi-Modal Variational Autoencoder

For effective human-robot teaming, it is important for the robots to be able to share their visual perception with the human operators. In a harsh remote collaboration setting, data compression techniques such as autoencoder can be utilized to obtain and transmit the data in terms of latent variables in a compact form. In addition, to ensure real-time runtime performance even under unstable environments, an anytime estimation approach is desired that can reconstruct the full contents from incomplete information. In this context, we propose a method for imputation of latent variables whose elements are partially lost. To achieve the anytime property with only a few dimensions of variables, exploiting prior information of the category-level is essential. A prior distribution used in variational autoencoders is simply assumed to be isotropic Gaussian regardless of the labels of each training datapoint. This type of flattened prior makes it difficult to perform imputation from the category-level distributions. We overcome this limitation by exploiting a category-specific multi-modal prior distribution in the latent space. The missing elements of the partially transferred data can be sampled, by finding a specific modal according to the remaining elements. Since the method is designed to use partial elements for anytime estimation, it can also be applied for data over-compression. Based on the experiments on the ModelNet and Pascal3D datasets, the proposed approach shows consistently superior performance over autoencoder and variational autoencoder up to 70% data loss. The software is open source and is available from our repository <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .

Design patterns for human-AI co-learning: A wizard-of-Oz evaluation in an urban-search-and-rescue task

The rapid advancement of technology empowered by artificial intelligence is believed to intensify the collaboration between humans and AI as team partners. Successful collaboration requires partners to learn about each other and about the task. This human-AI co-learning can be achieved by presenting situations that enable partners to share knowledge and experiences. In this paper we describe the development and implementation of a task context and procedures for studying co-learning. More specifically, we designed specific sequences of interactions that aim to initiate and facilitate the co-learning process. The effects of these interventions on learning were evaluated in an experiment, using a simplified virtual urban-search-and-rescue task for a human-robot team. The human participants performed a victim rescue- and evacuation mission in collaboration with a wizard-of-Oz (i.e., a confederate of the experimenter who executed the robot-behavior consistent with an ontology-based AI-model). The designed interaction sequences, formulated as Learning Design Patterns (LDPs), were intended to bring about co-learning. Results show that LDPs support the humans understanding and awareness of their robot partner and of the teamwork. No effects were found on collaboration fluency, nor on team performance. Results are used to discuss the importance of co-learning, the challenges of designing human-AI team tasks for research into this phenomenon, and the conditions under which co-learning is likely to be successful. The study contributes to our understanding of how humans learn with and from AI-partners, and our propositions for designing intentional learning (LDPs) provide directions for applications in future human-AI teams.

Trust in the Danger Zone: Individual Differences in Confidence in Robot Threat Assessments.

Effective human–robot teaming (HRT) increasingly requires humans to work with intelligent, autonomous machines. However, novel features of intelligent autonomous systems such as social agency and incomprehensibility may influence the human’s trust in the machine. The human operator’s mental model for machine functioning is critical for trust. People may consider an intelligent machine partner as either an advanced tool or as a human-like teammate. This article reports a study that explored the role of individual differences in the mental model in a simulated environment. Multiple dispositional factors that may influence the dominant mental model were assessed. These included the Robot Threat Assessment (RoTA), which measures the person’s propensity to apply tool and teammate models in security contexts. Participants (N = 118) were paired with an intelligent robot tasked with making threat assessments in an urban setting. A transparency manipulation was used to influence the dominant mental model. For half of the participants, threat assessment was described as physics-based (e.g., weapons sensed by sensors); the remainder received transparency information that described psychological cues (e.g., facial expression). We expected that the physics-based transparency messages would guide the participant toward treating the robot as an advanced machine (advanced tool mental model activation), while psychological messaging would encourage perceptions of the robot as acting like a human partner (teammate mental model). We also manipulated situational danger cues present in the simulated environment. Participants rated their trust in the robot’s decision as well as threat and anxiety, for each of 24 urban scenes. They also completed the RoTA and additional individual-difference measures. Findings showed that trust assessments reflected the degree of congruence between the robot’s decision and situational danger cues, consistent with participants acting as Bayesian decision makers. Several scales, including the RoTA, were more predictive of trust when the robot was making psychology-based decisions, implying that trust reflected individual differences in the mental model of the robot as a teammate. These findings suggest scope for designing training that uncovers and mitigates the individual’s biases toward intelligent machines.

An Intelligence Architecture for Grounded Language Communication with Field Robots

For humans and robots to collaborate effectively as teammates in unstructured environments, robots must be able to construct semantically rich models of the environment, communicate efficiently with teammates, and perform sequences of tasks robustly with minimal human intervention, as direct human guidance may be infrequent and/or intermittent. Contemporary architectures for human-robot interaction often rely on engineered human-interface devices or structured languages that require extensive prior training and inherently limit the kinds of information that humans and robots can communicate. Natural language, particularly when situated with a visual representation of the robot’s environment, allows humans and robots to exchange information about abstract goals, specific actions, and/or properties of the environment quickly and effectively. In addition, it serves as a mechanism to resolve inconsistencies in the mental models of the environment across the human-robot team. This article details a novel intelligence architecture that exploits a centralized representation of the environment to perform complex tasks in unstructured environments. The centralized environment model is informed by a visual perception pipeline, declarative knowledge, deliberate interactive estimation, and a multimodal interface. The language pipeline also exploits proactive symbol grounding to resolve uncertainty in ambiguous statements through inverse semantics. A series of experiments on three different, unmanned ground vehicles demonstrates the utility of this architecture through its robust ability to perform language-guided spatial navigation, mobile manipulation, and bidirectional communication with human operators. Experimental results give examples of component-level behaviors and overall system performance that guide a discussion on observed performance and opportunities for future innovation.

Diversity Training With Robots: Perspective-Taking Backfires, While Sterotype-Suppression Decreases Negative Attitudes Towards Robots.

The present research investigated the effects of a diversity training intervention on robot-related attitudes to test whether this could help to manage the diversity inherent in hybrid human-robot teams in the work context. Previous research in the human-human context has shown that stereotypes and prejudice, i.e., negative attitudes, may impair productivity and job satisfaction in teams high in diversity (e.g., regarding age, gender, or ethnicity). Relatedly, in hybrid human-robot teams, robots likely represent an “outgroup” to their human co-workers. The latter may have stereotypes towards robots and may hold negative attitudes towards them. Both aspects might have detrimental effects on subjective and objective performance in human-robot interactions (HRI). In an experiment, we tested the effect of an economic and easy to apply diversity training intervention for use in the work context: The so-called enlightenment approach. This approach utilizes perspective-taking to reduce prejudice and discrimination in human-human contexts. We adapted this intervention to the HRI context and explored its impact on participants’ implicit and explicit robot-related attitudes. However, contrary to our predictions, taking the perspective of a robot resulted in more negative robot-related attitudes, whereas actively suppressing stereotypes about social robots and their characteristics produced positive effects on robot attitudes. Therefore, we recommend considering potential pre-existing aversions against taking the perspective of a robot when designing interventions to improve human-robot collaboration at the workplace. Instead, it might be useful to provide information about existing stereotypes and their consequences, thereby making people aware of their potential biases against social robots.

Subgroup formation in human–robot teams: A multi‐study mixed‐method approach with implications for theory and practice

AbstractHuman–robot teams represent a challenging work application of artificial intelligence (AI). Building strong emotional bonds with robots is one solution to promoting teamwork in such teams, but does this come at a cost in the form of subgroups? Subgroups—smaller divisions within teams—in all human teams can undermine teamwork. Despite the importance of this question, it has received little attention. We employed a mixed‐methods approach by conducting a lab experiment and a qualitative online survey. We (a) examined the formation and impact of subgroups in human–robot teams and (b) obtained insights from workers currently adapting to robots in the workplace on mitigating impacts of subgroups. The experimental study (Study 1) with 44 human–robot teams found that robot identification (RID) and team identification (TID) are associated with increases and decreases in the likelihood of a subgroup formation, respectively. RID and TID moderated the impacts of subgroups on teamwork quality and subsequent performance in human–robot teams. Study 2 was a qualitative study with 112 managers and employees who worked collaboratively with robots. We derived practical insights from this study that help situate and translate what was learned in Study 1 into actual work practices.

Working With Robots: Human Resource Development Considerations in Human–Robot Interaction

Advancements in robotic technology have accelerated the adoption of collaborative robots in the workplace. The role of humans is not reduced, but robotic technology requires different high-level responsibilities in human–robot interaction (HRI). Based on a human-centered perspective, this literature review is to explore current knowledge on HRI through the lens of HRD and propose the roles of HRD in this realm. The review identifies HRD considerations that help implement effective HRI in three human-centered domains: human capabilities, collaboration configuration, and attributes related to contact. The eight HRD considerations include employees’ attitudes toward robots, their readiness for robot technology, communication with robots, human–robot team building, leading multiple robots, systemwide collaboration, safety interventions, and ethical issues. Theoretical implications, practical implications, and limitations are discussed. This paper contributes to HRD by introducing potential areas of multidisciplinary collaborations to help organizations implement robotic systems.