Human-robot Collaborative Task Research Articles

Evaluation of Biomechanical and Mental Workload During Human-Robot Collaborative Pollination Task.

The purpose of this study is to identify the potential biomechanical and cognitive workload effects induced by human robot collaborative pollination task, how additional cues and reliability of the robot influence these effects and whether interacting with the robot influences the participant's anxiety and attitude towards robots. Human-Robot Collaboration (HRC) could be used to alleviate pollinator shortages and robot performance issues. However, the effects of HRC for this setting have not been investigated. Sixteen participants were recruited. Four HRC modes, no cue, with cue, unreliable, and manual control were included. Three categories of dependent variables were measured: (1) spine kinematics (L5/S1, L1/T12, and T1/C7), (2) pupillary activation data, and (3) subjective measures such as perceived workload, robot-related anxiety, and negative attitudes towards robotics. HRC reduced anxiety towards the cobot, decreased joint angles and angular velocity for the L5/S1 and L1/T12 joints, and reduced pupil dilation, with the "with cue" mode producing the lowest values. However, unreliability was detrimental to these gains. In addition, HRC resulted in a higher flexion angle for the neck (i.e., T1/C7). HRC reduced the physical and mental workload during the simulated pollination task. Benefits of the additional cue were minimal compared to no cues. The increased joint angle in the neck and unreliability affecting lower and mid back joint angles and workload requires further investigation. These findings could be used to inform design decisions for HRC frameworks for agricultural applications that are cognizant of the different effects induced by HRC.

Implementation of hand‐object pose estimation using SSD and YOLOV5 model for object grasping by SCARA robot

AbstractEnforcement of advanced deep learning methods in hand‐object pose estimation is an imperative method for grasping the objects safely during the human–robot collaborative tasks. The position and orientation of a hand‐object from a two‐dimensional image is still a crucial problem under various circumstances like occlusion, critical lighting, and salient region detection and blur images. In this paper, the proposed method uses an enhanced MobileNetV3 with single shot detection (SSD) and YOLOv5 to ensure the improvement in accuracy and without compromising the latency in the detection of hand‐object pose and its orientation. To overcome the limitations of higher computation cost, latency and accuracy, the Network Architecture Search and NetAdapt Algorithm is used in MobileNetV3 that perform the network search for parameter tuning and adaptive learning for multiscale feature extraction and anchor box offset adjustment due to auto‐variance of weight in the level of each layers. The squeeze‐and‐excitation block reduces the computation and latency of the model. Hard‐swish activation function and feature pyramid networks are used to prevent over fitting the data and stabilizing the training. Based on the comparative analysis of MobileNetV3 with its predecessor and YOLOV5 are carried out, the obtained results are 92.8% and 89.7% of precision value, recall value of 93.1% and 90.2%, mAP value of 93.3% and 89.2%, respectively. The proposed methods ensure better grasping for robots by providing the pose estimation and orientation of hand‐objects with tolerance of −1.9 to 2.15 mm along x, −1.55 to 2.21 mm along y, −0.833 to 1.51 mm along z axis and −0.233° to 0.273° along z‐axis.

Hierarchical approximate optimal interaction control of human-centered modular robot manipulator systems: A Stackelberg differential game-based approach

A Stackelberg game-based approximate optimal interaction control approach is presented for human-centered modular robot manipulator (MRM) systems. Joint torque feedback (JTF) technique is utilized to form the MRM dynamic model. The major objective of optimal control with human–robot collaboration (HRC) is transformed into approximating Stackelberg equilibrium by adopting Stackelberg game governed between the human and the MRM that are regarded as players with different hierarchical level in interaction process. On the basis of the adaptive dynamic programming (ADP), the approximate optimal interaction control policy with HRC task is developed via critic neural network (NN)-based Stackelberg game manner for addressing Hamilton–Jacobian (HJ) equations. The position tracking error is ultimately uniformly bounded (UUB) according to the Lyapunov theory. Experiment results demonstrate the effectiveness of the proposed method.

Perception–Intention–Action Cycle in Human–Robot Collaborative Tasks: The Collaborative Lightweight Object Transportation Use-Case

AbstractThis study proposes to improve the reliability, robustness and human-like nature of Human–Robot Collaboration (HRC). For that, the classical Perception–Action cycle is extended to a Perception–Intention–Action (PIA) cycle, which includes an Intention stage at the same level as the Perception one, being in charge of obtaining both the implicit and the explicit intention of the human, opposing to classical approaches based on inferring everything from perception. This complete cycle is exposed theoretically including its use of the concept of Situation Awareness, which is shown as a key element for the correct understanding of the current situation and future action prediction. This enables the assignment of roles to the agents involved in a collaborative task and the building of collaborative plans. To visualize the cycle, a collaborative transportation task is used as a use-case. A force-based model is designed to combine the robot’s perception of its environment with the force exerted by the human and other factors in an illustrative way. Finally, a total of 58 volunteers participate in two rounds of experiments. In these, it is shown that the human agrees to explicitly state their intention without undue extra effort and that the human understands that this helps to minimize robot errors or misunderstandings. It is also shown that a system that correctly combines inference with explicit elicitation of the human’s intention is the best rated by the human on multiple parameters related to effective Human–Robot Interaction (HRI), such as perceived safety or trust in the robot.

Benchmarking human-robot collaborative assembly tasks

Manufacturing assembly tasks can vary in complexity and level of automation. Yet, achieving full automation can be challenging and inefficient, particularly due to the complexity of certain assembly operations. Human-robot collaborative work, leveraging the strengths of human labor alongside the capabilities of robots, can be a solution for enhancing efficiency. This paper introduces the CT benchmark, a benchmark and model set designed to facilitate the testing and evaluation of human-robot collaborative assembly scenarios. It was designed to compare manual and automatic processes using metrics such as the assembly time and human workload. The components of the model set can be assembled through the most common assembly tasks, each with varying levels of difficulty. The CT benchmark was designed with a focus on its applicability in human-robot collaborative environments, with the aim of ensuring the reproducibility and replicability of experiments. Experiments were carried out to assess assembly performance in three different setups (manual, automatic and collaborative), measuring metrics related to the assembly time and the workload on human operators. The results suggest that the collaborative approach takes longer than the fully manual assembly, with an increase of 70.8%. However, users reported a lower overall workload, as well as reduced mental demand, physical demand, and effort according to the NASA-TLX questionnaire.

Do Humans Trust Robots that Violate moral trust?

The increasing use of robots in social applications requires further research on human-robot trust. The research on human-robot trust needs to go beyond the conventional definition that mainly focuses on how human-robot relations are influenced by robot performance. The emerging field of social robotics considers optimizing a robot’s personality a critical factor in user perceptions of experienced human-robot interaction (HRI). Researchers have developed trust scales that account for different dimensions of trust in HRI. These trust scales consider one performance aspect (i.e., the trust in an agent’s competence to perform a given task and their proficiency in executing the task accurately) and one moral aspect (i.e., trust in an agent’s honesty in fulfilling their stated commitments or promises) for human-robot trust. The question that arises here is to what extent do these trust aspects affect human trust in a robot? The main goal of this study is to investigate whether a robot’s undesirable behavior due to the performance trust violation would affect human trust differently than another similar undesirable behavior due to a moral trust violation. We designed and implemented an online human-robot collaborative search task that allows distinguishing between performance and moral trust violations by a robot. We ran these experiments on Prolific and recruited 100 participants for this study. Our results showed that a moral trust violation by a robot affects human trust more severely than a performance trust violation with the same magnitude and consequences.

Towards the Legibility of Multi-Robot Systems

Communication is crucial for human-robot collaborative tasks. In this context, legibility studies movement as the means of implicit communication between robotic systems and a human observer. This concept has been explored mostly for manipulators and humanoid robots. In contrast, little information is available in the literature about legibility of multi-robot systems or swarms, where simplicity and non-anthropomorphism of robots, along with the complexity of their interactions and aggregated behavior impose different challenges that are not encountered in single-robot scenarios. This paper investigates legibility of multi-robot systems. Hence, we extend the definition of legibility, incorporating information about high-level goals in terms of the coordination objective of the group of robots, to previous results that focused solely on the legibility of spatial goals. A set of standard multi-robot algorithms corresponding to different coordination objectives are implemented and their legibility is evaluated in a user study, where participants observe the behavior of the multi-robot system in a virtual reality setup and are asked to identify the system’s spatial goal and coordination objective. The results of the study confirmed that coordination objectives are discernible by the users, hence multi-robot systems can be controlled to be legible, in terms of spatial goal and coordination objective.

FedHIP: Federated learning for privacy-preserving human intention prediction in human-robot collaborative assembly tasks

Human-robot collaboration is a promising solution to relieve construction workers from repetitive and physically demanding tasks, thus improving construction safety and productivity. Many studies have developed various deep learning models for human intention prediction, which will form the basis for proactive and adaptive robot planning and control to enable intelligent human-robot collaboration. However, there remain two challenges. First, most research only focuses on a single type of human intention, without a holistic understanding of multi-level intention, including both high-level intended actions and objects of interest, and low-level body movements. Second, conventional deep learning approaches train a centralized model with aggregated datasets, which requires the sharing of sensitive information (e.g., personal images and behavior data), posing broad privacy concerns in practical implementation. This study proposes a vision-based multi-task federated learning (FL) framework, FedHIP, for multi-level human intention prediction in human-robot collaborative assembly tasks. Specifically, taking body movements and assembly components as inputs, a long short-term memory based multi-task learning model was developed to simultaneously predict multi-level human intention in assembly tasks. FL was employed to train the model in a distributed and privacy-preserving way on local clients without the need of transmitting sensitive data. The results show that the proposal FedHIP without and with pre-train can achieve an accuracy of 80.1% and 85.7% in action prediction, 97.6% and 97.8% in object prediction, and an average displacement error of 12.7 and 11.7 pixels in motion prediction, respectively. Models trained from FedHIP were also compared with those obtained from traditional centralized training and local training. It was found that FL leads to compatible accuracy with centralized training and much higher accuracy than local training while preserving data privacy.

Human-robot collaborative task planning for assembly system productivity enhancement

PurposeIn the current era of Industry 4.0, the manufacturing industries are striving toward mass production with mass customization by considering human–robot collaboration. This study aims to propose the reconfiguration of assembly systems by incorporating multiple humans with robots using a human–robot task allocation (HRTA) to enhance productivity.Design/methodology/approachA human–robot task scheduling approach has been developed by considering task suitability, resource availability and resource selection through multicriteria optimization using the Linear Regression with Optimal Point and Minimum Distance Calculation algorithm. Using line-balancing techniques, the approach estimates the optimum number of resources required for assembly tasks operating by minimum idle time.FindingsThe task allocation schedule for a case study involving a punching press was solved using human–robot collaboration, and the approach incorporated the optimum number of appropriate resources to handle different types of proportion of resources.Originality/valueThis proposed work integrates the task allocation by human–robot collaboration and decrease the idle time of resource by integrating optimum number of resources.

Factors Affecting Workers' Mental Stress in Handover Activities During Human-Robot Collaboration.

This study investigated the effects of different approach directions, movement speeds, and trajectories of a co-robot's end-effector on workers' mental stress during handover tasks. Human-robot collaboration (HRC) is gaining attention in industry and academia. Understanding robot-related factors causing mental stress is crucial for designing collaborative tasks that minimize workers' stress. Mental stress in HRC tasks was measured subjectively through self-reports and objectively through galvanic skin response (GSR) and electromyography (EMG). Robot-related factors including approach direction, movement speed, and trajectory were analyzed. Movement speed and approach direction had significant effects on subjective ratings, EMG, and GSR. High-speed and approaching from one side consistently resulted in higher fear, lower comfort, and predictability, as well as increased EMG and GSR signals, indicating higher mental stress. Movement trajectory affected GSR, with the sudden stop condition eliciting a stronger response compared to the constrained trajectory. Interaction effects between speed and approach direction were observed for "surprise" and "predictability" subjective ratings. At high speed, approach direction did not significantly differ, but at low speeds, approaching from the side was found to be more surprising and unpredictable compared to approaching from the front. The mental stress of workers during HRC is lower when the robot's end effector (1) approaches a worker within the worker's field of view, (2) approaches at a lower speed, or (3) follows a constrained trajectory. The outcome of this study can serve as a guide to design HRC tasks with a low level of workers' mental stress.

Estimating 6D Object Poses with Temporal Motion Reasoning for Robot Grasping in Cluttered Scenes

6D object pose estimation is an essential task in vision-based robotic grasping and manipulation. Prior works extract object's 6D pose by regressing from single RGB-D frame without considering the occluded objects in the frame, limiting their performance in human-robot collaboration scenarios with heavy occlusion. In this paper, we present an end-to-end model named \textit{TemporalFusion}, which integrates the temporal motion information from RGB-D images for 6D object pose estimation. The core of proposed model is to embed and fuse the temporal motion information from multi-frame RGB-D sequences, which could handle heavy occlusion in human-robot collaboration tasks. Furthermore, the proposed deep model can also obtain stable pose sequences, which is essential for real-time robotic grasping tasks. We evaluated the proposed method in the YCB-Video dataset, and experimental results show our model outperforms state-of-the-art approaches. Our code is available at https://github.com/mufengjun260/H-MPose.

Diffusion Co-Policy for Synergistic Human-Robot Collaborative Tasks

Diffusion Co-Policy for Synergistic Human-Robot Collaborative Tasks

RTMN 2.0—An Extension of Robot Task Modeling and Notation (RTMN) Focused on Human–Robot Collaboration

This paper describes RTMN 2.0, an extension of the modeling language RTMN. RTMN combines process modeling and robot execution. Intuitive robot programming allows those without programming expertise to plan and control robots through easily understandable predefined modeling notations. These notations achieve no-code programming and serve as templates for users to create their processes via drag-and-drop functions with graphical representations. The design of the graphical user interface is based on a user survey and gaps identified in the literature We validate our survey through the most influential technology acceptance models, with two major factors: the perceived ease of use and perceived usefulness. While RTMN focuses on the ease of use and flexibility of robot programming by providing an intuitive modeling language, RTMN 2.0 concentrates on human–robot collaboration (HRC), which represents the current trend of the industry shift from “mass-production” to “mass-customization”. The biggest contribution that RTMN 2.0 makes is the creation of synergy between HRC modes (based on ISO standards) and HRC task types in the literature. They are modeled as five different HRC task notations: Coexistence Fence, Sequential Cooperation SMS, Teaching HG, Parallel Cooperation SSM, and Collaboration PFL. Both collaboration and safety criteria are defined for each notation. While traditional isolated robot systems in “mass-production” environments provide high payload capabilities and repeatability, they suffer from limited flexibility and dexterity in order to be adapted to the variability of customized products. Therefore, human–robot collaboration is a suitable arrangement to leverage the unique capabilities of both humans and robots for increased efficiency and quality in the new “mass-customization” industrial environments. HRC has made a great impact on the robotic industry: it leads to increased efficiency, reduced costs, and improved productivity, which can be adopted to make up for the skill gap of a shortage of workers in the manufacturing industry. The extension of RTMN 2.0 includes the following notations: HRC tasks, requirements, Key Performance Indicators (KPIs), condition checks and decision making, join/split, and data association. With these additional elements, RTMN 2.0 meets the full range of criteria for agile manufacturing—light-out manufacturing is a manufacturing philosophy that does not rely on human labor.

Task reallocation of human-robot collaborative production workshop based on a dynamic human fatigue model

As collaborative robots become more popular in industry, they are used to collaborate with human workers and free human workers from repeated and heavy work and reduce the risk of physical injuries. However, human workers need to finish tasks that require high cognitive abilities, which increases the risk of mental fatigue. Fatigue can negatively impact workers and even threaten the safety of workers. Currently, the dynamics of human fatigue model are rarely considered. Hence, it is crucial to develop a dynamic allocation method for human-robot collaborative tasks that considers dynamic human fatigue to ensure efficient and safe human-robot collaboration. This paper proposes a task reallocation method for human-robot collaborative production workshop based on a dynamic human fatigue model. A fatigue feature extraction method from multimodal data and a fatigue evaluation network are proposed to evaluate the current fatigue status of human workers. Based on the evaluation results, the human fatigue model is dynamically updated. If the worker’s fatigue state changes, a task reallocation scheme is generated by an improved genetic algorithm based on reinforcement learning and the dynamic human fatigue model. The experimental results show that the proposed model and algorithm are effective and efficient, and can provide innovative insights for realizing a human-centered manufacturing workshop.

Multi-modal interaction with transformers: bridging robots and human with natural language

AbstractThe language-guided visual robotic grasping task focuses on enabling robots to grasp objects based on human language instructions. However, real-world human-robot collaboration tasks often involve situations with ambiguous language instructions and complex scenarios. These challenges arise in the understanding of linguistic queries, discrimination of key concepts in visual and language information, and generation of executable grasping configurations for the robot’s end-effector. To overcome these challenges, we propose a novel multi-modal transformer-based framework in this study, which assists robots in localizing spatial interactions of objects using text queries and visual sensing. This framework facilitates object grasping in accordance with human instructions. Our developed framework consists of two main components. First, a visual-linguistic transformer encoder is employed to model multi-modal interactions for objects referred to in the text. Second, the framework performs joint spatial localization and grasping. Extensive ablation studies have been conducted on multiple datasets to evaluate the advantages of each component in our model. Additionally, physical experiments have been performed with natural language-driven human-robot interactions on a physical robot to validate the practicality of our approach.

Unified Human–Robot–Environment Interaction Control in Contact-Rich Collaborative Manipulation Tasks via Model-Based Reinforcement Learning

Many safety-critical or performance-demanding systems are human-in-the-loop, i.e., the robot interacts with human being and environment, in which human-in-the-loop control becomes a key research topic. In this paper, a unified optimal interaction control in joint space is presented for multi-point human-robot-environment interaction (HREI) problems, which are very common in human-robot collaborative manipulation tasks. Specifically, model-based reinforcement learning method is leveraged for obtaining optimal interaction control. For multi-point human-robot-environment interaction, the interaction forces exerted on each link are isolated and estimated via the backward generalized momentum observer method. In human-robot-environment interaction problems, the environmental as well as the human arm's dynamics parameters are usually unknown, stochastic, and time-varying. To obviate the dependence on these parameters, the GMM/GMR method is employed to learn the unknown external dynamics. It is noteworthy the interaction forces are considered as system states, the time derivatives of which are computed based on the GMM/GMR learning results via chain-rule. Then the iterative linear quadratic Gaussian with learned external dynamics (ILQG-LED) method is utilized to realize optimal multi-point human-robot-environment interaction control. The validity of the proposed method is verified through experimental studies.

Optimal resource allocation for multiple shop floor tasks in collaborative assembly

Industries are adopting advanced technologies to attain the goals of the manufacturing system and the demands of the current industrial era. A collaborative and flexible manufacturing system that integrates human intelligence with a robotic workforce leads to customized automation and reinforces the safety of the human workforce. This system creates a flexible platform to assemble the parts as per the requirements and it supports reconfigurability with respective upcoming technologies. The proposed work involves an approach of a human-robot collaborative task allocation (HRTA) system. The HRTA can concern the demand for new product variants in the market and the upgradation of new technologies in the existing manufacturing system. The proposed work is organized into two phases such as resource allocation, and resource estimation using an assembly line balancing approach. The resource allocation approach is developed to identify the appropriate resource(human/robot) for each assembly task using the Nelder-Mead simplex optimization approach. Further assembly line balancing technique is used to quantify the number of resources to minimize the total idle time. The performance of the proposed method is corroborated by implementing it on an industrial case study and the obtained results are discussed with suitable illustrations.

Interdisciplinary evaluation of a robot physically collaborating with workers.

Collaborative Robots-CoBots-are emerging as a promising technological aid for workers. To date, most CoBots merely share their workspace or collaborate without contact, with their human partners. We claim that robots would be much more beneficial if they physically collaborated with the worker, on high payload tasks. To move high payloads, while remaining safe, the robot should use two or more lightweight arms. In this work, we address the following question: to what extent can robots help workers in physical human-robot collaboration tasks? To find an answer, we have gathered an interdisciplinary group, spanning from an industrial end user to cognitive ergonomists, and including biomechanicians and roboticists. We drew inspiration from an industrial process realized repetitively by workers of the SME HANKAMP (Netherlands). Eleven participants replicated the process, without and with the help of a robot. During the task, we monitored the participants' biomechanical activity. After the task, the participants completed a survey with usability and acceptability measures; seven workers of the SME completed the same survey. The results of our research are the following. First, by applying-for the first time in collaborative robotics-Potvin's method, we show that the robot substantially reduces the participants' muscular effort. Second: we design and present an unprecedented method for measuring the robot reliability and reproducibility in collaborative scenarios. Third: by correlating the worker's effort with the power measured by the robot, we show that the two agents act in energetic synergy. Fourth: the participant's increasing level of experience with robots shifts his/her focus from the robot's overall functionality towards finer expectations. Last but not least: workers and participants are willing to work with the robot and think it is useful.

Eye-Tracking in Physical Human-Robot Interaction: Mental Workload and Performance Prediction.

In Physical Human-Robot Interaction (pHRI), the need to learn the robot's motor-control dynamics is associated with increased cognitive load. Eye-tracking metrics can help understand the dynamics of fluctuating mental workload over the course of learning. The aim of this study was to test eye-tracking measures' sensitivity and reliability to variations in task difficulty, as well as their performance-prediction capability, in physical human-robot collaboration tasks involving an industrial robot for object comanipulation. Participants (9M, 9F) learned to coperform a virtual pick-and-place task with a bimanual robot over multiple trials. Joint stiffness of the robot was manipulated to increase motor-coordination demands. The psychometric properties of eye-tracking measures and their ability to predict performance was investigated. Stationary Gaze Entropy and pupil diameter were the most reliable and sensitive measures of workload associated with changes in task difficulty and learning. Increased task difficulty was more likely to result in a robot-monitoring strategy. Eye-tracking measures were able to predict the occurrence of success or failure in each trial with 70% sensitivity and 71% accuracy. The sensitivity and reliability of eye-tracking measures was acceptable, although values were lower than those observed in cognitive domains. Measures of gaze behaviors indicative of visual monitoring strategies were most sensitive to task difficulty manipulations, and should be explored further for the pHRI domain where motor-control and internal-model formation will likely be strong contributors to workload. Future collaborative robots can adapt to human cognitive state and skill-level measured using eye-tracking measures of workload and visual attention.

A Survey of Deep Learning Techniques for Underwater Image Classification.

In recent years, there has been an enormous interest in using deep learning to classify underwater images to identify various objects, such as fishes, plankton, coral reefs, seagrass, submarines, and gestures of sea divers. This classification is essential for measuring the water bodies' health and quality and protecting the endangered species. Furthermore, it has applications in oceanography, marine economy and defense, environment protection, underwater exploration, and human-robot collaborative tasks. This article presents a survey of deep learning techniques for performing underwater image classification. We underscore the similarities and differences of several methods. We believe that underwater image classification is one of the killer application that would test the ultimate success of deep learning techniques. Toward realizing that goal, this survey seeks to inform researchers about state-of-the-art on deep learning on underwater images and also motivate them to push its frontiers forward.