Human-robot Collaborative Assembly Research Articles

A vision-language-guided robotic action planning approach for ambiguity mitigation in human–robot collaborative manufacturing

Human–robot collaboration (HRC) has been recognized as a potent pathway towards mass personalization in the manufacturing sector, by leveraging the synergy of human creativity and robotic precision. Previous approaches rely heavily on visual perception to autonomously comprehend the HRC environment. However, the inherent ambiguity in human–robot communication cannot be consistently neutralized by relying solely on visual cues. With the recently soaring popularity of large language models (LLMs), the consideration of language data as a complementary information source has increasingly drawn research attention, while the application of such large models, particularly within the context of HRC in manufacturing, remains largely under-explored. In response to this gap, a vision-language reasoning approach is proposed to mitigate the communication ambiguity prevalent in human–robot collaborative manufacturing scenarios. A referred object retrieval model is first designed to alleviate the object–reference ambiguity in the human language command. This model is then seamlessly integrated into an LLM-based robotic action planner to achieve an improved HRC performance. The effectiveness of the proposed approach is demonstrated empirically through a series of experiments conducted on the object retrieval model and its application in a human–robot collaborative assembly case.

A new XR-based human‐robot collaboration assembly system based on industrial metaverse

The industrial metaverse is an emerging paradigm of integrated information technologies. A human-robot collaboration assembly system is developed using extended reality (XR) and blockchain technologies. This system includes the physical, virtual reality (VR), augmented reality (AR) supporting, and blockchain systems. Virtual avatars represent physical operators and mobile robots in the XR environment. The VR system emulates real-time physical operations, facilitating pre-planning, online task optimization, and converting strategic decisions into executable instructions for physical robots. The AR system enhances the cognitive experience of intelligent operators, contributing to decision-making and VR system control through multi-modal interactions. The integration of the VR and AR systems in perception-decision-control enables real-time process optimization and the reverse control of physical operations. The system is implemented using a decentralized blockchain and end-edge-cloud collaborative network architecture to regulate interaction permissions for each scene characters, ensuring systematic functionality, robust communication, and comprehensive data management. A case study of a gearbox assembly is used to demonstrate the system’s development and efficacy. The results indicate the VR system's comprehensive perception and the AR system's detailed insights, yielding reliable information on global and local levels. The integration of artificial intelligence (AI) algorithms and spatiotemporal visualization during VR decision-making provides computational advantages. Incorporating human input into the AR system enables human-in-the-loop operational decision-making, providing optimal instructions to improve human-robot collaboration performance. Notably, the fusion of blockchain and end-edge-cloud architectures augments inter-system data exchange by efficiently managing and processing human-robot interactions and their diverse avatars. This system offers an innovative approach to human-robot collaboration assembly in manufacturing, with implications for analogous contexts.

An assembly sequence monitoring method based on workflow modeling for human–robot collaborative assembly

An assembly sequence monitoring method based on workflow modeling for human–robot collaborative assembly

Benchmarking human-robot collaborative assembly tasks

Manufacturing assembly tasks can vary in complexity and level of automation. Yet, achieving full automation can be challenging and inefficient, particularly due to the complexity of certain assembly operations. Human-robot collaborative work, leveraging the strengths of human labor alongside the capabilities of robots, can be a solution for enhancing efficiency. This paper introduces the CT benchmark, a benchmark and model set designed to facilitate the testing and evaluation of human-robot collaborative assembly scenarios. It was designed to compare manual and automatic processes using metrics such as the assembly time and human workload. The components of the model set can be assembled through the most common assembly tasks, each with varying levels of difficulty. The CT benchmark was designed with a focus on its applicability in human-robot collaborative environments, with the aim of ensuring the reproducibility and replicability of experiments. Experiments were carried out to assess assembly performance in three different setups (manual, automatic and collaborative), measuring metrics related to the assembly time and the workload on human operators. The results suggest that the collaborative approach takes longer than the fully manual assembly, with an increase of 70.8%. However, users reported a lower overall workload, as well as reduced mental demand, physical demand, and effort according to the NASA-TLX questionnaire.

Dynamic Affect-Based Motion Planning of a Humanoid Robot for Human-Robot Collaborative Assembly in Manufacturing

The objective was to investigate the impacts of the robot’s dynamic affective expressions in task-related scenarios on human–robot collaboration (HRC) and performance in human–robot collaborative assembly tasks in flexible manufacturing. A human–robot hybrid cell was developed to facilitate a human co-worker and a robot to collaborate to assemble a few parts into a final product. The collaborative robot was a humanoid manufacturing robot with the ability to display its affective states due to changes in task scenarios on its face. The assembly task was divided into several subtasks, and based on an optimization strategy, the subtasks were optimally allocated to the human and the robot. A computational model of the robot’s affective states was derived inspired by that of humans following the biomimetic approach, and an affect-based motion planning strategy for the robot was proposed to enable the robot to adjust its motions and behaviors with task situations and communicate (inform) the situations to the human co-worker through affective expressions. The HRC and the assembly performance for the affect-based motion planning were experimentally evaluated based on a comprehensive evaluation scheme and were compared with two alternative conditions: (i) motion planning that did not display affective states, and (ii) motion planning that displayed text messages instead of displaying affective states to communicate the situations to the human co-worker. The results clearly showed that the dynamic affect-based motion planning produced significantly better HRC and assembly performance than that produced by motion planning associated with the display of no affective states or text messages. The results encouraged employing manufacturing robots with dynamic affective expressions to collaborate with humans in flexible assembly in manufacturing to improve HRC and assembly performance.

Sensorless admittance control of 6-DoF parallel robot in human-robot collaborative assembly

This paper presents an adaptive admittance control to achieve human-robot collaborative (HRC) assembling of large, heavy components without using external sensors. This sensorless adaptive admittance control is constructed by the dynamic model of a six degree-of-freedom (DoF) parallel robot based on finite and instantaneous screw (FIS) theory. The dynamic model is intuitive and easy to be programmed. After getting stable actuation force by Kalman filter and identifying friction by LuGre model, accurate force estimator is built. Admittance controller is adopted to convert the change of estimated force to the change in position, velocity and acceleration. As a result, the 6-DoF parallel robot is capable of following the guidance of operator. Particularly, adaptive admittance control is design by using robot velocity as reference. A fast motion before contacting and a fine motion during contacting are allowed. Experiments are conducted to verify force estimator, admittance control and adaptive admittance control for HRC assembly. Results show that external force is accurately estimated. Robot complaint control in response to operator's force is achieved. The adaptive admittance control enhances execution time, accuracy of the HRC assembly.

A Multimodality Scene Graph Generation Approach for Robust Human–Robot Collaborative Assembly Visual Relationship Representation

A Multimodality Scene Graph Generation Approach for Robust Human–Robot Collaborative Assembly Visual Relationship Representation

Model and metaheuristic for human–robot collaboration assembly line worker assignment and balancing problem

Given the Industry 5.0 trend towards human-centric and resilient industries, workers continue to be considered as a valuable and irreplaceable resource. Human–robot collaboration (HRC) is a promising production mode that combines the advantages of both human workers and robots, resulting in improved productivity and reduced ergonomic risks for workers. In this study, we present one of the first attempts to address the human–robot collaboration assembly line worker assignment and balancing problem (HRCALWABP), which allows workers and robots to perform tasks in parallel or in collaboration. Workers are heterogeneous, leading to variations in task execution times. We formulate a mixed-integer linear programming (MILP) model that minimizes the cycle time and propose a tight lower/upper bound. A rebalancing process is further considered in this study. Due to the complexity of HRCALWABP, we develop an improved adaptive neighborhood simulated annealing algorithm (IASA). An adaptive mechanism is applied to the IASA to dynamically adjust the selection probability of the designed neighborhood structures and operators, and a restart mechanism is developed for improving the search capacity. Extensive computational experiments are conducted to evaluate the performance of both the MILP and IASA, and the results confirm the superiority of the IASA compared to the MILP and other existing algorithms.

FedHIP: Federated learning for privacy-preserving human intention prediction in human-robot collaborative assembly tasks

Human-robot collaboration is a promising solution to relieve construction workers from repetitive and physically demanding tasks, thus improving construction safety and productivity. Many studies have developed various deep learning models for human intention prediction, which will form the basis for proactive and adaptive robot planning and control to enable intelligent human-robot collaboration. However, there remain two challenges. First, most research only focuses on a single type of human intention, without a holistic understanding of multi-level intention, including both high-level intended actions and objects of interest, and low-level body movements. Second, conventional deep learning approaches train a centralized model with aggregated datasets, which requires the sharing of sensitive information (e.g., personal images and behavior data), posing broad privacy concerns in practical implementation. This study proposes a vision-based multi-task federated learning (FL) framework, FedHIP, for multi-level human intention prediction in human-robot collaborative assembly tasks. Specifically, taking body movements and assembly components as inputs, a long short-term memory based multi-task learning model was developed to simultaneously predict multi-level human intention in assembly tasks. FL was employed to train the model in a distributed and privacy-preserving way on local clients without the need of transmitting sensitive data. The results show that the proposal FedHIP without and with pre-train can achieve an accuracy of 80.1% and 85.7% in action prediction, 97.6% and 97.8% in object prediction, and an average displacement error of 12.7 and 11.7 pixels in motion prediction, respectively. Models trained from FedHIP were also compared with those obtained from traditional centralized training and local training. It was found that FL leads to compatible accuracy with centralized training and much higher accuracy than local training while preserving data privacy.

An Accurate Prediction Method of Human Assembly Motion for Human–Robot Collaboration

In the process of human–robot collaborative assembly, robots need to recognize and predict human behaviors accurately, and then perform autonomous control and work route planning in real-time. To support the judgment of human intervention behaviors and meet the need of real-time human–robot collaboration, the Fast Spatial–Temporal Transformer Network (FST-Trans), an accurate prediction method of human assembly actions, is proposed. We tried to maximize the symmetry between the prediction results and the actual action while meeting the real-time requirement. With concise and efficient structural design, FST-Trans can learn about the spatial–temporal interactions of human joints during assembly in the same latent space and capture more complex motion dynamics. Considering the inconsistent assembly rates of different individuals, the network is forced to learn more motion variations by introducing velocity–acceleration loss, realizing accurate prediction of assembly actions. An assembly dataset was collected and constructed for detailed comparative experiments and ablation studies, and the experimental results demonstrate the effectiveness of the proposed method.

Remarks from an experimental study on human-robot collaborative assembly

Human robot collaboration is becoming the norm in the workplace, due to the benefits robots can bring to efficiency and production. However, this creates highly complex and dynamic workplaces that human operators need to adapt to. Industry 5.0 promotes the use of robotics and smart technologies in a more human-centric way. However, research on how operators are affected by those changes is needed to better understand how to move towards human-centricity. As such, an experimental study was designed and performed on human robot collaborative assembly. The main aim was to investigate the correlation between cognitive load and quality due to collaboration. Here, the preliminary results of the experimental study are presented in order to remark relevant states influencing work allocation. The results showcased the need for better training and more knowledge for the operators, as well as involving operators in process and workplace design. This study helps contribute knowledge on robot implementation and process design for human robot collaboration for both researchers and operations management, as it showcases the need to involve operators in those steps due to the feedback they can provide due to their experience.

Virtual data generation for human intention prediction based on digital modeling of human-robot collaboration

Human intention prediction is vital for the efficiency of human-robot collaboration (HRC) and is usually modeled based on data-driven methods. However, due to the complexity and diverse nature of HRC, data collection for human intention prediction suffers from low sampling efficiency which restricts the application of HRC in manufacturing. Different from traditional real world data collection, a digital modeling method for HRC is proposed in this paper to generate virtual HRC data. The dynamic musculoskeletal model of human is adopted to simulate the musculoskeletal dynamics of human. The metabolic energy consumption of human is computed and used as an indicator to evaluate the reality of the generated virtual data. The virtual data are used to train human intention prediction model and compared with experimental data. Experimental results show the reality of virtual data and its effectiveness for human intention modeling in human-robot collaborative assembly. The proposed method has potential for reducing the cost of data collection compared with purely experiments.

Hybrid Convolutional Neural Network Approaches for Recognizing Collaborative Actions in Human–Robot Assembly Tasks

In the context of sustainable manufacturing, efficient collaboration between humans and machines is crucial for improving assembly quality and efficiency. However, traditional methods for action recognition and human–robot collaborative assembly often face challenges such as low efficiency, low accuracy, and poor robustness. To solve such problems, this paper proposes an assembly action-recognition method based on a hybrid convolutional neural network. Firstly, an assembly action-recognition model is proposed using skeletal sequences and a hybrid convolutional neural network model combining Spatial Temporal Graph Convolutional Networks (ST-GCNs) and One-Dimensional Convolutional Neural Networks (1DCNNs) to sense and recognize human behavior actions during the assembly process. This model combines the joint spatial relationship and temporal information extraction ability of the ST-GCN model with the temporal feature extraction ability of the 1DCNN model. By incorporating Batch Normalization (BN) layers and Dropout layers, the generalization performance of the model is enhanced. Secondly, the model is validated on a self-constructed dataset of assembly actions, and the results show that the recognition accuracy of the model can reach 91.7%, demonstrating its superiority. Finally, a digital workshop application system based on digital twins is developed. To test the effectiveness of the proposed method, three sets of control experiments were designed to evaluate both objective and subjective aspects and verify the feasibility of the method presented in this paper. Compared with traditional assembly systems, the proposed method optimizes the recognition of human–robot collaborative assembly actions and applies them to intelligent control systems using digital-twin technology. This intelligent assembly method improves assembly efficiency and saves assembly time. It enables efficient and sustainable collaboration between humans and robots in assembly, leading to a positive and sustainable impact on the manufacturing industry.

Pick the Right Co-Worker: Online Assessment of Cognitive Ergonomics in Human–Robot Collaborative Assembly

Human-robot collaborative assembly systems enhance the efficiency and productivity of the workplace but may increase the workers’ cognitive demand. This paper proposes an online and quantitative framework to assess the cognitive workload induced by the interaction with a co-worker, either a human operator or an industrial collaborative robot with different control strategies. The approach monitors the operator’s attention distribution and upper-body kinematics benefiting from the input images of a low-cost stereo camera and cutting-edge artificial intelligence algorithms (i.e. head pose estimation and skeleton tracking). Three experimental scenarios with variations in workstation features and interaction modalities were designed to test the performance of our online method against state-of-the-art offline measurements. Results proved that our vision-based cognitive load assessment has the potential to be integrated into the new generation of collaborative robotic technologies. The latter would enable human cognitive state monitoring and robot control strategy adaptation for improving human comfort, ergonomics, and trust in automation.

Difficulty and complexity definitions for assembly task allocation and assignment in human–robot collaborations: A review

This paper presents a literature review on the different aspects of task allocation and assignment problems in human–robot collaboration (HRC) tasks in industrial assembly environments. In future advanced industrial environments, robots and humans are expected to share the same workspace and collaborate to efficiently achieve shared goals. Difficulty- and complexity-aware HRC assembly is necessary for human-centric manufacturing, which is a goal of Industry 5.0. Therefore, the objective of this study is to clarify the definitions of difficulty and complexity used to encourage effective collaboration between humans and robots to leverage the adaptability of humans and the autonomy of robots. To achieve this goal, a systematic review of the following relevant databases for computer science was performed: IEEE Xplore, ScienceDirect, SpringerLink, ACM Digital Library, and ASME Digital Collection. The results extracted from 74 peer-reviewed research articles published until July 2022 were summarized and categorized into four taxonomies for 145 difficulty and complexity definitions from the perspectives of (1) definition-use objectives, (2) evaluation objectives, (3) evaluation factors, and (4) evaluation variables. Next, existing definitions were primarily classified according to the following two criteria to identify potential future studies on the formulation of new definitions for human-centric manufacturing: (1) agent specificity and (2) common aspects in manual and robotic assemblies.

The fuzzy human-robot collaboration assembly line balancing problem

Consideration is given to the assembly line balancing problem (ALBP) in production lines employed collaborative robots. Human–robot collaboration promises high advantages in manufacturing regarding automation, productivity, accuracy as well as flexibility in production. This technology is really challenging but requires redesigning of assembly lines to effectively involve the available human- and robot-resources. As robots can collaborate with humans sharing the assembly tasks in the same workstation, production managers experience an increasingly complex ALBP setting. Moreover, a crucial issue raised within this context is the variance and uncertainty in processing times for manual (human) work. Processing times for human work are subject to the nature and skills of the human worker. We explore the heuristic solution of this NP-hard problem using a problem-specific metaheuristic. Our aim is to investigate the effects of utilizing collaborate robots in manual assembly lines with respect to both production rate and workload smoothing. Simulation experiments on a real-life case study borrowed from the literature show how to tackle the problem under uncertainty (fuzziness) in task processing times.

Optimal layout planning for human robot collaborative assembly systems and visualization through immersive technologies

The deployment of Industry 4.0 emerging technologies such as Augmented reality (AR), Virtual reality (VR), and collaborative Robots enhances flexibility and precision in the manufacturing systems. A flexible, and collaborative manufacturing system with robots and human integrates the prominent attributes of customized automation and fortify the safety of the human workforce. The proposed work involves the approach of AR-assisted assembly layout configurations for the Human-Robot Collaboration (HRC) production system, which is a typical example of the smart manufacturing system. The reconfiguration/configuration of the assembly layout is triggered with respect to the demand of new variants into the market and the upgradation of new technologies in the existing system. A multi-objective algorithm is proposed to maximize the production outlet with effective utilization of floor space, minimum idle time and optimum resource allocation. The selected work is branched into three phases: To begin with, the approach of task allocation is constructed to identify the suitable resources (human, robot) for the appropriate assembly tasks in the HRC system. The outcomes such as resource allocation, number of resources, and footprint of resources are considered as input to the optimization model. Further, modified particle swarm optimization (MPSO) is deployed to generate a feasible assembly layout for the HRC manufacturing system. Additionally, the breath-first approach has been utilized to generate the optimal aisle as a conveyor in the layout. Notably, the comparisons studies have been carried out in this work, which facilitates to explore the robustness of proposed modified PSO algorithms for the selected applications. Subsequently, the virtual and AR tool is deployed for the layout planer to explore and validate the obtained layout from the PSO model in both the virtual and physical environment of the industrial workspace. Finally, an industrial case study of rotary gear pump assembly system is adopted for the implementation of the proposed framework.

The effect of human autonomy and robot work pace on perceived workload in human-robot collaborative assembly work.

Collaborative robots (in short: cobots) have the potential to assist workers with physically or cognitive demanding tasks. However, it is crucial to recognize that such assistance can have both positive and negative effects on job quality. A key aspect of human-robot collaboration is the interdependence between human and robotic tasks. This interdependence influences the autonomy of the operator and can impact the work pace, potentially leading to a situation where the human's work pace becomes reliant on that of the robot. Given that autonomy and work pace are essential determinants of job quality, design decisions concerning these factors can greatly influence the overall success of a robot implementation. The impact of autonomy and work pace was systematically examined through an experimental study conducted in an industrial assembly task. 20 participants engaged in collaborative work with a robot under three conditions: human lead (HL), fast-paced robot lead (FRL), and slow-paced robot lead (SRL). Perceived workload was used as a proxy for job quality. To assess the perceived workload associated with each condition was assessed with the NASA Task Load Index (TLX). Specifically, the study aimed to evaluate the role of human autonomy by comparing the perceived workload between HL and FRL conditions, as well as the influence of robot pace by comparing SRL and FRL conditions. The findings revealed a significant correlation between a higher level of human autonomy and a lower perceived workload. Furthermore, a decrease in robot pace was observed to result in a reduction of two specific factors measuring perceived workload, namely cognitive and temporal demand. These results suggest that interventions aimed at increasing human autonomy and appropriately adjusting the robot's work pace can serve as effective measures for optimizing the perceived workload in collaborative scenarios.

Task-oriented safety field for robot control in human-robot collaborative assembly based on residual learning

With the development of smart manufacturing, human-robot collaboration (HRC) is seen as the future of manufacturing. In a manufacturing environment with HRC, safety has been raising significant attention since the conventional separation of workspaces between robots and humans is removed. In this paper, a dynamic safety field is constructed that takes into account the collaborative assembly task requirements and the motions of both humans and robots. This safety field guides robot actions, ensuring that the robot carries out the required tasks while maintaining a safe distance from human workers. Based on the task requirements and safety considerations between humans and robots, a robot motion planning and control problem is formulated. To solve this problem, a hybrid RRL control scheme is proposed where a residual reinforcement learning (RRL) method is developed to combine the safety field-based control method with a deep reinforcement learning (DRL) method. Numerical studies are conducted to evaluate the performance of the proposed, which is compared with a pure deep Q-network (DQN) based method and a rapidly exploring random tree (RRT) method. The simulation results show that the proposed method can effectively optimize robot trajectories and outperforms the DQN-based and the RRT method in terms of computational efficiency.

A deep learning-enabled human-cyber-physical fusion method towards human-robot collaborative assembly

A deep learning-enabled human-cyber-physical fusion method towards human-robot collaborative assembly