Efficient Human-robot Collaboration Research Articles

Context‐aware hand gesture interaction for human–robot collaboration in construction

AbstractConstruction robots play a pivotal role in enabling intelligent processes within the construction industry. User‐friendly interfaces that facilitate efficient human–robot collaboration are essential for promoting robot adoption. However, most of the existing interfaces do not consider contextual information in the collaborative environment. The situation where humans and robots work together in the same jobsite creates a unique environmental context. Overlooking contextual information would limit the potential to optimize interaction efficiency. This paper proposes a novel context‐aware method that utilizes a two‐stream network to enhance human–robot interaction in construction settings. In the proposed network, the first‐person view‐based stream focuses on the relevant spatiotemporal regions for context extraction, while the motion sensory data‐based stream obtains features related to hand motions. By fusing the vision context and motion data, the method achieves gesture recognition for efficient communication between construction workers and robots. Experimental evaluation on a dataset from five construction sites demonstrates an overall classification accuracy of 92.6%, underscoring the practicality and potential benefits of the proposed method.

Praxis: a framework for AI-driven human action recognition in assembly

AbstractThe role of Artificial intelligence in achieving high performance in manufacturing systems has been explored over the years. However, with the increasing number of variants in the factories and the advances in digital technologies new opportunities arise for supporting operators in the factory. The hybrid production systems stipulate the efficient collaboration of the workers with the machines. Human action recognition is a major enabler for intuitive machines and robots to achieve more efficient interaction with workers. This paper discusses a software framework called Praxis, aiming to facilitate the deployment of human action recognition (HAR) in assembly. Praxis is designed to provide a flexible and scalable architecture for implementing human action recognition in assembly lines. The framework has been implemented in a real-world case study originating for showcasing and validating the effectiveness of Praxis in real-life applications. It is deployed in an assembly use case for an air compression production industry. This study highlights the potential of the Praxis framework for promoting efficient human–robot collaboration (HRC) in modern manufacturing environments through HAR.

Modeling and Analysis of Human Comfort in Human-Robot Collaboration.

The emergence and recent development of collaborative robots have introduced a safer and more efficient human-robot collaboration (HRC) manufacturing environment. Since the release of COBOTs, a great amount of research efforts have been focused on improving robot working efficiency, user safety, human intention detection, etc., while one significant factor-human comfort-has been frequently ignored. The comfort factor is critical to COBOT users due to its great impact on user acceptance. In previous studies, there is a lack of a mathematical-model-based approach to quantitatively describe and predict human comfort in HRC scenarios. Also, few studies have discussed the cases when multiple comfort factors take effect simultaneously. In this study, a multi-linear-regression-based general human comfort prediction model is proposed under human-robot collaboration scenarios, which is able to accurately predict the comfort levels of humans in multi-factor situations. The proposed method in this paper tackled these two gaps at the same time and also demonstrated the effectiveness of the approach with its high prediction accuracy. The overall average accuracy among all participants is 81.33%, while the overall maximum value is 88.94%, and the overall minimum value is 72.53%. The model uses subjective comfort rating feedback from human subjects as training and testing data. Experiments have been implemented, and the final results proved the effectiveness of the proposed approach in identifying human comfort levels in HRC.

Comparison of LSTM, Transformers, and MLP-mixer neural networks for gaze based human intention prediction.

Collaborative robots have gained popularity in industries, providing flexibility and increased productivity for complex tasks. However, their ability to interact with humans and adapt to their behavior is still limited. Prediction of human movement intentions is one way to improve the robots adaptation. This paper investigates the performance of using Transformers and MLP-Mixer based neural networks to predict the intended human arm movement direction, based on gaze data obtained in a virtual reality environment, and compares the results to using an LSTM network. The comparison will evaluate the networks based on accuracy on several metrics, time ahead of movement completion, and execution time. It is shown in the paper that there exists several network configurations and architectures that achieve comparable accuracy scores. The best performing Transformers encoder presented in this paper achieved an accuracy of 82.74%, for predictions with high certainty, on continuous data and correctly classifies 80.06% of the movements at least once. The movements are, in 99% of the cases, correctly predicted the first time, before the hand reaches the target and more than 19% ahead of movement completion in 75% of the cases. The results shows that there are multiple ways to utilize neural networks to perform gaze based arm movement intention prediction and it is a promising step toward enabling efficient human-robot collaboration.

Flexible sensor concept and an efficient integrated sensing controlling for an efficient human-robot collaboration using 3D local global sensing systems.

Human-robot collaboration with traditional industrial robots is a cardinal step towards agile manufacturing and re-manufacturing processes. These processes require constant human presence, which results in lower operational efficiency based on current industrial collision avoidance systems. The work proposes a novel local and global sensing framework, which discusses a flexible sensor concept comprising a single 2D or 3D LiDAR while formulating occlusion due to the robot body. Moreover, this work extends the previous local global sensing methodology to incorporate local (co-moving) 3D sensors on the robot body. The local 3D camera faces toward the robot occlusion area, resulted from the robot body in front of a single global 3D LiDAR. Apart from the sensor concept, this work also proposes an efficient method to estimate sensitivity and reactivity of sensing and control sub-systems The proposed methodologies are tested with a heavy-duty industrial robot along with a 3D LiDAR and camera. The integrated local global sensing methods allow high robot speeds resulting in process efficiency while ensuring human safety and sensor flexibility.

Prediction-Based Path Planning for Safe and Efficient Human–Robot Collaboration in Construction via Deep Reinforcement Learning

Prediction-Based Path Planning for Safe and Efficient Human–Robot Collaboration in Construction via Deep Reinforcement Learning

Control Techniques for Safe, Ergonomic, and Efficient Human-Robot Collaboration in the Digital Industry: A Survey

The fourth industrial revolution, also known as Industry 4.0, is reshaping the way individuals live and work while providing a substantial influence on the manufacturing scenario. The key enabling technology that has made Industry 4.0 a concrete reality is without doubt <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">collaborative robotics</i> , which is also evolving as a fundamental pillar of the next revolution, the so-called Industry 5.0. The improvement of employees’ safety and well-being, together with the increase of profitability and productivity, are indeed the main goals of human-robot collaboration (HRC) in the industrial setting. The robotic controller design and the analysis of existing decision and control techniques are crucially needed to develop innovative models and state-of-the-art methodologies for a safe, ergonomic, and efficient HRC. To this aim, this paper presents an accurate review of the most recent and relevant contributions to the related literature, focusing on the control perspective. All the surveyed works are carefully selected and categorized by target (i.e., safety, ergonomics, and efficiency), and then by problem and type of control, in presence or absence of optimization. Finally, the discussion of the achieved results and the analysis of the emerging challenges in this research field are reported, highlighting the identified gaps and the promising future developments in the context of the digital evolution. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The design and development of manufacturing systems are experiencing substantial changes towards full automation. This ongoing challenge is being tackled by academia and industrial practitioners with the adoption of collaborative robots, where the skills and peculiarities of humans (e.g., intelligence, creativity, adaptability, etc.) and robots (e.g., flexibility, pinpoint accuracy, tirelessness, etc.) are combined to better perform a variety of tasks. Nevertheless, due to their different characteristics, there is an emerging need for designing suitable decision and control techniques to ensure a safe and ergonomic HRC, while keeping the highest level of productivity. Against this background, the aim of this paper is to provide researchers and practitioners with a reference source in the related field, which can help them designing and developing suitable solutions to control problems in safe, ergonomic, and efficient collaborative robotics.

Collaborative Workplace Design: A Knowledge-Based Approach to Promote Human–Robot Collaboration and Multi-Objective Layout Optimization

The innovation-driven Industry 5.0 leads us to consider humanity in a prominent position as the center of the manufacturing field even more than Industry 4.0. This pushes us towards the hybridization of manufacturing plants promoting a full collaboration between humans and robots. However, there are currently very few workplaces where effective Human–Robot Collaboration takes place. Layout designing plays a key role in assuring safe and efficient Human–Robot Collaboration. The layout design, especially in the context of collaborative robotics, is a complex problem to face, since it is related to safety, ergonomics, and productivity aspects. In the current work, a Knowledge-Based Approach (KBA) is adopted to face the complexity of the layout design problem. The framework resulting from the KBA allows for developing a modeling paradigm that enables us to define a streamlined approach for the layout design. The proposed approach allows for placing resource within the workplace according to a defined optimization criterion, and also ensures compliance with various standards. This approach is applied to an industrial case study in order to prove its feasibility. A what-if analysis is performed by applying the proposed approach. Changing three control factors (i.e., minimum distance, robot speed, logistic space configuration) on three levels, in a Design of Experiments, 27 layout configurations of the same workplace are generated. Consequently, the inputs that most affect the layout design are identified by means of an Analysis of Variance (ANOVA). The results show that only one layout is eligible to be the best configuration, and only two out of three control factors are very significant for the designing of the HRC workplace layout. Hence, the proposed approach enables the designing of standard compliant and optimized HRC workplace layouts. Therefore, several alternatives of the layout for the same workplace can be easily generated and investigated in a systematic manner.

Function Block-Based Multimodal Control for Symbiotic Human–Robot Collaborative Assembly

AbstractIn human–robot collaborative assembly, robots are often required to dynamically change their preplanned tasks to collaborate with human operators in close proximity. One essential requirement of such an environment is enhanced flexibility and adaptability, as well as reduced effort on the conventional (re)programming of robots, especially for complex assembly tasks. However, the robots used today are controlled by rigid native codes that cannot support efficient human–robot collaboration. To solve such challenges, this article presents a novel function block-enabled multimodal control approach for symbiotic human–robot collaborative assembly. Within the context, event-driven function blocks as reusable functional modules embedded with smart algorithms are used for the encapsulation of assembly feature-based tasks/processes and control commands that are transferred to the controller of robots for execution. Then, multimodal control commands in the form of sensorless haptics, gestures, and voices serve as the inputs of the function blocks to trigger task execution and human-centered robot control within a safe human–robot collaborative environment. Finally, the performed processes of the method are experimentally validated by a case study in an assembly work cell on assisting the operator during the collaborative assembly. This unique combination facilitates programming-free robot control and the implementation of the multimodal symbiotic human–robot collaborative assembly with the enhanced adaptability and flexibility.

OMNIVIL-An Autonomous Mobile Manipulator for Flexible Production.

Flexible production is a key element in modern industrial manufacturing. Autonomous mobile manipulators can be used to execute various tasks: from logistics, to pick and place, or handling. Therefore, autonomous robotic systems can even increase the flexibility of existing production environments. However, the application of robotic systems is challenging due to their complexity and safety concerns. This paper addresses the design and implementation of the autonomous mobile manipulator OMNIVIL. A holonomic kinematic design provides high maneuverability and the implemented sensor setup with the underlying localization strategies are robust against typical static and dynamic uncertainties in industrial environments. For a safe and efficient human–robot collaboration (HRC), a novel workspace monitoring system (WMS) is developed to detect human co-workers and other objects in the workspace. The multilayer sensor setup and the parallel data analyzing capability provide superior accuracy and reliability. An intuitive zone-based navigation concept is implemented, based on the workspace monitoring system. Preventive behaviors are predefined for a conflict-free interaction with human co-workers. A workspace analyzing tool is implemented for adaptive manipulation, which significantly simplifies the determination of suitable platform positions for a manipulation task.

Human Motion Prediction Using Adaptable Recurrent Neural Networks and Inverse Kinematics

Human motion prediction, especially arm prediction, is critical to facilitate safe and efficient human-robot collaboration (HRC). This letter proposes a novel human motion prediction framework that combines a recurrent neural network (RNN) and inverse kinematics (IK) to predict human arm motion. A modified Kalman filter (MKF) is applied to adapt the model online. The proposed framework is tested on collected human motion data with up to 2 s prediction horizon. The experiments demonstrate that the proposed method improves the prediction accuracy by approximately 14% comparing to the state-of-art on seen situations. It stably adapts to unseen situations by keeping the maximum prediction error under 4 cm, which is 70% lower than other methods. Moreover, it is robust when the arm is partially occluded. The wrist prediction remains the same, while the elbow prediction has 20% less variation.

Towards Efficient Human-Robot Collaboration With Robust Plan Recognition and Trajectory Prediction

Human-robot collaboration (HRC) is becoming increasingly important as the paradigm of manufacturing is shifting from mass production to mass customization. The introduction of HRC can significantly improve the flexibility and intelligence of automation. To efficiently finish tasks in HRC systems, the robots need to not only predict the future movements of human, but also more high-level plans, i.e., the sequence of actions to finish the tasks. However, due to the stochastic and time-varying nature of human collaborators, it is quite challenging for the robot to efficiently and accurately identify such task plans and respond in a safe manner. To address this challenge, we propose an integrated human-robot collaboration framework. Both plan recognition and trajectory prediction modules are included for the generation of safe and efficient robotic motions. Such a framework enables the robots to perceive, predict and adapt their actions to the human's work plan and intelligently avoid collisions with the human. Moreover, by explicitly leveraging the hierarchical relationship between plans and trajectories, more robust plan recognition performance can be achieved. Physical experiments were conducted on an industrial robot to verify the proposed framework. The results show that the proposed framework could accurately recognize the human workers' plans and thus significantly improve the time efficiency of the HRC team even in the presence of motion classification noises.

Design of Human Robot Collaboration workstations – Two automotive case studies

Human Robot Collaboration assembly cells benefit the capabilities and skills of both industrial robots and humans, although fenceless coexistence leads to issues related to safety and human-system interaction. A series of methods for hybrid workstation design focusing on safe yet efficient Human Robot Collaboration is presented, where various layout configurations are illustrated and evaluated aiming industrial implementation. Two distinct automotive use cases, with different requirements and challenges, are used as reference for highlighting particularities on lightweight and high-payload robots, in addition to small and large size product assemblies.

Assessing worker performance using dynamic cost functions in human robot collaborative tasks

The aim of this research is to develop a framework to allow efficient human robot collaboration on manufacturing assembly tasks based on cost functions that quantify capabilities and performance of each element in a system and enable their efficient evaluation. A proposed cost function format is developed along with initial development of two example cost function variables, completion time and fatigue, obtained as each worker is completing assembly tasks. The cost function format and example variables were tested with two example tasks utilizing an ABB YuMi Robot in addition to a simulated human worker under various levels of fatigue. The total costs produced clearly identified the best worker to complete each task with these costs also clearly indicating when a human worker is fatigued to a greater or lesser degree than expected.

Safe and Efficient Human–Robot Collaboration Part II: Optimal Generalized Human-in-the-Loop Real-Time Motion Generation

The coexistence of humans and robots in fenceless robot cells requires robust safety precautions to prevent humans from being injured. Currently, safety is ensured by limiting the robot velocity, force, and power. This results in large cycle times and, hence, very inefficient industrial applications, where no amortization of the robotic system can be expected. In this letter, a novel method for improving the robot performance is presented that still complies with the international safety standards for collaborative robots. The approach of this letter is based on a projection of a human arm motion into the robot's path to estimate a possible collision with the robot. This idea is addressed in an optimization approach by minimizing the time needed by the robot to reach the goal position under human-in-the-loop constraints. The segmented path is optimized by solving a nonlinear programming problem, and the effect of crucial parameters is analyzed. To guarantee a flexible motion of the resulting optimized path, a generalization method using dynamic movement primitives and the compliance of constraints are proposed. Experiments validate this new method that significantly improves the efficiency of human–robot coexistence.

Design of Hybrid Cells to Facilitate Safe and Efficient Human–Robot Collaboration During Assembly Operations

This paper presents a framework to build hybrid cells that support safe and efficient human–robot collaboration during assembly operations. Our approach allows asynchronous collaborations between human and robot. The human retrieves parts from a bin and places them in the robot's workspace, while the robot picks up the placed parts and assembles them into the product. We present the design details of the overall framework comprising three modules—plan generation, system state monitoring, and contingency handling. We describe system state monitoring and present a characterization of the part tracking algorithm. We report results from human–robot collaboration experiments using a KUKA robot and a three-dimensional (3D)-printed mockup of a simplified jet-engine assembly to illustrate our approach.

Learning Semantics of Gestural Instructions for Human-Robot Collaboration.

Designed to work safely alongside humans, collaborative robots need to be capable partners in human-robot teams. Besides having key capabilities like detecting gestures, recognizing objects, grasping them, and handing them over, these robots need to seamlessly adapt their behavior for efficient human-robot collaboration. In this context we present the fast, supervised Proactive Incremental Learning (PIL) framework for learning associations between human hand gestures and the intended robotic manipulation actions. With the proactive aspect, the robot is competent to predict the human's intent and perform an action without waiting for an instruction. The incremental aspect enables the robot to learn associations on the fly while performing a task. It is a probabilistic, statistically-driven approach. As a proof of concept, we focus on a table assembly task where the robot assists its human partner. We investigate how the accuracy of gesture detection affects the number of interactions required to complete the task. We also conducted a human-robot interaction study with non-roboticist users comparing a proactive with a reactive robot that waits for instructions.

Survey on human–robot collaboration in industrial settings: Safety, intuitive interfaces and applications

Easy-to-use collaborative robotics solutions, where human workers and robots share their skills, are entering the market, thus becoming the new frontier in industrial robotics. They allow to combine the advantages of robots, which enjoy high levels of accuracy, speed and repeatability, with the flexibility and cognitive skills of human workers. However, to achieve an efficient human–robot collaboration, several challenges need to be tackled. First, a safe interaction must be guaranteed to prevent harming humans having a direct contact with the moving robot. Additionally, to take full advantage of human skills, it is important that intuitive user interfaces are properly designed, so that human operators can easily program and interact with the robot. In this survey paper, an extensive review on human–robot collaboration in industrial environment is provided, with specific focus on issues related to physical and cognitive interaction. The commercially available solutions are also presented and the main industrial applications where collaborative robotic is advantageous are discussed, highlighting how collaborative solutions are intended to improve the efficiency of the system and which the open issue are.

Efficient human-robot collaboration: When should a robot take initiative?

The promise of robots assisting humans in everyday tasks has led to a variety of research questions and challenges in human-robot collaboration. Here, we address the question of whether and when a robot should take initiative during joint human-robot task execution. We designed a robotic system capable of autonomously performing table-top manipulation tasks while monitoring the environmental state. Our system is able to predict future environmental states and the robot’s actions to reach them using a dynamic Bayesian network. To evaluate our system, we implemented three different initiative conditions to trigger the robot’s actions. Human-initiated help gives control of the robot action timing to the user; robot-initiated reactive help triggers robot assistance when it detects that the human needs help; robot-initiated proactive help makes the robot help whenever it can. We performed a user study (N=18) to compare the trigger mechanisms in terms of quality of interaction, system performance and perceived sociality of the robot. We found that people collaborate best with a proactive robot, yielding better team fluency and high subjective ratings. However, they prefer having control of when the robot should help, rather than working with a reactive robot that only helps when needed. We also found that participants gazed at the robot’s face more during the human-initiated help compared to the other conditions. This shows that asking for the robot’s help may lead to a more “social” interaction, without improving the quality of interaction or the system performance.

Goal Set Inverse Optimal Control and Iterative Replanning for Predicting Human Reaching Motions in Shared Workspaces

To enable safe and efficient human–robot collaboration in shared workspaces, it is important for the robot to predict how a human will move when performing a task. While predicting human motion for tasks not known a priori is very challenging, we argue that single-arm reaching motions for known tasks in collaborative settings (which are especially relevant for manufacturing) are indeed predictable. Two hypotheses underlie our approach for predicting such motions: First, that the trajectory the human performs is optimal with respect to an unknown cost function, and second, that human adaptation to their partner's motion can be captured well through iterative replanning with the above cost function. The key to our approach is thus to learn a cost function that “explains” the motion of the human. To do this, we gather example trajectories from pairs of participants performing a collaborative assembly task using motion capture. We then use inverse optimal control to learn a cost function from these trajectories. Finally, we predict reaching motions from the human's current configuration to a task-space goal region by iteratively replanning a trajectory using the learned cost function. Our planning algorithm is based on the trajectory optimizer: stochastic trajectory optimizer for motion planning [1] ; it plans for a 23-degree-of-freedom human kinematic model and accounts for the presence of a moving collaborator and obstacles in the environment. Our results suggest that in most cases, our method outperforms baseline methods when predicting motions. We also show that our method outperforms baselines for predicting human motion when a human and a robot share the workspace.