Human-robot Collaboration Scenarios Research Articles

Enhancing awareness of industrial robots in collaborative manufacturing

The diffusion of Human-Robot Collaborative cells is prevented by several barriers. Classical control approaches seem not yet fully suitable for facing the variability conveyed by the presence of human operators beside robots. The capabilities of representing heterogeneous knowledge representation and performing abstract reasoning are crucial to enhance the flexibility of control solutions. To this aim, the ontology SOHO (Sharework Ontology for Human-Robot Collaboration) has been specifically designed for representing Human-Robot Collaboration scenarios, following a context-based approach. This work brings several contributions. This paper proposes an extension of SOHO to better characterize behavioral constraints of collaborative tasks. Furthermore, this work shows a knowledge extraction procedure designed to automatize the synthesis of Artificial Intelligence plan-based controllers for realizing flexible coordination of human and robot behaviors in collaborative tasks. The generality of the ontological model and the developed representation capabilities as well as the validity of the synthesized planning domains are evaluated on a number of realistic industrial scenarios where collaborative robots are actually deployed.

Long Horizon Episodic Decision Making for Cognitively Inspired Robots

The Human decision-making process works by recollecting past sequences of observations and using them to decide the best possible action in the present. These past sequences of observations are stored in a derived form which only includes important information the brain thinks might be useful in the future, while forgetting the rest. we propose an architecture that tries to mimic the human brain and improve the memory efficiency of transformers by using a modified Transformer XL architecture which uses Automatic Chunking which only attends to the relevant chunks in the transformer block. On top of this, we use Forget Span which is technique to remove memories that do not contribute to learning. We also theorize the technique of Similarity based forgetting to remove repetitive memories. We test our model in various tasks that test the abilities required to perform well in a human-robot collaboration scenario.

On the Evaluation of Diverse Vision Systems towards Detecting Human Pose in Collaborative Robot Applications.

Tracking human operators working in the vicinity of collaborative robots can improve the design of safety architecture, ergonomics, and the execution of assembly tasks in a human-robot collaboration scenario. Three commercial spatial computation kits were used along with their Software Development Kits that provide various real-time functionalities to track human poses. The paper explored the possibility of combining the capabilities of different hardware systems and software frameworks that may lead to better performance and accuracy in detecting the human pose in collaborative robotic applications. This study assessed their performance in two different human poses at six depth levels, comparing the raw data and noise-reducing filtered data. In addition, a laser measurement device was employed as a ground truth indicator, together with the average Root Mean Square Error as an error metric. The obtained results were analysed and compared in terms of positional accuracy and repeatability, indicating the dependence of the sensors' performance on the tracking distance. A Kalman-based filter was applied to fuse the human skeleton data and then to reconstruct the operator's poses considering their performance in different distance zones. The results indicated that at a distance less than 3 m, Microsoft Azure Kinect demonstrated better tracking performance, followed by Intel RealSense D455 and Stereolabs ZED2, while at ranges higher than 3 m, ZED2 had superior tracking performance.

Digital twins for hand gesture-guided human-robot collaboration systems

Gesture control is one of the effective and flexible communication method between humans and robots. However, it always depends on complex hardware and configurations in human-robot collaboration systems. Simplifying the design of gesture-interaction systems and avoiding miscommunication are challenging problems. In this paper, we proposed a method that utilizes an RGB sensor to realize spatial human-robot collaboration. A random forest based depth estimator is presented to supplement the additional spatial information for hand gesture recognition. Additionally, we demonstrate the construction of secure human-robot collaboration scenarios in Unity and validate our approach in real-world settings, based on which a digital twin system oriented to human-machine collaboration is constructed to realize rapid human-machine task simulation, safety specification testing, and real-scene applications development.

Estimating 6D Object Poses with Temporal Motion Reasoning for Robot Grasping in Cluttered Scenes

6D object pose estimation is an essential task in vision-based robotic grasping and manipulation. Prior works extract object's 6D pose by regressing from single RGB-D frame without considering the occluded objects in the frame, limiting their performance in human-robot collaboration scenarios with heavy occlusion. In this paper, we present an end-to-end model named \textit{TemporalFusion}, which integrates the temporal motion information from RGB-D images for 6D object pose estimation. The core of proposed model is to embed and fuse the temporal motion information from multi-frame RGB-D sequences, which could handle heavy occlusion in human-robot collaboration tasks. Furthermore, the proposed deep model can also obtain stable pose sequences, which is essential for real-time robotic grasping tasks. We evaluated the proposed method in the YCB-Video dataset, and experimental results show our model outperforms state-of-the-art approaches. Our code is available at https://github.com/mufengjun260/H-MPose.

Emotional Experience in Human–Robot Collaboration: Suitability of Virtual Reality Scenarios to Study Interactions beyond Safety Restrictions

Today’s research on fenceless human–robot collaboration (HRC) is challenged by a relatively slow development of safety features. Simultaneously, design recommendations for HRC are requested by the industry. To simulate HRC scenarios in advance, virtual reality (VR) technology can be utilized and ensure safety. VR also allows researchers to study the effects of safety-restricted features like close distance during movements and events of robotic malfunctions. In this paper, we present a VR experiment with 40 participants collaborating with a heavy-load robot and compare the results to a similar real-world experiment to study transferability and validity. The participant’s proximity to the robot, interaction level, and occurring system failures were varied. State anxiety, trust, and intention to use were used as dependent variables, and valence and arousal values were assessed over time. Overall, state anxiety was low and trust and intention to use were high. Only simulated failures significantly increased state anxiety, reduced trust, and resulted in reduced valence and increased arousal. In comparison with the real-world experiment, non-significant differences in all dependent variables and similar progression of valence and arousal were found during scenarios without system failures. Therefore, the suitability of applying VR in HRC research to study safety-restricted features can be supported; however, further research should examine transferability for high-intensity emotional experiences.

A Path-Planning Method to Significantly Reduce Local Oscillation of Manipulators Based on Velocity Potential Field

The robotics industry and associated technology applications are a vital support for modern production and manufacturing. With the intelligent development of the manufacturing industry, the application of collaboration robots and human-robot collaboration technology is becoming more and more extensive. In a human-robot collaboration scenario, there are uncertainties such as dynamic impediments, especially in the human upper limb, which puts forward a higher assessment of the manipulator’s route planning technology. As one of the primary branches of the artificial potential field (APF), the velocity potential field (VPF) offers the advantages of good real-time performance and convenient mathematical expression. However, the traditional VPF algorithm is prone to local oscillation phenomena near obstacles, which degrades the smoothness of the movement of the manipulators. An improved velocity potential field algorithm is proposed in this paper. This method solves the problem of sudden velocity change when the manipulator enters and departs the region of the potential field by setting new functions for attraction velocity and repulsion velocity functions. A virtual target point construction method is given to overcome the local oscillation problem of the manipulators near obstacles. The simulation and practical findings of the manipulators reveal that the improved VPF algorithm can not only avoid collision but also effectively reduce the local oscillation problem when dealing with the human upper limb as a dynamic obstacle. The implementation of this algorithm can increase the safety and real-time performance of the human-robot collaboration process and ensure that the collaborative robot is safer and smoother in the working process.

Digital twin for human-robot collaboration enhancement in manufacturing systems: Literature review and direction for future developments

The integration of robots in manufacturing systems known as Human Robot Collaboration (HRC) can provide relevant opportunities in terms of productivity, safety and quality control although there are still many challenges such as integration costs, complexity, safety and workforce concerns, skills gaps and regulatory compliance. Digital Twins (DTs), which are virtual replicas of physical systems, are being explored as a promising solution to address these challenges. Therefore, this paper delves into the potential DTs-based approaches for HRC in manufacturing systems with a focus on their ability to provide a realistic simulation environment for testing and optimizing collaboration strategies. The first part of the study involves a comprehensive literature review devoted to understand how the scientific community perceives the application of DTs for HRC in manufacturing systems and the main trends of this research area are identified. The second part of the study presents a detailed analysis of the currently available commercial simulation software solutions for developing DT models in manufacturing systems for HRC scenarios. The analysis encompassed various aspects such as: the software's capabilities in human and robot simulation, ergonomic and time measurements analysis, data exchange and interoperability with the physical world as well as with other software. The results of the analysis are meant to provide practical insights to researchers and practitioners interested in implementing DTs to enhance HRC in manufacturing systems.

Modeling and Analysis of Human Comfort in Human-Robot Collaboration.

The emergence and recent development of collaborative robots have introduced a safer and more efficient human-robot collaboration (HRC) manufacturing environment. Since the release of COBOTs, a great amount of research efforts have been focused on improving robot working efficiency, user safety, human intention detection, etc., while one significant factor-human comfort-has been frequently ignored. The comfort factor is critical to COBOT users due to its great impact on user acceptance. In previous studies, there is a lack of a mathematical-model-based approach to quantitatively describe and predict human comfort in HRC scenarios. Also, few studies have discussed the cases when multiple comfort factors take effect simultaneously. In this study, a multi-linear-regression-based general human comfort prediction model is proposed under human-robot collaboration scenarios, which is able to accurately predict the comfort levels of humans in multi-factor situations. The proposed method in this paper tackled these two gaps at the same time and also demonstrated the effectiveness of the approach with its high prediction accuracy. The overall average accuracy among all participants is 81.33%, while the overall maximum value is 88.94%, and the overall minimum value is 72.53%. The model uses subjective comfort rating feedback from human subjects as training and testing data. Experiments have been implemented, and the final results proved the effectiveness of the proposed approach in identifying human comfort levels in HRC.

Stakeholders’ perspectives on safety-related human–robot collaborative scenarios

Safety plays a key role when implementing collaborative robots (cobots) in human-shared workspace scenarios. Nevertheless, the significance and understanding of safety-related aspects may significantly vary with different stakeholders’ perspectives. In particular, there is often a misconception that cobots are inherently safe with the result of significantly underestimating the need of adopting adequate safety measures and features for the deployment of collaborative robots in industrial and non-industrial frames. Accordingly, this article develops an in-depth analysis of the stakeholders and their stakes in human–robot collaboration in the manufacturing industry with particular attention to the safety aspects. The proposed approach consists of a field study, with an online questionnaire and structured interviews, directed at the main stakeholders. The field study results are presented and compared with insights from two recent explorative studies. Finally, careful analysis and discussion is provided. This research focuses on three topics: (i) the specific phases in the collaborative robot application cycle, (ii) the stakeholders that are involved in each phase, and (iii) the stakes that affect the stakeholders in general, and specifically the role of safety within these stakes. The findings show that awareness of safety issues is still jeopardized between stakeholders. In addition, key stakeholders still significantly underestimate safety-related issues when implementing cobots in new production lines. This implies the need for objective assessment tools that can systematically address and assess all safety-related aspects. This study provides industry practitioners with a broader view of how different stakeholders regard safety-related aspects in the cobot life cycle. This will enable more informed decision-making when implementing cobot solutions and potentially avoid unconsidered safety risks later on. This article also adds to the scant research on the role of each stakeholder in human–robot collaboration.

An adaptive indoor temperature control approach simultaneously improving thermal comfort and task performance

Air conditioning and mechanical ventilation systems help maintain the thermal comfort (TC) of occupants in a workplace. Indoor temperature determination that only aims to improve TC causes difficulty in fulfilling the occupant's task performance (TP). It is crucial to find an appropriate indoor temperature range that simultaneously accommodates occupants' TC and TP. In this study, we propose an adaptive temperature control approach that simultaneously improves the occupant's TC and TP. The relationship among the dissatisfaction rate, predictive mean vote, and indoor temperature was established and quantified. An investigation in a human-robot collaboration scenario was conducted by using the proposed approach. The preferable temperature range under various combinations of TC and TP is recommended. The proposed algorithm successfully achieves a significant energy-saving of 9.98% within the desired temperature range. By reducing the preferred upper limit of the temperature range by 3 °C compared to current standards, the algorithm further reduces energy consumption. To validate the effectiveness of the proposed model, we developed empirical formulas that establish the relationship between thermal comfort, task performance, and temperature, enabling the identification of optimal indoor temperature settings. Additionally, our model effectively captures the complex trade-offs and complementary nature between thermal comfort and task performance with respect to indoor temperature. Overall, this approach offers a solution for setting indoor temperatures in the workplace.

Coordinating human-robot collaboration by EEG-based human intention prediction and vigilance control.

In human-robot collaboration scenarios with shared workspaces, a highly desired performance boost is offset by high requirements for human safety, limiting speed and torque of the robot drives to levels which cannot harm the human body. Especially for complex tasks with flexible human behavior, it becomes vital to maintain safe working distances and coordinate tasks efficiently. An established approach in this regard is reactive servo in response to the current human pose. However, such an approach does not exploit expectations of the human's behavior and can therefore fail to react to fast human motions in time. To adapt the robot's behavior as soon as possible, predicting human intention early becomes a factor which is vital but hard to achieve. Here, we employ a recently developed type of brain-computer interface (BCI) which can detect the focus of the human's overt attention as a predictor for impending action. In contrast to other types of BCI, direct projection of stimuli onto the workspace facilitates a seamless integration in workflows. Moreover, we demonstrate how the signal-to-noise ratio of the brain response can be used to adjust the velocity of the robot movements to the vigilance or alertness level of the human. Analyzing this adaptive system with respect to performance and safety margins in a physical robot experiment, we found the proposed method could improve both collaboration efficiency and safety distance.

A framework for robotic arm pose estimation and movement prediction based on deep and extreme learning models

Human-robot collaboration has gained a notable prominence in Industry 4.0, as the use of collaborative robots increases efficiency and productivity in the automation process. However, it is necessary to consider the use of mechanisms that increase security in these environments, as the literature reports that risk situations may exist in the context of human-robot collaboration. One of the strategies that can be adopted is the visual recognition of the collaboration environment using machine learning techniques, which can automatically identify what is happening in the scene and what may happen in the future. In this work, we are proposing a new framework that is capable of detecting robotic arm keypoints commonly used in Industry 4.0. In addition to detecting, the proposed framework is able to predict the future movement of these robotic arms, thus providing relevant information that can be considered in the recognition of the human-robot collaboration scenario. The proposed framework has two main modules. The first one contains a convolutional neural network based on self-calibrated convolutions enabling better discriminative feature extraction and the support of extreme learning machine neural networks with different kernels for predicting robotic arm keypoints. The second module is composed of deep recurrent learning models, such as long short-term memory and gated recurrent unit. These models are able to predict future robotic arm keypoints. All experiments were evaluated using the mean squared error metric. Results show that the proposed framework is capable of detecting and predicting with low error, contributing to the mitigation of risks in human-robot collaboration. In addition, it was possible to verify that the use of convolutional neural networks in conjunction with extreme learning machines can offer a lower detection error in a regression task (e.g., keypoint detection), something that, as far as the authors are aware of, is not yet known, nor had been evaluated previously in the literature.

Learning from Demonstrations in Human–Robot Collaborative Scenarios: A Survey

Human–Robot Collaboration (HRC) is an interdisciplinary research area that has gained attention within the smart manufacturing context. To address changes within manufacturing processes, HRC seeks to combine the impressive physical capabilities of robots with the cognitive abilities of humans to design tasks with high efficiency, repeatability, and adaptability. During the implementation of an HRC cell, a key activity is the robot programming that takes into account not only the robot restrictions and the working space, but also human interactions. One of the most promising techniques is the so-called Learning from Demonstration (LfD), this approach is based on a collection of learning algorithms, inspired by how humans imitate behaviors to learn and acquire new skills. In this way, the programming task could be simplified and provided by the shop floor operator. The aim of this work is to present a survey of this programming technique, with emphasis on collaborative scenarios rather than just an isolated task. The literature was classified and analyzed based on: the main algorithms employed for Skill/Task learning, and the human level of participation during the whole LfD process. Our analysis shows that human intervention has been poorly explored, and its implications have not been carefully considered. Among the different methods of data acquisition, the prevalent method is physical guidance. Regarding data modeling, techniques such as Dynamic Movement Primitives and Semantic Learning were the preferred methods for low-level and high-level task solving, respectively. This paper aims to provide guidance and insights for researchers looking for an introduction to LfD programming methods in collaborative robotics context and identify research opportunities.

Influence of task decision autonomy on physical ergonomics and robot performances in an industrial human-robot collaboration scenario.

Adoption of human–robot collaboration is hindered by barriers in collaborative task design. A new approach for solving these problems is to empower operators in the design of their tasks. However, how this approach may affect user welfare or performance in industrial scenarios has not yet been studied. Therefore, in this research, the results of an experiment designed to identify the influences of the operator’s self-designed task on physical ergonomics and task performance are presented. At first, a collaborative framework able to accept operator task definition via parts’ locations and monitor the operator’s posture is presented. Second, the framework is used to tailor a collaborative experience favoring decision autonomy using the SHOP4CF architecture. Finally, the framework is used to investigate how this personalization influences collaboration through a user study with untrained personnel on physical ergonomics. The results from this study are twofold. On one hand, a high degree of decision autonomy was felt by the operators when they were allowed to allocate the parts. On the other hand, high decision autonomy was not found to vary task efficiency nor the MSD risk level. Therefore, this study emphasizes that allowing operators to choose the position of the parts may help task acceptance and does not vary operators’ physical ergonomics or task efficiency. Unfortunately, the test was limited to 16 participants and the measured risk level was medium. Therefore, this study also stresses that operators should be allowed to choose their own work parameters, but some guidelines should be followed to further reduce MSD risk levels.

Collaborating eye to eye: Effects of workplace design on the perception of dominance of collaboration robots.

The concept of Human-Robot Collaboration (HRC) describes innovative industrial work procedures, in which human staff works in close vicinity with robots on a shared task. Current HRC scenarios often deploy hand-guided robots or remote controls operated by the human collaboration partner. As HRC envisions active collaboration between both parties, ongoing research efforts aim to enhance the capabilities of industrial robots not only in the technical dimension but also in the robot’s socio-interactive features. Apart from enabling the robot to autonomously complete the respective shared task in conjunction with a human partner, one essential aspect lifted from the group collaboration among humans is the communication between both entities. State-of-the-art research has identified communication as a significant contributor to successful collaboration between humans and industrial robots. Non-verbal gestures have been shown to be contributing aspect in conveying the respective state of the robot during the collaboration procedure. Research indicates that, depending on the viewing perspective, the usage of non-verbal gestures in humans can impact the interpersonal attribution of certain characteristics. Applied to collaborative robots such as the Yumi IRB 14000, which is equipped with two arms, specifically to mimic human actions, the perception of the robots’ non-verbal behavior can affect the collaboration. Most important in this context are dominance emitting gestures by the robot that can reinforce negative attitudes towards robots, thus hampering the users’ willingness and effectiveness to collaborate with the robot. By using a 3 × 3 within-subjects design online study, we investigated the effect of dominance gestures (Akimbo, crossing arms, and large arm spread) working in a standing position with an average male height, working in a standing position with an average female height, and working in a seated position on the perception of dominance of the robot. Overall 115 participants (58 female and 57 male) with an average age of 23 years evaluated nine videos of the robot. Results indicated that all presented gestures affect a person’s perception of the robot in regards to its perceived characteristics and willingness to cooperate with the robot. The data also showed participants’ increased attribution of dominance based on the presented viewing perspective.

Mitigating the risk of musculoskeletal disorders during human robot collaboration: a reinforcement learning approach

Work-related musculoskeletal disorders (MSDs) are often observed in human-robot collaboration (HRC), a common work configuration in modern factories. In this study, we aim to reduce the risk of MSDs in HRC scenarios by developing a novel model-free reinforcement learning (RL) method to improve workers’ postures. Our approach follows two steps: first, we adopt a 3D human skeleton reconstruction method to calculate workers’ Rapid Upper Limb Assessment (RULA) scores; next, we devise an online gradient-based RL algorithm to dynamically improve the RULA score. Compared with previous model-based studies, the key appeals of the proposed RL algorithm are two-fold: (i) the model-free structure allows it to “learn” the optimal worker postures without need any specific biomechanical models of tasks or workers, and (ii) the data-driven nature makes it accustomed to arbitrary users by providing personalized work configurations. Results of our experiments confirm that the proposed method can significantly improve the workers’ postures.

A framework and method for Human-Robot cooperative safe control based on digital twin

Human-robot collaboration (HRC) combines the robot’s mechanical properties and predictability with human experience, logical thinking, and strain capabilities to alleviate production efficiency. However, ensuring the safety of the HRC process in-real time has become an urgent issue. Digital twin extends functions of virtual models in the design phase of the physical counterpart in the production phase through virtual-real interactive feedback, data fusion analysis, advanced computational features, etc. This paper proposes an HRC safety control framework and corresponding method based on the digital twin. In the design phase, virtual simulation and virtual reality technology are integrated to construct virtual twins of various HRC scenarios for testing and analyzing potential safety hazards. In the production phase, the safety distance between humans and robots of the HRC scene is monitored and calculated by an iterative algorithm according to machine vision and a convolutional neural network. Finally, the virtual twin is driven based on real-scene data, real-time online visual monitoring, and optimization of the HRC’s overall process. A case study using ABB-IRB1600 is presented to verify the feasibility of the proposed approach.

Coupled dynamic modeling and experimental validation of a collaborative industrial mobile manipulator with human-robot interaction

Dynamic modeling and analysis of a collaborative industrial mobile manipulator provide essential foundation and guidance for controlling it to achieve the desired physical interaction. However, it is a well-recognized challenge to precisely model the dynamics of a collaborative mobile manipulator due to its high-DOF (degree of freedom) structure, nonholonomic constraints, dynamic coupling between the manipulator and the mobile platform, and physical interaction with environments/humans. This paper presents a dynamic model of a new high-DOF nonholonomic collaborative mobile manipulator applied to physical human-robot collaboration scenarios, accounting for the dynamic coupling. A full Jacobian matrix introduces the dynamic coupling and is derived to solve the nonholonomic constraints on the velocity of the mobile platform. Besides, it facilitates the formulation of the dynamic model with independent generalized coordinates based on Lagrange equations. The comprehensive simulations with ADAMS are carried out along a pre-set trajectory to verify the coupled dynamic model. The dynamic modeling method is further validated by the human-robot interaction experiment. The simulation and testing results demonstrate the accuracy of the coupled dynamic model of the collaborative mobile manipulator.

A Resilient and Effective Task Scheduling Approach for Industrial Human-Robot Collaboration.

Effective task scheduling in human-robot collaboration (HRC) scenarios is one of the great challenges of collaborative robotics. The shared workspace inside an industrial setting brings a lot of uncertainties that cannot be foreseen. A prior offline task scheduling strategy is ineffective in dealing with these uncertainties. In this paper, a novel online framework to achieve a resilient and reliable task schedule is presented. The framework can deal with deviations that occur during operation, different operator skills, error by the human or robot, and substitution of actors, while maintaining an efficient schedule by promoting parallel human-robot work. First, the collaborative job and the possible deviations are represented by AND/OR graphs. Subsequently, the proposed architecture chooses the most suitable path to improve the collaboration. If some failures occur, the AND/OR graph is adapted locally, allowing the collaboration to be completed. The framework is validated in an industrial assembly scenario with a Franka Emika Panda collaborative robot.