Industrial Human-robot Collaboration Research Articles

Ergonomically optimized path-planning for industrial human–robot collaboration

This paper focuses on improving the ergonomics of industrial workers. It addresses the critical implications of poor ergonomics, which can lead to musculoskeletal disorders over time. A novel methodology for a path-planning algorithm designed for human–robot collaboration was introduced to tackle this challenge. The algorithm’s essential contribution lies in determining the most ergonomic path for a robot to guide a human’s hand during task execution, facilitating a transition toward an optimized body configuration. The algorithm effectively charts the ergonomic path by adopting a Cartesian path-planning approach and employing the cell decomposition method. The methodology was implemented on a dataset of ten individuals, representing a diverse group of male and female subjects aged between 20 and 35, with one participant being left-handed. The algorithm was applied to three different activities: “stacking an item,” “taking an object from a shelf,” and “assembling an object by sitting over a table.” The results demonstrated a significant improvement in the REBA score (as a measure of ergonomics condition) of the individuals after applying the algorithm. This outcome reinforces the efficacy of the methodology in enhancing the ergonomics of industrial workers. Furthermore, the study compared the performance of A* with three heuristic functions against Dijkstra’s algorithm, aiming to identify the most effective approach for achieving optimal ergonomic paths in human–robot collaboration. The findings revealed that A* with a specific heuristic function surpassed Dijkstra’s algorithm, underscoring its superiority in this context. The findings highlight the potential for optimizing human–robot collaboration and offer practical implications for designing more efficient industrial work environments.

Behavioral patterns in robotic collaborative assembly: comparing neurotypical and Autism Spectrum Disorder participants

IntroductionIn Industry 4.0, collaborative tasks often involve operators working with collaborative robots (cobots) in shared workspaces. Many aspects of the operator's well-being within this environment still need in-depth research. Moreover, these aspects are expected to differ between neurotypical (NT) and Autism Spectrum Disorder (ASD) operators.MethodsThis study examines behavioral patterns in 16 participants (eight neurotypical, eight with high-functioning ASD) during an assembly task in an industry-like lab-based robotic collaborative cell, enabling the detection of potential risks to their well-being during industrial human-robot collaboration. Each participant worked on the task for five consecutive days, 3.5 h per day. During these sessions, six video clips of 10 min each were recorded for each participant. The videos were used to extract quantitative behavioral data using the NOVA annotation tool and analyzed qualitatively using an ad-hoc observational grid. Also, during the work sessions, the researchers took unstructured notes of the observed behaviors that were analyzed qualitatively.ResultsThe two groups differ mainly regarding behavior (e.g., prioritizing the robot partner, gaze patterns, facial expressions, multi-tasking, and personal space), adaptation to the task over time, and the resulting overall performance.DiscussionThis result confirms that NT and ASD participants in a collaborative shared workspace have different needs and that the working experience should be tailored depending on the end-user's characteristics. The findings of this study represent a starting point for further efforts to promote well-being in the workplace. To the best of our knowledge, this is the first work comparing NT and ASD participants in a collaborative industrial scenario.

Deep Learning Framework for Controlling Work Sequence in Collaborative Human–Robot Assembly Processes

The human-robot collaboration (HRC) solutions presented so far have the disadvantage that the interaction between humans and robots is based on the human's state or on specific gestures purposely performed by the human, thus increasing the time required to perform a task and slowing down the pace of human labor, making such solutions uninteresting. In this study, a different concept of the HRC system is introduced, consisting of an HRC framework for managing assembly processes that are executed simultaneously or individually by humans and robots. This HRC framework based on deep learning models uses only one type of data, RGB camera data, to make predictions about the collaborative workspace and human action, and consequently manage the assembly process. To validate the HRC framework, an industrial HRC demonstrator was built to assemble a mechanical component. Four different HRC frameworks were created based on the convolutional neural network (CNN) model structures: Faster R-CNN ResNet-50 and ResNet-101, YOLOv2 and YOLOv3. The HRC framework with YOLOv3 structure showed the best performance, showing a mean average performance of 72.26% and allowed the HRC industrial demonstrator to successfully complete all assembly tasks within a desired time window. The HRC framework has proven effective for industrial assembly applications.

Robot learning towards smart robotic manufacturing: A review

Robotic equipment has been playing a central role since the proposal of smart manufacturing. Since the beginning of the first integration of industrial robots into production lines, industrial robots have enhanced productivity and relieved humans from heavy workloads significantly. Towards the next generation of manufacturing, this review first introduces the comprehensive background of smart robotic manufacturing within robotics, machine learning, and robot learning. Definitions and categories of robot learning are summarised. Concretely, imitation learning, policy gradient learning, value function learning, actor-critic learning, and model-based learning as the leading technologies in robot learning are reviewed. Training tools, benchmarks, and comparisons amongst different robot learning methods are delivered. Typical industrial applications in robotic grasping, assembly, process control, and industrial human-robot collaboration are listed and discussed. Finally, open problems and future research directions are summarised.

Influence of task decision autonomy on physical ergonomics and robot performances in an industrial human-robot collaboration scenario.

Adoption of human–robot collaboration is hindered by barriers in collaborative task design. A new approach for solving these problems is to empower operators in the design of their tasks. However, how this approach may affect user welfare or performance in industrial scenarios has not yet been studied. Therefore, in this research, the results of an experiment designed to identify the influences of the operator’s self-designed task on physical ergonomics and task performance are presented. At first, a collaborative framework able to accept operator task definition via parts’ locations and monitor the operator’s posture is presented. Second, the framework is used to tailor a collaborative experience favoring decision autonomy using the SHOP4CF architecture. Finally, the framework is used to investigate how this personalization influences collaboration through a user study with untrained personnel on physical ergonomics. The results from this study are twofold. On one hand, a high degree of decision autonomy was felt by the operators when they were allowed to allocate the parts. On the other hand, high decision autonomy was not found to vary task efficiency nor the MSD risk level. Therefore, this study emphasizes that allowing operators to choose the position of the parts may help task acceptance and does not vary operators’ physical ergonomics or task efficiency. Unfortunately, the test was limited to 16 participants and the measured risk level was medium. Therefore, this study also stresses that operators should be allowed to choose their own work parameters, but some guidelines should be followed to further reduce MSD risk levels.

User Experience and Physiological Response in Human-Robot Collaboration: A Preliminary Investigation

Within the context of Industry 4.0 and of the new emerging Industry 5.0, human factors are becoming increasingly important, especially in Human-Robot Collaboration (HRC). This paper provides a novel study focused on the human aspects involved in industrial HRC by exploring the effects of various HRC setting factors. In particular, this paper aims at investigating the impact of industrial HRC on user experience, affective state, and stress, assessed through both subjective measures (i.e., questionnaires) and objective ones (i.e., physiological signals). A collaborative assembly task was implemented with different configurations, in which the robot movement speed, the distance between the operator and the robot workspace, and the control of the task execution time were varied. Forty-two participants were involved in the study and provided feedbacks on interaction quality and their affective state. Participants’ physiological responses (i.e., electrodermal activity and heart rate) were also collected non-invasively to monitor the amount of stress generated by the interaction. Analysis of both subjective and objective responses revealed how the configuration factors considered influence them. Robot movement speed and control of the task execution time resulted to be the most influential factors. The results also showed the need for customization of HRC to improve ergonomics, both psychological and physical, and the well-being of the operator.

A Resilient and Effective Task Scheduling Approach for Industrial Human-Robot Collaboration.

Effective task scheduling in human-robot collaboration (HRC) scenarios is one of the great challenges of collaborative robotics. The shared workspace inside an industrial setting brings a lot of uncertainties that cannot be foreseen. A prior offline task scheduling strategy is ineffective in dealing with these uncertainties. In this paper, a novel online framework to achieve a resilient and reliable task schedule is presented. The framework can deal with deviations that occur during operation, different operator skills, error by the human or robot, and substitution of actors, while maintaining an efficient schedule by promoting parallel human-robot work. First, the collaborative job and the possible deviations are represented by AND/OR graphs. Subsequently, the proposed architecture chooses the most suitable path to improve the collaboration. If some failures occur, the AND/OR graph is adapted locally, allowing the collaboration to be completed. The framework is validated in an industrial assembly scenario with a Franka Emika Panda collaborative robot.

Human–robot collaboration in industrial environments: A literature review on non-destructive disassembly

Nowadays, numerous companies and industries introduce recycling processes in their production, aiming to increase the sustainable use of the planet’s natural resources. Nevertheless, these processes remain inefficient due to the high degree of complexity and variation in the products. In order to remedy this, industry stakeholders adopt the circular economy business model and introduce take-back programmes and remanufacturing processes for their End of Life products in their own supply chains. Take-back programmes enable the re-sourcing of sub-assemblies and components of previously manufactured products while remanufacturing processes encourage non-destructive disassembly. Due to the uncertain conditions of the re-sourced products, fully automated cells cannot cope with the demanding disassembly processes. Therefore, there is a need to establish hybrid disassembly robot cells where humans and robots work closely together in a process known as human–robot collaborative disassembly (HRCD). This paper examines the landscape of HRCD and reviews the progress in the field during the period 2009–2020. The analysis investigates principles and elements of human–robot collaboration in industrial environments such as safety standards and collaborative operation modes, HRI communication interfaces, and the design characteristics of a disassembly process. Additionally, the various technical challenges of HRCD are explored, and a review of existing systems supporting HRCD is presented. This review aims to support the robotics community in the future development of HRCD systems, discuss identified literature gaps, and suggest future research directions in this area. • This paper presents the first extensive review of the latest progress in the area of Industrial human–robot collaboration within the field of disassembly for the period 2009–2020. • The perspective is on how robot control schemes and human–robot interfaces enable human–robot collaborative disassembly tasks. • The outcome of this review identifies gaps in the practice of human–robot collaborative disassembly and provides the future trends and needs for robot cells focused on disassembly tasks.

Decision-Making Framework for Implementing Safer Human-Robot Collaboration Workstations: System Dynamics Modeling

Human-Robot Collaboration (HRC) systems are often implemented seeking for reducing risk of Work-related Musculoskeletal Disorders (WMSD) development and increasing productivity. The challenge is to successfully implement an industrial HRC to manage those factors, considering that non-linear behaviors of complex systems can produce counterintuitive effects. Therefore, the aim of this study was to design a decision-making framework considering the key ergonomic methods and using a computational model for simulations. It considered the main systemic influences when implementing a collaborative robot (cobot) into a production system and simulated scenarios of productivity and WMSD risk. In order to verify whether the computational model for simulating scenarios would be useful in the framework, a case study in a manual assembly workstation was conducted. The results show that both cycle time and WMSD risk depend on the Level of Collaboration (LoC). The proposed framework helps deciding which cobot to implement in a context of industrial assembly process. System dynamics were used to understand the actual behavior of all factors and to predict scenarios. Finally, the framework presented a clear roadmap for the future development of an industrial HRC system, drastically reducing risk management in decision-making.

A closed-loop brain-computer interface with augmented reality feedback for industrial human-robot collaboration

Industrial human-robot collaboration (HRC) aims to combine human intelligence and robotic capability to achieve higher productiveness. In industrial HRC, the communication between humans and robots is essential to enhance the understanding of the intent of each other to make a more fluent collaboration. Brain-computer interface (BCI) is a technology that could record the user’s brain activity that can be translated into interaction messages (e.g., control commands) to the outside world, which can build a direct and efficient communication channel between human and robot. However, due to lacking information feedback mechanisms, it is challenging for BCI to control robots with a high degree of freedom with a limited number of classifiable mental states. To address this problem, this paper proposes a closed-loop BCI with contextual visual feedback by an augmented reality (AR) headset. In such BCI, the electroencephalogram (EEG) patterns from the multiple voluntary eye blinks are considered the input and its online detection algorithm is proposed whose average accuracy can reach 94.31%. Moreover, an AR-enable information feedback interface is designed to achieve an interactive robotic path planning. A case study of an industrial HRC assembly task is also developed to show that the proposed closed-up BCI could shorten the time of user input in human-robot interaction.

Working with Industrial Cobots: The Influence of Reliability and Transparency on Perception and Trust

Industrial human-robot collaboration (HRC) is not yet widely spread but on the rise. This development raises the question about properties collaborative robots (cobots) need, to enable a pleasant and smooth interaction. Therefore, this study investigated the influence of transparency and reliability on perception of and trust towards cobots. A video-enhanced online study with 124 participants was conducted. Transparency was provided through the presentation of differing information, and reliability was manipulated through differing error rates. The results showed a positive effect of transparency on perceived safety and intelligence. Reliability had a positive effect on perceived intelligence, likeability and trust. The effect of reliability on trust was more pronounced for low transparent robots. The results indicate the relevance of carefully selected information to counteract negative effects of failures. Future research should transfer the study design into a real-life experiment with more fine-grained levels of transparency and reliability.

Deep reinforcement learning-based safe interaction for industrial human-robot collaboration using intrinsic reward function

Aiming at human-robot collaboration in manufacturing, the operator's safety is the primary issue during the manufacturing operations. This paper presents a deep reinforcement learning approach to realize the real-time collision-free motion planning of an industrial robot for human-robot collaboration. Firstly, the safe human-robot collaboration manufacturing problem is formulated into a Markov decision process, and the mathematical expression of the reward function design problem is given. The goal is that the robot can autonomously learn a policy to reduce the accumulated risk and assure the task completion time during human-robot collaboration. To transform our optimization object into a reward function to guide the robot to learn the expected behaviour, a reward function optimizing approach based on the deterministic policy gradient is proposed to learn a parameterized intrinsic reward function. The reward function for the agent to learn the policy is the sum of the intrinsic reward function and the extrinsic reward function. Then, a deep reinforcement learning algorithm intrinsic reward-deep deterministic policy gradient (IRDDPG), which is the combination of the DDPG algorithm and the reward function optimizing approach, is proposed to learn the expected collision avoidance policy. Finally, the proposed algorithm is tested in a simulation environment, and the results show that the industrial robot can learn the expected policy to achieve the safety assurance for industrial human-robot collaboration without missing the original target. Moreover, the reward function optimizing approach can help make up for the designed reward function and improve policy performance.

Risk assessment tools for industrial human-robot collaboration: Novel approaches and practical needs

According to the standard ISO 10218–2, industrial robot systems must be subjected to a risk assessment prior to commissioning. In current industrial practice, risk assessments are conducted on the basis of experience, expert knowledge, and simple tools such as checklists. However, the recent trend towards human-robot collaboration (HRC) significantly increases the complexity of robot systems and makes risk assessment more challenging. In response to this challenge, the scientific community has proposed various new tools and methods to support risk assessment of HRC applications. As of yet, only few of these novel approaches have found their way into industrial practice. In this paper, we review literature on novel approaches to HRC risk assessment. Furthermore, we evaluate interviews with professionals from the field of HRC to explore needs of industrial practitioners. We compare our findings from literature review and interviews, and discuss which challenges need to be addressed to successfully transfer novel approaches into industrial practice.

A Multimodal Approach to Human Safety in Collaborative Robotic Workcells

This article investigates the problem of controlling the speed of robots in collaborative workcells for automated manufacturing. The solution is tailored to robotic cells for cooperative assembly of aircraft fuselage panels, where only structural elements are present and robots and humans can share the same workspace, but no physical contact is allowed, unless it happens at zero robot speed. The proposed approach addresses the problem of satisfying the minimal set of requirements of an industrial human–robot collaboration (HRC) task: precision and reliability of human detection and tracking in the shared workspace; correct robot task execution with minimum cycle time while assuring safety for human operators. These requirements are often conflicting with each other. The former does not only concern with safety only but also with the need of avoiding unnecessary robot stops or slowdowns in case of false-positive human detection. The latter, according to the current regulations, concerns with the need of computing the minimum protective separation distance between the human operator and the robots by adjusting their speed when dangerous situations happen. This article proposes a novel fuzzy inference approach to control robot speed enforcing safety while maximizing the level of productivity of the robot minimizing cycle time as well. The approach is supported by a sensor fusion algorithm that merges the images acquired from different depth sensors with those obtained from a thermal camera, by using a machine learning approach. The methodology is experimentally validated in two experiments: the first one at a lab-scale and the second one performed on a full-scale robotic workcell for cooperative assembly of aeronautical structural parts. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article discusses a way to handle human safety specifications versus production requirements in collaborative robotized assembly systems. State-of-the-art (SoA) approaches cover only a few aspects of both human detection and robot speed scaling. The present research work proposes a complete pipeline that starts from a robust human tracking algorithm and scales the robot speed in real time. An innovative multimodal perception system composed of two depth cameras and a thermal camera monitors the collaborative workspace. The speed scaling algorithm is optimized to take on different human behaviors during less risky situations or more dangerous ones to guarantee both operator safety and minimum production time with the aim of better profitability and efficiency for collaborative workstations. The algorithm estimates the operator intention for real-time computation of the minimum protective distance according to the current safety regulations. The robot speed is smoothly changed for the psychological advantages of operators, both in the case of single and multiple workers. The result is a complete system, easily implementable on a standard industrial workcell.

Seven Levels of Detail to structure use cases and interaction mechanism for the development of industrial Virtual Reality applications within the context of planning and configuration of robot-based automation solutions

The term Virtual Reality (VR), is very well known in the consumer goods and entertainment sector. The visual presentation of the VR environment is typically achieved via stereoscopic head-mounted displays (HMD), and the interaction via a 3D position detection of HMD and additional input devices such as controllers for active user input. While the variety of applications in the entertainment industry is constantly growing and the technological possibilities become more extensive, this trend has marginally established itself in the industry. Even though powerful and affordable VR hardware is available, VR applications are often associated more with gaming than professional industrial applications. In addition, only few interaction mechanisms such as 3D viewing, moving, teleporting and rarely direct interaction capabilities are used in the most industrial VR solutions. The reason for this is often a lack of understanding and structure of use cases and the added value that VR applications and interactions create for companies and their customers. This unnecessarily limits the applicability of new VR applications for the industry. For a better structuring of VR use cases and required 3D objects for targeted user interaction, we introduce seven Levels of Detail. Along these, one VR use case setup is created, to provide examples for classical concept planning and a new knowledge-based process based on engineering Best Practices. For each, we derive adequate prototypical implemented demonstrator stations and necessary interaction mechanisms for VR development. To highlight further VR possibilities, we extend the examples by adding a second use case setup for VR planning and virtual commissioning of industrial human-robot collaboration solutions based on body and hand tracking. Hence, the contribution provides a structured compilation of potential and useful industrial VR planning use case setups and for these relevant interaction mechanisms in combination with concretely implemented examples.

Dynamic risk assessment and active response strategy for industrial human-robot collaboration

To enhance flexibility and sustainability, human-robot collaboration is becoming a major feature of next-generation robots. The safety assessment strategy is the first and crucial issue that needs to be considered due to the removal of the safety barrier. This paper determined the set of safety indicators and established an assessment model based on the latest safety-related ISO standards and manufacturing conditions. A dynamic modified SSM (speed and separation monitoring) method is presented for ensuring the safety of human-robot collaboration while maintaining productivity as high as possible. A prototype system including dynamic risk assessment and safe motion control is developed based on the virtual model of the robot and human skeleton point data from the vision sensor. The real-time risk status of the working robot can be known and the risk field around the robot which is visualized in an augmented reality environment so as to ensure safe human-robot collaboration. This system is experimentally validated on a human-robot collaboration cell using an industrial robot with six degrees of freedom.

A Framework for Realizing Industrial Human-Robot Collaboration through Virtual Simulation

Human-Robot Collaboration (HRC) could improve manual labor manufacturing processes by relieving workers of inappropriate operations, such as heavy, repetitive, or manual quality inspections. The literature shows that the greatest challenge for large scale implementation of Human-Robot Collaboration is safety, intuitive interfaces and design methods. We present a comprehensive framework that incorporates the use of virtual simulation for the implementation of Human-Robot Collaboration. The proposed framework defines a development process with five major steps, ending up with a virtual simulation model that can provide a foundation for the physical implementation. The framework has been developed using an automotive industrial case.

The development and evaluation of Robot Light Skin: A novel robot signalling system to improve communication in industrial human–robot collaboration

Abstract In a human–robot collaborative production system, the robot could make request for interaction or notify the human operator if an uncertainty arises. Conventional industrial tower lights were designed for generic machine signalling purposes which may not be the ultimate solution for robot signalling in a collaborative setting. In this type of system, human operators could be monitoring multiple robots while carrying out a manual task so it is important to minimise the diversion of their attention. This paper presents a novel robot signalling solution, the Robot Light Skin (RLS),which is an integrated signalling system that could be used on most articulated robots. Our experiment was conducted to validate this concept in terms of its effect on improving operator's reaction time, hit-rate, awareness and task performance. The results showed that participants reacted faster to the RLS as well as achieved higher hit-rate. An eye tracker was used in the experiment which shows a reduction in diversion away from the manual task when using the RLS. Future study should explore the effect of the RLS concept on large-scale systems and multi-robot systems.

The Design and Evaluation of an Ergonomic Contactless Gesture Control System for Industrial Robots

In industrial human-robot collaboration, variability commonly exists in the operation environment and the components, which induces uncertainty and error that require frequent manual intervention for rectification. Conventional teach pendants can be physically demanding to use and require user training prior to operation. Thus, a more effective control interface is required. In this paper, the design and evaluation of a contactless gesture control system using Leap Motion is described. The design process involves the use of RULA human factor analysis tool. Separately, an exploratory usability test was conducted to compare three usability aspects between the developed gesture control system and an off-the-shelf conventional touchscreen teach pendant. This paper focuses on the user-centred design methodology of the gesture control system. The novelties of this research are the use of human factor analysis tools in the human-centred development process, as well as the gesture control design that enable users to control industrial robot’s motion by its joints and tool centre point position. The system has potential to use as an input device for industrial robot control in a human-robot collaboration scene. The developed gesture control system was targeting applications in system recovery and error correction in flexible manufacturing environment shared between humans and robots. The system allows operators to control an industrial robot without the requirement of significant training.

A Morphology of Human Robot Collaboration Systems for Industrial Assembly

In recent years, the number of hybrid work systems using human robot collaboration (HRC) increased in industrial production environments – enhancing productivity while reducing work-related burden. Despite growing availability of HRC-suitable manipulation and safety technology, tools and techniques facilitating the design, planning and implementation process are still lacking. System engineers who strive to implement technically feasible, ergonomically meaningful and economically beneficial HRC applications need to make design and technology decisions in various subject areas, whereas the design alternatives per such subject area are plenty – combining aspects to a challenge of increased complexity. In this paper, the heuristic procedure of morphological analysis is applied to establish a description model that can serve as both a supporting design guideline for future HRC applications of value-adding, industrial quality as well as a tool to characterize and compare existing applications. It focuses on HRC within assembly processes, and illustrates the complexity of HRC applications in a comprehensible manner through its multi-dimensional structure. The morphology has been validated through its application on various existing industrial HRC applications, research demonstrators and interviews of experts from academia.