Human-robot Collaborative Environments Research Articles

A review of external sensors for human detection in a human robot collaborative environment

AbstractManufacturing industries are eager to replace traditional robot manipulators with collaborative robots due to their cost-effectiveness, safety, smaller footprint and intuitive user interfaces. With industrial advancement, cobots are required to be more independent and intelligent to do more complex tasks in collaboration with humans. Therefore, to effectively detect the presence of humans/obstacles in the surroundings, cobots must use different sensing modalities, both internal and external. This paper presents a detailed review of sensor technologies used for detecting a human operator in the robotic manipulator environment. An overview of different sensors installed locations, the manipulator details and the main algorithms used to detect the human in the cobot workspace are presented. We summarize existing literature in three categories related to the environment for evaluating sensor performance: entirely simulated, partially simulated and hardware implementation focusing on the ‘hardware implementation’ category where the data and experimental environment are physical rather than virtual. We present how the sensor systems have been used in various use cases and scenarios to aid human–robot collaboration and discuss challenges for future work.

Benchmarking human-robot collaborative assembly tasks

Manufacturing assembly tasks can vary in complexity and level of automation. Yet, achieving full automation can be challenging and inefficient, particularly due to the complexity of certain assembly operations. Human-robot collaborative work, leveraging the strengths of human labor alongside the capabilities of robots, can be a solution for enhancing efficiency. This paper introduces the CT benchmark, a benchmark and model set designed to facilitate the testing and evaluation of human-robot collaborative assembly scenarios. It was designed to compare manual and automatic processes using metrics such as the assembly time and human workload. The components of the model set can be assembled through the most common assembly tasks, each with varying levels of difficulty. The CT benchmark was designed with a focus on its applicability in human-robot collaborative environments, with the aim of ensuring the reproducibility and replicability of experiments. Experiments were carried out to assess assembly performance in three different setups (manual, automatic and collaborative), measuring metrics related to the assembly time and the workload on human operators. The results suggest that the collaborative approach takes longer than the fully manual assembly, with an increase of 70.8%. However, users reported a lower overall workload, as well as reduced mental demand, physical demand, and effort according to the NASA-TLX questionnaire.

Quality control in manufacturing – review and challenges on robotic applications

ABSTRACT Quality control methods and techniques have been investigated in different fields of manufacturing during the last decades. The introduction of robots in manufacturing processes has created a rapid deployment of robotic applications, leading to an increased research interest in the aspect of quality control in such industrial environments. This paper summarizes and presents a review of recent research and progress on quality control technologies, focusing on robot-based production systems. The role of different parameters affecting the control of quality in robotic tasks is also discussed, incorporating the impact of operator support systems in human robot collaborative environments. Research gaps and implications of quality control in robotic applications are also described, and a future outlook of research in the particular field is provided.

Multi-objective optimization of cycle time and robot energy expenditure in human-robot collaborated assembly lines

The recent Industry 4.0 trend, followed by the technological advancement of collaborative robots, has convinced many industries to shift towards semi-automated assembly lines with human-robot collaboration (HRC). In the HRC environment, robot agility can support human skill upon efficiently balancing tasks among the stations and operators. On the other hand, the robot energy consumption in today's energy crisis area demands that tasks be performed with as little energy utilization as possible by robots. In this context, the cycle time (CT) and total energy cost (TEC) of robots are among two conflicting objectives. Thus, this study balances HRC lines where a trade-off between CT and TEC of robots is sought. A mixed-integer linear programming model is proposed to formulate the problem. In addition, a multi-objective optimization approach based on ε-constraint is developed to address a case study from the automotive industry and a set of generated test problems. The computational results show that promising Pareto solutions in terms of CT and TEC can be obtained using the proposed approach.

Dynamic collision estimator for collaborative robots: A dynamic Bayesian network with Markov model for highly reliable collision detection

In this paper, we propose a highly reliable and accurate collision estimator for robot manipulators working in human–robot collaborative environments, based on the Bayesian approach for practical uses. We assume the robot collision as a dynamic Markov process, not a static event, to reflect the transient behavior of mechanical collisions. Thus, the collision estimator can integrate the prior belief on collision and the measurements on the robot state, to produce the current belief on the robot collision in a usual recursive form. An exponential form of observation model, serving as the likelihood function, is constructed on the projected observation domain by using the multi-variate statistical information of empirical models of collision and non-collision cases. The proposed method is validated by using a commercial 7 degree-of-freedom (DOF) collaborative robot arm with random impacts along the links while it is in motion. Results show that the proposed collision estimator achieves a compelling performance with collision estimation time of 8.86ms on average, an overall accuracy of 99.47%, and zero occurrence of false alarm.

Multimodal Assessment of Cognitive Workload Using Neural, Subjective and Behavioural Measures in Smart Factory Settings.

Collaborative robots (cobots) have largely replaced conventional industrial robots in today's workplaces, particularly in manufacturing setups, due to their improved performance and intelligent design. In the framework of Industry 5.0, humans are working alongside cobots to accomplish the required level of automation. However, human-robot interaction has brought up concerns regarding human factors (HF) and ergonomics. A human worker may experience cognitive stress as a result of cobots' irresponsive nature in unpredictably occurring situations, which adversely affects productivity. Therefore, there is a necessity to measure stress to enhance a human worker's performance in a human-robot collaborative environment. In this study, factory workers' mental workload was assessed using physiological, behavioural, and subjective measures. Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals were collected to acquire brain signals and track hemodynamic activity, respectively. The effect of task complexity, cobot movement speed, and cobot payload capacity on the mental stress of a human worker were observed for a task designed in the context of a smart factory. Task complexity and cobot speed proved to be more impactful. As physiological measures are unbiased and more authentic means to estimate stress, eventually they may replace the other conventional measures if they prove to correlate with the results of traditional ones. Here, regression and artificial neural networks (ANN) were utilised to determine the correlation between physiological data and subjective and behavioural measures. Regression performed better for most of the targets and the best correlation (rsq-adj = 0.654146) was achieved for predicting missed beeps, a behavioural measure, using a combination of multiple EEG and fNIRS predictors. The k-nearest neighbours (KNN) algorithm was used to evaluate the accuracy of correlation between traditional measures and physiological variables, with the highest accuracy of 77.8% achieved for missed beeps as the target. Results show that physiological measures can be more insightful and have the tendency to replace other biased parameters.

Exploring Embodied Resources in Gaze in Human-Robot Collaborative Environments

Among various types of embodied resources in humans, gaze, started with mutual gaze, plays a major role in embodied cognition. In addition to establishing relationships during interactions, gaze further portrays information about the level of engagement in a dyadic interaction. Hence the gaze and gaze-related behaviors such as averted gaze can be used as cues to make decisions regarding an interaction. This holds true for a human and a robot during human-robot interaction (HRI) as well. Hence proactive robots could evaluate human gaze as a parameter to achieve situation-awareness. In this work, we present the outcomes of several experiments aimed at evaluating such gaze behavior of human-human nonverbal interactions and other behaviors initiated as a result during dyadic interactions. The possibility of evaluating situations by such behavioral responses of individuals as cues, is also examined. We further compared the relationships between gaze behavior of humans during HRI and human-human interaction (HHI). We considered the properties of existence and aversion of gaze, as gaze behaviors during this study. Results of these experiments indicate interesting tendencies in verbal and nonverbal human behavior in initiating of an interaction in both HHI and HRI. The behavioral patterns related to gaze, observed during the study were analyzed using statistical methods and critical observations are highlighted. The potential of analyzing gaze behavior in displaying messages to the outside world during HRI is discussed.

Integration of acoustic compliance and noise mitigation in path planning for drones in human–robot collaborative environments

This work presents a framework aimed at mitigating adverse effects of high-amplitude drone noise ranging from hearing loss to reduced productivity in human–robot collaborative environments by infusing acoustic awareness in a path planning algorithm without imposing any additional design layers or hardware to an operational drone. Following a detailed outline of the proposed approach, it is shown that a significant reduction of noise levels perceived by human workers at noise-sensitive locations is realized via a path planner which generates optimal paths ranging from quietest to shortest paths. The approach is then augmented with a path-correction mechanism which accounts for noise exposure duration to ensure the aforementioned optimal paths are compliant with a given industrial/environmental standard. The correction mechanism enforces an adjustment of subsets of the planned paths inside quiet zones designated around noise-sensitive locations. The presented concepts were verified using numerical simulations conducted for a 2-dimensional rasterized obstacle field followed by a statistical design of experiments. The proposed framework is highly versatile and integrable with widely used industrial path planners, rendering it a highly valuable tool for noisy collaborative workplaces.

A graph-based reinforcement learning-enabled approach for adaptive human-robot collaborative assembly operations

In today’s prevailing manufacturing paradigm of mass personalization, neither human operators nor robots alone can perform all assembly tasks efficiently. To overcome it, human-robot collaborative assembly shows its great potentials to ensure the flexibility of human operations with high reliability of robot assistance. However, it is often challenging to achieve harmonious coexistence between humans and robots to complete the tasks safely and efficiently. In this regard, this research provides a detailed description of the human-robot coexisting environment and further introduces key issues in collaborative assembly. A part-behavior assembly and/or graph based on process requirements is proposed to represent the assembly task of complex products. Moreover, the human behavior prediction network based on self-attention can achieve higher accuracy. Combined with the robustness of Soft Actor-Critic (SAC), the collaborative system improves the self-decision ability of the robot in the dynamic scene. Finally, the effectiveness of the method is verified through experimental analysis. The results indicate that the accuracy of the proposed behavior recognition based on self-attention method is 91%. At the same time, it is proved that the reinforcement learning method is theoretically feasible to provide adaptive decision-making for robots in human-machine collaboration. In addition, the convergence speed of the reward function proves the feasibility of SAC for adaptive decision-making in a human-robot collaborative environment.

Synthesis and Prototyping of a 6-dof Parallel Robot for the Automatic Performance of Assembly Tasks

This paper presents the development of a 6-dof parallel robot for the performance of assembly tasks in a human-robot collaborative environment. The architecture and design of the robot are selected such that the robot is mechanically backdrivable. Thereby, the robot can physically interact with an environment or with humans without requiring the use of a force/torque sensor, which is the main objective of this work. The architecture of the robot is first described and its kinematic model is established. The Jacobian matrices are derived and an algorithm is presented for the determination of its workspace. The force capabilities of the robot are then established based on a proposed formulation. A prototype of the robot is presented and control schemes are developed, including a controller based on a vision system. Finally, a video demonstrating the experimental validation of the robot accompanies this paper. The video qualitatively demonstrates the interaction capabilities of the robot.

Operator Future Trajectory Prediction in Human-Robot Collaborative Environments Using Artificial Intelligence

Operator Future Trajectory Prediction in Human-Robot Collaborative Environments Using Artificial Intelligence

Status Recognition Using Pre-Trained YOLOv5 for Sustainable Human-Robot Collaboration (HRC) System in Mold Assembly

Molds are still assembled manually because of frequent demand changes and the requirement for comprehensive knowledge related to their high flexibility and adaptability in operation. We propose the application of human-robot collaboration (HRC) systems to improve manual mold assembly. In the existing HRC systems, humans control the execution of robot tasks, and this causes delays in the operation. Therefore, we propose a status recognition system to enable the early execution of robot tasks without human control during the HRC mold assembly operation. First, we decompose the mold assembly operation into task and sub-tasks, and define the actions representing the status of sub-tasks. Second, we develop status recognition based on parts, tools, and actions using a pre-trained YOLOv5 model, a one-stage object detection model. We compared four YOLOv5 models with and without a freezing backbone. The YOLOv5l model without a freezing backbone gave the optimal performance with a mean average precision (mAP) value of 84.8% and an inference time of 0.271 s. Given the success of the status recognition, we simulated the mold assembly operations in the HRC environment and reduced the assembly time by 7.84%. This study improves the sustainability of the mold assembly from the point of view of human safety, with reductions in human workload and assembly time.

Identifying Interaction Patterns of Tangible Co-Adaptations in Human-Robot Team Behaviors.

As robots become more ubiquitous, they will increasingly need to behave as our team partners and smoothly adapt to the (adaptive) human team behaviors to establish successful patterns of collaboration over time. A substantial amount of adaptations present themselves through subtle and unconscious interactions, which are difficult to observe. Our research aims to bring about awareness of co-adaptation that enables team learning. This paper presents an experimental paradigm that uses a physical human-robot collaborative task environment to explore emergent human-robot co-adaptions and derive the interaction patterns (i.e., the targeted awareness of co-adaptation). The paradigm provides a tangible human-robot interaction (i.e., a leash) that facilitates the expression of unconscious adaptations, such as “leading” (e.g., pulling the leash) and “following” (e.g., letting go of the leash) in a search-and-navigation task. The task was executed by 18 participants, after which we systematically annotated videos of their behavior. We discovered that their interactions could be described by four types of adaptive interactions: stable situations, sudden adaptations, gradual adaptations and active negotiations. From these types of interactions we have created a language of interaction patterns that can be used to describe tacit co-adaptation in human-robot collaborative contexts. This language can be used to enable communication between collaborating humans and robots in future studies, to let them share what they learned and support them in becoming aware of their implicit adaptations.

Metrological Evaluation of Human-Robot Collaborative Environments Based on Optical Motion Capture Systems.

In the context of human–robot collaborative shared environments, there has been an increase in the use of optical motion capture (OMC) systems for human motion tracking. The accuracy and precision of OMC technology need to be assessed in order to ensure safe human–robot interactions, but the accuracy specifications provided by manufacturers are easily influenced by various factors affecting the measurements. This article describes a new methodology for the metrological evaluation of a human–robot collaborative environment based on optical motion capture (OMC) systems. Inspired by the ASTM E3064 test guide, and taking advantage of an existing industrial robot in the production cell, the system is evaluated for mean error, error spread, and repeatability. A detailed statistical study of the error distribution across the capture area is carried out, supported by a Mann–Whitney U-test for median comparisons. Based on the results, optimal capture areas for the use of the capture system are suggested. The results of the proposed method show that the metrological characteristics obtained are compatible and comparable in quality to other methods that do not require the intervention of an industrial robot.

Function Block-Based Multimodal Control for Symbiotic Human–Robot Collaborative Assembly

AbstractIn human–robot collaborative assembly, robots are often required to dynamically change their preplanned tasks to collaborate with human operators in close proximity. One essential requirement of such an environment is enhanced flexibility and adaptability, as well as reduced effort on the conventional (re)programming of robots, especially for complex assembly tasks. However, the robots used today are controlled by rigid native codes that cannot support efficient human–robot collaboration. To solve such challenges, this article presents a novel function block-enabled multimodal control approach for symbiotic human–robot collaborative assembly. Within the context, event-driven function blocks as reusable functional modules embedded with smart algorithms are used for the encapsulation of assembly feature-based tasks/processes and control commands that are transferred to the controller of robots for execution. Then, multimodal control commands in the form of sensorless haptics, gestures, and voices serve as the inputs of the function blocks to trigger task execution and human-centered robot control within a safe human–robot collaborative environment. Finally, the performed processes of the method are experimentally validated by a case study in an assembly work cell on assisting the operator during the collaborative assembly. This unique combination facilitates programming-free robot control and the implementation of the multimodal symbiotic human–robot collaborative assembly with the enhanced adaptability and flexibility.

Operator support in human–robot collaborative environments using AI enhanced wearable devices

Nowadays, in order to cover the needs of market for product mass customization, industries have started to move to hybrid production cells, involving both robots and human operators. Research has been done during previous years to promote and improve the collaboration between humans and robots, trying to address topics such as safety, awareness and cognitive support in form of Augmented Reality based instructions. Results of previous research show bottlenecks related to the way of interaction of the operators with such supportive systems though. Direct interaction approach with the use of push buttons or indirect-gesture based interaction, which are most often adopted by the researchers, require operators to constantly occupy their hands performing the relevant button presses or gestures. Moreover, previous approaches are hardware dependent and need a lot of customization to work with different hardware. This work tries to address these bottlenecks proposing the usage of wearable devices enhanced with AI in order to support the interaction of human operators with robots in human-robot collaborative environments in a seamless and non-intrusive way, wrapped around a framework called “Operator Support Module” (OSM). Among others, OSM supports a variety of hardware to easily fit in various industrial scenarios. Two case studies will be presented to demonstrate the approach.

Augmented Reality (AR) based framework for supporting human workers in flexible manufacturing

This paper presents an Augmented Reality (AR) application that aims to facilitate the operator’s work in an industrial, human-robot collaboration environment with mobile robots. In such a flexible environment, with robots and humans working and moving in the same area, the ease of communication between the two sides is critical and prerequisite. The developed application provides the user with handy tools to interact with the mobile platform, give direct instructions to it and receive information about the robot’s and the broader system’s state, through an AR headset. The communication between the headset and the robot is achieved through a ROS based system, that interconnects the resources. The discussed tool has been deployed and tested in a use case inspired from the automotive industry, assisting the operators during the collaborative assembly tasks.

In Control or under Control? Human-Robot Collaboration in Warehouse Order Picking

Warehouse operations are rapidly becoming more robotized to increase performance and cut costs. One important new way of working is in collaboration between humans and robots. However, even though humans and their behavior remain essential for operational performance, human factors in human-robot collaboration are still underrepresented in Operations Management literature. To contribute to the scarce knowledge in this domain, we conducted a unique real-effort experiment in a warehouse especially erected for this study. In this experiment we compare the objective outcomes of collaborative productivity, collaborative accuracy, and human pick speed between a scenario with the human leading the robot and a scenario with the human following the robot. Furthermore, we investigate the behavioral mechanism that governs the reaction of humans to the novel collaboration with robots. We find that human leading allows for superior collaborative order picking productivity compared to human following. In contrast, human following allows for greater collaborative order picking accuracy compared to human leading. We additionally establish prevention regulatory focus as the behavioral mechanism through which workers may accelerate their pick speed to bridge the productivity gap between the setups. Finally, we confirm that competence in manual order picking is directly transferable to the human-robot collaborative environment.

A new approach to the anthropocentric design of human–robot collaborative environments

<p class="Abstract">This paper deals with collaborative robotics by highlighting the main issues linked to the interaction between humans and robots. A critical study of the standards in force on human–robot interaction and the current principles on workplace design for human–robot collaboration (HRC) are presented. The paper focuses on an anthropocentric paradigm in which the human becomes the core of the workplace in combination with the robot, and it presents a basis for designing workplaces through two key concepts: (i) the introduction of human and robot spaces as elementary spaces and (ii) the dynamic variations of the elementary spaces in shape, size and position. According to this paradigm, the limitations of a safety-based approach, introduced by the standards, are overcome by positioning the human and the robot inside the workplace and managing their interaction through the elementary spaces. The introduced concepts, in combination with the safety prescriptions, have been organised by means of a multi-level graph for driving the HRC design phase. The collaborative workplace is separated into sublevels. The main elements of a collaborative workplace are identified and their relationships presented by means of digraphs. </p>

Robust human position estimation in cooperative robotic cells

Robots are increasingly present in our lives, sharing the workspace and tasks with human co-workers. However, existing interfaces for human-robot interaction / cooperation (HRI/C) have limited levels of intuitiveness to use and safety is a major concern when humans and robots share the same workspace. Many times, this is due to the lack of a reliable estimation of the human pose in space which is the primary input to calculate the human-robot minimum distance (required for safety and collision avoidance) and HRI/C featuring machine learning algorithms classifying human behaviours / gestures. Each sensor type has its own characteristics resulting in problems such as occlusions (vision) and drift (inertial) when used in an isolated fashion. In this paper, it is proposed a combined system that merges the human tracking provided by a 3D vision sensor with the pose estimation provided by a set of inertial measurement units (IMUs) placed in human body limbs. The IMUs compensate the gaps in occluded areas to have tracking continuity. To mitigate the lingering effects of the IMU offset we propose a continuous online calculation of the offset value. Experimental tests were designed to simulate human motion in a human-robot collaborative environment where the robot moves away to avoid unexpected collisions with de human. Results indicate that our approach is able to capture the human’s position, for example the forearm, with a precision in the millimetre range and robustness to occlusions.