Effective Human-robot Collaboration Research Articles

A Deep Learning-Based Robot Analysis Model for Semantic Context Capturing by Using Predictive Models in Public Management

In the realm of robotics, the ability to comprehend intricate semantic contexts within diverse environments is paramount for autonomous decision-making and effective human-robot collaboration. This article delves into the realm of enhancing robotic semantic understanding through the fusion of deep learning techniques. This work presents a pioneering approach: integrating several neural network models to analyze robot images, thereby capturing nuanced environmental semantic contexts. The authors augment this analysis with predictive models, enabling the robot to adapt the changing contexts intelligently. Through rigorous experimentation, our model demonstrated a substantial 25% increase in accuracy when compared to conventional methods, showcasing its robustness in real-world applications. This research marks a significant stride toward imbuing robots with sophisticated visual comprehension, paving the way for more seamless human-robot interactions and a myriad of practical applications in the evolving landscape of robotics.

An Incremental Learning Approach to Detect Muscular Fatigue in Human– Robot Collaboration

Human–robot collaboration aims to join the distinctive strengths of humans and robots to compensate for the weaknesses associated with each party and, thus, to enable synergetic effects. Robots are characteristically considered fatigue-proof. Hence, they are utilized to assist human operators during heavy pushing and pulling activities. To detect physical fatigue or high payloads held by a human operator, wearable sensors, such as electromyographys (EMGs), are deployed. The EMG data are typically processed via machine learning, which includes training models offline before an application in an online system. However, these approaches often demonstrate varying performances between offline and online applications due to subject-specific characteristics within the data. An opportunity to tackle this challenge can be found in incremental learning, as these models purely learn online and constantly fine-tune the model's structure. In this article, a Mondrian Forest is applied to predict payloads and physical fatigue of human operators during an assistance scenario with a collaborative robot. An experiment was conducted with a total of 12 participants, where the payload was increased until participants initiated an assistance request from a Universal Robots model 10 cobot. This allowed for testing whether the Mondrian Forest can accurately predict the payload and fatigue levels from the acquired EMG signals. Overall, the approach demonstrates a promising potential toward higher awareness when an operator might require assistance from a robot and ultimately toward a more effective human–robot collaboration.

Leveraging explainability for understanding object descriptions in ambiguous 3D environments.

For effective human-robot collaboration, it is crucial for robots to understand requests from users perceiving the three-dimensional space and ask reasonable follow-up questions when there are ambiguities. While comprehending the users' object descriptions in the requests, existing studies have focused on this challenge for limited object categories that can be detected or localized with existing object detection and localization modules. Further, they have mostly focused on comprehending the object descriptions using flat RGB images without considering the depth dimension. On the other hand, in the wild, it is impossible to limit the object categories that can be encountered during the interaction, and 3-dimensional space perception that includes depth information is fundamental in successful task completion. To understand described objects and resolve ambiguities in the wild, for the first time, we suggest a method leveraging explainability. Our method focuses on the active areas of an RGB scene to find the described objects without putting the previous constraints on object categories and natural language instructions. We further improve our method to identify the described objects considering depth dimension. We evaluate our method in varied real-world images and observe that the regions suggested by our method can help resolve ambiguities. When we compare our method with a state-of-the-art baseline, we show that our method performs better in scenes with ambiguous objects which cannot be recognized by existing object detectors. We also show that using depth features significantly improves performance in scenes where depth data is critical to disambiguate the objects and across our evaluation dataset that contains objects that can be specified with and without the depth dimension.

Adaptive speed and separation monitoring based on switching of safety zones for effective human robot collaboration

• Design of dynamically switched safety zones which are used to monitor human presence • Implementation of safety architecture using safety sensors, safety PLC and robot's controller • Use the status of the safety zones to adapt robot behaviour based on the signals received • Ensure safe co-existence and collaboration of human operators with industrial robots This paper presents the design and deployment of a system that enables the safe cooperation and interaction of human operators with industrial robots. Dynamically switched safety zones are used to monitor human presence and respectively adapt robot behavior, namely speed and movement status. An overhead safety camera system is used to identify moving objects within each zone and produce warning and stop signals which are used to regulate the behavior of the robot, slowing it down and stopping it respectively. Based on the real time position reporting of the robot, continuous switching between zone configurations is achieved. This leads to a more efficient speed and separation monitoring process, as defined by ISO/TS 15066, EN-ISO 10218-1 and EN ISO 10218-2, compared to a fixed fence solution. Unlike the case of virtual fences which are currently implemented using light barriers, this approach allows humans to use more workspace from the robot while ensuring separation. The control architecture design and the implementation of the system on a pilot cell inspired from the automotive industry are presented in this paper, as well..

Heterogeneous human–robot task allocation based on artificial trust

Effective human–robot collaboration requires the appropriate allocation of indivisible tasks between humans and robots. A task allocation method that appropriately makes use of the unique capabilities of each agent (either a human or a robot) can improve team performance. This paper presents a novel task allocation method for heterogeneous human–robot teams based on artificial trust from a robot that can learn agent capabilities over time and allocate both existing and novel tasks. Tasks are allocated to the agent that maximizes the expected total reward. The expected total reward incorporates trust in the agent to successfully execute the task as well as the task reward and cost associated with using that agent for that task. Trust in an agent is computed from an artificial trust model, where trust is assessed along a capability dimension by comparing the belief in agent capabilities with the task requirements. An agent’s capabilities are represented by a belief distribution and learned using stochastic task outcomes. Our task allocation method was simulated for a human–robot dyad. The team total reward of our artificial trust-based task allocation method outperforms other methods both when the human’s capabilities are initially unknown and when the human’s capabilities belief distribution has converged to the human’s actual capabilities. Our task allocation method enables human–robot teams to maximize their joint performance.

Alignment of the attitude of teleoperators with that of a semi-autonomous android

Studies on social robots that can communicate with humans are increasingly important. In particular, semi-aautonomous robots have shown potential for practical applications in which robot autonomy and human teleoperation are jointly used to accomplish difficult tasks. However, it is unknown how the attitude represented in the autonomous behavior of the robots affects teleoperators. Previous studies reported that when humans play a particular role, their attitudes align with that role. The teleoperators of semi-autonomous robots also play the role given to the robots and may assimilate their autonomous expression. We hypothesized that the attitude of teleoperators may align with that of robots through teleoperation. To verify this, we conducted an experiment with conditions under which a participant operated a part of the body of an android robot that autonomously expressed a preferential attitude toward a painting and a condition under which they did not. Experimental results demonstrated that the preferential attitude of participants who teleoperated the android aligned statistically significantly more with that of the robot in comparison to those who did not teleoperate it, thereby supporting our hypothesis. This finding is novel regarding attitude change in teleoperators of semi-autonomous robots and can support the implementation of effective human-robot collaboration systems.

An integrated mixed reality system for safety-aware human-robot collaboration using deep learning and digital twin generation

For human-robot collaboration (HRC), one of the most practical methods to ensure human safety with a vision-based system is establishing a minimum safe distance. This study proposes a novel integrated mixed reality (MR) system for safety-aware HRC using deep learning and digital twin generation. The proposed approach can accurately measure the minimum safe distance in real-time and provide MR-based task assistance to the human operator. The approach integrates MR with safety-related monitoring by tracking the shared workplace and providing user-centric visualization through smart MR glasses for safe and effective HRC. Two RGB-D sensors are used to reconstruct and track the working environment. One sensor scans one area of the physical environment through 3D point cloud data. The other also scans another area of the environment and tracks the user's 3D skeletal information. In addition, the two partially scanned environments are registered together by applying a fast global registration method to two sets of the 3D point cloud. Furthermore, deep learning-based instance segmentation is applied to the target object's 3D point cloud to increase the registration between the real robot and its virtual robot, the digital twin of the real robot. While only 3D point cloud data are widely used in previous studies, this study proposes a simple yet effective 3D offset-based safety distance calculation method based on the robot's digital twin and the human skeleton. The 3D offset-based method allows for real-time applicability without sacrificing the accuracy of safety distance calculation for HRI. In addition, two comparative evaluations were conducted to confirm the originality and advantage of the proposed MR-based HRC.

Collaborative Workplace Design: A Knowledge-Based Approach to Promote Human–Robot Collaboration and Multi-Objective Layout Optimization

The innovation-driven Industry 5.0 leads us to consider humanity in a prominent position as the center of the manufacturing field even more than Industry 4.0. This pushes us towards the hybridization of manufacturing plants promoting a full collaboration between humans and robots. However, there are currently very few workplaces where effective Human–Robot Collaboration takes place. Layout designing plays a key role in assuring safe and efficient Human–Robot Collaboration. The layout design, especially in the context of collaborative robotics, is a complex problem to face, since it is related to safety, ergonomics, and productivity aspects. In the current work, a Knowledge-Based Approach (KBA) is adopted to face the complexity of the layout design problem. The framework resulting from the KBA allows for developing a modeling paradigm that enables us to define a streamlined approach for the layout design. The proposed approach allows for placing resource within the workplace according to a defined optimization criterion, and also ensures compliance with various standards. This approach is applied to an industrial case study in order to prove its feasibility. A what-if analysis is performed by applying the proposed approach. Changing three control factors (i.e., minimum distance, robot speed, logistic space configuration) on three levels, in a Design of Experiments, 27 layout configurations of the same workplace are generated. Consequently, the inputs that most affect the layout design are identified by means of an Analysis of Variance (ANOVA). The results show that only one layout is eligible to be the best configuration, and only two out of three control factors are very significant for the designing of the HRC workplace layout. Hence, the proposed approach enables the designing of standard compliant and optimized HRC workplace layouts. Therefore, several alternatives of the layout for the same workplace can be easily generated and investigated in a systematic manner.

Human–Robot Collaboration in Smart Manufacturing: Robot Reactive Behavior Intelligence

Abstract To enable safe and effective human–robot collaboration (HRC) in smart manufacturing, seamless integration of sensing, cognition, and prediction into the robot controller is critical for real-time awareness, response, and communication inside a heterogeneous environment (robots, humans, and equipment). The specific research objective is to provide the robot Proactive Adaptive Collaboration Intelligence (PACI) and switching logic within its control architecture in order to give the robot the ability to optimally and dynamically adapt its motions, given a priori knowledge and predefined execution plans for its assigned tasks. The challenge lies in augmenting the robot’s decision-making process to have greater situation awareness and to yield smart robot behaviors/reactions when subject to different levels of human–robot interaction, while maintaining safety and production efficiency. Robot reactive behaviors were achieved via cost function-based switching logic activating the best suited high-level controller. The PACI’s underlying segmentation and switching logic framework is demonstrated to yield a high degree of modularity and flexibility. The performance of the developed control structure subjected to different levels of human–robot interactions was validated in a simulated environment. Open-loop commands were sent to the physical e.DO robot to demonstrate how the proposed framework would behave in a real application.

FMG- and RNN-Based Estimation of Motor Intention of Upper-Limb Motion in Human-Robot Collaboration.

Research on human-robot interactions has been driven by the increasing employment of robotic manipulators in manufacturing and production. Toward developing more effective human-robot collaboration during shared tasks, this paper proposes an interaction scheme by employing machine learning algorithms to interpret biosignals acquired from the human user and accordingly planning the robot reaction. More specifically, a force myography (FMG) band was wrapped around the user's forearm and was used to collect information about muscle contractions during a set of collaborative tasks between the user and an industrial robot. A recurrent neural network model was trained to estimate the user's hand movement pattern based on the collected FMG data to determine whether the performed motion was random or intended as part of the predefined collaborative tasks. Experimental evaluation during two practical collaboration scenarios demonstrated that the trained model could successfully estimate the category of hand motion, i.e., intended or random, such that the robot either assisted with performing the task or changed its course of action to avoid collision. Furthermore, proximity sensors were mounted on the robotic arm to investigate if monitoring the distance between the user and the robot had an effect on the outcome of the collaborative effort. While further investigation is required to rigorously establish the safety of the human worker, this study demonstrates the potential of FMG-based wearable technologies to enhance human-robot collaboration in industrial settings.

Analysis of Visuo Motor Control between Dominant Hand and Non-Dominant Hand for Effective Human-Robot Collaboration.

The human-in-the-loop technology requires studies on sensory-motor characteristics of each hand for an effective human–robot collaboration. This study aims to investigate the differences in visuomotor control between the dominant (DH) and non-dominant hands in tracking a target in the three-dimensional space. We compared the circular tracking performances of the hands on the frontal plane of the virtual reality space in terms of radial position error (ΔR), phase error (Δθ), acceleration error (Δa), and dimensionless squared jerk (DSJ) at four different speeds for 30 subjects. ΔR and Δθ significantly differed at relatively high speeds (ΔR: 0.5 Hz; Δθ: 0.5, 0.75 Hz), with maximum values of ≤1% compared to the target trajectory radius. DSJ significantly differed only at low speeds (0.125, 0.25 Hz), whereas Δa significantly differed at all speeds. In summary, the feedback-control mechanism of the DH has a wider range of speed control capability and is efficient according to an energy saving model. The central nervous system (CNS) uses different models for the two hands, which react dissimilarly. Despite the precise control of the DH, both hands exhibited dependences on limb kinematic properties at high speeds (0.75 Hz). Thus, the CNS uses a different strategy according to the model for optimal results.

Trust-Aware Decision Making for Human-Robot Collaboration

Trust in autonomy is essential for effective human-robot collaboration and user adoption of autonomous systems such as robot assistants. This article introduces a computational model that integrates trust into robot decision making. Specifically, we learn from data a partially observable Markov decision process (POMDP) with human trust as a latent variable. The trust-POMDP model provides a principled approach for the robot to (i) infer the trust of a human teammate through interaction, (ii) reason about the effect of its own actions on human trust, and (iii) choose actions that maximize team performance over the long term. We validated the model through human subject experiments on a table clearing task in simulation (201 participants) and with a real robot (20 participants). In our studies, the robot builds human trust by manipulating low-risk objects first. Interestingly, the robot sometimes fails intentionally to modulate human trust and achieve the best team performance. These results show that the trust-POMDP calibrates trust to improve human-robot team performance over the long term. Further, they highlight that maximizing trust alone does not always lead to the best performance.

Probabilistic Human Intent Recognition for Shared Autonomy in Assistive Robotics.

Effective human-robot collaboration in shared autonomy requires reasoning about the intentions of the human partner. To provide meaningful assistance, the autonomy has to first correctly predict, or infer, the intended goal of the human collaborator. In this work, we present a mathematical formulation for intent inference during assistive teleoperation under shared autonomy. Our recursive Bayesian filtering approach models and fuses multiple non-verbal observations to probabilistically reason about the intended goal of the user without explicit communication. In addition to contextual observations, we model and incorporate the human agent's behavior as goal-directed actions with adjustable rationality to inform intent recognition. Furthermore, we introduce a user-customized optimization of this adjustable rationality to achieve user personalization. We validate our approach with a human subjects study that evaluates intent inference performance under a variety of goal scenarios and tasks. Importantly, the studies are performed using multiple control interfaces that are typically available to users in the assistive domain, which differ in the continuity and dimensionality of the issued control signals. The implications of the control interface limitations on intent inference are analyzed. The study results show that our approach in many scenarios outperforms existing solutions for intent inference in assistive teleoperation, and otherwise performs comparably. Our findings demonstrate the benefit of probabilistic modeling and the incorporation of human agent behavior as goal-directed actions where the adjustable rationality model is user customized. Results further show that the underlying intent inference approach directly affects shared autonomy performance, as do control interface limitations.

Toward Ergonomic Risk Prediction via Segmentation of Indoor Object Manipulation Actions Using Spatiotemporal Convolutional Networks

Automated real-time prediction of the ergonomic risks of manipulating objects is a key unsolved challenge in developing effective human-robot collaboration systems for logistics and manufacturing applications. We present a foundational paradigm to address this challenge by formulating the problem as one of action segmentation from RGB-D camera videos. Spatial features are first learned using a deep convolutional model from the video frames, which are then fed sequentially to temporal convolutional networks to semantically segment the frames into a hierarchy of actions, which are either ergonomically safe, require monitoring, or need immediate attention. For performance evaluation, in addition to an open-source kitchen dataset, we collected a new dataset comprising 20 individuals picking up and placing objects of varying weights to and from cabinet and table locations at various heights. Results show very high (87%-94%) F1 overlap scores among the ground truth and predicted frame labels for videos lasting over 2 min and consisting of a large number of actions.

Symbiotic human-robot collaborative assembly

In human-robot collaborative assembly, robots are often required to dynamically change their pre-planned tasks to collaborate with human operators in a shared workspace. However, the robots used today are controlled by pre-generated rigid codes that cannot support effective human-robot collaboration. In response to this need, multi-modal yet symbiotic communication and control methods have been a focus in recent years. These methods include voice processing, gesture recognition, haptic interaction, and brainwave perception. Deep learning is used for classification, recognition and context awareness identification. Within this context, this keynote provides an overview of symbiotic human-robot collaborative assembly and highlights future research directions.

Recursive Bayesian Human Intent Recognition in Shared-Control Robotics.

Effective human-robot collaboration in shared control requires reasoning about the intentions of the human user. In this work, we present a mathematical formulation for human intent recognition during assistive teleoperation under shared autonomy. Our recursive Bayesian filtering approach models and fuses multiple non-verbal observations to probabilistically reason about the intended goal of the user. In addition to contextual observations, we model and incorporate the human agent's behavior as goal-directed actions with adjustable rationality to inform the underlying intent. We examine human inference on robot motion and furthermore validate our approach with a human subjects study that evaluates autonomy intent inference performance under a variety of goal scenarios and tasks, by novice subjects. Results show that our approach outperforms existing solutions and demonstrates that the probabilistic fusion of multiple observations improves intent inference and performance for shared-control operation.