Physical Human-robot Collaboration Research Articles

Prediction of cognitive conflict during unexpected robot behavior under different mental workload conditions in a physical human–robot collaboration

Objective. Brain–computer interface (BCI) technology is poised to play a prominent role in modern work environments, especially a collaborative environment where humans and machines work in close proximity, often with physical contact. In a physical human robot collaboration (pHRC), the robot performs complex motion sequences. Any unexpected robot behavior or faulty interaction might raise safety concerns. Error-related potentials, naturally generated by the brain when a human partner perceives an error, have been extensively employed in BCI as implicit human feedback to adapt robot behavior to facilitate a safe and intuitive interaction. However, the integration of BCI technology with error-related potential for robot control demands failure-free integration of highly uncertain electroencephalography (EEG) signals, particularly influenced by the physical and cognitive state of the user. As a higher workload on the user compromises their access to cognitive resources needed for error awareness, it is crucial to study how mental workload variations impact the error awareness as it might raise safety concerns in pHRC. In this study, we aim to study how cognitive workload affects the error awareness of a human user engaged in a pHRC. Approach. We designed a blasting task with an abrasive industrial robot and manipulated the mental workload with a secondary arithmetic task of varying difficulty. EEG data, perceived workload, task and physical performance were recorded from 24 participants moving the robot arm. The error condition was achieved by the unexpected stopping of the robot in 33% of trials. Main results. We observed a diminished amplitude for the prediction error negativity (PEN) and error positivity (Pe), indicating reduced error awareness with increasing mental workload. We further observed an increased frontal theta power and increasing trend in the central alpha and central beta power after the unexpected robot stopping compared to when the robot stopped correctly at the target. We also demonstrate that a popular convolution neural network model, EEGNet, could predict the amplitudes of PEN and Pe from the EEG data prior to the error. Significance. This prediction model could be instrumental in developing an online prediction model that could forewarn the system and operators of the diminished error awareness of the user, alluding to a potential safety breach in error-related potential-based BCI system for pHRC. Therefore, our work paves the way for embracing BCI technology in pHRC to optimally adapt the robot behavior for personalized user experience using real-time brain activity, enriching the quality of the interaction.

Imposing Motion Variability for Ergonomic Human-Robot Collaboration

OCCUPATIONAL APPLICATIONS “Overassistive” robots can adversely impact long-term human-robot collaboration in the workplace, leading to risks of worker complacency, reduced workforce skill sets, and diminished situational awareness. Ergonomics practitioners should thus be cautious about solely targeting widely adopted metrics for improving human-robot collaboration, such as user trust and comfort. By contrast, introducing variability and adaptation into a collaborative robot’s behavior could prove vital in preventing the negative consequences of overreliance and overtrust in an autonomous partner. This work reported here explored how instilling variability into physical human-robot collaboration can have a measurably positive effect on ergonomics in a repetitive task. A review of principles related to this notion of “stimulating” robot behavior is also provided to further inform ergonomics practitioners of existing human-robot collaboration frameworks.

Vision-and Tactile-Based Continuous Multimodal Intention and Attention Recognition for Safer Physical Human–Robot Interaction

Employing skin-like tactile sensors on robots enhances both the safety and usability of collaborative robots by adding the capability to detect human contact. Unfortunately, simple binary tactile sensors alone cannot determine the context of the human contact—whether it is a deliberate interaction or an unintended collision that requires safety manoeuvres. Many published methods classify discrete interactions using more advanced tactile sensors or by analysing joint torques. Instead, we propose to augment the intention recognition capabilities of simple binary tactile sensors by adding a robot-mounted camera for human posture analysis. Different interaction characteristics, including touch location, human pose, and gaze direction, are used to train a supervised machine learning algorithm to classify whether a touch is intentional or not with an F1-score of 86%. We demonstrate that multimodal intention recognition is significantly more accurate than monomodal analyses with the collaborative robot Baxter. Furthermore, our method can also continuously monitor interactions that fluidly change between intentional or unintentional by gauging the user’s attention through gaze. If a user stops paying attention mid-task, the proposed intention and attention recognition algorithm can activate safety features to prevent unsafe interactions. We also employ a feature reduction technique that reduces the number of inputs to five to achieve a more generalized low-dimensional classifier. This simplification both reduces the amount of training data required and improves real-world classification accuracy. It also renders the method potentially agnostic to the robot and touch sensor architectures while achieving a high degree of task adaptability. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Whenever a user interacts physically with a robot, such as in collaborative manufacturing, the robot may respond to unintended touch inputs from the user. This may be through body collisions or that the user is suddenly distracted and is no longer paying attention to what they are doing. We propose an easy-to-implement method to augment safety of physical human-robot collaboration by determining whether the touch from a user is intentional or not through the use of robot-mounted basic touch sensors and computer vision. The algorithm examines the location of the user’s hands relative to the touched sensors in addition to observing where the user is looking. Machine learning is then used to classify in real-time, with an F1-score of 86%, whether a touch is intentional or not such that the robot can react accordingly. The method is particularly applicable in collaborative manufacturing contexts, but can also be applied anywhere where a user physically interacts with a robot. We demonstrate the utility of the method in enhancing safety during human-robot collaboration through a simulated collaborative manufacturing scenario with the robot Baxter, but the method can easily be adapted to other systems.

Interdisciplinary evaluation of a robot physically collaborating with workers.

Collaborative Robots-CoBots-are emerging as a promising technological aid for workers. To date, most CoBots merely share their workspace or collaborate without contact, with their human partners. We claim that robots would be much more beneficial if they physically collaborated with the worker, on high payload tasks. To move high payloads, while remaining safe, the robot should use two or more lightweight arms. In this work, we address the following question: to what extent can robots help workers in physical human-robot collaboration tasks? To find an answer, we have gathered an interdisciplinary group, spanning from an industrial end user to cognitive ergonomists, and including biomechanicians and roboticists. We drew inspiration from an industrial process realized repetitively by workers of the SME HANKAMP (Netherlands). Eleven participants replicated the process, without and with the help of a robot. During the task, we monitored the participants' biomechanical activity. After the task, the participants completed a survey with usability and acceptability measures; seven workers of the SME completed the same survey. The results of our research are the following. First, by applying-for the first time in collaborative robotics-Potvin's method, we show that the robot substantially reduces the participants' muscular effort. Second: we design and present an unprecedented method for measuring the robot reliability and reproducibility in collaborative scenarios. Third: by correlating the worker's effort with the power measured by the robot, we show that the two agents act in energetic synergy. Fourth: the participant's increasing level of experience with robots shifts his/her focus from the robot's overall functionality towards finer expectations. Last but not least: workers and participants are willing to work with the robot and think it is useful.

Eye-Tracking in Physical Human-Robot Interaction: Mental Workload and Performance Prediction.

In Physical Human-Robot Interaction (pHRI), the need to learn the robot's motor-control dynamics is associated with increased cognitive load. Eye-tracking metrics can help understand the dynamics of fluctuating mental workload over the course of learning. The aim of this study was to test eye-tracking measures' sensitivity and reliability to variations in task difficulty, as well as their performance-prediction capability, in physical human-robot collaboration tasks involving an industrial robot for object comanipulation. Participants (9M, 9F) learned to coperform a virtual pick-and-place task with a bimanual robot over multiple trials. Joint stiffness of the robot was manipulated to increase motor-coordination demands. The psychometric properties of eye-tracking measures and their ability to predict performance was investigated. Stationary Gaze Entropy and pupil diameter were the most reliable and sensitive measures of workload associated with changes in task difficulty and learning. Increased task difficulty was more likely to result in a robot-monitoring strategy. Eye-tracking measures were able to predict the occurrence of success or failure in each trial with 70% sensitivity and 71% accuracy. The sensitivity and reliability of eye-tracking measures was acceptable, although values were lower than those observed in cognitive domains. Measures of gaze behaviors indicative of visual monitoring strategies were most sensitive to task difficulty manipulations, and should be explored further for the pHRI domain where motor-control and internal-model formation will likely be strong contributors to workload. Future collaborative robots can adapt to human cognitive state and skill-level measured using eye-tracking measures of workload and visual attention.

Detection and Estimation of Cognitive Conflict During Physical Human–Robot Collaboration

Detection and Estimation of Cognitive Conflict During Physical Human–Robot Collaboration

Model Predictive Variable Impedance Control of Manipulators for Adaptive Precision-Compliance Tradeoff

Impedance control (IC) is widely used in contact-rich manipulator tasks since it provides manipulators with both operation precision and contact compliance, and trades them off by impedance parameters. However, fixed impedance parameters limit the applicability of IC in complex tasks during which the focus on the operation precision or the contact compliance is variable, such as physical human–robot collaboration and complex assembly tasks. This article presents model predictive variable impedance control approaches for adaptive precision-compliance tradeoff to satisfy variable task requirements. Specifically, we establish a novel impedance model, which transforms the variable impedance law design problem into a control law design problem, and allows us to consider novel impedance constraints that can determine manipulators' extreme precision and compliance properties. According to whether state constraints are further considered, one-step (OS) and multistep (MS) model predictive control approaches are proposed to solve these transformed control problems, and the corresponding optimization problem of the OS MPC can be established as a QP problem due to the special form of the novel impedance model. A tank-based approach is further proposed to correct the variable impedance laws for the system passivity concern. Various comparative experiments are conducted to validate the effectiveness of the presented approaches.

PREDICTOR: A Physical emulatoR enabling safEty anD ergonomICs evaluation and Training of physical human-rObot collaboRation.

Safety and ergonomics of Physical Human-Robot Collaboration (PHRC) are crucial to make human-robot collaborative systems trustworthy and make a significant impact in real-world applications. One big obstacle to the development of relevant research is the lack of a general platform for evaluating the safety and ergonomics of proposed PHRC systems. This paper aims to create a Physical emulatoR enabling safEty anD ergonomICs evaluation and Training of physical human-rObot collaboRation (PREDICTOR). PREDICTOR consists of a dual-arm robot system and a VR headset as its hardware and contains physical simulation, haptic rendering and visual rendering modules as its software. The dual-arm robot system is used as an integrated admittance-type haptic device, which senses the force/torque applied by a human operator as an input to drive the simulation of a PHRC system and constrains the handles' motion to match their virtual counterparts in the simulation. The motion of the PHRC system in the simulation is fed back to the operator through the VR headset. PREDICTOR combines haptics and VR to emulate PHRC tasks in a safe environment since the interactive forces are monitored to avoid any risky events. PREDICTOR also brings flexibility as different PHRC tasks can be easily set up by changing the PHRC system model and the robot controller in the simulation. The effectiveness and performance of PREDICTOR were evaluated by experiments.

Adaptive Admittance Control for Safety-Critical Physical Human Robot Collaboration

Physical human-robot collaboration requires strict safety guarantees, due to the fact that robots and humans work in a shared workspace. This paper presents a novel control framework to handle safety-critical position-based constraints for human-robot physical interaction. The proposed methodology is based on admittance control, exponential control barrier functions (ECBFs), and quadratic program (QP) to achieve compliance during the force interaction between human and robot, while simultaneously guaranteeing safety constraints. In particular, the formulation of admittance control is formulated as a second-order nonlinear control system, and the interaction forces between humans and robots are regarded as the control input. A virtual force feedback for admittance control is provided in real-time by using the ECBFs-QP framework as a compensator of the external human forces. A safe trajectory is therefore derived from the proposed adaptive admittance control scheme for a low-level controller to track. The main innovation of the proposed approach is the ability to enable the robot to naturally comply with human forces without violating any safety constraints, even when external human forces incidentally force the robot to do so. The effectiveness of our approach is demonstrated in simulation studies on a two-link planar robot manipulator.

Bounded Energy Collisions in Human–Robot Cooperative Transportation

In this article, we introduce the notion of bounded energy collision for physical human–robot collaboration to allow safe interaction with the environment, by keeping the collision impact within predetermined bounds. The proposed method is based on energy dissipation, implemented by a variable damping strategy along the direction of imminent collisions with the environment, and fully complies with any physical human–robot interaction strategy that combines an indirect force control (e.g., impedance control) with an estimation of the human motion intention. Finally, extensive experimental studies clarify and verify the proposed scheme.

Cooperative Transportation With Mobile Manipulator: A Capability Map-Based Framework for Physical Human–Robot Collaboration

In the cooperative transportation task with a mobile manipulator (MM), the mobile robot and the manipulator must move simultaneously to adapt to the human motion. In addition, the motion of the MM is underconstrained due to redundancy, which makes MM real-time motion planning challenging. In this article, a capability map-based framework was proposed to enable real-time motion planning of the MM. In the motion planner, the dexterity of the MM, the formation of the human–robot system, and obstacle avoidance of the mobile robot are considered to get safe and human-like robot motion. Moreover, to make optimization faster and avoid local minimum, capability map, which represents the manipulability distribution of the MM in its workspace, is queried to determine the seed of the optimization algorithm. The proposed motion planner is self-contained and can be extended to the transportation task with multiple MMs. The effectiveness of this proposed framework is validated both in simulations and real-world experiments.

Effect of active and passive protective soft skins on collision forces in human–robot collaboration

Soft electronic skins are one of the means to turn an industrial manipulator into a collaborative robot. For manipulators that are already fit for physical human-robot collaboration, soft skins can make them safer. In this work, we study the after impact behavior of two collaborative manipulators (UR10e and KUKA LBR iiwa) and one classical industrial manipulator (KUKA Cybertech), in presence or absence of an industrial protective skin (AIRSKIN). In addition, we isolate the effects of the passive padding and the active contribution of the sensor to robot reaction. We present a total of 2250 collision measurements and study the impact force, contact duration, clamping force, and impulse. The dataset is publicly available. We summarize our results as follows. For transient collisions, the passive skin properties lowered the impact forces by about 40 %. During quasi-static contact, the effect of skin covers -- active or passive -- cannot be isolated from the collision detection and reaction by the collaborative robots. Important effects of the stop categories triggered by the active protective skin were found. We systematically compare the different settings and the empirically established safe velocities with prescriptions by the ISO/TS 15066. In some cases, up to the quadruple of the ISO/TS 15066 prescribed velocity can comply with the impact force limits and thus be considered safe. We propose an extension of the formulas relating impact force and permissible velocity that take into account the stiffness and compressible thickness of the protective cover, leading to better predictions of the collision forces. At the same time, this work emphasizes the need for in situ measurements as all the factors we studied -- presence of active/passive skin, safety stop settings, robot collision reaction, impact direction, and, of course, velocity -- have effects on the force evolution after impact.

Measuring Interaction Bandwidth During Physical Human-Robot Collaboration

An assistive robot can augment human performance by providing physical assistance or motion guidance. In human-robot collaboration, it is important to know the bandwidth of an individual's ability to generate motion in response to external stimuli, such as visual or haptic cuing. This becomes particularly relevant in timing-sensitive tasks, such as walking or catching a falling object. In this work, we propose a frequency-based assessment of motion that enables us to measure the bandwidth of physical human-robot interaction (pHRI)—quantifying how fast individuals can respond to stimuli on a continuous basis. We introduce a robot-assisted virtual dynamic task with a tunable resonant frequency. A human subject study with seven participants shows that our task can elicit a dynamic response in a participant at frequencies of 0.5 Hz, 1 Hz, 1.5 Hz and 2.5 Hz at the arm. Using the virtual task, we test whether haptic cues improve motion timing. At all tested frequencies, we find that haptic stimuli help guide timing of dynamic movement and improve performance compared to visual-only cuing. By quantifying the interaction bandwidth for other pHRI systems—particularly when the human collaborators have neuromotor impairments—our method can help assistive robots adapt to an individual. Moreover, our results highlight the importance of incorporating haptic feedback into pHRI for dynamic tasks—haptics can provide guidance around motion timing, such as in assistive robots used for assessment and physical rehabilitation.

An online human-robot collaborative grinding state recognition approach based on contact dynamics and LSTM.

Collaborative state recognition is a critical issue for physical human–robot collaboration (PHRC). This paper proposes a contact dynamics-based state recognition method to identify the human–robot collaborative grinding state. The main idea of the proposed approach is to distinguish between the human–robot contact and the robot–environment contact. To achieve this, dynamic models of both these contacts are first established to identify the difference in dynamics between the human–robot contact and the robot–environment contact. Considering the reaction speed required for human–robot collaborative state recognition, feature selections based on Spearman's correlation and random forest recursive feature elimination are conducted to reduce data redundancy and computational burden. Long short-term memory (LSTM) is then used to construct a collaborative state classifier. Experimental results illustrate that the proposed method can achieve a recognition accuracy of 97% in a period of 5 ms and 99% in a period of 40 ms.

Q-Learning-based model predictive variable impedance control for physical human-robot collaboration

Physical human-robot collaboration is increasingly required in many contexts (such as industrial and rehabilitation applications). The robot needs to interact with the human to perform the target task while relieving the user from the workload. To do that, the robot should be able to recognize the human's intentions and guarantee safe and adaptive behavior along the intended motion directions. The robot-control strategies with such attributes are particularly demanded in the industrial field, where the operator guides the robot manually to manipulate heavy parts (e.g., while teaching a specific task). With this aim, this work proposes a Q-Learning-based Model Predictive Variable Impedance Control (Q-LMPVIC) to assist the operators in a physical human-robot collaboration (pHRC) tasks. A Cartesian impedance control loop is designed to implement a decoupled compliant robot dynamics. The impedance control parameters (i.e., setpoint and damping parameters) are then optimized online in order to maximize the performance of the pHRC. For this purpose, an ensemble of neural networks is designed to learn the modeling of the human-robot interaction dynamics while capturing the associated uncertainties. The derived modeling is then exploited by the model predictive controller (MPC), enhanced with the stability guarantees by means of Lyapunov constraints. The MPC is solved by making use of a Q-Learning method that, in its online implementation, uses an actor-critic algorithm to approximate the exact solution. Indeed, the Q-learning method provides an accurate and highly efficient solution (in terms of computational time and resources). The proposed approach has been validated through experimental tests, in which a Franka EMIKA panda robot has been used as a test platform. Each user was asked to interact with the robot along the controlled vertical z Cartesian direction. The proposed controller has been compared with a model-based reinforcement learning variable impedance controller (MBRLC) previously developed by some of the authors in order to evaluate the performance. As highlighted in the achieved results, the proposed controller is able to improve the pHRC performance. Additionally, two industrial tasks (a collaborative assembly and a collaborative deposition task) have been demonstrated to prove the applicability of the proposed solution in real industrial scenarios.

Combining human guidance and structured task execution during physical human–robot collaboration

Combining human guidance and structured task execution during physical human–robot collaboration

Coupled dynamic modeling and experimental validation of a collaborative industrial mobile manipulator with human-robot interaction

Dynamic modeling and analysis of a collaborative industrial mobile manipulator provide essential foundation and guidance for controlling it to achieve the desired physical interaction. However, it is a well-recognized challenge to precisely model the dynamics of a collaborative mobile manipulator due to its high-DOF (degree of freedom) structure, nonholonomic constraints, dynamic coupling between the manipulator and the mobile platform, and physical interaction with environments/humans. This paper presents a dynamic model of a new high-DOF nonholonomic collaborative mobile manipulator applied to physical human-robot collaboration scenarios, accounting for the dynamic coupling. A full Jacobian matrix introduces the dynamic coupling and is derived to solve the nonholonomic constraints on the velocity of the mobile platform. Besides, it facilitates the formulation of the dynamic model with independent generalized coordinates based on Lagrange equations. The comprehensive simulations with ADAMS are carried out along a pre-set trajectory to verify the coupled dynamic model. The dynamic modeling method is further validated by the human-robot interaction experiment. The simulation and testing results demonstrate the accuracy of the coupled dynamic model of the collaborative mobile manipulator.

Dynamic Primitives Limit Human Force Regulation during Motion.

Humans excel at physical interaction despite long feedback delays and low-bandwidth actuators. Yet little is known about how humans manage physical interaction. A quantitative understanding of how they do is critical for designing machines that can safely and effectively interact with humans, e.g. amputation prostheses, assistive exoskeletons, therapeutic rehabilitation robots, and physical human-robot collaboration. To facilitate applications, this understanding should be in the form of a simple mathematical model that not only describes humans' capabilities but also their limitations. In robotics, hybrid control allows simultaneous, independent control of both motion and force and it is often assumed that humans can modulate force independent of motion as well. This paper experimentally tested that assumption. Participants were asked to apply a constant 5N force on a robot manipulandum that moved along an elliptical path. After initial improvement, force errors quickly plateaued, despite practice and visual feedback. Within-trial analyses revealed that force errors varied with position on the ellipse, rejecting the hypothesis that humans have independent control of force and motion. The findings are consistent with a feed-forward motion command composed of two primitive oscillations acting through mechanical impedance to evoke force.

Computational Model of Robot Trust in Human Co-Worker for Physical Human-Robot Collaboration

Trust is key to achieving successful Human-Robot Interaction (HRI). Besides trust of the human co-worker in the robot, trust of the robot in its human co-worker should also be considered. A computational model of a robot&#x2019;s trust in its human co-worker for physical human-robot collaboration (pHRC) is proposed. The trust model is a function of the human co-worker&#x2019;s performance which can be characterized by factors including safety, robot singularity, smoothness, physical performance and cognitive performance. Experiments with a collaborative robot are conducted to verify the developed trust model.

Preference-Based Optimization of a Human-Robot Collaborative Controller

Preference-based optimization is a powerful tool to improve the performance of a system in an intuitive way. Such a methodology allows for solving optimization problems in which the decision-maker cannot evaluate the objective function related to the target problem, but rather can only express a preference such as “this is better than that” between two candidate decision vectors. In such a way, the target cost function can be easily defined and implemented, avoiding complex formulations and the use of external sources of data. Such a methodology is nowadays deeply investigated to reduce the complexity related to systems/applications tuning in real applications, providing the user with a natural procedure capable to exploit his/her knowledge and preferences. Physical human-robot collaboration (HRC) is an important field of application considering the preference-based optimization topic. Physical HRC is increasingly important in many different domains. Properly tuning the robot behavior is important to achieve satisfactory performance from the human's perspective. However, a common tuning for all the possible subjects is not feasible due to their different skills, different backgrounds, and different expected performance. However, ad hoc tuning of the robot behavior is not trivial. To address this issue, the here presented contribution applies preference-based optimization to the tuning of a physical HRC controller. In such a way, an ad hoc tuning of the robot behavior based on the perceived interaction is achieved for each subject. Experimental results show the capabilities of the method, being able to optimize the robot behavior in limited optimization trials.