Human-robot Co-manipulation Research Articles

Classification of Co-Manipulation Modus with Human-Human Teams for Future Application to Human-Robot Systems

Despite the existence of robots that can physically lift heavy loads, robots that can collaborate with people to move heavy objects are not readily available. This article makes progress toward effective human-robot co-manipulation by studying 30 human-human dyads that collaboratively manipulated an object weighing \(27 \mathrm{kg}\) without being co-located (i.e., participants were at either end of the extended object). Participants maneuvered around different obstacles with the object while exhibiting one of four modi–the manner or objective with which a team moves an object together–at any given time. Using force and motion signals to classify modus or behavior was the primary objective of this work. Our results showed that two of the originally proposed modi were very similar, such that one could effectively be removed while still spanning the space of common behaviors during our co-manipulation tasks. The three modi used in classification were quickly , smoothly and avoiding obstacles . Using a deep convolutional neural network (CNN), we classified three modi with up to 89% accuracy from a validation set. The capability to detect or classify modus during co-manipulation has the potential to greatly improve human-robot performance by helping to define appropriate robot behavior or controller parameters depending on the objective or modus of the team.

Human-robot planar co-manipulation of extended objects: data-driven models and control from human-human dyads.

Human teams are able to easily perform collaborative manipulation tasks. However, simultaneously manipulating a large extended object for a robot and human is a difficult task due to the inherent ambiguity in the desired motion. Our approach in this paper is to leverage data from human-human dyad experiments to determine motion intent for a physical human-robot co-manipulation task. We do this by showing that the human-human dyad data exhibits distinct torque triggers for a lateral movement. As an alternative intent estimation method, we also develop a deep neural network based on motion data from human-human trials to predict future trajectories based on past object motion. We then show how force and motion data can be used to determine robot control in a human-robot dyad. Finally, we compare human-human dyad performance to the performance of two controllers that we developed for human-robot co-manipulation. We evaluate these controllers in three-degree-of-freedom planar motion where determining if the task involves rotation or translation is ambiguous.

Robust variable admittance control for human–robot co-manipulation of objects with unknown load

Over the last years, physical Human–Robot Interaction (pHRI) has become a particularly interesting topic for industrial tasks. An important issue is allowing people and robots to collaborate in an useful, simple and safe manner. In this work, we propose a new framework that allows the person to collaborate with a robot manipulator, while the robot has its own predefined task. To allow the robot to smoothly switch from its own task to be a compliant collaborator for the person, a variable admittance control is developed. Furthermore, in general the task to accomplish requires the robot to carry variable, unknown, loads at the end-effector. To include this feature in our framework, a robust control is also included to preserve the performance of the robot despite uncertainties coming from the unknown load. To validate our approach, experiments were carried out with a Kuka LBR iiwa 14 R820, first to validate both parts of the controller, and finally, to study a use-case scenario similar to an industrial production line. Results show the efficiency of this approach to allow the person to collaborate at any moment while the robot is capable of performing another task. This flexible framework for object co-manipulation also allows unknown loads up to 2 kg to be handled without making the task more difficult for the person.

Level position and obstacle avoidance control for human-robot co-manipulation of a rigid object

Level position and obstacle avoidance control for human-robot co-manipulation of a rigid object

Human–Robot Interaction: A Review and Analysis on Variable Admittance Control, Safety, and Perspectives

Human–robot interaction (HRI) is a broad research topic, which is defined as understanding, designing, developing, and evaluating the robotic system to be used with or by humans. This paper presents a survey on the control, safety, and perspectives for HRI systems. The first part of this paper reviews the variable admittance (VA) control for human–robot co-manipulation tasks, where the virtual damping, inertia, or both are adjusted. An overview of the published research for the VA control approaches, their methods, the accomplished collaborative co-manipulation tasks and applications, and the criteria for evaluating them are presented and compared. Then, the performance of various VA controllers is compared and investigated. In the second part, the safety of HRI systems is discussed. The various methods for detection of human–robot collisions (model-based and data-based) are investigated and compared. Furthermore, the criteria, the main aspects, and the requirements for the determination of the collision and their thresholds are discussed. The performance measure and the effectiveness of each method are analyzed and compared. The third and final part of the paper discusses the perspectives, necessity, influences, and expectations of the HRI for future robotic systems.

Robust trocar identification and its application in robotic minimally invasive surgery.

In minimally invasive surgery (MIS), instruments pass through trocars which are installed through the incision points. This forms a fulcrum effect and imposes significant constraint. For robotic manipulative operations, the real-time trocar information is a prerequisite. Systems acquire this knowledge either with a prior registration procedure or through coordinated control of their joints. A robust and real-time trocar identification algorithm based on least square (LS) algorithm was proposed in the context of human-robot co-manipulation scenario. Both in vitro and in vivo experiments were performed to verify the effectiveness of the proposed algorithm. The estimated trocar coordinates expressed in the robot base frame were further leveraged to implement an instrument gravity compensation function. An LS based approach can be employed to robustly estimate the real-time trocar information so as to implement more practical robotic functions.

On deformable object handling: Model-based motion planning for human-robot co-manipulation

Despite extensive automation in multiple industrial sectors, manufacturing operations involving deformable objects are mostly performed manually. Challenges originating from flexible objects’ dynamic distortion underline handicaps in robot cognition and dexterity. This paper presents a model-based motion planner for deformable object co-manipulation. The developed closed-loop controlling framework interprets manipulation inputs into appropriate handling actions by simulating fabric's distortion through a mass-spring model. The planner incorporates tools for rapid system commissioning and reconfiguration, grasping point planning, and monitoring of human actions. Inspired by automotive composite industry, two experimental setups are used for validating the system's performance during translational and rotational co-manipulation.

Towards a balancing safety against performance approach in human–robot co-manipulation for door-closing emergencies

Telemanipulation in power stations commonly require robots first to open doors and then gain access to a new workspace. However, the opened doors can easily close by disturbances, interrupt the operations, and potentially lead to collision damages. Although existing telemanipulation is a highly efficient master–slave work pattern due to human-in-the-loop control, it is not trivial for a user to specify the optimal measures to guarantee safety. This paper investigates the safety-critical motion planning and control problem to balance robotic safety against manipulation performance during work emergencies. Based on a dynamic workspace released by door-closing, the interactions between the workspace and robot are analyzed using a partially observable Markov decision process, thereby making the balance mechanism executed as belief tree planning. To act the planning, apart from telemanipulation actions, we clarify other three safety-guaranteed actions: on guard, defense and escape for self-protection by estimating collision risk levels to trigger them. Besides, our experiments show that the proposed method is capable of determining multiple solutions for balancing robotic safety and work efficiency during telemanipulation tasks.

Trends in Haptic Communication of Human-Human Dyads: Toward Natural Human-Robot Co-manipulation

In this paper, we analyze and report on observable trends in human-human dyads performing collaborative manipulation (co-manipulation) tasks with an extended object (object with significant length). We present a detailed analysis relating trends in interaction forces and torques with other metrics and propose that these trends could provide a way of improving communication and efficiency for human-robot dyads. We find that the motion of the co-manipulated object has a measurable oscillatory component. We confirm that haptic feedback alone represents a sufficient communication channel for co-manipulation tasks, however we find that the loss of visual and auditory channels has a significant effect on interaction torque and velocity. The main objective of this paper is to lay the essential groundwork in defining principles of co-manipulation between human dyads. We propose that these principles could enable effective and intuitive human-robot collaborative manipulation in future co-manipulation research.

Towards robust human-robot mobile co-manipulation for tasks involving the handling of non-rigid materials using sensor-fused force-torque, and skeleton tracking data

Over the past decades, robots have been extensively deployed in multiple industries. More recently, industrial robots have been taken out of their cages, being more present in dynamic, uncertain environments interacting in the vicinity of humans. Traditionally, robots have been mainly developed to perform pre-programmed tasks. However, some tasks are too complex or expensive to be performed by a robotic system alone. One of these hard tasks is the handling of large fibre sheets in composite part production. This paper presents a force-based approach towards human-robot co-manipulation of tasks involving the handling of non-rigid materials, such as composite fibres. Our approach fuses the data of a force-torque sensor and skeleton tracking data from a 2D camera to control the mobile manipulator in an intuitive manner, by using the intelligence of the operator as much as possible. By using this approach, tools such as path planning, high-level task planning, or modelling of objects to be manipulated are not essential to obtain the results. The overall approach is illustrated with a co-manipulation proof-of-concept experiment in which a mobile manipulator robot handles a flexible textile sheet together with a human operator.

Human–Robot co-manipulation during surface tooling: A general framework based on impedance control, haptic rendering and discrete geometry

Despite the advancements in machine learning and artificial intelligence, there are many tooling tasks with cognitive aspects that are rather challenging for robots to handle in full autonomy, thus still requiring a certain degree of interaction with a human operator. In this paper, we propose a theoretical framework for both planning and execution of robot-surface contact tasks whereby interaction with a human operator can be accommodated to a variable degree.

Toward an Intuitive and Iterative 6D Virtual Guide Programming Framework for Assisted Human–Robot Comanipulation

SUMMARYIn human–robot comanipulation, virtual guides are an important tool used to assist the human worker as they constrain the movement of the robot to improve the task accuracy and to avoid undesirable effects, such as collisions with the environment. Consequently, the physical effort and cognitive overload are reduced during accomplishment of comanipulative tasks. However, the construction of virtual guides often requires expert knowledge and modeling of the task, which restricts the usefulness of virtual guides to scenarios with fixed constraints. Moreover, few approaches have addressed the implementation of virtual guides enforcing orientation constraints and, when done, these approaches have treated translation and orientation separately, and consequently there is no synchronization of the translational and rotational motions. To overcome these challenges and enhance the programming flexibility of virtual guides, we present a new framework that allows the user to create 6D virtual guides through XSplines which we define as a combination of Akima splines for the translation component and spherical cubic interpolation of quaternions for the orientation component. For complex tasks, the user is able to initially define a 3D virtual guide and then use this assistance in translational motion to concentrate only on defining the orientations along the path. It is also possible for the user to modify a particular point or portion of a guide while being assisted by it. We demonstrate in an industrial scenario that these innovations provide an intuitive solution to extend the use of virtual guides to 6 degrees of freedom and increase the human worker’s comfort during the programming phase of these guides in an assisted human–robot comanipulation context.

Reference Trajectory Reshaping Optimization and Control of Robotic Exoskeletons for Human-Robot Co-Manipulation.

For human-robot co-manipulation by robotic exoskeletons, the interaction forces provide a communication channel through which the human and the robot can coordinate their actions. In this article, an optimization approach for reshaping the physical interactive trajectory is presented in the co-manipulation tasks, which combines impedance control to enable the human to adjust both the desired and the actual trajectories of the robot. Different from previous studies, the proposed method significantly reshapes the desired trajectory during physical human-robot interaction (pHRI) based on force feedback, without requiring constant human guidance. The proposed scheme first formulates a quadratically constrained programming problem, which is then solved by neural dynamics optimization to obtain a smooth and minimal-energy trajectory similar to the natural human movement. Then, we propose an adaptive neural-network controller based on the barrier Lyapunov function (BLF), which enables the robot to handle the uncertain dynamics and the joint space constraints directly. To validate the proposed method, we perform experiments on the exoskeleton robot with human operators for co-manipulation tasks. The experimental results demonstrate that the proposed controller could complete the co-manipulation tasks effectively.

A Unified Architecture for Physical and Ergonomic Human–Robot Collaboration

SUMMARYIndustrial applications that involve working on and moving a heavy load or that constrain the operator to work in uncomfortable positions can take advantage of the assistance of a robotic assistant. In this paper, we propose an architecture for an ergonomic human–robot co-manipulation of objects of various shapes and weight. The object is carried by the robot and, thanks to an ergonomic planner, is positioned in the most comfortable way for the user. Furthermore, thanks to an admittance control with payload compensation, the user can easily adjust the position of the object for working on different parts of it. The proposed architecture is experimentally validated in a robotic cell including an ABB industrial robot.

A selective muscle fatigue management approach to ergonomic human-robot co-manipulation

Abstract In this paper, we propose a method for selective monitoring and management of human muscle fatigue in human-robot co-manipulation scenarios. The proposed approach uses a machine learning technique to learn the complex relationship between individual human muscle forces, arm configuration and arm endpoint force that are provided by a sophisticated offline musculoskeletal model. The estimated muscle forces are used in the fatigue model to estimate the individual muscle fatigue levels online. Two fatigue management protocols are proposed that enable the robot to handle and reduce the human fatigue by altering the configuration of task execution. The first protocol uses optimisation technique to find the optimal position for task execution, where the fatigue-related endurance time can be maximised. The second protocol divides the arm muscles into groups and then alters the direction of endpoint force so that the fatigued muscle group can relax and the relaxed muscle group becomes active. The proposed method has a potential to enable the robot to facilitate safer and more ergonomic working conditions for the human coworker. The main advantage of this approach is that it can operate online, and that all the measurements can be performed by the robot sensory system, which can significantly increase the applicability in real world scenarios. To validate the proposed method, we performed multiple experiments with two collaborative tasks (polishing and drilling) under different conditions.

Calibrating intuitive and natural human–robot interaction and performance for power-assisted heavy object manipulation using cognition-based intelligent admittance control schemes

In the first step, a one degree of freedom power assist robotic system is developed for lifting lightweight objects. Dynamics for human–robot co-manipulation is derived that includes human cognition, for example, weight perception. A novel admittance control scheme is derived using the weight perception–based dynamics. Human subjects lift a small-sized, lightweight object with the power assist robotic system. Human–robot interaction and system characteristics are analyzed. A comprehensive scheme is developed to evaluate the human–robot interaction and performance, and a constrained optimization algorithm is developed to determine the optimum human–robot interaction and performance. The results show that the inclusion of weight perception in the control helps achieve optimum human–robot interaction and performance for a set of hard constraints. In the second step, the same optimization algorithm and control scheme are used for lifting a heavy object with a multi-degree of freedom power assist robotic system. The results show that the human–robot interaction and performance for lifting the heavy object are not as good as that for lifting the lightweight object. Then, weight perception–based intelligent controls in the forms of model predictive control and vision-based variable admittance control are applied for lifting the heavy object. The results show that the intelligent controls enhance human–robot interaction and performance, help achieve optimum human–robot interaction and performance for a set of soft constraints, and produce similar human–robot interaction and performance as obtained for lifting the lightweight object. The human–robot interaction and performance for lifting the heavy object with power assist are treated as intuitive and natural because these are calibrated with those for lifting the lightweight object. The results also show that the variable admittance control outperforms the model predictive control. We also propose a method to adjust the variable admittance control for three degrees of freedom translational manipulation of heavy objects based on human intent recognition. The results are useful for developing controls of human friendly, high performance power assist robotic systems for heavy object manipulation in industries.

Robot adaptation to human physical fatigue in human–robot co-manipulation

In this paper, we propose a novel method for human---robot collaboration, where the robot physical behaviour is adapted online to the human motor fatigue. The robot starts as a follower and imitates the human. As the collaborative task is performed under the human lead, the robot gradually learns the parameters and trajectories related to the task execution. In the meantime, the robot monitors the human fatigue during the task production. When a predefined level of fatigue is indicated, the robot uses the learnt skill to take over physically demanding aspects of the task and lets the human recover some of the strength. The human remains present to perform aspects of collaborative task that the robot cannot fully take over and maintains the overall supervision. The robot adaptation system is based on the Dynamical Movement Primitives, Locally Weighted Regression and Adaptive Frequency Oscillators. The estimation of the human motor fatigue is carried out using a proposed online model, which is based on the human muscle activity measured by the electromyography. We demonstrate the proposed approach with experiments on real-world co-manipulation tasks: material sawing and surface polishing.

A Human-Robot Co-Manipulation Approach Based on Human Sensorimotor Information.

This paper aims to improve the interaction and coordination between the human and the robot in cooperative execution of complex, powerful, and dynamic tasks. We propose a novel approach that integrates online information about the human motor function and manipulability properties into the hybrid controller of the assistive robot. Through this human-in-the-loop framework, the robot can adapt to the human motor behavior and provide the appropriate assistive response in different phases of the cooperative task. We experimentally evaluate the proposed approach in two human-robot co-manipulation tasks that require specific complementary behavior from the two agents. Results suggest that the proposed technique, which relies on a minimum degree of task-level pre-programming, can achieve an enhanced physical human-robot interaction performance and deliver appropriate level of assistance to the human operator.

Online Stability in Human-Robot Cooperation with Admittance Control.

In the design of a compliant admittance controller for physical human-robot interaction, it is necessary to ensure stable and effective cooperation. The stability of the admittance controller is mainly threatened by a stiff environment. Many methods that guarantee stability in arbitrary environments, impose conservative control gains that limit the effectiveness of the cooperation. Inspired by previous work in frequency domain stability observers, a method is proposed in this paper to detect unstable behavior and stabilize the robot with online adaptation of the admittance control gains. The introduced instability index is based on frequency domain analysis, which very quickly detects unstable behavior by monitoring high frequency oscillation in the force signal. To treat the instability, an adaptation scheme of the admittance parameters is proposed, that relaxes conservative gains and improves the cooperation by considering the effect of variable admittance on the operators' effort. We investigate two human-robot co-manipulation tasks; cooperation within a zero stiffness environment and cooperation in contact with a stiff double-wall virtual environment. The proposed methods are validated experimentally with a number of subjects in cooperation with an LWR manipulator.

Variable Impedance Control of Redundant Manipulators for Intuitive Human–Robot Physical Interaction

This paper presents an experimental study on human-robot comanipulation in the presence of kinematic redundancy. The objective of the work is to enhance the performance during human-robot physical interaction by combining Cartesian impedance modulation and redundancy resolution. Cartesian impedance control is employed to achieve a compliant behavior of the robot's end effector in response to forces exerted by the human operator. Different impedance modulation strategies, which take into account the human's behavior during the interaction, are selected with the support of a simulation study and then experimentally tested on a 7-degree-of-freedom KUKA LWR4. A comparative study to establish the most effective redundancy resolution strategy has been made by evaluating different solutions compatible with the considered task. The experiments have shown that the redundancy, when used to ensure a decoupled apparent inertia at the end effector, allows enlarging the stability region in the impedance parameters space and improving the performance. On the other hand, the variable impedance with a suitable modulation strategy for parameters' tuning outperforms the constant impedance, in the sense that it enhances the comfort perceived by humans during manual guidance and allows reaching a favorable compromise between accuracy and execution time.