Human-in-the-Loop Robot Control for Human-Robot Collaboration: Human Intention Estimation and Safe Trajectory Tracking Control for Collaborative Tasks

Abstract

This article provides a perspective on estimation and control problems in cyberphysical human systems (CPHSs) that work at the intersection of cyberphysical systems and human systems. The article also discusses solutions to some of the problems in CPHSs. One example of a CPHS is a close-proximity human-robot collaboration (HRC) in a manufacturing setting. The issue of the joint operation's efficiency and human factors, such as safety, attention, mental states, and comfort, naturally arise in the HRC context. By considering human factors, robots' actions can be controlled to achieve objectives, including safe operations and human comfort. Alternately, questions arise when robot factors are considered. For example, can we provide direct inputs and information to humans about an environment and the robots in the area such that the objectives of safety, efficiency, and comfort can be satisfied by considering the robots' current capabilities? The article discusses specific problems involved in HRC related to controlling a robot's motion by taking the current actions of the human in the loop with the robot's control system. To this end, two main challenges are discussed: 1) inferring the intention behind human actions by analyzing a person's motion as observed through skeletal tracking and gaze data and 2) a controller design that keeps robot motion constrained to a boundary in a 3D space by using control barrier functions. The intention inference method fuses skeleton-joint tracking data obtained using the Microsoft Kinect sensor and human gaze data gathered from red-green-blue Kinect images. The direction of a human's hand-reaching motion and a goal-reaching point is estimated while performing a joint pick-and-place task. The trajectory of the hand is estimated forward in time based on the gaze and hand motion data at the current time instance. A barrier function method is applied to generate safe robot trajectories along with forecast hand movements to complete the collaborative displacement of an object by a person and a robot. An adaptive controller is then used to track the reference trajectories using the Baxter robot, which is tested in a Gazebo simulation environment.