Probabilistic Movement Primitives Research Articles

Demonstration-enhanced policy search for space multi-arm robot collaborative skill learning

The increasing complexity of on-orbit tasks imposes great demands on the flexible operation of space robotic arms, prompting the development of space robots from single-arm manipulation to multi-arm collaboration. In this paper, a combined approach of Learning from Demonstration (LfD) and Reinforcement Learning (RL) is proposed for space multi-arm collaborative skill learning. The combination effectively resolves the trade-off between learning efficiency and feasible solution in LfD, as well as the time-consuming pursuit of the optimal solution in RL. With the prior knowledge of LfD, space robotic arms can achieve efficient guided learning in high-dimensional state-action space. Specifically, an LfD approach with Probabilistic Movement Primitives (ProMP) is firstly utilized to encode and reproduce the demonstration actions, generating a distribution as the initialization of policy. Then in the RL stage, a Relative Entropy Policy Search (REPS) algorithm modified in continuous state-action space is employed for further policy improvement. More importantly, the learned behaviors can maintain and reflect the characteristics of demonstrations. In addition, a series of supplementary policy search mechanisms are designed to accelerate the exploration process. The effectiveness of the proposed method has been verified both theoretically and experimentally. Moreover, comparisons with state-of-the-art methods have confirmed the outperformance of the approach.

Non-parametric Gaussian process movement primitive with via-point constraint for effective and safe robot skill learning

Learning from demonstration (LFD) algorithms has been proven to be an effective way to encode basic human skills, such as probabilistic movement primitives (ProMPs). However, such algorithms are based on parametric methods, which means that more effort is required to expand to multi-dimensional and high reproduction accuracy. In this article, a novel non-parametric LFD algorithm is proposed, named Gaussian process movement primitive based on Gaussian mixture regression (GPGR), which has lower computational complexity and high accuracy. We encapsulate the variability of the demonstration set into the prior of Gaussian processes and propose a necessary and sufficient condition to ensure that the generated trajectory can pass the expected via point with 100% probability, which is different from existing ProMPs and their variants. In addition, we propose a novel trajectory obstacle avoidance method. The method allows efficient and safe robot obstacle avoidance by solving for a small number of via points. The effectiveness of the algorithm is validated through a series of experiments on the LASA dataset and the Franka Emika Panda robot.

Probabilistic movement primitives based multi-task learning framework

With the increasing complexity of industrial production and manufacturing tasks, industrial robots are expected to learn intricate operations from simple actions easily and quickly with adaption to dynamic environment. In this paper, a task-parameterized multi-task learning framework is proposed to facilitate rapid learning of operational skills for industrial robots. In this framework, a conditional Probabilistic Movement Primitives (ProMP) is firstly employed to the single-task learning. Using the conditional probability calculation, the extrapolation issue in Learning from Demonstration (LfD) is addressed, enabling robots to learn beyond teaching. Subsequently, the single-task is extended to multi-task scenario by proposing a multi-task learning approach where each single task executes an extrapolation learning. The learned skill can meet the multiple task requirements through an iterative modulation manner. The effectiveness of the proposed framework is validated through both the simulation and a 7-DoF Franka-Emika robot experiment in a predefined task scenario. Furthermore, the outperformance of the proposed method is demonstrated by comparing with the state-of-art movement primitives based learning method.

Improved Generalization of Probabilistic Movement Primitives for Manipulation Trajectories

Improved Generalization of Probabilistic Movement Primitives for Manipulation Trajectories

AL-ProMP: Force-relevant skills learning and generalization method for robotic polishing

Skill learning in robot polishing is gaining attention and becoming a hot issue. Current studies on skill learning in robot polishing are mainly about trajectory skills, and force-relevant skills learning models are less studied. A skill learning method with good generalization and robustness is one of the elements worth investigating. In this study, a force-relevant skills learning method called arc-length probabilistic movement primitives (AL-ProMP) is proposed to improve the efficiency of robot polishing force planning. AL-ProMP learns the mapping between the contact force and polishing trajectory, and the temporal scaling factor and force scaling factor in AL-ProMP enable better robustness of force planning in speed scaling tasks and polishing tasks in different scenarios. Speed scaling is an important property for adaptation of the polishing policy. For the generalization of polishing skills to different polishing tools in robotics disc polishing tasks of unknown geometric model workpieces, a novel force scaling factor for different polishing discs is proposed according to the contact force model. In addition, polishing contact position learning provides the basis for polishing trajectory generalization. Finally, it is experimentally verified that the proposed method is effective in learning and generalizing the demonstrated skills and improving the polishing surface quality of the workpiece with unknown geometric model.

Rule-Based Safe Probabilistic Movement Primitive Control via Control Barrier Functions

In this paper, we develop a novel and safe control design approach that takes demonstrations provided by a human teacher to enable a robot to accomplish complex manipulation scenarios in dynamic environments. First, an overall task is divided into multiple simpler subtasks that are more appropriate for learning and control objectives. Then, by collecting human demonstrations, the subtasks that require robot movement are modeled by probabilistic movement primitives (ProMPs). We also study two strategies for modifying the ProMPs to avoid collisions with environmental obstacles. Finally, we introduce a rule-base control technique by utilizing a finite-state machine along with a unique means of control design for ProMPs. For the ProMP controller, we propose control barrier and Lyapunov functions to guide the system along a trajectory within the distribution defined by a ProMP while guaranteeing that the system state never leaves more than a desired distance from the distribution mean. This allows for better performance on nonlinear systems and offers solid stability and known bounds on the system state. A series of simulations and experimental studies demonstrate the efficacy of our approach and show that it can run in real time. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper is motivated by the need to create a teach-by-demonstration framework that captures the strengths of movement primitives and verifiable, safe control. We provide a framework that learns safe control laws from a probability distribution of robot trajectories through the use of advanced nonlinear control that incorporates safety constraints. Typically, such distributions are stochastic, making it difficult to offer any guarantees on safe operation. Our approach ensures that the distribution of allowed robot trajectories is within an envelope of safety and allows for robust operation of a robot. Furthermore, using our framework various probability distributions can be combined to represent complex scenarios in the environment. It will benefit practitioners by making it substantially easier to test and deploy accurate, efficient, and safe robots in complex real-world scenarios. The approach is currently limited to scenarios involving static obstacles, with dynamic obstacle avoidance an avenue of future effort.

Gaussian process movement primitive

Using probabilistic movement primitives (ProMPs) has been shown as an effective way to encode basic robot movements. However, ProMPs are learned based on parametric approaches, which increases the demands for human efforts, and also limits the reproduction accuracy. In this technical communique, we present a novel non-parametric movement primitive, named Gaussian process movement primitive (GP-MP), which is totally data-driven and more accurate. Moreover, different from existing ProMP or its variants, we propose a sufficient condition for GP-MP to ensure that the generated trajectory can pass desired via-points with 100% probability. We also show that different GP-MPs can be analytically blended like ProMPs. Comparative simulations on the LASA data set are conducted to show the advantages of the presented GP-MP.

ProDMP: A Unified Perspective on Dynamic and Probabilistic Movement Primitives

Movement Primitives (MPs) are a well-known concept to represent and generate modular trajectories. MPs can be broadly categorized into two types: (a) dynamics-based approaches that generate smooth trajectories from any initial state, e. g., Dynamic Movement Primitives (DMPs), and (b) probabilistic approaches that capture higher-order statistics of the motion, e. g., Probabilistic Movement Primitives (ProMPs). To date, however, there is no MP method that unifies both, i. e. that can generate smooth trajectories from an arbitrary initial state while capturing higher-order statistics. In this paper, we introduce a unified perspective of both approaches by solving the ODE underlying the DMPs. We convert expensive online numerical integration of DMPs into position and velocity basis functions that can be used to represent trajectories or trajectory distributions similar to ProMPs while maintaining all the properties of dynamical systems. Since we inherit the properties of both methodologies, we call our proposed model Probabilistic Dynamic Movement Primitives (ProDMPs). Additionally, we embed ProDMPs in deep neural network architecture and propose a new cost function for efficient end-to-end learning of higher-order trajectory statistics. To this end, we leverage Bayesian Aggregation for non-linear iterative conditioning on sensory inputs. Our proposed model achieves smooth trajectory generation, goal-attractor convergence, correlation analysis, non-linear conditioning, and online re-planing in one framework. Our code can be found in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ALRhub/ProDMP_RAL</uri> .

Interaction learning control with movement primitives for lower limb exoskeleton.

Research on robotic exoskeletons both in the military and medical fields has rapidly expanded over the previous decade. As a human-robot interaction system, it is a challenge to develop an assistive strategy that makes the exoskeleton supply efficient and natural assistance following the user's intention. This paper proposed a novel interaction learning control strategy for the lower extremity exoskeleton. A powerful representative tool probabilistic movement primitives (ProMPs) is adopted to model the motion and generate the desired trajectory in real-time. To adjust the trajectory by the user's real-time intention, a compensation term based on human-robot interaction force is designed and merged into the ProMPs model. Then, compliant impedance control is adopted as a low-level control where the desired trajectory is put into. Moreover, the model will be dynamically adapted online by penalizing both the interaction force and trajectory mismatch, with all the parameters that can be further learned by learning algorithm PIBB. The experimental results verified the effectiveness of the proposed control framework.

Constrained Probabilistic Movement Primitives for Robot Trajectory Adaptation

Placing robots outside controlled conditions requires versatile movement representations that allow robots to learn new tasks and adapt them to environmental changes. The introduction of obstacles or the placement of additional robots in the workspace, the modification of the joint range due to faults or range-of-motion constraints are typical cases where the adaptation capabilities play a key role for safely performing the robot's task. Probabilistic movement primitives (ProMPs) have been proposed for representing adaptable movement skills, which are modelled as Gaussian distributions over trajectories. These are analytically tractable and can be learned from a small number of demonstrations. However, both the original ProMP formulation and the subsequent approaches only provide solutions to specific movement adaptation problems, e.g., obstacle avoidance, and a generic, unifying, probabilistic approach to adaptation is missing. In this paper we develop a generic probabilistic framework for adapting ProMPs. We unify previous adaptation techniques, for example, various types of obstacle avoidance, via-points, mutual avoidance, in one single framework and combine them to solve complex robotic problems. Additionally, we derive novel adaptation techniques such as temporally unbound via-points and mutual avoidance. We formulate adaptation as a constrained optimisation problem where we minimise the Kullback-Leibler divergence between the adapted distribution and the distribution of the original primitive while we constrain the probability mass associated with undesired trajectories to be low. We demonstrate our approach on several adaptation problems on simulated planar robot arms and 7-DOF Franka-Emika robots in a dual robot arm setting.

Safe Robot Trajectory Control Using Probabilistic Movement Primitives and Control Barrier Functions.

In this paper, we present a novel means of control design for probabilistic movement primitives (ProMPs). Our proposed approach makes use of control barrier functions and control Lyapunov functions defined by a ProMP distribution. Thus, a robot may move along a trajectory within the distribution while guaranteeing that the system state never leaves more than a desired distance from the distribution mean. The control employs feedback linearization to handle nonlinearities in the system dynamics and real-time quadratic programming to ensure a solution exists that satisfies all safety constraints while minimizing control effort. Furthermore, we highlight how the proposed method may allow a designer to emphasize certain safety objectives that are more important than the others. A series of simulations and experiments demonstrate the efficacy of our approach and show it can run in real time.

A Shared Control Method for Collaborative Human-Robot Plug Task

Humans are superior to robots in performing complex assembly tasks primarily due to their dexterity and sensorimotor abilities. When humans and robots collaborate to complete an assembly task, it is critical to provide the robot with sufficiently precise manipulation capabilities and formulate a shared control method that coordinates human and robot actions. In this study, we are investigating human-robot collaboration in completing an assembly task, namely the plug task. The plug task, motivated by the 2015 DARPA Robotics Challenge Finals, involves inserting a plug connected with a cable into a paired socket. The human holds the socket while the robot is grasping the cable and manages to insert the plug. We estimate an initial socket pose based on human subjects' data. With the statistically calculated initial guess, the robot could act instantly from the beginning to move the plug towards an initial target. The actual socket pose is updated using the Kalman filter in real-time. Besides, the system continuously updates the cable model and tracks the plug to enable visual servoing to complete the plug task. The result of an extensive experimental study demonstrates that the proposed approach demonstrates fast and accurate performance as well as flexibility when compared to a state-of-the-art method based on interactive probabilistic movement primitives.

An Improvement of Robot Stiffness-Adaptive Skill Primitive Generalization Using the Surface Electromyography in Human–Robot Collaboration

Learning from Demonstration in robotics has proved its efficiency in robot skill learning. The generalization goals of most skill expression models in real scenarios are specified by humans or associated with other perceptual data. Our proposed framework using the Probabilistic Movement Primitives (ProMPs) modeling to resolve the shortcomings of the previous research works; the coupling between stiffness and motion is inherently established in a single model. Such a framework can request a small amount of incomplete observation data to infer the entire skill primitive. It can be used as an intuitive generalization command sending tool to achieve collaboration between humans and robots with human-like stiffness modulation strategies on either side. Experiments (human–robot hand-over, object matching, pick-and-place) were conducted to prove the effectiveness of the work. Myo armband and Leap motion camera are used as surface electromyography (sEMG) signal and motion capture sensors respective in the experiments. Also, the experiments show that the proposed framework strengthened the ability to distinguish actions with similar movements under observation noise by introducing the sEMG signal into the ProMP model. The usage of the mixture model brings possibilities in achieving automation of multiple collaborative tasks.

Using Probabilistic Movement Primitives in Analyzing Human Motion Differences Under Transcranial Current Stimulation.

In medical tasks such as human motion analysis, computer-aided auxiliary systems have become the preferred choice for human experts for their high efficiency. However, conventional approaches are typically based on user-defined features such as movement onset times, peak velocities, motion vectors, or frequency domain analyses. Such approaches entail careful data post-processing or specific domain knowledge to achieve a meaningful feature extraction. Besides, they are prone to noise and the manual-defined features could hardly be re-used for other analyses. In this paper, we proposed probabilistic movement primitives (ProMPs), a widely-used approach in robot skill learning, to model human motions. The benefit of ProMPs is that the features are directly learned from the data and ProMPs can capture important features describing the trajectory shape, which can easily be extended to other tasks. Distinct from previous research, where classification tasks are mostly investigated, we applied ProMPs together with a variant of Kullback-Leibler (KL) divergence to quantify the effect of different transcranial current stimulation methods on human motions. We presented an initial result with 10 participants. The results validate ProMPs as a robust and effective feature extractor for human motions.

Learning robot anomaly recovery skills from multiple time-driven demonstrations

Robots are prone to making anomalies when performing manipulation tasks in unstructured environments, it is often desirable to rapidly adapt the robotic behavior to avoid environmental changes by learning from experts’ demonstrations. We propose a framework for learning robot anomaly recovery skills from time-driven demonstrations based on a Gaussian process regression with prior mean derived by Gaussian mixture regression, named as mean-prior GPR (MP-GPR), which allows an end-user to adjust the anomalous trajectory intuitively by simultaneously considering the variability of the demonstrations and the adaptation of recovery skills. Evaluations are divided into two phases, a benchmarking dataset with robot reaching, pushing, writing, and pressing tasks are first used to verify the path accuracy and variability, and then a real-time robot bin-picking task for evaluating the adaptation of the framework. Our method has a fair comparison with probabilistic-based methods in the field of robot learning from demonstrations, including Gaussian mixture regression, probabilistic movement primitives, and kernelized movement primitives. The results indicate that our proposed method can efficiently encode the variability from multiple demonstrations and rapidly anomaly recovery skills learning by modulating a learned trajectory to safe via-points.

Probabilistic Adaptive Control for Robust Behavior Imitation

In the context of learning from demonstration (LfD), trajectory policy representations such as probabilistic movement primitives (ProMPs) allow for rich modeling of demonstrated skills. To reproduce a learned skill with a real robot, a feedback controller is required to cope with perturbations and to react to dynamic changes in the environment. In this letter, we propose a generalized probabilistic control approach that merges the probabilistic modeling of the demonstrated movements and the feedback control action for reproducing the demonstrated behavior. We show that our controller can be easily employed, outperforming both original controller and a controller with constant feedback gains. Furthermore, we show that the proposed approach is able to solve dynamically changing tasks by modeling the demonstrated behavior as Gaussian mixtures and by introducing context variables. We demonstrate the capability of the approach with experiments in simulation and by teaching a 7-axis Franka Emika Panda robot to drop a ball into a moving box with only few demonstrations.

Active Learning of Bayesian Probabilistic Movement Primitives

Learning from Demonstration permits non-expert users to easily and intuitively reprogram robots. Among approaches embracing this paradigm, probabilistic movement primitives (ProMPs) are a well-established and widely used method to learn trajectory distributions. However, providing or requesting useful demonstrations is not easy, as quantifying what constitutes a good demonstration in terms of generalization capabilities is not trivial. In this letter, we propose an active learning method for contextual ProMPs for addressing this problem. More specifically, we learn the trajectory distributions using a Bayesian Gaussian mixture model (BGMM) and then leverage the notion of epistemic uncertainties to iteratively choose new context query points for demonstrations. We show that this approach reduces the required number of human demonstrations. We demonstrate the effectiveness of the approach on a pouring task, both in simulation and on a real 7-DoF Franka Emika robot.

Predictive Exoskeleton Control for Arm-Motion Augmentation Based on Probabilistic Movement Primitives Combined With a Flow Controller

There are many work-related repetitive tasks where the application of exoskeletons could significantly reduce the physical effort by assisting the user in moving the arms towards the desired location in space. To make such control more user acceptable, the controller should be able to predict the motion of the user and act accordingly. This letter presents an exoskeleton control method that utilizes probabilistic movement primitives to generate predictions of user movements in real-time. These predictions are used in a flow controller, which represents a novel velocity-field-based exoskeleton control approach to provide assistance to the user in a predictive way. We evaluated our approach with a haptic robot, where a group of twelve participants had to perform movements towards different target locations in the frontal plane. We tested whether we could generalize the predictions for new and unknown target locations whilst providing assistance to the user without changing their kinematic parameters. The evaluation showed that we could accurately predict user movement intentions while at the same time significantly decrease the overall physical effort exerted by the participants to achieve the task.

Adaptive Multi-Task Human-Robot Interaction Based on Human Behavioral Intention

Learning from demonstrations with Probabilistic Movement Primitives (ProMPs) has been widely used in robot skill learning, especially in human-robot collaboration. Although ProMP has been extended to multi-task situations inspired by the Gaussian mixture model, it still treats each task independently. ProMP ignores the common scenario that robots conduct adaptive switching of the collaborative tasks in order to align with the instantaneous change of human intention. To solve this problem, we proposed an alternate learning-based parameter estimation method and an empirical minimum variation-based decomposition strategy with projection points, combining with linear interpolation strategy for weights, based on a Gaussian mixture model framework. Alternate learning of weights and parameters in multi-task ProMP (MTProMP) allows the robot to obtain a smooth composite trajectory planning which crosses expected via points. Decomposition strategy reflects how the desired via point state is projected onto the individual ProMP component, rendering the minimum total sum of deviations between each projection point with the respective prior. Linear interpolation is used to adjust the weights among sequential via points automatically. The proposed method and strategy are successfully extended to multi-task interaction ProMPs (MTiProMP). With MTProMP and MTiProMP, the robot can be applied to multiple tasks in industrial factories and collaborate with the worker to switch from one task to another according to changing intentions of the human. Classical via points trajectory planning experiments and human-robot collaboration experiments are performed on the Sawyer robot. The results of experiments show that MTProMP and MTiProMP with the proposed method and strategy perform better.

A Framework for Recognition and Prediction of Human Motions in Human-Robot Collaboration Using Probabilistic Motion Models

This letter presents a framework for recognition and prediction of ongoing human motions. The predictions generated by this framework could be used in a controller for a robotic device, enabling the emergence of intuitive and predictable interactions between humans and a robotic collaborator. The framework includes motion onset detection, phase speed estimation, intent estimation and conditioning. For recognition and prediction of a motion, the framework makes use of a motion model database. This database contains several motion models learned using the probabilistic Principal Component Analysis (PPCA) method. The proposed framework is evaluated with joint angle trajectories of eight subjects performing squatting, stooping and lifting tasks. The motion onset and phase speed estimation modules are first evaluated separately. Next, an evaluation of the full framework provides more insight in the current challenges regarding motion prediction. A brief comparison between PPCA and the Probabilistic Movement Primitives (ProMP) method for learning motion models is made based on the influence of both methodologies on the performance of the framework. Both PPCA and ProMP motion models are able to predict motions over a short time horizon but struggle to predict motions over a longer horizon.