Complex Manipulation Tasks Research Articles

Geometric Task Networks: Learning Efficient and Explainable Skill Coordination for Object Manipulation

Complex manipulation tasks can contain various execution branches of primitive skills in sequence or in parallel under different scenarios. Manual specifications of such branching conditions and associated skill parameters are not only error-prone due to corner cases, but also quickly untraceable given a large number of objects and skills. On the other hand, learning from demonstration has increasingly shown to be an intuitive and effective way to program such skills for industrial robots. Parameterized skill representations allow generalization over new scenarios, which however makes the planning process much slower thus unsuitable for online applications. In this article, we propose a hierarchical and compositional planning framework that learns a geometric task network (GTN) from exhaustive planners, without any manual inputs. A GTN is a goal-dependent task graph that encapsulates both the transition relations among skill representations and the geometric constraints underlying these transitions. This framework has shown to improve dramatically the offline learning efficiency, the online performance, and the transparency of decision process, by leveraging the task-parameterized models. We demonstrate the approach on a 7-DoF robot arm both in simulation and on hardware solving various manipulation tasks.

Contact-consistent visual object pose estimation for contact-rich robotic manipulation tasks

PurposeThe purpose of this paper is to estimate the contact-consistent object poses during contact-rich manipulation tasks based only on visual sensors.Design/methodology/approachThe method follows a four-step procedure. Initially, the raw object poses are retrieved using the available object pose estimation method and filtered using Kalman filter with nominal model; second, a group of particles are randomly generated for each pose and evaluated the corresponding object contact state using the contact simulation software. A probability guided particle averaging method is proposed to balance the accuracy and safety issues; third, the independently estimated contact states are fused in a hidden Markov model to remove the abnormal contact state observations; finally, the object poses are refined by averaging the contact state consistent particles.FindingsThe experiments are performed to evaluate the effectiveness of the proposed methods. The results show that the method can achieve smooth and accurate pose estimation results and the estimated contact states are consistent with ground truth.Originality/valueThis paper proposes a method to obtain contact-consistent poses and contact states of objects using only visual sensors. The method tries to recover the true contact state from inaccurate visual information by fusing contact simulations results and contact consistency assumptions. The method can be used to extract pose and contact information from object manipulation tasks by just observing the demonstration, which can provide a new way for the robot to learn complex manipulation tasks.

Learning-Based Fabric Folding and Box Wrapping

Manipulation of deformable objects is an essential task in surgery, the textile industry, and household tasks, such as washing, hanging, and folding clothes. However, current studies on fabric manipulation, have rarely considered the scenes in which rigid objects have to be wrapped by fabrics. This type of operation is widely adopted in the logistic packaging and packaging of surgical instrument baskets. In this study, we propose a method to perform this operation, which can be used in fabric folding or wrapping a box with fabric. Owing to the complex dynamics and configuration spaces of fabric, our method is based on deep imitation learning to estimate pick-place points and the phase of the manipulation process. The dataset was completely generated from an open-source physical simulator. To make the data as realistic as that in actual scenarios, we adopted domain randomization and rendered the texture of fabric and box from the real world to simulation data. This not only helps in transferring the learned policies to a physical robot, but also allows the robot to wrap a box with complex patterns. The experiments demonstrate the efficiency of the developed method for accomplishing complex manipulation tasks. The results also showed that it could be generalized to fabrics with different colors and boxes with different sizes, textures, or geometric shapes.

Correct Me If I am Wrong: Interactive Learning for Robotic Manipulation

Learning to solve complex manipulation tasks from visual observations is a dominant challenge for real-world robot learning. Although deep reinforcement learning algorithms have recently demonstrated impressive results in this context, they still require an impractical amount of time-consuming trial-and-error iterations. In this work, we consider the promising alternative paradigm of interactive learning in which a human teacher provides feedback to the policy during execution, as opposed to imitation learning where a pre-collected dataset of perfect demonstrations is used. Our proposed CEILing (Corrective and Evaluative Interactive Learning) framework combines both corrective and evaluative feedback from the teacher to train a stochastic policy in an asynchronous manner, and employs a dedicated mechanism to trade off human corrections with the robot’s own experience. We present results obtained with our framework in extensive simulation and real-world experiments to demonstrate that CEILing can effectively solve complex robot manipulation tasks directly from raw images in less than one hour of real-world training.

Semantic coupling of path planning and a primitive action of a task plan for the simulation of manipulation tasks in a virtual 3D environment

• Simulation of complex manipulation tasks under strong geometric constraints . • Use ontologies to model the environment, by involving high abstraction level data. • Use ontologies reasoning capacities to define task-related geometric constraints on path planning queries. • Improvement of motion planning through better semantic control. This work deals with the simulation of complex manipulation tasks in virtual environments. Validating such complex tasks, possibly to be performed under strong geometric constraints, requires considering task and path planning jointly. The contribution of this work focuses on using task-related information at the path planning level. We propose an ontology-based approach to a) model the 3D environment where the simulated task is executed, based on an original multi-level environment model involving higher abstraction level data than the purely geometric models traditionally used, and b) automatically define path planning queries for the primitive actions of a task plan, together with task-related geometric constraints on these queries. This approach allows the improvement of the state of the art from two points of view. First, our joint task and path planning approach allows the improvement of path planning through better semantic control of the path planning process. Second, if compared to hard-coded geometric constraints, the proposed ontology-based approach introduces a more flexible way of defining geometric constraints through an inference process, and can be adapted to different applications of manipulation tasks.

Receding Horizon Task and Motion Planning in Changing Environments

Complex manipulation tasks require careful integration of symbolic reasoning and motion planning. This problem, commonly referred to as Task and Motion Planning (TAMP), is even more challenging if the workspace is non-static, e.g. due to human interventions and perceived with noisy non-ideal sensors. This work proposes an online approximated TAMP method that combines a geometric reasoning module and a motion planner with a standard task planner in a receding horizon fashion. Our approach iteratively solves a reduced planning problem over a receding window of a limited number of future actions during the implementation of the actions. Thus, only the first action of the horizon is actually scheduled at each iteration, then the window is moved forward, and the problem is solved again. This procedure allows to naturally take into account potential changes in the scene while ensuring good runtime performance. We validate our approach within extensive experiments in a simulated environment. We showed that our approach is able to deal with unexpected changes in the environment while ensuring comparable performance with respect to other recent TAMP approaches in solving traditional static benchmarks. We release with this paper the open-source implementation of our method.

Global motion planning and redundancy resolution for large objects manipulation by dual redundant robots with closed kinematics

AbstractThe multi-arm robotic systems consisting of redundant robots are able to conduct more complex and coordinated tasks, such as manipulating large or heavy objects. The challenges of the motion planning and control for such systems mainly arise from the closed-chain constraint and redundancy resolution problem. The closed-chain constraint reduces the configuration space to lower-dimensional subsets, making it difficult for sampling feasible configurations and planning path connecting them. A global motion planner is proposed in this paper for the closed-chain systems, and motions in different disconnected manifolds are efficiently bridged by two type regrasping moves. The regrasping moves are automatically chosen by the planner based on cost-saving principle, which greatly improve the success rate and efficiency. Furthermore, to obtain the optional inverse kinematic solutions satisfying joint physical limits (e.g., joint position, velocity, acceleration limits) in the planning, the redundancy resolution problem for dual redundant robots is converted into a unified quadratic programming problem based on the combination of two diff erent-level optimizing criteria, i.e. the minimization velocity norm (MVN) and infinity norm torque-minimization (INTM). The Dual-MVN-INTM scheme guarantees smooth velocity, acceleration profiles, and zero final velocity at the end of motion. Finally, the planning results of three complex closed-chain manipulation task using two Franka Emika Panda robots and two Kinova Jaco2 robots in both simulation and experiment demonstrate the effectiveness and efficiency of the proposed method.

Tactile Sensor Data Interpretation for Estimation of Wire Features

At present, the tactile perception is essential for robotic applications when performing complex manipulation tasks, e.g., grasping objects of different shapes and sizes, distinguishing between different textures, and avoiding slips by grasping an object with a minimal force. Considering Deformable Linear Object manipulation applications, this paper presents an efficient and straightforward method to allow robots to autonomously work with thin objects, e.g., wires, and to recognize their features, i.e., diameter, by relying on tactile sensors developed by the authors. The method, based on machine learning algorithms, is described in-depth in the paper to make it easily reproducible by the readers. Experimental tests show the effectiveness of the approach that is able to properly recognize the considered object’s features with a recognition rate up to 99.9%. Moreover, a pick and place task, which uses the method to classify and organize a set of wires by diameter, is presented.

Control of robotic object pivoting based on tactile sensing

Robotic manipulation of objects using the sole tactile feedback is a challenge. If the contact between the robot end effector and the manipulated object is distributed, the robot can exchange both friction forces and torques with it. The friction highly influences the motion of the object. By controlling the friction it is possible to perform complex manipulation tasks, such as moving the object with respect to the end effector by executing a controlled sliding motion. If the motion is a rotation with respect to the end effector, the corresponding maneuver is called pivoting. Control of the pivoting motion is considerably difficult, especially without any visual feedback. This paper proposes a novel method to regulate the object angular position, by means of a pivoting maneuver, through a parallel gripper endowed with force/tactile sensors. The strategy is based on a novel nonlinear observer that estimates the sliding velocity from force/torque measurements and a model of the sliding dynamics. We exploit the Limit Surface concept and the LuGre friction model to build a dynamic model of a planar slider. We show, through experimental results, that simple parallel grippers are able to execute such maneuvers that correspond to adding a virtual joint between the fingers, thus enlarging the robot workspace.

Distinct Functional and Structural Connectivity of the Human Hand-Knob Supported by Intraoperative Findings.

Fine motor skills rely on the control of hand muscles exerted by a region of primary motor cortex (M1) that has been extensively investigated in monkeys. Although neuroimaging enables the exploration of this system also in humans, indirect measurements of brain activity prevent causal definitions of hand motor representations, which can be achieved using data obtained during brain mapping in tumor patients. High-frequency direct electrical stimulation delivered at rest (HF-DES-Rest) on the hand-knob region of the precentral gyrus has identified two sectors showing differences in cortical excitability. Using quantitative analysis of motor output elicited with HF DES-Rest, we characterized two sectors based on their excitability, higher in the posterior and lower in the anterior sector. We studied whether the different cortical excitability of these two regions reflected differences in functional connectivity (FC) and structural connectivity (SC). Using healthy adults from the Human Connectome Project (HCP), we computed FC and SC of the anterior and the posterior hand-knob sectors identified within a large cohort of patients. The comparison of FC of the two seeds showed that the anterior hand-knob, relative to the posterior hand-knob, showed stronger functional connections with a bilateral set of parietofrontal areas responsible for integrating perceptual and cognitive hand-related sensorimotor processes necessary for goal-related actions. This was reflected in different patterns of SC between the two sectors. Our results suggest that the human hand-knob is a functionally and structurally heterogeneous region organized along a motor-cognitive gradient.SIGNIFICANCE STATEMENT The capability to perform complex manipulative tasks is one of the major characteristics of primates and relies on the fine control of hand muscles exerted by a highly specialized region of the precentral gyrus, often termed the "hand-knob" sector. Using intraoperative brain mapping, we identify two hand-knob sectors (posterior and anterior) characterized by differences in cortical excitability. Based on resting-state functional connectivity (FC) and tractography in healthy subjects, we show that posterior and anterior hand-knob sectors differ in their functional connectivity (FC) and structural connectivity (SC) with frontoparietal regions. Thus, anteroposterior differences in cortical excitability are paralleled by differences in FC and SC that likely reflect a motor (posterior) to cognitive (anterior) organization of this cortical region.

A probabilistic framework for learning geometry-based robot manipulation skills

Programming robots to perform complex manipulation tasks is difficult because many tasks require sophisticated controllers that may rely on data such as manipulability ellipsoids, stiffness/damping and inertia matrices. Such data are naturally represented as Symmetric Positive Definite (SPD) matrices to capture specific geometric characteristics of the data, which increases the complexity of hard-coding them. To alleviate this difficulty, the Learning from Demonstration (LfD) paradigm can be used in order to learn robot manipulation skills with specific geometric constraints encapsulated in SPD matrices.Learned skills often need to be adapted when they are applied to new situations. While existing techniques can adapt Cartesian and joint space trajectories described by various desired points, the adaptation of motion skills encapsulated in SPD matrices remains an open problem. In this paper, we introduce a new LfD framework that can learn robot manipulation skills encapsulated in SPD matrices from expert demonstrations and adapt them to new situations defined by new start-, via- and end-matrices. The proposed approach leverages Kernelized Movement Primitives (KMPs) to generate SPD-based robot manipulation skills that smoothly adapt the demonstrations to conform to new constraints. We validate the proposed framework using a couple of simulations in addition to a real experiment scenario.

A review on manipulation skill acquisition through teleoperation‐based learning from demonstration

Manipulation skill learning and generalisation have gained increasing attention due to the wide applications of robot manipulators and the spurt of robot learning techniques. Especially, the learning from demonstration method has been exploited widely and successfully in the robotic community, and it is regarded as a promising direction to realise the manipulation skill learning and generalisation. In addition to the learning techniques, the immersive teleoperation enables the human to operate a remote robot with an intuitive interface and achieve the telepresence. Thus, it is a promising way to transfer manipulation skills from humans to robots by combining the learning methods and teleoperation, and adapting the learned skills to different tasks in new situations. This review, therefore, aims to provide an overview of immersive teleoperation for skill learning and generalisation to deal with complex manipulation tasks. To this end, the key technologies, for example, manipulation skill learning, multimodal interfacing for teleoperation and telerobotic control, are introduced. Then, an overview is given in terms of the most important applications of immersive teleoperation platform for robot skill learning. Finally, this survey discusses the remaining open challenges and promising research topics.

Discrimination Improvement Through Undesirable Feedback in Coupling Object Manipulation Tasks.

Subjective effort can significantly affect the ability of humans to act optimally in dynamic manipulation tasks. In a previous study, we designed a complex object coupling manipulation task that required tight performance and induced high cognitive workload. We hypothesize that strong-effort-related physiological reactivity during the dynamic manipulation task improves the user performance in an undesired task feedback situation. To test this hypothesis, using the motor intentions' discrimination from electroencephalogram (EEG) measurements, we evaluate the effort expended by 20 participants in a controlling task with constraints involving complex coupling objects. Specifically, the finer motor decisions are obtained from the controlling information in EEG by using two fingers from the same hand rather than two hands. The motor intention is decoded from a task-dependent EEG through a regularized discriminant analysis, and the area under the curve is [Formula: see text]. Furthermore, we compare the undesired and desired task feedback conditions along with the individual's effort dynamic adjustment, and investigate whether the undesired task feedback improved the discrimination of the motor activities. A stronger effort to attain the desired feedback state corresponds to improved motor activity discrimination from the EEG in the undesired task feedback scenario. The differences in the brain activities under the undesired and desired task feedback conditions are analyzed using brain-network-based topographical scalp maps. Our experiment provides preliminary evidence that inducing strong effort can improve discrimination performance during highly demanding tasks. This finding can advance our understanding of human attention, potentially improve the accuracy of intention recognition, and may inspire better EEG acquisition contexts.

Task-Adaptive Robot Learning From Demonstration With Gaussian Process Models Under Replication

Learning from Demonstration (LfD) is a paradigm that allows robots to learn complex manipulation tasks that can not be easily scripted, but can be demonstrated by a human teacher. One of the challenges of LfD is to enable robots to acquire skills that can be adapted to different scenarios. In this paper, we propose to achieve this by exploiting the variations in the demonstrations to retrieve an adaptive and robust policy, using Gaussian Process (GP) models. Adaptability is enhanced by incorporating task parameters into the model, which encode different specifications within the same task. With our formulation, these parameters can be either real, integer, or categorical. Furthermore, we propose a GP design that exploits the structure of replications, i.e., repeated demonstrations with identical conditions within data. Our method significantly reduces the computational cost of model fitting in complex tasks, where replications are essential to obtain a robust model. We illustrate our approach through several experiments on a handwritten letter demonstration dataset.

Manipulation of Articulated Objects Using Dual-arm Robots via Answer Set Programming

AbstractThe manipulation of articulated objects is of primary importance in Robotics and can be considered as one of the most complex manipulation tasks. Traditionally, this problem has been tackled by developing ad hoc approaches, which lack flexibility and portability. In this paper, we present a framework based on answer set programming (ASP) for the automated manipulation of articulated objects in a robot control architecture. In particular, ASP is employed for representing the configuration of the articulated object for checking the consistency of such representation in the knowledge base and for generating the sequence of manipulation actions. The framework is exemplified and validated on the Baxter dual-arm manipulator in the first, simple scenario. Then, we extend such scenario to improve the overall setup accuracy and to introduce a few constraints in robot actions execution to enforce their feasibility. The extended scenario entails a high number of possible actions that can be fruitfully combined together. Therefore, we exploit macro actions from automated planning in order to provide more effective plans. We validate the overall framework in the extended scenario, thereby confirming the applicability of ASP also in more realistic Robotics settings and showing the usefulness of macro actions for the robot-based manipulation of articulated objects.

Manipulating deformable objects by interleaving prediction, planning, and control

We present a framework for deformable object manipulation that interleaves planning and control, enabling complex manipulation tasks without relying on high-fidelity modeling or simulation. The key question we address is when should we use planning and when should we use control to achieve the task? Planners are designed to find paths through complex configuration spaces, but for highly underactuated systems, such as deformable objects, achieving a specific configuration is very difficult even with high-fidelity models. Conversely, controllers can be designed to achieve specific configurations, but they can be trapped in undesirable local minima owing to obstacles. Our approach consists of three components: (1) a global motion planner to generate gross motion of the deformable object; (2) a local controller for refinement of the configuration of the deformable object; and (3) a novel deadlock prediction algorithm to determine when to use planning versus control. By separating planning from control we are able to use different representations of the deformable object, reducing overall complexity and enabling efficient computation of motion. We provide a detailed proof of probabilistic completeness for our planner, which is valid despite the fact that our system is underactuated and we do not have a steering function. We then demonstrate that our framework is able to successfully perform several manipulation tasks with rope and cloth in simulation, which cannot be performed using either our controller or planner alone. These experiments suggest that our planner can generate paths efficiently, taking under a second on average to find a feasible path in three out of four scenarios. We also show that our framework is effective on a 16-degree-of-freedom physical robot, where reachability and dual-arm constraints make the planning more difficult.

Fast Model-Based Contact Patch and Pose Estimation for Highly Deformable Dense-Geometry Tactile Sensors

Modeling deformable contact is a well-known problem in soft robotics and is particularly challenging for compliant interfaces that permit large deformations. We present a model for the behavior of a highly deformable dense geometry sensor in its interaction with objects; the forward model predicts the elastic deformation of a mesh given the pose and geometry of a contacting rigid object. We use this model to develop a fast approximation to solve the inverse problem: estimating the contact patch when the sensor is deformed by arbitrary objects. This inverse model can be easily identified through experiments and is formulated as a sparse Quadratic Program (QP) that can be solved efficiently online. The proposed model serves as the first stage of a pose estimation pipeline for robot manipulation. We demonstrate the proposed inverse model through real-time estimation of contact patches on a contact-rich manipulation problem in which oversized fingers screw a nut onto a bolt, and as part of a complete pipeline for pose-estimation and tracking based on the Iterative Closest Point (ICP) algorithm. Our results demonstrate a path towards realizing soft robots with highly compliant surfaces that perform complex real-world manipulation tasks.

Mirroring without Overimitation: Learning Functionally Equivalent Manipulation Actions

This paper presents a mirroring approach, inspired by the neuroscience discovery of the mirror neurons, to transfer demonstrated manipulation actions to robots. Designed to address the different embodiments between a human (demonstrator) and a robot, this approach extends the classic robot Learning from Demonstration (LfD) in the following aspects:i) It incorporates fine-grained hand forces collected by a tactile glove in demonstration to learn robot’s fine manipulative actions; ii) Through model-free reinforcement learning and grammar induction, the demonstration is represented by a goal-oriented grammar consisting of goal states and the corresponding forces to reach the states, independent of robot embodiments; iii) A physics-based simulation engine is applied to emulate various robot actions and mirrors the actions that are functionally equivalent to the human’s in the sense of causing the same state changes by exerting similar forces. Through this approach, a robot reasons about which forces to exert and what goals to achieve to generate actions (i.e., mirroring), rather than strictly mimicking demonstration (i.e., overimitation). Thus the embodiment difference between a human and a robot is naturally overcome. In the experiment, we demonstrate the proposed approach by teaching a real Baxter robot with a complex manipulation task involving haptic feedback—opening medicine bottles.

Human-Inspired Online Path Planning and Biped Walking Realization in Unknown Environment.

The focus of research in biped locomotion has moved toward real-life scenario applications, like walking on uneven terrain, passing through doors, climbing stairs and ladders. As a result, we are witnessing significant advances in the locomotion of biped robots, enabling them to move in hazardous environments while simultaneously accomplishing complex manipulation tasks. Yet, considering walking in an unknown environment, the efficiency of humanoid robots is still far from being comparable with the human. Currently, bipeds are very sensitive to external changes and they have severe constraints for adaptation of walk to conditions from such a complex environment. Promising approaches for efficient generation and realization of walking in a complex environment are based on biological solutions that have been developed for many years of evolution. This work presents one such human-inspired methodology for path planning and realization of biped walk appropriate for motion in a complex unfamiliar environment. Path planning results in calculating clothoid curves that represent well the human-like walking path. The robot walk is realized by the composition of parametric motion primitives. Such an approach enables on-line modification of planned path and walk parameters at any moment, instantly. To establish the relationship between high-level path planner and the low-level joint motion realization, we had to find a way to extract the parameters of the clothoid paths that can be linked with the parameters of the walk and consequently to motion primitive parameters. This enabled the robot to adopt its walking for avoiding the obstacles and for a smooth transition between different paths. In this paper we provide a complete framework that integrates the following components: (i) bio-inspired online path planning, (ii) path-dependent automatic calculation of high-level gait parameters (step length, walking speed, direction, and the height of the foot sole), and (iii) automatic calculation of low-level joint movements and corresponding control terms (driving motor voltage) through the adaptation of motion primitives which realize walking pattern and preserves the dynamic balance of the robot.

Shared control-based bimanual robot manipulation.

Human-centered environments provide affordances for and require the use of two-handed, or bimanual, manipulations. Robots designed to function in, and physically interact with, these environments have not been able to meet these requirements because standard bimanual control approaches have not accommodated the diverse, dynamic, and intricate coordinations between two arms to complete bimanual tasks. In this work, we enabled robots to more effectively perform bimanual tasks by introducing a bimanual shared-control method. The control method moves the robot's arms to mimic the operator's arm movements but provides on-the-fly assistance to help the user complete tasks more easily. Our method used a bimanual action vocabulary, constructed by analyzing how people perform two-hand manipulations, as the core abstraction level for reasoning about how to assist in bimanual shared autonomy. The method inferred which individual action from the bimanual action vocabulary was occurring using a sequence-to-sequence recurrent neural network architecture and turned on a corresponding assistance mode, signals introduced into the shared-control loop designed to make the performance of a particular bimanual action easier or more efficient. We demonstrate the effectiveness of our method through two user studies that show that novice users could control a robot to complete a range of complex manipulation tasks more successfully using our method compared to alternative approaches. We discuss the implications of our findings for real-world robot control scenarios.