Robotic Manipulation Tasks Research Articles

Prioritized Hindsight with Dual Buffer for Meta-Reinforcement Learning

Sharing prior knowledge across multiple robotic manipulation tasks is a challenging research topic. Although the state-of-the-art deep reinforcement learning (DRL) algorithms have shown immense success in single robotic tasks, it is still challenging to extend these algorithms to be applied directly to resolve multi-task manipulation problems. This is mostly due to the problems associated with efficient exploration in high-dimensional state and continuous action spaces. Furthermore, in multi-task scenarios, the problem of sparse reward and sample inefficiency of DRL algorithms is exacerbated. Therefore, we propose a method to increase the sample efficiency of the soft actor-critic (SAC) algorithm and extend it to a multi-task setting. The agent learns a prior policy from two structurally similar tasks and adapts the policy to a target task. We propose a prioritized hindsight with dual experience replay to improve the data storage and sampling technique, which, in turn, assists the agent in performing structured exploration that leads to sample efficiency. The proposed method separates the experience replay buffer into two buffers to contain real trajectories and hindsight trajectories to reduce the bias introduced by the hindsight trajectories in the buffer. Moreover, we utilize high-reward transitions from previous tasks to assist the network in easily adapting to the new task. We demonstrate the proposed method based on several manipulation tasks using a 7-DoF robotic arm in RLBench. The experimental results show that the proposed method outperforms vanilla SAC in both a single-task setting and multi-task setting.

Using meta-reasoning for incremental repairs in multi-object robot manipulation tasks

Robots tasked with object assembly by manipulation of parts require not only a high-level plan for order of placement of parts but also detailed low-level information on how to place and pick the part based on its state. This is a complex multi-level problem prone to failures at various levels. This paper employs meta reasoning architecture along with robotics principles and proposes dual encoding of state expectations during the progression of task to ground nominal scenarios. We present our results on table-top scenario using perceptual expectations based in the concept of occupancy grids and key point representations. Our results in a constrained manipulation setting suggest using low-level information or high-level expectations alone the system performs worse than if the architecture uses them both. We then outline a complete architecture and system which tackles this problem for repairing more generic assembly plans with objects moving in spaces with 6 degrees of freedom.

Complex Robotic Manipulation via Graph-Based Hindsight Goal Generation.

Reinforcement learning algorithms, such as hindsight experience replay (HER) and hindsight goal generation (HGG), have been able to solve challenging robotic manipulation tasks in multigoal settings with sparse rewards. HER achieves its training success through hindsight replays of past experience with heuristic goals but underperforms in challenging tasks in which goals are difficult to explore. HGG enhances HER by selecting intermediate goals that are easy to achieve in the short term and promising to lead to target goals in the long term. This guided exploration makes HGG applicable to tasks in which target goals are far away from the object's initial position. However, the vanilla HGG is not applicable to manipulation tasks with obstacles because the Euclidean metric used for HGG is not an accurate distance metric in such an environment. Although, with the guidance of a handcrafted distance grid, grid-based HGG can solve manipulation tasks with obstacles, a more feasible method that can solve such tasks automatically is still in demand. In this article, we propose graph-based hindsight goal generation (G-HGG), an extension of HGG selecting hindsight goals based on shortest distances in an obstacle-avoiding graph, which is a discrete representation of the environment. We evaluated G-HGG on four challenging manipulation tasks with obstacles, where significant enhancements in both sample efficiency and overall success rate are shown over HGG and HER. Videos can be viewed at https://videoviewsite.wixsite.com/ghgg.

Review of Learning-Based Robotic Manipulation in Cluttered Environments.

Robotic manipulation refers to how robots intelligently interact with the objects in their surroundings, such as grasping and carrying an object from one place to another. Dexterous manipulating skills enable robots to assist humans in accomplishing various tasks that might be too dangerous or difficult to do. This requires robots to intelligently plan and control the actions of their hands and arms. Object manipulation is a vital skill in several robotic tasks. However, it poses a challenge to robotics. The motivation behind this review paper is to review and analyze the most relevant studies on learning-based object manipulation in clutter. Unlike other reviews, this review paper provides valuable insights into the manipulation of objects using deep reinforcement learning (deep RL) in dense clutter. Various studies are examined by surveying existing literature and investigating various aspects, namely, the intended applications, the techniques applied, the challenges faced by researchers, and the recommendations adopted to overcome these obstacles. In this review, we divide deep RL-based robotic manipulation tasks in cluttered environments into three categories, namely, object removal, assembly and rearrangement, and object retrieval and singulation tasks. We then discuss the challenges and potential prospects of object manipulation in clutter. The findings of this review are intended to assist in establishing important guidelines and directions for academics and researchers in the future.

End-to-End Learning to Grasp via Sampling From Object Point Clouds

The ability to grasp objects is an essential skill that enables many robotic manipulation tasks. Recent works have studied point cloud-based methods for object grasping by starting from simulated datasets and have shown promising performance in real-world scenarios. Nevertheless, many of them still rely on ad-hoc geometric heuristics to generate grasp candidates, which fail to generalize to objects with significantly different shapes with respect to those observed during training. Several approaches exploit complex multi-stage learning strategies and local neighborhood feature extraction while ignoring semantic global information. Furthermore, they are inefficient in terms of number of training samples and time required for inference. In this letter, we propose an end-to-end learning solution to generate 6-DOF parallel-jaw grasps starting from the 3D partial view of the object. Our Learning to Grasp (L2G) method gathers information from the input point cloud through a new procedure that combines a differentiable sampling strategy to identify the visible contact points, with a feature encoder that leverages local and global cues. Overall, L2G is guided by a multi-task objective that generates a diverse set of grasps by optimizing contact point sampling, grasp regression, and grasp classification. With a thorough experimental analysis, we show the effectiveness of L2G as well as its robustness and generalization abilities.

3D Contact Point Cloud Reconstruction From Vision-Based Tactile Flow

With the growing interest in vision-based tactile sensors, various types of sensors that utilize digital imaging are being developed. Among them, a group of sensors captures the tactile flow resulted from the contact deformation using the optical flow algorithm from computer vision and achieves full resolution deformation tracking on the tactile surface. In this work, a novel 3D contact reconstruction algorithm is proposed and evaluated. It exploits the contact geometry and projection relationship in the tactile flow, which are versatile for vision-based tactile sensors, unique for tactile perception but not inherited from computer vision. The resulted 3D contact point cloud representation is consistent with the tactile flow constraint, scale estimation, and contact edge estimation. It can be directly manipulated in downstream applications such as contact force estimation and contact pose estimation. Experiments and examples are provided that indicate the potential for the proposed tactile processing algorithm to connect tactile perception to tactile enabled robotic manipulation tasks.

Learning From Demonstrations Via Multi-Level and Multi-Attention Domain-Adaptive Meta-Learning

Despite significant advances in few-shot classification, object detection, or speech recognition in recent years, training an effective robot to adapt to previously unseen environments in a small data regime is still a long-lasting problem for learning from demonstrations (LfD). A promising solution is meta-learning. However, we notice that simply constructing a model with a more complicated and deeper network via previous meta-learning methods does not perform well as we expected. One possible reason is that the shallow features are gradually lost as the network deepens, while these shallow features play an essential role in the adaptation process of meta-learning. Thus, we present a novel yet effective Multi-Level and Multi-Attention Domain-Adaptive Domain-Adaptive Meta-Learning (MLMA-DAML) framework, which meta-learns multiple visual features via different attention heads to update the model policy. Once the model is updated, our MLMA-DAML predicts robot actions (e.g., positions of end-effectors) via fully connected layers (FCL). As we notice that directly converting visual signals to robot actions via FCL following prior methods is not robust to perform robot manipulation tasks, we further extend our MLMA-DAML to MLMA-DAML++. The proposed MLMA-DAML++ learns an effective representation of manipulation tasks via an extra goal prediction network with convolutional layers (CL) to predict more reliable robot actions (represented by feature pixels/grids). Extensive experiments on a UR5 robot arm demonstrate that our proposed methods significantly outperform current related state-of-the-art methods in different real-world placing settings.

Spatial and temporal features unified self-supervised representation learning networks

Robot manipulation tasks can be carried out effectively, provided the state representation is satisfactorily detailed. Embodiment difference, Viewpoint difference, and Domain difference are some of the challenges in learning from human demonstration. This work proposes a self-supervised and multi-viewpoint spatial and temporal features unified representation learning method. The algorithm consists of two components: (a) Spatial Component, which learns the setting of the environment, i.e., on which pixels to focus on most to get the best representation of the image regardless of point of view, and (b) Temporal Component that learns how snapshots taken from multiple viewpoints simultaneously (i.e., at the same time-step but from a different viewpoint) are similar and how these snaps are different from snaps taken at a different time-step but same viewpoint. Further, these representations are integrated with the Reinforcement Learning (RL) framework to learn accurate behaviors from videos of humans performing the manipulation task. The effectiveness of this approach is illustrated by training the robots to learn various manipulation tasks i.e., (a) grab objects (b) lift objects (c) open and close drawers from expert demonstrations provided by humans. The algorithm shows great promise and is highly successful across all the manipulation tasks. The robot learns to pick up objects of various shapes, sizes and colors having different orientations and placements on the table. The robot also successfully learns how to open and close drawers. The method is highly sample efficient and addresses the challenges of embodiment, viewpoint, and domain difference.

CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks

General-purpose robots coexisting with humans in their environment must learn to relate human language to their perceptions and actions to be useful in a range of daily tasks. Moreover, they need to acquire a diverse repertoire of general-purpose skills that allow composing long-horizon tasks by following unconstrained language instructions. In this letter, we present Composing Actions from Language and Vision (CALVIN) ( <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</u> omposing <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</u> ctions from <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</u> anguage and <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Vi</u> sio <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</u> ), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites. We evaluate the agents in zero-shot to novel language instructions and to novel environments. We show that a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents that learn to relate human language to their world models with this benchmark.

Object affordance detection with boundary-preserving network for robotic manipulation tasks

Object affordance detection with boundary-preserving network for robotic manipulation tasks

Scalable Fabrication of Bioinspired Controllable Dry Adhesive by Roll‐to‐Roll Slitting

Controllable dry adhesives have attracted significant interest; however, their development and application remain limited owing to their high cost, complexity, and machine precision required for their fabrication. Herein, a scalable roll‐to‐roll technique for fabricating wedge‐shaped controllable dry adhesives for use in robotic grasping tasks is developed. Wedge‐shaped structures with various geometrical dimensions are manufactured, and the structure with optimal wedge dimension (50 μm width and 30° tip angle) demonstrates peak and stable normal adhesion values of 14 and 6.8 kPa, respectively. Moreover, parallel grippers are designed to grasp everyday objects that have flat surfaces and a variety of geometries and surface textures. This novel scalable, low‐cost, and simple method can be used to manufacture a straight and continuous wedge structure in large scale for controllable dry adhesives that can be effectively applied in robotic manipulation tasks.

Manipulation Task Planning and Motion Control Using Task Relaxations

This paper proposes a robotic manipulation task relaxation method applied to a task and motion planning framework adapted from the literature. The task relaxation method consists of defining regions of interest instead of defining the end-effector pose, which can potentially increase the robot’s redundancy with respect to the manipulation task. Tasks are relaxed by controlling the end-effector distance to a target plane while respecting suitable constraints in the task-space. We formulate the problem as a constrained control problem that enforces both equality and inequality constraints while being reactive to changes in the workspace. We evaluate the adapted framework in a simulated pick-and-place task with similar complexity to the one evaluated in the original framework. The number of plan nodes that our framework generates is 54% smaller than the one in the original framework and our framework is faster both in planning and execution time. Also, the end-effector remains within the regions of interest and moves toward the target region while satisfying additional constraints.

Parameterized Particle Filtering for Tactile-Based Simultaneous Pose and Shape Estimation

Object state and shape estimation is essential in many robotic manipulation tasks (e.g., in-hand manipulation, insertion). While such estimation is typically relied on visual perception, for tasks to be carried out in a vision-degraded or vision-denied environment, haptics becomes the reliable source of perception. In this letter, we propose the use of parameterized particle filtering to estimate object pose and shape in 3D space using tactile feedback. This approach is able to estimate with high accuracy using contact information of the object with a collision surface from a rough initial estimation. In comparison to conventional particle filtering, this approach significantly reduces the number of particles required for a satisfactory estimation, making it applicable for pose and shape estimation, where the number of degrees of freedom is high or even uncertain. Moreover, the proposed method can automatically choose the fastest-convergent contact action during the pose estimation stage to shorten the time required. A set of experiments in both simulation and on a real-world robot have been conducted to validate the proposed method and compare against the state-of-the-art approach in the literature. Results from both sets of experiments show that the proposed method can determine the pose and shape of the objects with very high accuracy within a small number of iterations.

Bottom-Up Skill Discovery From Unsegmented Demonstrations for Long-Horizon Robot Manipulation

We tackle real-world long-horizon robot manipulation tasks through skill discovery. We present a bottom-up approach to learning a library of reusable skills from unsegmented demonstrations and use these skills to synthesize prolonged robot behaviors. Our method starts with constructing a hierarchical task structure from each demonstration through agglomerative clustering. From the task structures of multi-task demonstrations, we identify skills based on the recurring patterns and train goal-conditioned sensorimotor policies with hierarchical imitation learning. Finally, we train a meta controller to compose these skills to solve long-horizon manipulation tasks. The entire model can be trained on a small set of human demonstrations collected within 30 minutes without further annotations, making it amendable to real-world deployment. We systematically evaluated our method in simulation environments and on a real robot. Our method has shown superior performance over state-of-the-art imitation learning methods in multi-stage manipulation tasks. Furthermore, skills discovered from multi-task demonstrations boost the average task success by 8% compared to those discovered from individual tasks.

Q-Attention: Enabling Efficient Learning for Vision-Based Robotic Manipulation

Despite the success of reinforcement learning methods, they have yet to have their breakthrough moment when applied to a broad range of robotic manipulation tasks. This is partly due to the fact that reinforcement learning algorithms are notoriously difficult and time consuming to train, which is exacerbated when training from images rather than full-state inputs. As humans perform manipulation tasks, our eyes closely monitor every step of the process with our gaze focusing sequentially on the objects being manipulated. With this in mind, we present our Attention-driven Robotic Manipulation (ARM) algorithm, which is a general manipulation algorithm that can be applied to a range of sparse-rewarded tasks, given only a small number of demonstrations. ARM splits the complex task of manipulation into a 3 stage pipeline: (1) a Q-attention agent extracts relevant pixel locations from RGB and point cloud inputs, (2) a next-best pose agent that accepts crops from the Q-attention agent and outputs poses, and (3) a control agent that takes the goal pose and outputs joint actions. We show that current learning algorithms fail on a range of RLBench tasks, whilst ARM is successful. Videos found at: <uri>https://sites.google.com/view/q-attention</uri>.

Correct Me If I am Wrong: Interactive Learning for Robotic Manipulation

Learning to solve complex manipulation tasks from visual observations is a dominant challenge for real-world robot learning. Although deep reinforcement learning algorithms have recently demonstrated impressive results in this context, they still require an impractical amount of time-consuming trial-and-error iterations. In this work, we consider the promising alternative paradigm of interactive learning in which a human teacher provides feedback to the policy during execution, as opposed to imitation learning where a pre-collected dataset of perfect demonstrations is used. Our proposed CEILing (Corrective and Evaluative Interactive Learning) framework combines both corrective and evaluative feedback from the teacher to train a stochastic policy in an asynchronous manner, and employs a dedicated mechanism to trade off human corrections with the robot’s own experience. We present results obtained with our framework in extensive simulation and real-world experiments to demonstrate that CEILing can effectively solve complex robot manipulation tasks directly from raw images in less than one hour of real-world training.

SKP: Semantic 3D Keypoint Detection for Category-Level Robotic Manipulation

As human-assisting robots are starting to play an important role in the food service and hospitality field, service vision for robot manipulation on intra-category objects, which have variations in shape, size, and appearance, is an inevitable and valuable task. Previous works typically rely on estimating the pose of the objects, such methods are not sufficient to generalize well to category-level. Hence, we propose Semantic Keypoint Detection Network (SKP) for category-level robotic manipulation tasks, that 1) uses semantic keypoint representation to expand the perception module to category level and drive the entire grasping pipeline to handle shape variations and dynamic environment, 2) leverages the underlying geometric constraints via jointly optimize instance-aware features with the self-selection mechanism, 3) utilizes multi-scale multi-layer features for 3D keypoint position regression and keypoint association and 4) train with synthetic data only with domain randomization. Extensive experiments on synthetic and real-world datasets show that our work improves accuracy for category-level keypoint detection and demonstrates the generalizability and practicality in robotics grasping tasks.

3D-Printed Soft Sensors for Adaptive Sensing with Online and Offline Tunable Stiffness.

The stiffness of a soft robot with structural cavities can be regulated by controlling the pressure of a fluid to render predictable changes in mechanical properties. When the soft robot interacts with the environment, the mediating fluid can also be considered an inherent information pathway for sensing. This approach to using structural tuning to improve the efficacy of a sensing task with specific states has not yet been well studied. A tunable stiffness soft sensor also renders task-relevant contact dynamics in soft robotic manipulation tasks. This article proposes a type of adaptive soft sensor that can be directly three-dimensional printed and controlled using pneumatic pressure. The tunability of such a sensor helps to adjust the sensing characteristics to better capturing specific tactile features, demonstrated by detecting texture with different frequencies. We present the design, modeling, Finite Element Simulation, and experimental characterization of a single unit of such a tunable stiffness sensor. How the sensing characteristics are affected by adjusting its stiffness is studied in depth. In addition to the tunability, the results show that such types of adaptive sensors exhibit good sensitivity (up to 2.6 KPa/N), high sensor repeatability (average std <0.008 KPa/N), low hysteresis (<6%), and good manufacturing repeatability (average std = 0.0662 KPa/N).

An Enhanced GRU Model With Application to Manipulator Trajectory Tracking

Service robots, e.g. massage robots, have attracted more and more attention in recent years and the most popular study within this field is trajectory tracking. Due to the actual demand for service robots, the solution of trajectory tracking requires fast convergence and high accuracy. In order to solve the above issues, this paper proposed an enhanced Gated recurrent unit (GRU) to deal with trajectory tracking tasks of robot manipulators. The main feature of enhanced GRU is utilizing cell states as well as various gate units to build a novel neural cell. Besides, the presented enhanced GRU resolves the problem of the general neural network model and large memory occupancy. Then the derivations about the computational process of cell state and mixed hidden state of the proposed model have been illustrated. Finally, three trajectory tracking applications, comparison, and visual simulation have verified feasibility as well as the superiority of the enhanced GRU model.

Learning Robotic Manipulation Tasks via Task Progress Based Gaussian Reward and Loss Adjusted Exploration

Multi-step manipulation tasks in unstructured environments are extremely challenging for a robot to learn. Such tasks interlace high-level reasoning that consists of the expected states that can be attained to achieve an overall task and low-level reasoning that decides what actions will yield these states. We propose a model-free deep reinforcement learning method to learn multi-step manipulation tasks. We introduce a Robotic Manipulation Network (RoManNet) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , which is a vision-based model architecture, to learn the action-value functions and predict manipulation action candidates. We define a Task Progress based Gaussian (TPG) reward function that computes the reward based on actions that lead to successful motion primitives and progress towards the overall task goal. To balance the ratio of exploration/exploitation, we introduce a Loss Adjusted Exploration (LAE) policy that determines actions from the action candidates according to the Boltzmann distribution of loss estimates. We demonstrate the effectiveness of our approach by training RoManNet to learn several challenging multi-step robotic manipulation tasks in both simulation and real-world. Experimental results show that our method outperforms the existing methods and achieves state-of-the-art performance in terms of success rate and action efficiency. The ablation studies show that TPG and LAE are especially beneficial for tasks like multiple block stacking.