Manipulation Of Deformable Objects Research Articles

Reactive human–robot collaborative manipulation of deformable linear objects using a new topological latent control model

Real-time reactive manipulation of deformable linear objects is a challenging task that requires robots to quickly and adaptively respond to changes in the object’s deformed shape that result from external forces. In this paper, a novel approach is proposed for real-time reactive deformable linear object manipulation in the context of human–robot collaboration. The proposed approach combines a topological latent representation and a fixed-time sliding mode controller to enable seamless interaction between humans and robots. The introduced topological control model offers a framework for controlling the dynamic shape of deformable objects. By leveraging the topological representation, our approach captures the connectivity and structure of the objects’ shapes within a latent space. This enables improved generalization and performance when handling complex deformable shapes. A fixed-time sliding mode controller ensures that the object is manipulated in real-time, while also ensuring that it remains accurate and stable during the manipulation process. To validate our proposed framework, we first conduct motor-robot experiments to simulate fixed human interaction processes, enabling straightforward comparisons with other approaches. We then follow up with human–robot experiments to demonstrate the effectiveness of our approach.

Untangling Multiple Deformable Linear Objects in Unknown Quantities With Complex Backgrounds

The manipulation of deformable objects, especially deformable linear objects (DLOs), represents an open challenge in robotics. The keys to solving the problem are accurately recognizing the topological model describing the entwined state of DLO(s) and a manipulation strategy based on it. The situation becomes more challenging in practical applications since the DLO(s) may be placed under complex and unencountered environments, making distinguishing between the targets and the background difficult. In addition, with the information obtained from a sensor system with a limited field of sense, a robot has to treat the encountered DLO(s) as multiple ones of an unknown quantity. This paper proposes a solution based on deep learning techniques for these complicated scenarios. The approach can derive a topological model describing the entangled structure of single or multiple DLOs. Based on the model, we proposed a strategy for untangling the DLO(s), considering both the possible self-tangling in one DLO and the tangling between multiple DLOs. The strategy ensures that the entangled DLO(s) can be arranged to be the neatest state given a limited field of view. The feasibility and effectiveness of the proposed solution were verified by untangling experiments utilizing a dual-arm robot system. Even if the exact quantity of the DLO(s) is unknown, the robots can still untangle the DLO(s). Moreover, the proposed approach performed robustness to unfamiliar background textures, which is preferable in practical applications. Dataset used in this paper can be found at https://github.com/lancexz/dlos-dataset. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article proposes an automatic solution for untangling an unknown quantity of DLO(s) using robots. Issues concerned with complex backgrounds, an unknown quantity of DLOs, and limits of field of view, which are easily encountered in practical applications, are solved. Experiments demonstrated the feasibility and robustness of the approach in different scenarios. Potential applications for the proposed method include pipeline bin-picking tasks, rope-like object manipulation in household robots, and cable assembly in industrial areas. Moreover, the topological state recognition process can also be utilized in knot-tying studies and producing datasets. As the current system can not deal with scenes where DLO segments or crossings severely overlap, practitioners should integrate real-time tracking, deformation control method, and unqualified cases detection to avoid the appearance of such cases during manipulation. Future works should also consider the rebound caused by the elastic potential of DLOs.

Deformable Object Manipulation in Caregiving Scenarios: A Review

This paper reviews the robotic manipulation of deformable objects in caregiving scenarios. Deformable objects like clothing, food, and medical supplies are ubiquitous in care tasks, yet pose modeling, control, and sensing challenges. This paper categorises caregiving deformable objects and analyses their distinct properties influencing manipulation. Key sections examine progress in simulation, perception, planning, control, and system designs for deformable object manipulation, along with end-to-end deep learning’s potential. Hybrid analytical data-driven modeling shows promise. While laboratory successes have been achieved, real-world caregiving applications lag behind. Enhancing safety, speed, generalisation, and human compatibility is crucial for adoption. The review synthesises critical technologies, capabilities, and limitations, while also pointing to open challenges in deformable object manipulation for robotic caregiving. It provides a comprehensive reference for researchers tackling this socially valuable domain. In conclusion, multi-disciplinary innovations combining analytical and data-driven methods are needed to advance real-world robot performance and safety in deformable object manipulation for patient care.

Iterative residual policy: For goal-conditioned dynamic manipulation of deformable objects

This paper tackles the task of goal-conditioned dynamic manipulation of deformable objects. This task is highly challenging due to its complex dynamics (introduced by object deformation and high-speed action) and strict task requirements (defined by a precise goal specification). To address these challenges, we present Iterative Residual Policy (IRP), a general learning framework applicable to repeatable tasks with complex dynamics. IRP learns an implicit policy via delta dynamics—instead of modeling the entire dynamical system and inferring actions from that model, IRP learns delta dynamics that predict the effects of delta action on the previously observed trajectory. When combined with adaptive action sampling, the system can quickly optimize its actions online to reach a specified goal. We demonstrate the effectiveness of IRP on two tasks: whipping a rope to hit a target point and swinging a cloth to reach a target pose. Despite being trained only in simulation on a fixed robot setup, IRP is able to efficiently generalize to noisy real-world dynamics, new objects with unseen physical properties, and even different robot hardware embodiments, demonstrating its excellent generalization capability relative to alternative approaches.

Cyber-physical systems in non-rigid assemblies: A methodology for the calibration of deformable object reconstruction models

Despite the advances in robot agent cognition and systems’ decision-making, under the prism of cyber-physical systems and industrial metaverse, the manufacturing processes involving the handling of non-rigid product assemblies present a delay in the adoption of smart automation. Model-based planning and control can address the particularities of deformable object manipulation; however, their competence is heavily dependent on the models’ accuracy and reconstruction frequencies. Despite the many breakthroughs that have been achieved in real-time modelling and behavior prediction of deformable objects, the calibration of such models and the measurement of their accuracy remain a significant challenge. In this paper, a method for the definition of the physics parameters of flexible material reconstruction models is presented. The proposed systematic approach, employing a number of optimization algorithms, fine-tunes the model’s parameters for the real-world deformable object, as captured by the perception system, to be aligned with its digital twin. A mass-spring model for the reconstruction of two-dimensional fabric objects is used as an application paradigm. An experimental setup in an industrially relevant environment validates the applicability of the proposed approach and is used for assessing alternative sensing practices and optimization algorithms.

Robotic Manipulation of Deformable Rope-Like Objects Using Differentiable Compliant Position-Based Dynamics

Robot manipulation of rope-like objects is an interesting problem with some critical applications, such as autonomous robotic suturing. Solving for and controlling rope is difficult due to the complexity of rope physics and the challenge of building fast and accurate models of deformable materials. While more data-driven approaches have become more popular for finding controllers that learn to do a single task, there is still a strong motivation for a model-based method that could solve many optimization problems. Towards this end, we introduced compliant position-based dynamics (XPBD) to model rope-like objects. Using geometric constraints, the model can represent the coupling of shear/stretch and bend/twist effects. Of crucial importance is that our formulation is differentiable, which can solve parameter estimation problems and improve the matching of rope physics to real-life scenarios (i.e., the real-to-sim problem). For the generality of rope-like objects, two different solvers are proposed to handle the inextensible and extensible effects of varied material stiffness for the rope. We demonstrate our framework's robustness and accuracy on real-to-sim experimental setups using the Baxter robot and the da Vinci research kit (DVRK) [1]. Our work leads to a new path for robotic manipulation of the deformable rope-like object taking advantage of the ready-to-use gradients.

Automated Orientation Control of Motile Deformable Cells

Automated manipulation of deformable objects is challenging due to the object&#x2019;s deformation behavior. Different from still deformable objects such as wires and cloth, biological organisms such as sperm and worms are both deformable and motile, requiring the control of both deformation and motion. This paper reports automated orientation control of live sperm, as an example of motile deformable cells. Robotic manipulation of human sperm was performed by using a glass micropipette, which is a standard clinical tool, to rotate individual motile sperm. Sperm rotation must be performed before immobilization, as required in clinical cell surgery for infertility treatment. To control tail deformation during sperm rotation, a path planner was designed based on kinematic analysis and manipulation point update. To deal with the intrinsic motion of a motile sperm, a motorized stage was controlled to compensate for sperm swimming motion, and an observer was designed to decouple sperm orientation from its wiggling motion. A sliding mode controller was designed to cope with stiffness variances along the sperm tail and among different sperm. Deep neural networks were developed for robust sperm tail detection, and Kalman filter was used to predict tail motion. Experimental results demonstrated that automated sperm manipulation achieved an orientation error of 0.8<inline-formula> <tex-math notation="LaTeX">$^{\circ}$</tex-math> </inline-formula> and operation time of 6.8 s, both significantly less than those of manual operation. The designed observer was effective to reduce sperm orientation error by reducing the disturbance from sperm wiggling motion. The developed sliding mode controller outperformed the PID controller in operation time, reducing the time of oocyte exposure to the ambient environment. <i>Note to Practitioners</i>&#x2014;This work tackled the challenge of rotating a fast-swimming and deformable sperm in clinical cell surgeries. Automated manipulation of deformable objects has wide applications in industrial and service settings such as manipulating wires and folding cloth. However, the intrinsic motion of a motile sperm and the lack of a rotational degree of freedom in standard micromanipulators pose difficulties to automated sperm manipulation. In this paper, we propose automation techniques for sperm orientation control. For sperm tail detection, deep learning was used to handle the variances of shape and length among different sperm. A path planning strategy and a controller were designed to achieve automated rotation of motile sperm, with its deformation and motion both controlled. The developed methods can be generalized to the manipulation of other deformable objects such as wires, cables and cloth. These objects exhibit significant variance of mechanical properties, and calibration is often time-consuming. The designed controller can be used to manipulate deformable objects with robustness to varied mechanical parameters. Path planning was designed by updating the manipulation point based on the object&#x2019;s deformation behavior, and is suitable in manipulation where constraints are imposed such as the object&#x2019;s strain.

A review on reinforcement learning for contact-rich robotic manipulation tasks

Research and application of reinforcement learning in robotics for contact-rich manipulation tasks have exploded in recent years. Its ability to cope with unstructured environments and accomplish hard-to-engineer behaviors has led reinforcement learning agents to be increasingly applied in real-life scenarios. However, there is still a long way ahead for reinforcement learning to become a core element in industrial applications. This paper examines the landscape of reinforcement learning and reviews advances in its application in contact-rich tasks from 2017 to the present. The analysis investigates the main research for the most commonly selected tasks for testing reinforcement learning algorithms in both rigid and deformable object manipulation. Additionally, the trends around reinforcement learning associated with serial manipulators are explored as well as the various technological challenges that this machine learning control technique currently presents. Lastly, based on the state-of-the-art and the commonalities among the studies, a framework relating the main concepts of reinforcement learning in contact-rich manipulation tasks is proposed. The final goal of this review is to support the robotics community in future development of systems commanded by reinforcement learning, discuss the main challenges of this technology and suggest future research directions in the domain.

Learning Graph Dynamics With External Contact for Deformable Linear Objects Shape Control

This paper focuses on the shape control manipulation of deformable linear objects (DLO) with a dual-arm robotic system. One significant challenge of DLO shape control is the underactuated control system, which means that finite robotic manipulators can not fully control DLO's shape due to the lack of sufficient constraints on DLO. We propose a novel DLO shape control framework aiming to stably control DLO's shape in a wide deformation range, which innovatively provides additional constraints on DLO via external contact. The proposed framework consists of a learning module named <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"/> <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>Learning Graph Dynamics with External Contact for deformable linear objects</b></i> <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"/> (LG-DEC) and an action generation module. The LG-DEC uses a graph convolution network to approximate the dynamics between DLO and contactable objects based on keypoint-based representation. In the action generation module, the learned graph dynamics is used to predict whether the randomly sampled actions can drive DLO closer to the desired shape and output the best action for DLO shape control. In simulation, the proposed framework accomplishes the DLO shape control task with external contact in six scenarios. Furthermore, the proposed framework can be well applied to a physical robotic system for the shape control on different types of DLO. Videos are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://sites.google.com/view/lg-dec</uri> .

Multidimensional Deformable Object Manipulation Based on DN-Transporter Networks

In the process of transportation, the handling and loading methods of rigid objects are becoming more and more perfect. However, whether in today’s transportation system or in daily life, such as packing objects or sorting cables before transportation, the manipulation of deformable objects has been always inevitable and has attracted more and more attention. Due to the super degrees of freedom and the unpredictable physical state of deformed objects. It is difficult for robots to complete tasks under the environment of the deformable object. Therefore, we present a method based on imitation learning. In the generated expert demonstration, the agent is offered to learn the state sequence, and then imitate the expert’s trajectory sequence which avoid the above-mentioned difficulties. In addition, compared with the baseline method, our proposed DN-Transporter Networks are more competitive in a simulation environment involving cloth, ropes or bags.

Interaction Control of a Robotic Manipulator With the Surface of Deformable Object

Robotic manipulation of deformable objects has drawn the attention of researchers over the past few years and is associated with a large spectrum of new application perspectives. In this article, we present an efficient integrated motion planning framework to effectively and accurately control a robotic manipulator executing interactive tasks on the surface of a deformable object. The proposed interactive motion planning framework is based on a mesh representation of the object, integrating three efficient preprocessing algorithmic steps, including visual object segmentation, finite element method deformation tracking, and local mesh parameterization. The use of barycentric coordinates, defined on the mesh triangles, enables the establishment of bijective transformations between the deformable part of an object surface and its planar (static and dynamic) parameterized mapping. By merging these spatial transformations with the preprocessing steps, in combination with an active stiffness scheme for robot manipulator control, we are able to achieve accurate and reactive motion planning of interactive trajectories, even under large and persistent visual occlusions (such as due to the presence of the robot in the visual scene). An extensive experimental evaluation study is presented, involving a robotic manipulator in interaction with a hemispherical model of controllable periodic active deformation, which permits precise ground truth derivation. Motion planning accuracy is evaluated in comparison with our previous direct vision-based approach, showing clearly superior performance of the proposed approach under all experimental conditions. The performance of the proposed framework is also further highlighted in tasks involving physical point tracking, interactive programming by human demonstration, as well as contact force regulation.

A versatile end effector for grabbing and spreading of flaky deformable object manipulation

Abstract. It is a conundrum to use robots to complete the intelligent operation of leather and clothing fabrics; this is mainly reflected in the design of an end effector that can realize this function. We have determined the design requirements and objectives of the end effector by observing the action of manually grabbing and spreading leather, and put forward an end effector composed of a reconfigurable multi-link mechanism and roller-type fingertip, which can be used to grab and spread flaky deformable object manipulation. At the same time, the reconfigurable multi-link mechanism and roller-type fingertip are introduced and analyzed in detail. By setting up the prototype of the end effector and completing the grabbing experiment of three kinds of soft pieces, we verified and tested the rationality of the prototype design. The result shows that the grabbing success rate will be affected by many factors, and further exploration is needed to further improve the performance of the end effector.

Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects

Manipulating cloth-like deformable objects (CDOs) is a long-standing problem in the robotics community. CDOs are flexible (non-rigid) objects that do not show a detectable level of compression strength while two points on the article are pushed towards each other and include objects such as ropes (1D), fabrics (2D) and bags (3D). In general, CDOs' many degrees of freedom (DoF) introduce severe self-occlusion and complex state-action dynamics as significant obstacles to perception and manipulation systems. These challenges exacerbate existing issues of modern robotic control methods such as imitation learning (IL) and reinforcement learning (RL). This review focuses on the application details of data-driven control methods on four major task families in this domain: cloth shaping, knot tying/untying, dressing and bag manipulation. Furthermore, we identify specific inductive biases in these four domains that present challenges for more general IL and RL algorithms.

Task-Oriented Grasping Position Selection in Deformable Object Manipulation

Grasping mechanisms in robotic deformable object manipulation (DOM) remain under-investigated due to the intrinsic complexity of DOM. For grasping in DOM, grasping position selection is crucial since external forces are applied to grasping locations on the deformable object (DO) to make it deform. The feasibility and efficiency of performing a specific DOM task thus highly depend on where grasps are placed. This letter investigates the grasping position selection problem in DOM. In particular, we tackle it in a task-oriented manner to facilitate robot deforming operations. For common deformation description methods (image feature and point cloud) and grasping scenarios (single and dual grasps), we develop grasping position selection methods that take into account the properties of both the task and DO. Specifically, metrics are first constructed to quantify grasping position quality when conducting the deforming process. Algorithms are then proposed based on metrics, which enable robots to select appropriate grasping positions in DOM. We evaluated our methods in both simulations and hardware experiments.

Enabling Visual Action Planning for Object Manipulation Through Latent Space Roadmap

In this article, we present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a latent space roadmap (LSR) for task planning, which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of the following three parts. First, a mapping module (MM) that maps observations is given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states. Second, the LSR, which builds and connects clusters containing similar states in order to find the latent plans between start and goal states, extracted by MM. Third, the action proposal module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on simulated box stacking and rope/box manipulation tasks, and a folding task executed on a real robot.

Sim-to-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery

Automation holds the potential to assist surgeons in robotic interventions, shifting their mental work load from visuomotor control to high level decision making. Reinforcement learning has shown promising results in learning complex visuomotor policies, especially in simulation environments where many samples can be collected at low cost. A core challenge is learning policies in simulation that can be deployed in the real world, thereby overcoming the sim-to-real gap. In this letter, we bridge the visual sim-to-real gap with an image-based reinforcement learning pipeline based on pixel-level domain adaptation and demonstrate its effectiveness on an image-based task in deformable object manipulation. We choose a tissue retraction task because of its importance in clinical reality of precise cancer surgery. After training in simulation on domain-translated images, our policy requires no retraining to perform tissue retraction with a 50% success rate on the real robotic system using raw RGB images. Furthermore, our sim-to-real transfer method makes no assumptions on the task itself and requires no paired images. This letter introduces the first successful application of visual sim-to-real transfer for robotic manipulation of deformable objects in the surgical field, which represents a notable step towards the clinical translation of cognitive surgical robotics.

Global Model Learning for Large Deformation Control of Elastic Deformable Linear Objects: An Efficient and Adaptive Approach

Robotic manipulation of deformable linear objects (DLOs) has broad application prospects in many fields. However, a key issue is to obtain the exact deformation models (i.e., how robot motion affects DLO deformation), which are hard to theoretically calculate and vary among different DLOs. Thus, shape control of DLOs is challenging, especially for large deformation control which requires global and more accurate models. In this paper, we propose a coupled offline and online data-driven method for efficiently learning a global deformation model, allowing for both accurate modeling through offline learning and further updating for new DLOs via online adaptation. Specifically, the model approximated by a neural network is first trained offline on random data, then seamlessly migrated to the online phase, and further updated online during actual manipulation. Several strategies are introduced to improve the model's efficiency and generalization ability. We propose a convex-optimization-based controller, and analyze the system's stability using the Lyapunov method. Detailed simulations and real-world experiments demonstrate that our method can efficiently and precisely estimate the deformation model, and achieve large deformation control of untrained DLOs in 2D and 3D dual-arm manipulation tasks better than the existing methods. It accomplishes all 24 tasks with different desired shapes on different DLOs in the real world, using only simulation data for the offline learning.

Cooperative Manipulation of Deformable Objects by Single-Leader–Dual-Follower Teleoperation

This article proposes a method for single-leader–dual-follower teleoperation, where one robot (direct-follower robot, DFR) is directly teleoperated and the other robot (assisting-follower robot, AFR) can autonomously cooperate with the DFR to hold and move a deformable object, with the contact force regulated to a desired value. Since the AFR does not know its partner’s movement, first, it achieves position alignment with the DFR by using the contact force. Second, we develop an adaptive movement estimation algorithm according to the Lyapunov theory, such that DFR’s position is estimated in the presence of dynamic uncertainties. Finally, we adopt the impedance control to generate a reference trajectory for the AFR to track, in order to maintain the desired contact force. Several simulations are conducted on a platform with two three-degrees-of-freedom robots. Experiments are performed with a dual-arm robot (Baxter), results of which demonstrate the feasibility and effectiveness of the proposed method.

Performance measures to benchmark the grasping, manipulation, and assembly of deformable objects typical to manufacturing applications.

The National Institute of Standards and Technology is developing performance tests and associated artifacts to benchmark research in the area of robotic assembly. Sets of components consistent with mechanical assemblies including screws, gears, electrical connectors, wires, and belts are configured for assembly or disassembly using a task board concept. Test protocols accompany the task boards and are designed to mimic low-volume, high-mixture assembly challenges typical to small and medium sized manufacturers. In addition to the typical rigid components found in assembled products, the task boards include many non-rigid component operations representative of wire harness and belt drive assemblies to support research in the area of grasping and manipulation of deformable objects, an area still considered to be an emerging research problem in robotics. A set of four primary task boards as well as competition task boards are presented as benchmarks along with scoring metrics and a method to compare robot system assembly times with human performance. Competitions are used to raise awareness to these benchmarks. Tools to progress and compare research are described along with emphasis placed on system competition-based solutions to grasp and manipulate deformable task board components.

Autonomous Learning of Page Flipping Movements via Tactile Feedback

Robotic manipulation is challenging when both the objects being manipulated and the tactile sensors are deformable. In this article, we addressed the interplay between the manipulation of deformable objects, tactile sensing, and model-free reinforcement learning on a real robot. We showed how a real robot can learn to manipulate a deformable, thin-shell object via feedback from deformable, multimodal tactile sensors. We addressed the learning of a page flipping task using a two-stage approach. For the first stage, we learned nominal page flipping trajectories for two page sizes by constructing a reward function that quantifies functional task performance from the perspective of tactile sensing. For the second stage, we learned adapted trajectories using tactile-driven perceptual coupling, with an intuitive assumption that, while the page flipping trajectories for different task contexts (page sizes) might differ, similar tactile feedback should be expected from functional trajectories for each context. We also investigated the quality of information encoded by two different representations of tactile sensing data: one based on the artificial apical tuft of bio-inspired tactile sensors, and another based on principal component analysis eigenvalues. The results and effectiveness of our learning framework were demonstrated on a real seven-degree of freedom robot arm and gripper outfitted with tactile sensors.