Deformable Linear Objects Research Articles

Rearranging Deformable Linear Objects for Implicit Goals with Self‐Supervised Planning and Control

The robotic manipulation of deformable linear objects is a frontier problem with many potential applications in diverse industries. However, most existing research in this area focuses on shape control for a provided explicit goal and does not consider physical constraints, which limits its applicability in many real‐world scenarios. In this study, a self‐supervised planning and control approach are proposed to address the challenge of rearranging deformable linear objects for implicit goals. Specifically, the context of making both ends of the object reachable (inside the robotic access range) and graspable (outside potential collision regions) by dual‐arm robots is considered. Firstly, the object is described with sequential keypoints and the correspondence‐based action is parameterized. Secondly, a generator capable of producing multiple explicit targets is developed, which adhere to implicit conditions. Thirdly, value models are learnt to assign the most promising explicit target as guidance and determine the goal‐conditioned action. All models within the policy are trained in a self‐supervised manner based on data collected from simulations. Importantly, the learned policy can be directly applied to real‐world settings since we do not rely on accurate dynamic models. The performance of the new method is validated with simulations and real‐world experiments.

Obstacle avoidance shape control of deformable linear objects with online parameters adaptation based on differentiable simulation

The manipulation of deformable linear objects (DLOs) such as ropes, cables, and hoses by robots has promising applications in various fields such as product assembly and surgical suturing. However, DLOs are more difficult to manipulate than rigid objects because their shape changes during manipulation. Furthermore, preventing a DLO from colliding with the environment is important to prevent it from becoming entangled and causing shape control to fail. In this paper, we proposed an obstacle avoidance and shape control scheme for DLOs based on differentiable simulation that does not require prior data or a specialized controller. First, we established a dynamic model of the DLO that allows for both forward dynamics transfer and error backpropagation to obtain gradients. Then, we employed model predictive control to optimize the embedded neural network for predicting the actions that would manipulate the DLO. Finally, the control scheme was made applicable to DLOs with different material properties by allowing online adaptation of the model parameters essential to deformation during manipulation. Simulations and real-world experiments demonstrate that the proposed control scheme could manipulate the DLO stably and accurately to avoid obstacles and achieve the goal state. In addition, the online adaptation of parameters helped mitigate the sim-to-real gap.

Generalizable whole-body global manipulation of deformable linear objects by dual-arm robot in 3-D constrained environments

Generalizable whole-body global manipulation of deformable linear objects by dual-arm robot in 3-D constrained environments

Contrastive Prediction and Estimation of Deformable Objects based on Improved Resnet

Because the dynamic model of deformable linear object is complex, the learning based on visual model is difficult, and the feature information extraction is insufficient. Therefore, we propose a joint visual representation model using contrast learning of optimized encoder. We start with the encoder, add the residual structure to the encoder, optimize the extraction and compression of its feature information, and control its parameters to 3 million. In this way, we can not only obtain excellent feature information, but also have good efficiency. In the rope experiment, we collect information from the simulated environment without manual marking, extract features through the encoder and transmit them to the downstream task. Experiments show that the evaluation of our model at 135 ° and 45 ° is improved by about 50%.

Shape Control of Elastic Deformable Linear Objects for Robotic Cable Assembly

Currently, cables are manually installed in aircraft manufacturing scenarios. The utilization of robots in cable assemblies can enhance efficiency and ensure quality, presenting great promise for the future. However, the assembly process must control the shape of the cables, which is a significant challenge for robots. On the one hand, cables have high degrees of freedom in space, making accurate modeling of cable dynamics for robotic arm manipulation difficult; on the other hand, cable deformation has uncertainty, and the applied forces cause simultaneous deformation and motion, which is difficult to control. To address these problems, this study proposes a cable‐shape control method based on graph neural networks and online visual shape‐servoing. The method first approximates cable dynamics with a graph neural network. Then, in practice, the learnt model is used alongside visual‐based shape‐servoing to generate optimal robotic arm movements, controlling the cable to attain the desired shape. In the experiments, precise control of three different cable types is realized, and an example of a completed cable assembly is shown.

Robotic co-manipulation of deformable linear objects for large deformation tasks

This research addresses the challenge of large/complex deformation in the shape control tasks of Deformable Linear Objects (DLO). We propose a collaborative approach using two manipulators to achieve shape control of a DLO in 2D workspace. The proposed methodology introduces an innovative Intermediary Shapes Generation (ISG) algorithm which outputs a series of intermediary shapes to guide the DLO towards the desired shape. The robot controller is formulated as an optimization problem, where the main objective is to minimize the error between the current shape and the desired shape, while ensuring the diminishing rigidity property of the DLO as a constraint. We conduct extensive simulations and real-life experiments to evaluate the effectiveness of our approach. We consider various scenarios of basic shapes, as well as complex deformations with opposite concavities between initial and final shapes. The outcomes demonstrate the robustness and high accuracy of the proposed system in achieving complex deformations. This capability represents the primary contribution of our research. The optimization-based control framework, coupled with the ISG algorithm, enables effective shape control without the need for extensive modeling nor training, and offers a promising solution for practical applications requiring precise shape control of DLOs. Moreover, we carry out a thorough review and comparative analysis encompassing the latest literature in DLO shape control, and the techniques for DLO modeling.

Manipulation Planning for Cable Shape Control

The control of deformable linear objects (DLOs) such as cables presents a significant challenge for robotic systems due to their unpredictable behavior during manipulation. This paper introduces a novel approach for cable shape control using dual robotic arms on a two–dimensional plane. A discrete point model is utilized for the cable, and a path generation algorithm is developed to define intermediate cable shapes, facilitating the transformation of the cable into the desired profile through a formulated optimization problem. The problem aims to minimize the discrepancy between the cable configuration and the targeted shape to ensure an accurate and stable deformation process. Moreover, a cable dynamic model is developed in which the manipulation approach is validated using this model. Additionally, the approach is tested in a simulation environment in which a framework of two manipulators grasps a cable. The results demonstrate the feasibility and accuracy of the proposed method, offering a promising direction for robotic manipulation of cables.

Reactive human–robot collaborative manipulation of deformable linear objects using a new topological latent control model

Real-time reactive manipulation of deformable linear objects is a challenging task that requires robots to quickly and adaptively respond to changes in the object’s deformed shape that result from external forces. In this paper, a novel approach is proposed for real-time reactive deformable linear object manipulation in the context of human–robot collaboration. The proposed approach combines a topological latent representation and a fixed-time sliding mode controller to enable seamless interaction between humans and robots. The introduced topological control model offers a framework for controlling the dynamic shape of deformable objects. By leveraging the topological representation, our approach captures the connectivity and structure of the objects’ shapes within a latent space. This enables improved generalization and performance when handling complex deformable shapes. A fixed-time sliding mode controller ensures that the object is manipulated in real-time, while also ensuring that it remains accurate and stable during the manipulation process. To validate our proposed framework, we first conduct motor-robot experiments to simulate fixed human interaction processes, enabling straightforward comparisons with other approaches. We then follow up with human–robot experiments to demonstrate the effectiveness of our approach.

Untangling Multiple Deformable Linear Objects in Unknown Quantities With Complex Backgrounds

The manipulation of deformable objects, especially deformable linear objects (DLOs), represents an open challenge in robotics. The keys to solving the problem are accurately recognizing the topological model describing the entwined state of DLO(s) and a manipulation strategy based on it. The situation becomes more challenging in practical applications since the DLO(s) may be placed under complex and unencountered environments, making distinguishing between the targets and the background difficult. In addition, with the information obtained from a sensor system with a limited field of sense, a robot has to treat the encountered DLO(s) as multiple ones of an unknown quantity. This paper proposes a solution based on deep learning techniques for these complicated scenarios. The approach can derive a topological model describing the entangled structure of single or multiple DLOs. Based on the model, we proposed a strategy for untangling the DLO(s), considering both the possible self-tangling in one DLO and the tangling between multiple DLOs. The strategy ensures that the entangled DLO(s) can be arranged to be the neatest state given a limited field of view. The feasibility and effectiveness of the proposed solution were verified by untangling experiments utilizing a dual-arm robot system. Even if the exact quantity of the DLO(s) is unknown, the robots can still untangle the DLO(s). Moreover, the proposed approach performed robustness to unfamiliar background textures, which is preferable in practical applications. Dataset used in this paper can be found at https://github.com/lancexz/dlos-dataset. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article proposes an automatic solution for untangling an unknown quantity of DLO(s) using robots. Issues concerned with complex backgrounds, an unknown quantity of DLOs, and limits of field of view, which are easily encountered in practical applications, are solved. Experiments demonstrated the feasibility and robustness of the approach in different scenarios. Potential applications for the proposed method include pipeline bin-picking tasks, rope-like object manipulation in household robots, and cable assembly in industrial areas. Moreover, the topological state recognition process can also be utilized in knot-tying studies and producing datasets. As the current system can not deal with scenes where DLO segments or crossings severely overlap, practitioners should integrate real-time tracking, deformation control method, and unqualified cases detection to avoid the appearance of such cases during manipulation. Future works should also consider the rebound caused by the elastic potential of DLOs.

Identification of Deformable Linear Object Dynamics from Input-output Measurements in 3D Space

Controlling deformable linear objects requires reliable models that capture their complex and high-dimensional dynamics. This paper aims to obtain accurate state-space models of such objects from input-output measurements only, tailored for the use in a model predictive control setting. The proposed identification approach is initialized by a physics-inspired baseline model which approximates a deformable linear object by a serial chain of rigid bodies connected via elastic joints. The nonlinear residual dynamics of the baseline model are then modeled by an artificial neural network, where a key element is the so-called guided residual search that first infers the latent residual and state trajectories in the time domain, in this way facilitating supervised learning of the neural network. The efficacy of the modeling approach is demonstrated on experimental data of the 3D end-point motion of a flexible aluminum rod, excited by a 7-axis Franka Panda robot arm. Compared to a physics-inspired baseline model, the proposed method achieves a 14% prediction performance increase, and compared to a fully neural network-based approach with a similar number of parameters, it achieves a 58% improvement, while also reporting a training time that is more than 5 times faster.

DLO Perceiver: Grounding Large Language Model for Deformable Linear Objects Perception

DLO Perceiver: Grounding Large Language Model for Deformable Linear Objects Perception

Closed-loop Shape Control of Deformable Linear Objects Based on Cosserat Model

Closed-loop Shape Control of Deformable Linear Objects Based on Cosserat Model

Cut-paste image generation for instance segmentation for robotic picking of industrial parts

Vision-based robotic picking enables automation of commissioning and sortation of disordered parts. To locate parts for grasping, state-of-the-art approaches rely on convolutional neural networks for instance segmentation in 2D images. However, this requires sufficiently large training datasets, which are expensive to capture and annotate. Therefore, training with synthetic data is promising as the data can be generated automatically. We present an approach for the cut-paste method to create synthetic images for industrial use cases. With this approach, an end-user first prepares the image generation with just a smartphone and about 20 minutes of manual effort. Then, a versatile dataset with instance segmentation labels is generated automatically. In addition, a procedure for grasp pose computation is applied to enable robotic picking based on instance segmentation. For evaluation, training data is generated for a wide range of rigid parts and deformable linear objects. Testing with real-world data and practical experiments demonstrates the effectiveness of the proposed cut-paste method for industrial applications.

Sim2Real Neural Controllers for Physics-Based Robotic Deployment of Deformable Linear Objects

Deformable linear objects (DLOs), such as rods, cables, and ropes, play important roles in daily life. However, manipulation of DLOs is challenging as large geometrically nonlinear deformations may occur during the manipulation process. This problem is made even more difficult as the different deformation modes (e.g., stretching, bending, and twisting) may result in elastic instabilities during manipulation. In this paper, we formulate a physics-guided data-driven method to solve a challenging manipulation task—accurately deploying a DLO (an elastic rod) onto a rigid substrate along various prescribed patterns. Our framework combines machine learning, scaling analysis, and physical simulations to develop a physics-based neural controller for deployment. We explore the complex interplay between the gravitational and elastic energies of the manipulated DLO and obtain a control method for DLO deployment that is robust against friction and material properties. Out of the numerous geometrical and material properties of the rod and substrate, we show that only three non-dimensional parameters are needed to describe the deployment process with physical analysis. Therefore, the essence of the controlling law for the manipulation task can be constructed with a low-dimensional model, drastically increasing the computation speed. The effectiveness of our optimal control scheme is shown through a comprehensive robotic case study comparing against a heuristic control method for deploying rods for a wide variety of patterns. In addition to this, we also showcase the practicality of our control scheme by having a robot accomplish challenging high-level tasks such as mimicking human handwriting, cable placement, and tying knots.

RT-DLO: Real-Time Deformable Linear Objects Instance Segmentation

Deformable Linear Objects (DLOs) such as cables, wires, ropes, and elastic tubes are numerously present both in domestic and industrial environments. Unfortunately, robotic systems handling DLOs are rare and have limited capabilities due to the challenging nature of perceiving them. Hence, we propose a novel approach named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RT-DLO</i> for real-time instance segmentation of DLOs. First, the DLOs are semantically segmented from the background. Afterward, a novel method to separate the DLO instances is applied. It employs the generation of a graph representation of the scene given the semantic mask where the graph nodes are sampled from the DLOs center-lines whereas the graph edges are selected based on topological reasoning. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RT-DLO</i> is experimentally evaluated against both DLO-specific and general-purpose instance segmentation deep learning approaches, achieving overall better performances in terms of accuracy and inference time.

An approach based on machine vision for the identification and shape estimation of deformable linear objects

The automation of processes that handle deformable materials, and in particular Deformable Linear Objects (DLOs), such as cables, ropes, and sutures; is a challenging task. Due to their properties, it is very difficult to predict the shape of these objects, making indispensable the use of perception systems for their manipulation. However, the detection of a DLO is a non-trivial task, and it can be even more complicated when additional considerations are made, such as detecting multiple DLOs, with small distances between them or even adjacent to each other, and with occlusions and entanglements between them. In this paper, a novel machine vision approach for estimating the shape of DLOs is proposed to address all these challenges. This approach processes the different DLOs in the image sequentially, repeating the following procedure for each of them. First, the DLO is segmented by examining the colors and edges in the image. Next, the remaining pixels are analyzed using evaluation windows to identify a series of points along the DLO’s skeleton. These points are then employed to model the DLO’s shape using a polynomial function. Finally, the output is evaluated by an unsupervised self-critique module, which validates the results, or fine-tunes the system’s parameters and repeats the process. The performance of the system was tested with several wiring harnesses, detecting all their cables in homogeneous and complex backgrounds, with adjacent cables, and with occlusions. The results show an outstanding performance, with a successful shape estimation rate of more than 90% for some of the system configurations.

Mass spring model for non-uniformed deformable linear object toward dexterous manipulation

Mass spring model for non-uniformed deformable linear object toward dexterous manipulation

Robotic Perception-Motion Synergy for Novel Rope Wrapping Tasks

This paper introduces a novel and general method to address the problem of using a general-purpose robot manipulator with a parallel gripper to wrap a deformable linear object (DLO), called a rope, around a rigid object, called a rod, autonomously. Such a robotic wrapping task has broad potential applications in automotive, electromechanical industries construction manufacturing, etc., but has hardly been studied. Our method does not require prior knowledge of the physical and geometrical properties of the objects but enables the robot to use real-time RGB-D perception to determine the wrapping state and feedback control to achieve high-quality results. As such, it provides the robot manipulator with the general capabilities to handle wrapping tasks of different rods or ropes. We tested our method on 6 combinations of 3 different ropes and 2 rods. The result shows that the wrapping quality improved and converged within 5 wraps for all test cases.

Learning Graph Dynamics With External Contact for Deformable Linear Objects Shape Control

This paper focuses on the shape control manipulation of deformable linear objects (DLO) with a dual-arm robotic system. One significant challenge of DLO shape control is the underactuated control system, which means that finite robotic manipulators can not fully control DLO's shape due to the lack of sufficient constraints on DLO. We propose a novel DLO shape control framework aiming to stably control DLO's shape in a wide deformation range, which innovatively provides additional constraints on DLO via external contact. The proposed framework consists of a learning module named <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"/> <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>Learning Graph Dynamics with External Contact for deformable linear objects</b></i> <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"/> (LG-DEC) and an action generation module. The LG-DEC uses a graph convolution network to approximate the dynamics between DLO and contactable objects based on keypoint-based representation. In the action generation module, the learned graph dynamics is used to predict whether the randomly sampled actions can drive DLO closer to the desired shape and output the best action for DLO shape control. In simulation, the proposed framework accomplishes the DLO shape control task with external contact in six scenarios. Furthermore, the proposed framework can be well applied to a physical robotic system for the shape control on different types of DLO. Videos are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://sites.google.com/view/lg-dec</uri> .

Grasp Analysis for the Robot-Based Manipulation of Pre-Assembled Cables with Electrical Connectors

The mounting of pre-assembled cables with electrical connectors is mainly carried out manually in industry today. An exemplary application is the interconnection of battery modules. Automation of such assembly tasks offers the potential for increasing efficiency but requires the design of suitable gripper systems. This is challenging as the cable induces state-dependent forces and torques on the gripper system, which must be transmitted via the complex surface geometries of the plugs. Currently, the required grasp force cannot be determined in advance but only after prototypes have been manufactured and with elaborate physical experiments. To overcome these drawbacks, we present a methodology for the grasp analysis of pre-assembled cables with electrical connectors. The novelty of this approach is to combine a physics simulation for deformable linear objects with a contact model for non-planar grasping surfaces. The results indicate that the cable deformation significantly affects the required grasp force. In addition, each combination of contact surface and dynamic cable deformation results in an individual grasp force course. The methodology enables comparison of different electrical connectors and their grasping surfaces, as well as cables and their manipulation paths, efficiently and with little expert knowledge.