Manipulation Of Deformable Linear Objects Research Articles

Rearranging Deformable Linear Objects for Implicit Goals with Self‐Supervised Planning and Control

The robotic manipulation of deformable linear objects is a frontier problem with many potential applications in diverse industries. However, most existing research in this area focuses on shape control for a provided explicit goal and does not consider physical constraints, which limits its applicability in many real‐world scenarios. In this study, a self‐supervised planning and control approach are proposed to address the challenge of rearranging deformable linear objects for implicit goals. Specifically, the context of making both ends of the object reachable (inside the robotic access range) and graspable (outside potential collision regions) by dual‐arm robots is considered. Firstly, the object is described with sequential keypoints and the correspondence‐based action is parameterized. Secondly, a generator capable of producing multiple explicit targets is developed, which adhere to implicit conditions. Thirdly, value models are learnt to assign the most promising explicit target as guidance and determine the goal‐conditioned action. All models within the policy are trained in a self‐supervised manner based on data collected from simulations. Importantly, the learned policy can be directly applied to real‐world settings since we do not rely on accurate dynamic models. The performance of the new method is validated with simulations and real‐world experiments.

Obstacle avoidance shape control of deformable linear objects with online parameters adaptation based on differentiable simulation

The manipulation of deformable linear objects (DLOs) such as ropes, cables, and hoses by robots has promising applications in various fields such as product assembly and surgical suturing. However, DLOs are more difficult to manipulate than rigid objects because their shape changes during manipulation. Furthermore, preventing a DLO from colliding with the environment is important to prevent it from becoming entangled and causing shape control to fail. In this paper, we proposed an obstacle avoidance and shape control scheme for DLOs based on differentiable simulation that does not require prior data or a specialized controller. First, we established a dynamic model of the DLO that allows for both forward dynamics transfer and error backpropagation to obtain gradients. Then, we employed model predictive control to optimize the embedded neural network for predicting the actions that would manipulate the DLO. Finally, the control scheme was made applicable to DLOs with different material properties by allowing online adaptation of the model parameters essential to deformation during manipulation. Simulations and real-world experiments demonstrate that the proposed control scheme could manipulate the DLO stably and accurately to avoid obstacles and achieve the goal state. In addition, the online adaptation of parameters helped mitigate the sim-to-real gap.

Generalizable whole-body global manipulation of deformable linear objects by dual-arm robot in 3-D constrained environments

Generalizable whole-body global manipulation of deformable linear objects by dual-arm robot in 3-D constrained environments

Reactive human–robot collaborative manipulation of deformable linear objects using a new topological latent control model

Real-time reactive manipulation of deformable linear objects is a challenging task that requires robots to quickly and adaptively respond to changes in the object’s deformed shape that result from external forces. In this paper, a novel approach is proposed for real-time reactive deformable linear object manipulation in the context of human–robot collaboration. The proposed approach combines a topological latent representation and a fixed-time sliding mode controller to enable seamless interaction between humans and robots. The introduced topological control model offers a framework for controlling the dynamic shape of deformable objects. By leveraging the topological representation, our approach captures the connectivity and structure of the objects’ shapes within a latent space. This enables improved generalization and performance when handling complex deformable shapes. A fixed-time sliding mode controller ensures that the object is manipulated in real-time, while also ensuring that it remains accurate and stable during the manipulation process. To validate our proposed framework, we first conduct motor-robot experiments to simulate fixed human interaction processes, enabling straightforward comparisons with other approaches. We then follow up with human–robot experiments to demonstrate the effectiveness of our approach.

Sim2Real Neural Controllers for Physics-Based Robotic Deployment of Deformable Linear Objects

Deformable linear objects (DLOs), such as rods, cables, and ropes, play important roles in daily life. However, manipulation of DLOs is challenging as large geometrically nonlinear deformations may occur during the manipulation process. This problem is made even more difficult as the different deformation modes (e.g., stretching, bending, and twisting) may result in elastic instabilities during manipulation. In this paper, we formulate a physics-guided data-driven method to solve a challenging manipulation task—accurately deploying a DLO (an elastic rod) onto a rigid substrate along various prescribed patterns. Our framework combines machine learning, scaling analysis, and physical simulations to develop a physics-based neural controller for deployment. We explore the complex interplay between the gravitational and elastic energies of the manipulated DLO and obtain a control method for DLO deployment that is robust against friction and material properties. Out of the numerous geometrical and material properties of the rod and substrate, we show that only three non-dimensional parameters are needed to describe the deployment process with physical analysis. Therefore, the essence of the controlling law for the manipulation task can be constructed with a low-dimensional model, drastically increasing the computation speed. The effectiveness of our optimal control scheme is shown through a comprehensive robotic case study comparing against a heuristic control method for deploying rods for a wide variety of patterns. In addition to this, we also showcase the practicality of our control scheme by having a robot accomplish challenging high-level tasks such as mimicking human handwriting, cable placement, and tying knots.

Learning Graph Dynamics With External Contact for Deformable Linear Objects Shape Control

This paper focuses on the shape control manipulation of deformable linear objects (DLO) with a dual-arm robotic system. One significant challenge of DLO shape control is the underactuated control system, which means that finite robotic manipulators can not fully control DLO's shape due to the lack of sufficient constraints on DLO. We propose a novel DLO shape control framework aiming to stably control DLO's shape in a wide deformation range, which innovatively provides additional constraints on DLO via external contact. The proposed framework consists of a learning module named <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"/> <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>Learning Graph Dynamics with External Contact for deformable linear objects</b></i> <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"/> (LG-DEC) and an action generation module. The LG-DEC uses a graph convolution network to approximate the dynamics between DLO and contactable objects based on keypoint-based representation. In the action generation module, the learned graph dynamics is used to predict whether the randomly sampled actions can drive DLO closer to the desired shape and output the best action for DLO shape control. In simulation, the proposed framework accomplishes the DLO shape control task with external contact in six scenarios. Furthermore, the proposed framework can be well applied to a physical robotic system for the shape control on different types of DLO. Videos are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://sites.google.com/view/lg-dec</uri> .

Global Model Learning for Large Deformation Control of Elastic Deformable Linear Objects: An Efficient and Adaptive Approach

Robotic manipulation of deformable linear objects (DLOs) has broad application prospects in many fields. However, a key issue is to obtain the exact deformation models (i.e., how robot motion affects DLO deformation), which are hard to theoretically calculate and vary among different DLOs. Thus, shape control of DLOs is challenging, especially for large deformation control which requires global and more accurate models. In this paper, we propose a coupled offline and online data-driven method for efficiently learning a global deformation model, allowing for both accurate modeling through offline learning and further updating for new DLOs via online adaptation. Specifically, the model approximated by a neural network is first trained offline on random data, then seamlessly migrated to the online phase, and further updated online during actual manipulation. Several strategies are introduced to improve the model's efficiency and generalization ability. We propose a convex-optimization-based controller, and analyze the system's stability using the Lyapunov method. Detailed simulations and real-world experiments demonstrate that our method can efficiently and precisely estimate the deformation model, and achieve large deformation control of untrained DLOs in 2D and 3D dual-arm manipulation tasks better than the existing methods. It accomplishes all 24 tasks with different desired shapes on different DLOs in the real world, using only simulation data for the offline learning.

Robotic Manipulation of Sperm as a Deformable Linear Object

The robotic manipulation of deformable linear objects is a classic and challenging topic. Apart from synthetic objects, such as wires and cables, linear objects are also commonly found in biological cells and organisms. Biomanipulation of such objects is hampered by difficulties, such as limited degrees of freedom of micromanipulators and varied mechanical properties of the biological entities to manipulate. This article presents a robotic manipulation of human sperm, which are deformable cells with a linear shape. The shape and movement of the cell are recapitulated by our developed geometric and kinematic models. Under unfixed constraints between the end-effector and the cell, path planning is designed to update the manipulation point to control cell deformation. A state transition function is formulated in path planning to handle the stiffness variations of sperm without force sensing. A model-predictive controller is designed to minimize the orientation error and manipulation path length. To detect sperm tail for visual feedback, an accuracy of 98% was achieved via deep neural networks. The robotic manipulation of human sperm was performed using a standard clinical setup of a glass micropipette to rotate a sperm to the target orientation. Experimental results showed that robotic sperm rotation achieved an orientation error of 0.8 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{\circ }$</tex-math></inline-formula> , a tail curvedness of 0.14 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mu$</tex-math></inline-formula> m <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{-1}$</tex-math></inline-formula> , and an operation time of 5.6 s, all significantly less than those of the manual approach. The less orientation error and tail curvedness after robotic rotation led to a significantly lower speed of sperm entering the micropipette during sperm aspiration, resulting in a higher success rate of 97% (versus 76% after manual rotation) for aspiration control.

Dynamic Modeling and Control of Deformable Linear Objects for Single-Arm and Dual-Arm Robot Manipulations

Robotic manipulation of deformable linear objects (DLOs) is important in many applications such as the assembly of deformable wire harnesses and cables in manufacturing. Despite some relevant work in modeling, few studies have studied the comprehensive dynamic modeling and precise automatic control of DLOs using robots due to their high degrees of freedom and high flexibility. To fill this gap, the precise single-arm and dual-arm robot manipulation control of DLOs is studied in this article. First, a more comprehensive dynamic model of DLOs is established based on a discrete elastic rod model, which takes into account the twisting deformation of DLOs. The collisions, contacts, and frictions between the DLOs and the plane, as well as the kinematic constraints of both ends of the DLOs, are also considered in the model. Second, practical dynamic control schemes of robots are proposed to realize the precise control of DLOs for both the single-arm control and dual-arm control. For validations, we built an experimental platform using an ABB Yumi robot to implement and validate the proposed approaches in addition to simulations. Finally, various DLO manipulation tasks are conducted and the results for both single-arm and dual-arm manipulations validate the proposed modeling and control approaches.

Keypoint-Based Planar Bimanual Shaping of Deformable Linear Objects Under Environmental Constraints With Hierarchical Action Framework

This letter addresses the problem of contact-based manipulation of deformable linear objects (DLOs) towards desired shapes with a dual-arm robotic system. To alleviate the burden of high-dimensional continuous state-action spaces, we model DLOs as kinematic multibody systems via our proposed keypoint encoding network. This novel encoding is trained on a synthetic labeled image dataset without requiring any manual annotations and can be directly transferred to real manipulation scenarios.Our goal-conditioned policy efficiently rearranges the configuration of the DLO based on the keypoints. The proposed hierarchical action framework tackles the manipulation problem in a coarse-to-fine manner (with high-level task planning and low-level motion control) by leveraging two action primitives. The identification of deformation properties is bypassed since the algorithm replans its motion after each bimanual execution. The conducted experimental results reveal that our method achieves high performance in state representation and shaping manipulation of the DLO under environmental constraints.

A novel cable-grasping planner for manipulator based on the operation surface

The manipulation of deformable linear objects is a hot spot and difficult problem in robotics research. Cables, as one kind of deformable linear objects, are characterized by elongation and uncertain shape. Although previous research has paid attention to deformable linear object’s manipulation, it still remains a challenge to detect and grasp cables in unstructured environments. In this paper, a new cable-grasping method framework by analysis of cable characteristics is proposed. The framework consists of cable pose measurement and cable-grasping path planning. In the cable pose measurement mission, we design a symmetric edge feature according to the cable image projection principle. Then, a semantic segmentation based edge detection technique and a center line fitting method based on sorting by principal axis are applied to measure the cable pose accurately. In the cable-grasping path planning mission, the grasping pose is determined considering the force directional manipulability . The cable operation surface is constructed after the characteristic analysis of the cable, on which basis, a two-phase path planner is proposed, with the second phase planner BG-RRT guided by motion directional manipulability to further optimize the planning path. The performance of the approach is verified by experiments in both the simulation environment and the actual environment. • Proposing a symmetric edge feature based on the cable image projection principle. • Designing an edge detection method based on semantic segmentation as well as a center line fitting method based on sorting by principal axis. • Constructing the cable operation surface according to the characteristics of cable. • A two-phase path planner based on operation surface for reducing collisions. • A new BG-RRT algorithm guided by motion directional manipulability for further improving the efficiency of cable grasping.

New model-based manipulation technique for reshaping deformable linear objects

In this article, we consider the problem of reshaping a deformable linear object (DLO) like wires, cables, ropes, and surgical sutures. The solution to this problem would be useful for many fields, especially industrial manufacturing, where the DLO manipulation is still frequently carried out by human workers. In this work, a new model-based manipulation technique for reshaping a DLO is addressed employing a sequence of grasping and releasing primitives performed by a single-armed robot equipped with a gripper. A decision process selects the optimal grasping point exploiting an error minimization approach and chooses the related releasing point. This decision process performs a spline interpolation between the error values obtained from candidate grasping points and chooses the optimal point that owns a minimum error. The multivariate dynamic spline model of the DLO is exploited for selecting the optimal grasping point and predicting the DLO behavior during the manipulation process. Because of its advantages over other integration methods, the symplectic integrator is utilized for iteratively solving the DLO dynamic model. Simulation results of reshaping a DLO lying on a table are presented to evaluate the proposed technique. These results illustrate the intermediate deformation steps which lead the DLO from its starting state to the desired one. They demonstrate that our proposed technique can efficiently manipulate the DLO into various shapes in few steps.

Proximity Sensor for Thin Wire Recognition and Manipulation

In robotic grasping and manipulation, the knowledge of a precise object pose represents a key issue. The point acquires even more importance when the objects and, then, the grasping areas become smaller. This is the case of Deformable Linear Object manipulation application where the robot shall autonomously work with thin wires which pose and shape estimation could become difficult given the limited object size and possible occlusion conditions. In such applications, a vision-based system could not be enough to obtain accurate pose and shape estimation. In this work the authors propose a Time-of-Flight pre-touch sensor, integrated with a previously designed tactile sensor, for an accurate estimation of thin wire pose and shape. The paper presents the design and the characterization of the proposed sensor. Moreover, a specific object scanning and shape detection algorithm is presented. Experimental results support the proposed methodology, showing good performance. Hardware design and software applications are freely accessible to the reader.

An algorithm based on bidirectional searching and geometric constrained sampling for automatic manipulation planning in aircraft cable assembly

Robotic manipulation of deformable linear objects has potential application in aircraft cable assembly. However, it is difficult to be implemented in real tasks due to requiring prediction of the object’s deformation and obstacle-free manipulation planning with high efficiency. Aiming at exploring automatic assembly planning for aircraft cables assembly in narrow cabins with obstacles, this paper proposes a novel planning algorithm named RRT-BwC (Bi-direction with Constrain). Firstly, formulation of the question and the manipulation objects are presented with geometric definitions. Then a bi-RRT-tree searching method is developed to design the planner for overcoming obstacles in the high dimensional planning space. The numerical distance between configurations in the cable shape space are defined to measure the demands for their transition in consideration of the manipulation logics. And the sampling zone constrains to the shape configuration nodes are also discussed. Finally, the algorithm presents a valid manipulation sequence for robotic manipulation. The functionality and performance of the improved RRT approach are demonstrated with a simulated real-world problem of aircraft cable assembly, exhibiting computation efficiency promotion.

Planning for Manipulation of Interlinked Deformable Linear Objects With Applications to Aircraft Assembly

Manipulation of deformable linear objects (DLOs) has potential applications in the fields of aerospace and automotive assembly. In this paper, we introduce a problem formulation for attaching a set of interlinked DLOs to a support structure using a set of clamping points. The formulation describes the manipulation planning problem in terms of known clamp poses; predetermined ideal clamping locations on the cables, called “reference points;” and a set of finite gripping points on the DLOs. We also present a prototype algorithm that generates a solution in terms of primitive manipulation actions. The algorithm guarantees that no interlink constraints are violated at any stage of manipulation. We incorporate gravity in the computation of a DLO shape and propose a property linking geometrically similar cable shapes across the space of cable length and stiffness. This property allows for the computation of solutions for unit length and scaling of these solutions to appropriate length, potentially resulting in faster shape computation.

A Framework for Manipulating Deformable Linear Objects by Coherent Point Drift

Manipulation of deformable linear objects is a challenging task for robots. These objects have infinite-dimensional configuration space and are computational-expensive to model, making it difficult for real-time tracking, planning and control. To deal with these challenges, a uniform framework that includes state estimation, task planning, and trajectory planning is proposed in this letter based on the concept of coherent point drift (CPD). A real-time observer is proposed to estimate the states of deformable objects from the perceived point clouds. An online task planner is then developed to recognize the manipulation step according to the state estimation result. For trajectory planning, human operators first train robots example trajectories given several object states. In the test stage, a new feasible trajectory can be autonomously generated by a smooth transformation from training scenarios to test scenarios. A series of rope manipulation experiments on a dual-arm robotic platform are performed to validate the effectiveness of the proposed methods.

Robotized recognition of a wire harness utilizing tracing operation

In assembly lines, deformation of the objects makes many efforts on automating the corresponding processes difficult. Mating of a wire harness onto a car body is among the representative tasks. Academically, this kind of problems are known as automatic manipulation of deformable linear objects. Previous technical approaches for solving the problem is characterized by employment of multiple contactless sensors and complex sensor based algorithms. These attempts are verified to be feasible only for simplified environments. The dependence of the approaches on real-time measurement to the state of the deformable targets limits their application in practical factory environment. In this paper, it is proposed to utilize wire tracing operation in recognizing the wire harness for the purpose of automatic mating. This method is inspired from the behaviors of humans demonstrated when they deal with similar problems. The proposed method is implemented and verified in a robot system. The feasibility and effectiveness demonstrated in the preliminary experiments show the potential of the proposed method for substantially simplifying the wire recognition problems which is thought to be crucial for practices in factory environments. HighlightsA new method for realizing automatic wire harness mating is proposed. In this method, recognition to the wire harness is proposed to be achieved by using a gripping which can move along the wire harness.The proposed solution does not necessitate complex sensing means and algorithms thus its extension has the potential to be applied in factory environment.The proposed approach has been implemented using three arms robot manipulator.Preliminary experiments are verifying the feasibility of the proposed approach.

Vibration damping in manipulation of deformable linear objects using sliding mode control

In this paper, we proposed a position-based control strategy for eliminating the vibration at the end of deformable linear objects (DLOs) during its manipulation. Using Schur decomposition of matrices and linear transform of variables, actuated and underactuated parts of the DLO dynamic model are separated. Based on the decoupled dynamic model of a DLO system, a sliding mode control with exponential approach law is designed to force the state variables to converge to an equilibrium and to allow vibration at the end of the DLO to be damped quickly. The DLO system, subjected to control input saturation, is further studied to solve the input saturation problem. An adaptive sliding mode control law is designed to suppress the damping at the end of the DLO. Proposed control strategies are verified by numerical simulations. The simulation results show that proposed methods can effectively damp the vibration at the end of the DLO.

Manipulation of deformable linear objects using knot invariants to classify the object condition based on image sensor information

Using a topological model and knot theory, we propose a method for describing the condition of a rope. We also propose a recognition method based on the image information obtained from the charge-coupled device cameras to obtain the structure of the rope when manipulated by a robot. This method will help solve the difficulties of robots manipulating deformable objects by providing a theoretical framework of error recovery for deformable object manipulation. We confirm the effectiveness of the methods through experiments