Robotic Object Manipulation Research Articles

A feedback-based manoeuvre planner for nonprehensile magnetic micromanipulation of large microscopic biological objects

This paper reports a feedback-based manoeuvre planning approach for automated nonprehensile selective micromanipulation of large, microscopic biological objects (∼100μm). We employ ferromagnetic micro-particles as microrobots actuated via a global magnetic field produced by electromagnetic coils placed in quadrupole configuration. The microrobot motion is programmed to push the target object to the goal location. We employ a three-step approach comprising: (a) generate a collision-free optimal path between the initial and the commanded goal location, (b) generate a manoeuvre planning algorithm that invokes one of the three motion manoeuvres, namely, ‘approach’, ‘push’, and ‘align’ depending upon the instantaneous locations of the microrobot, target object, and the desired waypoint, and (c) deploy a simple proportional controller that determines the currents required in the electromagnetic coils that can produce a suitable magnetic field for executing the manoeuvre invoked by the manoeuvre planner. This paper reports a number of validation experiments conducted on both zebrafish, i.e., Danio rerio embryos and silica beads as target objects. We envisage that the developed inexpensive approach can be useful in robotic manipulation of biological objects with sizes in hundreds of microns including large biological cells, polyploid giant cancer cells (PGCC), multicellular spheroids, Dictyostelium slug, human oocytes, and autophagy candidates. We also believe that functionalizing the microrobots with living cells or suitable chemicals will make it possible to perform on-chip biological experiments in future.

Algorithmic Selection of Sliding–Sticking Contacts in Robotic In-hand Manipulation

Abstract The paper describes a kinematic method for robotic in-hand manipulation of objects. The method focuses on repositioning the object using a combination of sticking and sliding robotic contacts. Two fingertips with sliding contacts are fixed in space while the remaining two fingertips actively manipulate the object without a change in the point of contact with the object. When sliding over two fixed contacts, the object is constrained to a “three-parameter twist space” if it is not programmed to rotate about the line joining the two fixed contacts. A gradient-descent-based kinematic algorithm is developed to project the desired twist to the allowable twist space, generating a movement sequence of robotic fingertips. The transition from fixed support vis-á-vis the sticking contacts for manipulating the object also emerges from the algorithm.

Control of robotic object pivoting based on tactile sensing

Robotic manipulation of objects using the sole tactile feedback is a challenge. If the contact between the robot end effector and the manipulated object is distributed, the robot can exchange both friction forces and torques with it. The friction highly influences the motion of the object. By controlling the friction it is possible to perform complex manipulation tasks, such as moving the object with respect to the end effector by executing a controlled sliding motion. If the motion is a rotation with respect to the end effector, the corresponding maneuver is called pivoting. Control of the pivoting motion is considerably difficult, especially without any visual feedback. This paper proposes a novel method to regulate the object angular position, by means of a pivoting maneuver, through a parallel gripper endowed with force/tactile sensors. The strategy is based on a novel nonlinear observer that estimates the sliding velocity from force/torque measurements and a model of the sliding dynamics. We exploit the Limit Surface concept and the LuGre friction model to build a dynamic model of a planar slider. We show, through experimental results, that simple parallel grippers are able to execute such maneuvers that correspond to adding a virtual joint between the fingers, thus enlarging the robot workspace.

An algorithm based on bidirectional searching and geometric constrained sampling for automatic manipulation planning in aircraft cable assembly

Robotic manipulation of deformable linear objects has potential application in aircraft cable assembly. However, it is difficult to be implemented in real tasks due to requiring prediction of the object’s deformation and obstacle-free manipulation planning with high efficiency. Aiming at exploring automatic assembly planning for aircraft cables assembly in narrow cabins with obstacles, this paper proposes a novel planning algorithm named RRT-BwC (Bi-direction with Constrain). Firstly, formulation of the question and the manipulation objects are presented with geometric definitions. Then a bi-RRT-tree searching method is developed to design the planner for overcoming obstacles in the high dimensional planning space. The numerical distance between configurations in the cable shape space are defined to measure the demands for their transition in consideration of the manipulation logics. And the sampling zone constrains to the shape configuration nodes are also discussed. Finally, the algorithm presents a valid manipulation sequence for robotic manipulation. The functionality and performance of the improved RRT approach are demonstrated with a simulated real-world problem of aircraft cable assembly, exhibiting computation efficiency promotion.

Robotic Manipulation of Deformable Cells for Orientation Control

Robotic manipulation of deformable objects has been a classic topic in robotics. Compared to synthetic deformable objects such as rubber balls and clothes, biological cells are highly deformable and more prone to damage. This article presents robotic manipulation of deformable cells for orientation control (both out-of-plane and in-plane), which is required in both clinical (e.g., in vitro fertilization) and biomedical (e.g., clone) applications. Compared to manual cell orientation control based on empirical experience, the robotic approach, based on modeling and path planning, effectively rotates a cell, while consistently maintaining minimal cell deformation to avoid cell damage. A force model is established to determine the minimal force applied by the micropipette to rotate a spherical or, more generally, ellipsoidal oocyte. The force information is translated into indentation through a contact mechanics model, and the manipulation path of the micropipette is formed by connecting the indentation positions on the oocyte. An optimal controller is designed to compensate for the variations of mechanical properties across oocytes. The polar body of an oocyte is detected by deep neural networks with robustness to shape and size differences. In experiments, the system achieved an accuracy of 97.6% in polar body detection and an accuracy of 0.7 $^{\circ }$ in oocyte orientation control with maximum oocyte deformation of 2.70 $\mu$ m throughout the orientation control process.

A review on modeling of flexible deformable object for dexterous robotic manipulation

In this article, we present a review on the recent advancement in flexible deformable object modeling for dexterous manipulation in robotic system. Flexible deformable object is one of the most research topics in computer graphic, computer vision, and robotic literature. The deformable models are known as the construction of object with material parameters in virtual environment to describe the deformation behavior. Existing modeling techniques and different types of deformable model are described. Various approaches of deformable object modeling have been used in robotic recognition and manipulation in order to reduce the time and cost to obtain more accurate result. In robotic manipulation, object detection, classification, and recognition of deformable objects are always a challenging problem and required as a first step to imbue the robot to able handle these deformable objects. Furthermore, the dexterity of robot control is also another essential key in handling of deformable object which its manipulation strategies need to plan intelligently for each sequence process. We also discuss some deserving direction for further research based on most current contribution.

From Visual Understanding to Complex Object Manipulation

Planning and executing object manipulation requires integrating multiple sensory and motor channels while acting under uncertainty and complying with task constraints. As the modern environment is tuned for human hands, designing robotic systems with similar manipulative capabilities is crucial. Research on robotic object manipulation is divided into smaller communities interested in, e.g., motion planning, grasp planning, sensorimotor learning, and tool use. However, few attempts have been made to combine these areas into holistic systems. In this review, we aim to unify the underlying mechanics of grasping and in-hand manipulation by focusing on the temporal aspects of manipulation, including visual perception, grasp planning and execution, and goal-directed manipulation. Inspired by human manipulation, we envision that an emphasis on the temporal integration of these processes opens the way for human-like object use by robots.

Multi-Modal Sensing and Robotic Manipulation of Non-Rigid Objects: A Survey

This paper aims to provide a comprehensive survey of recent advancements in modelling and autonomous manipulation of non-rigid objects. It first summarizes the recent advances in sensing and modelling of such objects with a focus on describing the methods and technologies used to measure their shape and estimate their material and physical properties. Formal representations considered to predict the deformation resulting from manipulation of non-rigid objects are then investigated. The third part provides a survey of planning and control strategies exploited to operate dexterous robotic systems while performing various tasks on objects made of different non-rigid materials.

Model-Driven Feedforward Prediction for Manipulation of Deformable Objects

Robotic manipulation of deformable objects is a difficult problem especially because of the complexity of the many different ways an object can deform. Searching such a high-dimensional state space makes it difficult to recognize, track, and manipulate deformable objects. In this paper, we introduce a predictive, model-driven approach to address this challenge, using a precomputed, simulated database of deformable object models. Mesh models of common deformable garments are simulated with the garments picked up in multiple different poses under gravity, and stored in a database for fast and efficient retrieval. To validate this approach, we developed a comprehensive pipeline for manipulating clothing as in a typical laundry task. First, the database is used for category and the pose estimation is used for a garment in an arbitrary position. A fully featured 3-D model of the garment is constructed in real time, and volumetric features are then used to obtain the most similar model in the database to predict the object category and pose. Second, the database can significantly benefit the manipulation of deformable objects via nonrigid registration, providing accurate correspondences between the reconstructed object model and the database models. Third, the accurate model simulation can also be used to optimize the trajectories for the manipulation of deformable objects, such as the folding of garments. Extensive experimental results are shown for the above tasks using a variety of different clothings. Note to Practitioners —This paper provides an open source, extensible, 3-D database for dissemination to the robotics and graphics communities. Model-driven methods are proliferating, and they need to be applied, tested, and validated in real environments. A key idea we have exploited is to have an innovative and novel use of simulation. This database will serve as infrastructure for developing advanced robotic machine learning algorithms. We want to address this machine learning idea ourselves, but we expect the dissemination of the database to other researchers with different agendas and task applications, which will bring wide progress in this area. Our proposed methods, as mentioned earlier, can be easily applied to interrelated areas. One example is that the 3-D shape-based matching algorithm can be used for other objects, such as bottles, papers, and food. After integrating with other robotic systems, the use of the robot can be easily extended to other tasks, such as making food, cleaning room, and fetching objects, to assist our daily life.

Fast Ellipse Detection via Gradient Information for Robotic Manipulation of Cylindrical Objects

Robotic manipulation of objects requires a fast recognition from image stream. For many cylindrical object (e.g., cans, cups, pipes, bottles, etc.) this is possible through detection of ellipse depicting the circular top of the cylinder. Growing industrial and warehouse applications of robots drive the demand for fast and reliable detection of ellipses, while state-of-the-art methods are lacking in either speed or accuracy strength. We present a novel algorithm to perform fast and robust ellipse detection. First, the method utilizes the information of edge curvature to split curves into arcs. Next, the arc convexity–concavity is used to classify arcs into different quadrants of ellipses. Then, based on multiple geometric constraints the arcs can be grouped at low computational cost. Our method is compared with six state-of-the-art methods using three public image datasets. The comparison results show that the proposed algorithm outperforms other methods with high detection accuracy and fast detection speed. Lastly, the algorithm is applied to identifying cylindrical objects in real-time for arranging and tracking purposes.

Robust shape estimation for 3D deformable object manipulation

Existing shape estimation methods for deformable object manipulation suffer from the drawbacks of being off-line, model dependent, noise-sensitive or occlusion-sensitive, and thus are not appropriate for manipulation tasks requiring high precision. In this paper, we present a real-time shape estimation approach for autonomous robotic manipulation of 3D deformable objects. Our method fulfills all the requirements necessary for the high-quality deformable object manipulation in terms of being real-time, model-free and robust to noise and occlusion. These advantages are accomplished using a joint tracking and reconstruction framework, in which we track the object deformation by aligning a reference shape model with the stream input from the RGB-D camera, and simultaneously upgrade the reference shape model according to the newly captured RGB-D data. We have evaluated the quality and robustness of our real-time shape estimation pipeline on a set of deformable manipulation tasks implemented on physical robots. Videos are available at https://lifeisfantastic.github.io/DeformShapeEst/

An Accurate Perception Method for Low Contrast Bright Field Microscopy in Heterogeneous Microenvironments

Automated optical tweezers-based robotic manipulation of microscale objects requires real-time visual perception for estimating the states, i.e., positions and orientations, of the objects. Such visual perception is particularly challenging in heterogeneous environments comprising mixtures of biological and colloidal objects, such as cells and microspheres, when the popular imaging modality of low contrast bright field microscopy is used. In this paper, we present an accurate method to address this challenge. Our method combines many well-established image processing techniques such as blob detection, histogram equalization, erosion, and dilation with a convolutional neural network in a novel manner. We demonstrate the effectiveness of our processing pipeline in perceiving objects of both regular and irregular shapes in heterogeneous microenvironments of varying compositions. The neural network, in particular, helps in distinguishing the individual microspheres present in dense clusters.

Design and analysis of an origami-based three-finger manipulator

SUMMARYThis paper describes a new robotic manipulator with three fingers based on an origami twisted tower design. The design specifications, kinematic description, and results from the stiffness and durability tests for the selected origami design are presented. The robotic arm is made of a 10-layer twisted tower, actuated by four cables with pulleys driven by servo motors. Each finger is made of a smaller 11-layer tower and uses a single cable directly attached to a servo motor. The current hardware setup supports vision-based autonomous control and internet-based remote control in real time. For preliminary evaluation of the robot's object manipulation capabilities, arbitrary objects with varying weights, sizes, and shapes (i.e., a shuttlecock, an egg shell, a paper cub, and a cubic block) were selected and the rate of successful grasping and lifting for each object was measured. In addition, an experiment comparing a rigid gripper and the new origami-based manipulator revealed that the origami structure in the fingers absorbs the excessive force applied to the object through force distribution and structural deformation, demonstrating its potential applications for effective manipulation of fragile objects.

Single and Multiple View Support Order Prediction in Clutter for Manipulation

Robotic manipulation of objects in clutter remains a challenging problem to date. The challenge is posed by various levels of complexity involved in interaction among objects. Understanding these semantic interactions among different objects is important to manipulate in complex settings. It can play a significant role in extending the scope of manipulation to cluttered environment involving generic objects, and both direct and indirect physical contact. In our work, we aim at learning semantic interaction among objects of generic shapes and sizes lying in clutter involving physical contact. We infer three types of support relationships: "support from below", "support from side", and "containment". Subsequently, the learned semantic interaction or support relationship is used to derive a sequence or order in which the objects surrounding the object of interest should be removed without causing damage to the environment. The generated sequence is called support order. We also extend understanding of semantic interaction from single view to multiple views and predict support order in multiple views. Using multiple views addresses those cases that are not handled when using single view such as scenarios of occlusion or missing support relationships. We have created two RGBD datasets for our experiments on support order prediction in single view and multiple views respectively. The datasets contains RGB images, point clouds and depth maps of various objects used in day-to-day life present in clutter with physical contact and overlap. We captured many different cluttered settings involving different kinds of object-object interaction and successfully learned support relationship and performed Support Order Prediction in these settings.

Robotic manipulation of multiple objects as a POMDP

This paper investigates manipulation of multiple unknown objects in a crowded environment. Because of incomplete knowledge due to unknown objects and occlusions in visual observations, object observations are imperfect and action success is uncertain, making planning challenging. We model the problem as a partially observable Markov decision process (POMDP), which allows a general reward based optimization objective and takes uncertainty in temporal evolution and partial observations into account. In addition to occlusion dependent observation and action success probabilities, our POMDP model also automatically adapts object specific action success probabilities. To cope with the changing system dynamics and performance constraints, we present a new online POMDP method based on particle filtering that produces compact policies. The approach is validated both in simulation and in physical experiments in a scenario of moving dirty dishes into a dishwasher. The results indicate that: 1) a greedy heuristic manipulation approach is not sufficient, multi-object manipulation requires multi-step POMDP planning, and 2) on-line planning is beneficial since it allows the adaptation of the system dynamics model based on actual experience.

Efficient Optimization for Autonomous Robotic Manipulation of Natural Objects

Manipulating natural objects of irregular shapes, such as rocks, is an essential capability of robots operating in outdoor environments. Physics-based simulators are commonly used to plan stable grasps for man-made objects. However, planning is an expensive process that is based on simulating hand and object trajectories in different configurations, and evaluating the outcome of each trajectory. This problem is particularly concerning when the objects are irregular or cluttered, because the space of feasible grasps is significantly smaller, and more configurations need to be evaluated before finding a good one. In this paper, we first present a learning technique for fast detection of an initial set of potentially stable grasps in a cluttered scene. The best detected grasps are further optimized by fine-tuning the configuration of the hand in simulation. To reduce the computational burden of this last operation, we model the outcomes of the grasps as a Gaussian Process, and use an entropy-search method in order to focus the optimization on regions where the best grasp is most likely to be. This approach is tested on the task of clearing piles of real, unknown, rock debris with an autonomous robot. Empirical results show a clear advantage of the proposed approach when the time window for decision is short.

Object Recognition Based on Dual Tree Complex Wavelet Transform

Automated recognition of objects from images plays an important role in many computer vision systems such as robot navigation, object manipulation and content based image retrieval. In this study, an approach for object recognition based on Dual Tree Complex Wavelet Transform (DTCWT) is proposed. The proposed approach attempts to extract the detailed information of objects from the multi scale representation by DTCWT. The proposed system is tested on Columbia Object Image Library (COIL-100). All the objects are considered for the classification based on nearest neighbor classifier. The results show that the maximum recognition accuracy achieved by the proposed approach is 97.03%.

A van der Waals Force-Based Adhesion Model for Micromanipulation

The robotic manipulation of microscopic objects is disturbed directly by the adhesion between the end-effector and the objects. In the microscale, no reliable model of adhesion is available and currently the behaviour of the micro-objects cannot be predicted before experiments. This paper proposes a new model of adhesion based on the analytical resolution of the coupling between the mechanical deformation of the micro-objects and van der Waals forces. In the nanoscale, the impact of the deformation can be neglected and the proposed model is thus similar to the classical expression for van der Waals forces. In the microscale, the deformation induces van der Waals forces to increase significantly and a new analytical expression is proposed. The limit of validity of this 'deformable van der Waals forces' is also discussed. This result can be used as an alternative to classical adhesion–deformation models in literature (Johnson–Kendall–Roberts (JKR) or Derjaguin–Muller–Toporov (DMT)), which have been validated at the macroscale but are not sufficient to describ the interaction forces in the microscale (typically from 100 nm to 500 μm).

Tracking rigid objects using integration of model-based and model-free cues

Model-based 3-D object tracking has earned significant importance in areas such as augmented reality, surveillance, visual servoing, robotic object manipulation and grasping. Key problems to robust and precise object tracking are the outliers caused by occlusion, self-occlusion, cluttered background, reflections and complex appearance properties of the object. Two of the most common solutions to the above problems have been the use of robust estimators and the integration of visual cues. The tracking system presented in this paper achieves robustness by integrating model-based and model-free cues together with robust estimators. As a model-based cue, a wireframe edge model is used. As model-free cues, automatically generated surface texture features are used. The particular contribution of this work is the integration framework where not only polyhedral objects are considered. In particular, we deal also with spherical, cylindrical and conical objects for which the complete pose cannot be estimated using only wireframe models. Using the integration with the model-free features, we show how a full pose estimate can be obtained. Experimental evaluation demonstrates robust system performance in realistic settings with highly textured objects and natural backgrounds.

Vision for robotic object manipulation in domestic settings

In this paper, we present a vision system for robotic object manipulation tasks in natural, domestic environments. Given complex fetch-and-carry robot tasks, the issues related to the whole detect-approach-grasp loop are considered. Our vision system integrates a number of algorithms using monocular and binocular cues to achieve robustness in realistic settings. The cues are considered and used in connection to both foveal and peripheral vision to provide depth information, segmentation of the object(s) of interest, object recognition, tracking and pose estimation. One important property of the system is that the step from object recognition to pose estimation is completely automatic combining both appearance and geometric models. Experimental evaluation is performed in a realistic indoor environment with occlusions, clutter, changing lighting and background conditions.