Robotic Grasping Research Articles

A Two-Stream CNN With Simultaneous Detection and Segmentation for Robotic Grasping

The manipulating robots receive much attention by offering better services, where object grasping is still challenging especially under background interferences. In this article, a novel two-stream grasping convolutional neural network (CNN) with simultaneous detection and segmentation is proposed. The proposed method is cascaded by an improved simultaneous detection and segmentation network BlitzNet and a two-stream grasping CNN TsGNet. The improved BlitzNet introduces the channel-based attention mechanism, and achieves an improvement of detection accuracy and segmentation accuracy with the combination of the learning of multitask loss weightings and background suppression. Based on the obtained bounding box and the segmentation mask of the target object, the target object is separated from the background, and the corresponding depth map and grayscale map are sent to TsGNet. By adopting depthwise separable convolution and designed global deconvolution network, TsGNet achieves the best grasp detection with only a small amount of network parameters. This best grasp in the pixel coordinate system is converted to a desired 6-D pose for the robot, which drives the manipulator to execute grasping. The proposed method combines a grasping CNN with simultaneous detection and segmentation to achieve the best grasp with a good adaptability to background. With the Cornell grasping dataset, the image-wise accuracy and object-wise accuracy of the proposed TsGNet are 93.13% and 92.99%, respectively. The effectiveness of the proposed method is verified by the experiments.

Target Positioning and Grasping of NAO Robot Based on Monocular Stereo Vision

With the continuous advancement of science and technology and the rapid development of robotics, it has become an inevitable trend for domestic robots to enter thousands of households. In order to solve the inconvenience problem of the elderly and people with special needs, because the elderly and other people in need may need the help of domestic robots due to inconvenient legs and feet, the research of the robot target position based on monocular stereo vision and the understanding of the robot NAO are very important. Research and experiments are carried out on the target recognition and positioning in the process of NAO robot grasping. This paper proposes a recognition algorithm corresponding to quantitative component statistical information. First, extract the area of interest that contains the purpose from the image. After that, to eliminate interference in various fields and achieve target recognition, the robot cameras have almost no common field of view and can only use one camera at the same time. Therefore, this article uses the monocular vision principle to locate the target, and the detection algorithm is based on the structure of the robot head material, establishes the relationship between the height change of the machine head and the tilt angle, and improves the monocular vision NAO robot detection algorithm. According to experiments, the accuracy of the robot at close range can be controlled below 1 cm. This article completes the robot’s grasping and transmission of the target. About 80% of the external information that humans can perceive comes from vision. In addition, there are advantages such as high efficiency and good stability.

Faster than Real-Time Surface Pose Estimation with Application to Autonomous Robotic Grasping

Motivated by grasp planning applications within cluttered environments, this paper presents a novel approach to performing real-time surface segmentations of never-before-seen objects scattered across a given scene. This approach utilizes an input 2D depth map, where a first principles-based algorithm is utilized to exploit the fact that continuous surfaces are bounded by contours of high gradient. From these regions, the associated object surfaces can be isolated and further adapted for grasp planning. This paper also provides details for extracting the six-DOF pose for an isolated surface and presents the case of leveraging such a pose to execute planar grasping to achieve both force and torque closure. As a consequence of the highly parallel software implementation, the algorithm is shown to outperform prior approaches across all notable metrics and is also shown to be invariant to object rotation, scale, orientation relative to other objects, clutter, and varying degree of noise. This allows for a robust set of operations that could be applied to many areas of robotics research. The algorithm is faster than real time in the sense that it is nearly two times faster than the sensor rate of 30 fps.

A Novel Generative Convolutional Neural Network for Robot Grasp Detection on Gaussian Guidance

The vision-based grasp detection method is an important research direction in the field of robotics. However, due to the rectangle metric of the grasp detection rectangle's limitation, a false-positive grasp occurs, resulting in the failure of the real-world robot grasp task. In this paper, we propose a novel generative convolutional neural network model to improve the accuracy and robustness of robot grasp detection in real-world scenes. First, a Gaussian-based guided training method is used to encode the quality of the grasp point and grasp angle in the grasp pose, highlighting the highest-quality grasp point position and grasp angle and reducing the generation of false-positive grasps. Simultaneously, deformable convolution is used to obtain the shape features of the object in order to guide the subsequent network to the position. Furthermore, a global-local feature fusion method is introduced in order to efficiently obtain finer features during the feature reconstruction stage, allowing the network to focus on the features of the grasped objects. On the Cornell Grasping Datasets and Jacquard Datasets, our method achieves excellent performance of 99.0$\%$ and 95.9$\%$, respectively. Finally, the proposed method is put to the test in a real-world robot grasping scenario.

OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulation

In this paper, we propose a cloud-based benchmark for robotic grasping and manipulation, called the OCRTOC benchmark. The benchmark focuses on the object rearrangement problem, specifically table organization tasks. We provide a set of identical real robot setups and facilitate remote experiments of standardized table organization scenarios in varying difficulties. In this workflow, users upload their solutions to our remote server and their code is executed on the real robot setups and scored automatically. After each execution, the OCRTOC team resets the experimental setup manually. We also provide a simulation environment that researchers can use to develop and test their solutions. With the OCRTOC benchmark, we aim to lower the barrier of conducting reproducible research on robotic grasping and manipulation and accelerate progress in this field. Executing standardized scenarios on identical real robot setups allows us to quantify algorithm performances and achieve fair comparisons. Using this benchmark we held a competition in the 2020 International Conference on Intelligence Robots and Systems (IROS 2020). In total, 59 teams took part in this competition worldwide. We present the results and our observations of the 2020 competition, and discuss our adjustments and improvements for the upcoming OCRTOC 2021 competition. The homepage of the OCRTOC competition is <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">www.ocrtoc.org</uri> , and the OCRTOC software package is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/OCRTOC/OCRTOC_software_package</uri> .

Genetic Algorithms for Stable Robot Grasping

Genetic Algorithms for Stable Robot Grasping

Improved Kernel Correlation Filter Based Moving Target Tracking for Robot Grasping

Tracking and grasping moving objects is a hot topic in the field of robots, which provide great potential in the industrial scene and human-computer cooperation. Based on kernel correlation filter and vision 3D reconstruction, this paper proposes a visual-based tracking and grasping method for moving targets. An improved algorithm based on kernel correlation filter is proposed for object tracking. A scale pool is constructed and a scale filter is trained to solve the problem of algorithm scale adaptation. At the same time, the judgment mechanism of tracking results, secondary detection, and modification update mechanisms are added to improve the robustness of the algorithm. Combined with RGB-D camera, the target is reconstructed to obtain the 3D pose the target. The tracking and intercepting strategy is adopted to grasp the moving target. The proposed method is proven to have good performance through data set comparison tests and experiments on real robot systems.

Deep Robotic Grasping Prediction with Hierarchical RGB-D Fusion

Deep Robotic Grasping Prediction with Hierarchical RGB-D Fusion

Lightweight Pixel-Wise Generative Robot Grasping Detection Based on RGB-D Dense Fusion

Grasping detection is one of the essential tasks for robots to achieve automation and intelligence. The existing grasp detection mainly relies on data-driven discriminative and generative strategies. Generative strategies have significant advantages over discriminative strategies in terms of efficiency. RGB and Depth data are widely used in grasping data sources due to the sufficient amount of information and low cost of acquisition. RGB-D fusion has shown advantages over only using RGB or Depth. However, existing research has mainly focused on early fusion and late fusion, which is challenging to utilize information from both modalities fully. Improving the accuracy of grasping while leveraging the knowledge of both modalities, while ensuring lightweight and real-time is crucial. Therefore, this paper proposes a pixel-wise RGB-D dense fusion method based on a generative strategy. The technique is doubly experimentally validated on public datasets and real robot platform. Accuracy rates of 98.9% and 94.0% are achieved on Cornell and Jacquard datasets, and efficiency of only 15ms is achieved for single image processing. The average success rate of the AUBO i5 robotic platform with DHAG-95 parallel gripper reached 94.0% for single-object scenes, 86.7% for three-object scenes, and 84% for five-object scenes. Our approach has outperformed existing state-of-the-art methods.

Deep Reinforcement Learning-Based Robotic Grasping in Clutter and Occlusion

In robotic manipulation, object grasping is a basic yet challenging task. Dexterous grasping necessitates intelligent visual observation of the target objects by emphasizing the importance of spatial equivariance to learn the grasping policy. In this paper, two significant challenges associated with robotic grasping in both clutter and occlusion scenarios are addressed. The first challenge is the coordination of push and grasp actions, in which the robot may occasionally fail to disrupt the arrangement of the objects in a well-ordered object scenario. On the other hand, when employed in a randomly cluttered object scenario, the pushing behavior may be less efficient, as many objects are more likely to be pushed out of the workspace. The second challenge is the avoidance of occlusion that occurs when the camera itself is entirely or partially occluded during a grasping action. This paper proposes a multi-view change observation-based approach (MV-COBA) to overcome these two problems. The proposed approach is divided into two parts: 1) using multiple cameras to set up multiple views to address the occlusion issue; and 2) using visual change observation on the basis of the pixel depth difference to address the challenge of coordinating push and grasp actions. According to experimental simulation findings, the proposed approach achieved an average grasp success rate of 83.6%, 86.3%, and 97.8% in the cluttered, well-ordered object, and occlusion scenarios, respectively.

Object Pose Estimation via Pruned Hough Forest With Combined Split Schemes for Robotic Grasp

Robotic grasp in complex open-world scenarios requires an effective and generalizable perception. Estimating object’s pose is needed in a variety of practical grasping scenarios. Here we present a novel approach of pose estimation of textureless and textured objects. The algorithm utilizes a single RGB-D image to exploit depth invariant, oriented point pair feature as well as local contextual sensitivity in cluttered environments. To enhance the performance of the voting process and improve learning efficiency, we employ a global pruning algorithm that reduces the risk of overfitting and simplifies the structure of decision trees after compensating for the complementary information among multiple trees by optimizing a designed global objective function. Finally, we also refine the pose obtained from the above stage. The proposed approach of estimating 6-D (degree of freedom) poses of textured and textureless objects is evaluated on publicly available data sets against the recent works under various conditions. It illustrates that our framework is superior to these recent works. Further, we perform extensive qualitative experiments of robotic grasp to illustrate the proposed approach can be applied to practical scenarios. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article is motivated by the problem of the pose estimation of textured and textureless objects in clutter environments. It is difficult for conventional works to address the issue of estimating textured or textureless objects’ poses in such scenarios. We considered that a novel system should be able to obtain the 6-D poses of objects. Therefore, we investigate the combined use of multiple split functions with different characteristics. Learning the model based on Hough forests always cost much computational resource; therefore, we construct a novel pruned Hough forest for solving this issue. Through the comparison and robotic grasp verifications, the behavior of our system can be used in practical applications. In future, we will deploy the proposed system in robotic assembling tasks.

A Flexible Robotic Assembly System Combining CAD Based Localization, Compliance Control, and a Multi-Modal Gripper

As manufacturing trends shift towards customized production, the demand for agile automation systems capable of efficient adaptation to rapidly changing task requirements is rising. This work presents a flexible robotic assembly system that combines CAD based component localization, compliance control, and a multi-modal gripper to enable robust and efficient programming of complex tasks. The process can be easily configured for novel assemblies through a dedicated Graphical User Interface (GUI), which facilitates component identification and task sequencing. Component poses are extracted from 3D CAD models in reference to the assembly origin, while the active compliance scheme compensates for minor positioning errors. The gripper incorporates a parallel jaw element, a rotating module, and an electromagnet to minimize retooling delays. The first iteration of the proposed system placed first in the manufacturing track of the Robotic Grasping and Manipulation Competition of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), experimentally validating its efficiency.

EPM–MRE: Electropermanent Magnet–Magnetorheological Elastomer for Soft Actuation System and Its Application to Robotic Grasping

Conventional soft robotic grippers have been developed based on soft pneumatic actuators or using smart soft materials, but they need external pressure sources, are easily deteriorated or used in millimetric-scale, which leads to increase of the independency, instability, and vulnerability. In this study, we propose an EPM-MRE (Electropermanent Magnet-Magnetorheological elastomer) actuation system, which strengthens the independency and stability of soft actuation by non-contact magnetic force. We established its fundamental principle and investigated parameter design through developing a suction cup as a robotic gripper. We prototyped an EPM-MRE actuated suction cup based on magnetic-charge and tensile modeling as well as optimized the structure of EPM by introducing axisymmetric EPM with frustum pole and suction cup by introducing bi-silicone structure to uniformly activate MRE membrane. The activation of EPM by different current pulses and suction force affected by contact shapes and air gaps were tested. Evaluations showed that the suction cup could be activated with 10 ms and generate a maximum suction force of 9.2 N at a steady state with 0 J energy consumption. In conclusion, the EPM-MRE soft actuation system can be used as a robotic gripper.

HSV Color-Space-Based Automated Object Localization for Robot Grasping without Prior Knowledge

We propose a simple and robust HSV color-space-based algorithm that can automatically extract object position information without human intervention or prior knowledge. In manufacturing sites with high variability, it is difficult to recognize products through robot machine vision, especially in terms of extracting object information accurately, owing to various environmental factors such as the noise around objects, shadows, light reflections, and illumination interferences. The proposed algorithm, which does not require users to reset the HSV color threshold value whenever a product is changed, uses ROI referencing method to solve this problem. The algorithm automatically identifies the object’s location by using the HSV color-space-based ROI random sampling, ROI similarity comparison, and ROI merging. The proposed system utilizes an IoT device with several modules for the detection, analysis, control, and management of object data. The experimental results show that the proposed algorithm is very useful for industrial automation applications under complex and highly variable manufacturing environments.

Graph-Based Visual Manipulation Relationship Reasoning Network for Robotic Grasping.

To grasp the target object stably and orderly in the object-stacking scenes, it is important for the robot to reason the relationships between objects and obtain intelligent manipulation order for more advanced interaction between the robot and the environment. This paper proposes a novel graph-based visual manipulation relationship reasoning network (GVMRN) that directly outputs object relationships and manipulation order. The GVMRN model first extracts features and detects objects from RGB images, and then adopts graph convolutional network (GCN) to collect contextual information between objects. To improve the efficiency of relation reasoning, a relationship filtering network is built to reduce object pairs before reasoning. The experiments on the Visual Manipulation Relationship Dataset (VMRD) show that our model significantly outperforms previous methods on reasoning object relationships in object-stacking scenes. The GVMRN model is also tested on the images we collected and applied on the robot grasping platform. The results demonstrated the generalization and applicability of our method in real environment.

Irregular Depth Tiles: Automatically Generated Data Used for Network-based Robotic Grasping in 2D Dense Clutter

Irregular Depth Tiles: Automatically Generated Data Used for Network-based Robotic Grasping in 2D Dense Clutter

Finger‐Skin‐Inspired Flexible Optical Sensor for Force Sensing and Slip Detection in Robotic Grasping

AbstractThe finger skin of human plays a key role in amplifying and transferring tactile signals to sensory receptors, and it provides design guidelines for next‐generation flexible tactile sensors. To enable robotic applications such as dexterous manipulation and tactile feedback, the functions of force sensing and slip detection are crucial for flexible tactile sensors. Here, a finger‐skin inspired flexible optical (FIFO) sensor based on optical microfiber is reported, and this sensor has fingerprint‐like surface with parallel ridges and resin/polydimethylsiloxane multilayer structure with different stiffness, mimicking the structural characteristics of human finger skin. With these design features, the sensor is capable of measuring force with high durability and detecting slip with high sensitivity and short response time. Time‐resolved response curve and its wavelet transform provide useful information related to grasping and slipping when the sensor is in contact with objects with different shapes and materials. As a proof‐of‐concept demonstration, the sensor directly attached onto a robotic gripper is capable of measuring grasping force and monitoring object slippage during manipulation tasks, indicating its potential applications in advanced robotics.

On-Orbit Robotic Grasping of a Spent Rocket Stage: Grasp Stability Analysis and Experimental Results.

The increased complexity of the tasks that on-orbit robots have to undertake has led to an increased need for manipulation dexterity. Space robots can become more dexterous by adopting grasping and manipulation methodologies and algorithms from terrestrial robots. In this paper, we present a novel methodology for evaluating the stability of a robotic grasp that captures a piece of space debris, a spent rocket stage. We calculate the Intrinsic Stiffness Matrix of a 2-fingered grasp on the surface of an Apogee Kick Motor nozzle and create a stability metric that is a function of the local contact curvature, material properties, applied force, and target mass. We evaluate the efficacy of the stability metric in a simulation and two real robot experiments. The subject of all experiments is a chasing robot that needs to capture a target AKM and pull it back towards the chaser body. In the V-REP simulator, we evaluate four grasping points on three AKM models, over three pulling profiles, using three physics engines. We also use a real robotic testbed with the capability of emulating an approaching robot and a weightless AKM target to evaluate our method over 11 grasps and three pulling profiles. Finally, we perform a sensitivity analysis to demonstrate how a variation on the grasping parameters affects grasp stability. The results of all experiments suggest that the grasp can be stable under slow pulling profiles, with successful pulling for all targets. The presented work offers an alternative way of capturing orbital targets and a novel example of how terrestrial robotic grasping methodologies could be extended to orbital activities.

Robot Grasping System and Grasp Stability Prediction Based on Flexible Tactile Sensor Array

As an essential perceptual device, the tactile sensor can efficiently improve robot intelligence by providing contact force perception to develop algorithms based on contact force feedback. However, current tactile grasping technology lacks high-performance sensors and high-precision grasping prediction models, which limits its broad application. Herein, an intelligent robot grasping system that combines a highly sensitive tactile sensor array was constructed. A dataset that can reflect the grasping contact force of various objects was set up by multiple grasping operation feedback from a tactile sensor array. The stability state of each grasping operation was also recorded. On this basis, grasp stability prediction models with good performance in grasp state judgment were proposed. By feeding training data into different machine learning algorithms and comparing the judgment results, the best grasp prediction model for different scenes can be obtained. The model was validated to be efficient, and the judgment accuracy was over 98% in grasp stability prediction with limited training data. Further, experiments prove that the real-time contact force input based on the feedback of the tactile sensor array can periodically control robots to realize stable grasping according to the real-time grasping state of the prediction model.

Accurate Instance-Based Segmentation for Boundary Detection in Robot Grasping Application

With the rapid development of cameras and deep learning technologies, computer vision tasks such as object detection, object segmentation and object tracking are being widely applied in many fields of life. For robot grasping tasks, object segmentation aims to classify and localize objects, which helps robots to be able to pick objects accurately. The state-of-the-art instance segmentation network framework, Mask Region-Convolution Neural Network (Mask R-CNN), does not always perform an excellent accurate segmentation at the edge or border of objects. The approach using 3D camera, however, is able to extract the entire (foreground) objects easily but can be difficult or require a large amount of computation effort to classify it. We propose a novel approach, in which we combine Mask R-CNN with 3D algorithms by adding a 3D process branch for instance segmentation. Both outcomes of two branches are contemporaneously used to classify the pixels at the edge objects by dealing with the spatial relationship between edge region and mask region. We analyze the effectiveness of the method by testing with harsh cases of object positions, for example, objects are closed, overlapped or obscured by each other to focus on edge and border segmentation. Our proposed method is about 4 to 7% higher and more stable in IoU (intersection of union). This leads to a reach of 46% of mAP (mean Average Precision), which is a higher accuracy than its counterpart. The feasibility experiment shows that our method could be a remarkable promoting for the research of the grasping robot.