Bin-picking Research Articles

MetaGraspNetV2: All-in-One Dataset Enabling Fast and Reliable Robotic Bin Picking via Object Relationship Reasoning and Dexterous Grasping

MetaGraspNetV2: All-in-One Dataset Enabling Fast and Reliable Robotic Bin Picking via Object Relationship Reasoning and Dexterous Grasping

Development of Annulus-Object Random Bin Picking System based on Rapid Establishment of RGB-D Images

With the development of the automation industry, robotic arms, and vision applications are no longer limited to fixed actions of the past. Production lines increasingly require the recognition and grasping of objects in complex environments, emphasizing quick setup and stability. In this paper, a rapidly constructed eye-hand system for robotic arm grasping, which enables fast and efficient object manipulation, particularly for stacking, is introduced. Initially, images were captured using a camera to generate extensive datasets from a limited number of images. Objects were subsequently segmented and categorized using deep learning networks for object detection and instance segmentation. Three-dimensional position information was obtained from an RGB-D camera. Finally, object poses were determined based on plane normal vectors, and gripping positions were manually marked. This reduced the time required for grab-point identification, model training, and pose localization. Based on experimental results, the grasping procedure proposed in this paper is suitable for various object-grasping scenarios. It achieved impressive picking success rates of 96% for unstacked annular objects and 90.86% for random bin annular objects, respectively. In the final experiment, following depth information filtering, a success rate of 95.1% was attained with random bin annular object picking.

Filtering Organized 3D Point Clouds for Bin Picking Applications.

In robotic bin-picking applications, autonomous robot action is guided by a perception system integrated with the robot. Unfortunately, many perception systems output data contaminated by spurious points that have no correspondence to the real physical objects. Such spurious points in 3D data are the outliers that may spoil obstacle avoidance planning executed by the robot controller and impede the segmentation of individual parts in the bin. Thus, they need to be removed. Many outlier removal procedures have been proposed that work very well on unorganized 3D point clouds acquired for different, mostly outdoor, scenarios, but these usually do not transfer well to the manufacturing domain. This paper presents a new filtering technique specifically designed to deal with the organized 3D point cloud acquired from a cluttered scene, which is typical for a bin-picking task. The new procedure was tested on six different datasets (bins filled with different parts) and its performance was compared with the generic statistical outlier removal procedure. The new method outperforms the general procedure in terms of filtering efficacy, especially on datasets heavily contaminated by numerous outliers.

A Novel Robotic Grasp Detection Framework Using Low-Cost RGB-D Camera for Industrial Bin Picking

A Novel Robotic Grasp Detection Framework Using Low-Cost RGB-D Camera for Industrial Bin Picking

A Deep Learning-Enhanced Stereo Matching Method and Its Application to Bin Picking Problems Involving Tiny Cubic Workpieces

This paper proposes a stereo matching method enhanced by object detection and instance segmentation results obtained through the use of a deep convolutional neural network. Then, this method is applied to generate a picking plan to solve bin picking problems, that is, to automatically pick up objects with random poses in a stack using a robotic arm. The system configuration and bin picking process flow are suggested using the proposed method, and it is applied to bin picking problems, especially those involving tiny cubic workpieces. The picking plan is generated by applying the Harris corner detection algorithm to the point cloud in the generated three-dimensional map. In the experiments, two kinds of stacks consisting of cubic workpieces with an edge length of 10 mm or 5 mm are tested for bin picking. In the first bin picking problem, all workpieces are successfully picked up, whereas in the second, the depths of the workpieces are obtained, but the instance segmentation process is not completed. In future work, not only cubic workpieces but also other arbitrarily shaped workpieces should be recognized in various types of bin picking problems.

Autonomous Robotic Bin Picking Platform Generated From Human Demonstration and YOLOv5

Abstract Vision-based robots have been utilized for pick-and-place operations by their ability to find object poses. As they progress into handling a variety of objects with cluttered state, more flexible and lightweight operations have been presented. In this paper, an autonomous robotic bin-picking platform is proposed. It combines human demonstration with a collaborative robot for the flexibility of the objects and YOLOv5 neural network model for faster object localization without prior computer-aided design models or dataset in the training. After a simple human demonstration of which target object to pick and place, the raw color and depth images were refined, and the one on top of the bin was utilized to create synthetic images and annotations for the YOLOv5 model. To pick up the target object, the point cloud was lifted using the depth data corresponding to the result of the trained YOLOv5 model, and the object pose was estimated by matching them with Iterative Closest Points (ICP) algorithm. After picking up the target object, the robot placed it where the user defined it in the previous human demonstration stage. From the result of experiments with four types of objects and four human demonstrations, it took a total of 0.5 s to recognize the target object and estimate the object pose. The success rate of object detection was 95.6%, and the pick-and-place motion of all the found objects was successful.

Learning to Dexterously Pick or Separate Tangled-Prone Objects for Industrial Bin Picking

Learning to Dexterously Pick or Separate Tangled-Prone Objects for Industrial Bin Picking

Bin Picking for Ship-Building Logistics Using Perception and Grasping Systems

Bin picking is a challenging task involving many research domains within the perception and grasping fields, for which there are no perfect and reliable solutions available that are applicable to a wide range of unstructured and cluttered environments present in industrial factories and logistics centers. This paper contributes with research on the topic of object segmentation in cluttered scenarios, independent of previous object shape knowledge, for textured and textureless objects. In addition, it addresses the demand for extended datasets in deep learning tasks with realistic data. We propose a solution using a Mask R-CNN for 2D object segmentation, trained with real data acquired from a RGB-D sensor and synthetic data generated in Blender, combined with 3D point-cloud segmentation to extract a segmented point cloud belonging to a single object from the bin. Next, it is employed a re-configurable pipeline for 6-DoF object pose estimation, followed by a grasp planner to select a feasible grasp pose. The experimental results show that the object segmentation approach is efficient and accurate in cluttered scenarios with several occlusions. The neural network model was trained with both real and simulated data, enhancing the success rate from the previous classical segmentation, displaying an overall grasping success rate of 87.5%.

Learning Efficient Policies for Picking Entangled Wire Harnesses: An Approach to Industrial Bin Picking

Wire harnesses are essential connecting components in manufacturing industry but are challenging to be automated in industrial tasks such as bin picking. They are long, flexible and tend to get entangled when randomly placed in a bin. This makes it difficult for the robot to grasp a single one in dense clutter. Besides, training or collecting data in simulation is challenging due to the difficulties in modeling the combination of deformable and rigid components for wire harnesses. In this work, instead of directly lifting wire harnesses, we propose to grasp and extract the target following a circle-like trajectory until it is untangled. We learn a policy from real-world data that can infer grasps and separation actions from visual observation. Our policy enables the robot to efficiently pick and separate entangled wire harnesses by maximizing success rates and reducing execution time. To evaluate our policy, we present a set of real-world experiments on picking wire harnesses. Our policy achieves an overall 84.6% success rate compared with 49.2% in baseline. We also evaluate the effectiveness of our policy under different clutter scenarios using unseen types of wire harnesses. Results suggest that our approach is feasible for handling wire harnesses in industrial bin picking.

Localization and Grasp Planning for Bin Picking of Deformable Linear Objects

Localization and Grasp Planning for Bin Picking of Deformable Linear Objects

Using Simulation to Evaluate a Tube Perception Algorithm for Bin Picking

Bin picking is a challenging problem that involves using a robotic manipulator to remove, one-by-one, a set of objects randomly stacked in a container. In order to provide ground truth data for evaluating heuristic or machine learning perception systems, this paper proposes using simulation to create bin picking environments in which a procedural generation method builds entangled tubes that can have curvatures throughout their length. The output of the simulation is an annotated point cloud, generated by a virtual 3D depth camera, in which the tubes are assigned with unique colors. A general metric based on micro-recall is proposed to compare the accuracy of point cloud annotations with the ground truth. The synthetic data is representative of a high quality 3D scanner, given that the performance of a tube modeling system when given 640 simulated point clouds was similar to the results achieved with real sensor data. Therefore, simulation is a promising technique for the automated evaluation of solutions for bin picking tasks.

A Sim-to-Real Object Recognition and Localization Framework for Industrial Robotic Bin Picking

We present a generic and robust sim-to-real deep-learning-based framework, namely S2R-Pick, for fast and accurate object recognition and localization in industrial robotic bin picking. Unlike existing works designed for general everyday environments, objects for industrial bin picking are often texture-less, metallic, and also suffer from severe occlusion and cluttering. In this work, we first develop an automated synthetic data generation pipeline to produce large-scale photo-realistic data with precise annotations. We then formulate an instance segmentation network for object recognition and a 6D pose estimation network for localization. We cascade and train the two networks only on our generated synthetic data and can directly transfer them for processing real inputs without needing to retrain them on real samples. Extensive experiments show the effectiveness and superiority of our framework over state-of-the-art methods.

Design of Two-sided Gripper for Bin Picking

Design of Two-sided Gripper for Bin Picking

Scooping Manipulation Via Motion Control With a Two-Fingered Gripper and Its Application to Bin Picking

This letter introduces a new method for scooping manipulation in constrained environments. The presented technique lends itself to relatively thin objects, with a large width-to-thickness ratio, which are commonly seen in many domestic and industrial tasks yet can cause considerable difficulty in handling. We present a novel approach to scooping based on a mobility analysis that leads to a new way to perform the operation via motion control with a two-fingered gripper. Our method thus expands the ways in which the manipulation of scooping can be performed, without necessitating underactuated control or a compliant mechanism. Our variable-length digit is suggested as a key facilitator for realizing the motion-controlled scooping maneuver. We present an extensive set of experiments showing the effectiveness of our approach in various scenarios featuring a constrained workspace, including bin picking from clutter.

Dig-Grasping via Direct Quasistatic Interaction Using Asymmetric Fingers: An Approach to Effective Bin Picking

This letter introduces a new method for simultaneously singulating and picking objects from clutter. The method can lead to effective robotic bin picking, which still remains elusive despite its importance in many industrial and domestic applications, especially for objects with a thin profile. We leverage planar quasistatic pushing manipulation as a way of standardized physical interaction between a robot and the object to pick. A gripper designed with digit asymmetry, realized as a two-fingered gripper with different finger lengths, is suggested as the key to successful singulating and picking through the controlled pushing maneuver. A detailed account of the manipulation process and design principles will be presented. An extensive set of experiments validate the effectiveness of our approach in three-dimensional bin picking tasks. Beyond picking, more complex manipulation capabilities such as autonomous pick-and-place/pack will also be presented.

Separating Entangled Workpieces in Random Bin Picking using Deep Reinforcement Learning

Entangled workpiece situations often occur in random bin picking of chaotically stored objects and are a common source of problem in the bin picking process. Previous methods for averting this problem, such as randomly shaking the gripper over the bin, lead to decreasing production efficiency and an increase in cycle time. A promising new strategy uses supervised learning and deep neural networks to learn the separation. However, this approach requires a large amount of labeled data. To overcome this issue, this paper proposes a deep reinforcement learning approach to separate entangled workpieces and to minimize the setup effort.

Sheet-Based Gripper Featuring Passive Pull-In Functionality for Bin Picking and for Picking Up Thin Flexible Objects

This letter presents a novel versatile parallel gripper equipped with a sheet-embedded belt that enables passive pull-in operations. The belt is wrapped around each fingertip of the gripper, and it passively moves with the opening and closing motions of the gripper. The fingertip is made of plastic, and it is deflected when a grasping force is applied to the fingertip. This deflection enables the belt to roll up with the closing motion of the gripper, thereby passively pulling the grasped workpiece towards the gripper. This pull-in mechanism is effective in performing bin-picking tasks as well as in picking up thin, flexible objects one at a time. The softness of the sheet-embedded belt surface provides high adaptability to the shape of the object; therefore, many kinds of workpieces can be picked up. The effectiveness of the proposed gripper is experimentally demonstrated in this study.

Detecting and Solving Tube Entanglement in Bin Picking Operations

Manufacturing and production industries are increasingly turning to robots to carry out repetitive picking operations in an efficient manner. This paper focuses on tackling the novel challenge of automating the bin picking process for entangled objects, for which there is very little research. The chosen case study are sets of freely curved tubes, which are prone to occlusions and entanglement. The proposed algorithm builds a representation of the tubes as an ordered list of cylinders and joints using a point cloud acquired by a 3D scanner. This representation enables the detection of occlusions in the tubes. The solution also performs grasp planning and motion planning, by evaluating post-grasp trajectories via simulation using Gazebo and the ODE physics engine. A force/torque sensor is used to determine how many items were picked by a robot gripper and in which direction it should rotate to solve cases of entanglement. Real-life experiments with sets of PVC tubes and rubber radiator hoses showed that the robot was able to pick a single tube on the first try with success rates of 99% and 93%, respectively. This study indicates that using simulation for motion planning is a promising solution to deal with entangled objects.

Increasing the Robustness of Random Bin Picking by Avoiding Grasps of Entangled Workpieces

In bin picking applications, a robot often picks workpieces that have a complex geometry. This complex geometry can cause entanglements between workpieces resulting in failed grips. This paper presents a machine learning approach to avoid these situations and therefore improves the calculation of suitable grips. Using the depth map of the workpieces and their surrounding neighborhood, a convolutional neural network, which is trained on simulated data, predicts whether an entanglement is present. This information is used to select and calculate the most reliable grip. By avoiding such entangled workpiece situations the robustness of random bin picking increases.

Construction of Virtual Datasets for Bin Picking

Bin Picking is a typical problem for identifying and sorting complex stacked objects, the difficulty of which lies in the identification of objects. This paper uses deep learning method to identify objects preliminarily. Because the deep learning method requires high-quality datasets, the traditional manual annotation method is too difficult to complete under the industrial production environment. Therefore, this paper proposes a method to build virtual datasets based on virtual simulation technology. It uses OpenGL technology and Bullet physics engine to build virtual scenes. Then through camera loading and scene drawing, the datasets including RGB images, segmentation annotation images and so on are generated. Experimental results show that the model trained on virtual datasets by deep learning can achieve good accuracy, so the feasibility and practical application value of the method are verified.