Bin-picking Tasks Research Articles

Latent Space Search-Based Adaptive Template Generation for Enhanced Object Detection in Bin-Picking Applications.

Template matching is a common approach in bin-picking tasks. However, it often struggles in complex environments, such as those with different object poses, various background appearances, and varying lighting conditions, due to the limited feature representation of a single template. Additionally, during the bin-picking process, the template needs to be frequently updated to maintain detection performance, and finding an adaptive template from a vast dataset poses another challenge. To address these challenges, we propose a novel template searching method in a latent space trained by a Variational Auto-Encoder (VAE), which generates an adaptive template dynamically based on the current environment. The proposed method was evaluated experimentally under various conditions, and in all scenarios, it successfully completed the tasks, demonstrating its effectiveness and robustness for bin-picking applications. Furthermore, we integrated our proposed method with YOLO, and the experimental results indicate that our method effectively improves YOLO's detection performance.

Sparse Convolution-Based 6D Pose Estimation for Robotic Bin-Picking With Point Clouds

Abstract Estimating the orientation and position of objects is a crucial step in robotic bin-picking tasks. The challenge lies in the fact that, in real-world scenarios, a diverse array of objects is often randomly stacked, resulting in significant occlusion. This study introduces an innovative approach aimed at predicting 6D poses by processing point clouds through a two-stage neural network. In the initial stage, a network for scenes with low-textured environments is designed. Its purpose is to perform instance segmentation and provide an initial pose estimation. Entering the second stage, a pose refinement network is suggested. This network is intended to enhance the precision of pose prediction, building upon the output from the first stage. To tackle the challenge of resource-intensive annotation, a simulation technique is employed to generate a synthetic dataset. Additionally, a dedicated software tool has been developed to annotate real point cloud datasets. In practical experiments, our method demonstrated superior performance compared to baseline methods such as PointGroup and Iterative Closest Point. This superiority is evident in both segmentation accuracy and pose refinement. Moreover, practical grasping experiments have underscored the method's efficacy in real-world industrial robot bin-picking applications. The results affirm its capability to successfully address the challenges produced by occluded and randomly stacked objects.

Filtering Organized 3D Point Clouds for Bin Picking Applications.

In robotic bin-picking applications, autonomous robot action is guided by a perception system integrated with the robot. Unfortunately, many perception systems output data contaminated by spurious points that have no correspondence to the real physical objects. Such spurious points in 3D data are the outliers that may spoil obstacle avoidance planning executed by the robot controller and impede the segmentation of individual parts in the bin. Thus, they need to be removed. Many outlier removal procedures have been proposed that work very well on unorganized 3D point clouds acquired for different, mostly outdoor, scenarios, but these usually do not transfer well to the manufacturing domain. This paper presents a new filtering technique specifically designed to deal with the organized 3D point cloud acquired from a cluttered scene, which is typical for a bin-picking task. The new procedure was tested on six different datasets (bins filled with different parts) and its performance was compared with the generic statistical outlier removal procedure. The new method outperforms the general procedure in terms of filtering efficacy, especially on datasets heavily contaminated by numerous outliers.

Untangling Multiple Deformable Linear Objects in Unknown Quantities With Complex Backgrounds

The manipulation of deformable objects, especially deformable linear objects (DLOs), represents an open challenge in robotics. The keys to solving the problem are accurately recognizing the topological model describing the entwined state of DLO(s) and a manipulation strategy based on it. The situation becomes more challenging in practical applications since the DLO(s) may be placed under complex and unencountered environments, making distinguishing between the targets and the background difficult. In addition, with the information obtained from a sensor system with a limited field of sense, a robot has to treat the encountered DLO(s) as multiple ones of an unknown quantity. This paper proposes a solution based on deep learning techniques for these complicated scenarios. The approach can derive a topological model describing the entangled structure of single or multiple DLOs. Based on the model, we proposed a strategy for untangling the DLO(s), considering both the possible self-tangling in one DLO and the tangling between multiple DLOs. The strategy ensures that the entangled DLO(s) can be arranged to be the neatest state given a limited field of view. The feasibility and effectiveness of the proposed solution were verified by untangling experiments utilizing a dual-arm robot system. Even if the exact quantity of the DLO(s) is unknown, the robots can still untangle the DLO(s). Moreover, the proposed approach performed robustness to unfamiliar background textures, which is preferable in practical applications. Dataset used in this paper can be found at https://github.com/lancexz/dlos-dataset. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article proposes an automatic solution for untangling an unknown quantity of DLO(s) using robots. Issues concerned with complex backgrounds, an unknown quantity of DLOs, and limits of field of view, which are easily encountered in practical applications, are solved. Experiments demonstrated the feasibility and robustness of the approach in different scenarios. Potential applications for the proposed method include pipeline bin-picking tasks, rope-like object manipulation in household robots, and cable assembly in industrial areas. Moreover, the topological state recognition process can also be utilized in knot-tying studies and producing datasets. As the current system can not deal with scenes where DLO segments or crossings severely overlap, practitioners should integrate real-time tracking, deformation control method, and unqualified cases detection to avoid the appearance of such cases during manipulation. Future works should also consider the rebound caused by the elastic potential of DLOs.

Minimizing the Effect of Specular Reflection on Object Detection and Pose Estimation of Bin Picking Systems Using Deep Learning

The rapid evolution towards industrial automation has widened the usage of industrial applications, such as robot arm manipulation and bin picking. The performance of these applications relies on object detection and pose estimation through visual data. In fact, the clarity of those data significantly influences the accuracy of object detection and pose estimation. However, a majority of visual data corresponding to metal or glossy surfaces tend to have specular reflections that reduce the accuracy. Hence, this work aims to improve the performance of industrial bin-picking tasks by reducing the effects of specular reflections. This work proposes a deep learning (DL)-based neural network model named SpecToPoseNet to improve object detection and pose estimation accuracy by intelligently removing specular reflections. The proposed work implements a synthetic data generator to train and test the SpecToPoseNet. The conceptual breakthrough of this work is its ability to remove specular reflections from scenarios with multiple objects. With the use of the proposed method, we could reduce the fail rate of object detection to 7%, which is much less compared to specular images (27%), U-Net (20%), and the basic SpecToPoseNet model (11%). Thus, it is claimable that the performance improvements gained are positive influences of the proposed DL-based contexts such as bin-picking.

Data-Driven Object Pose Estimation in a Practical Bin-Picking Application.

This paper addresses the problem of pose estimation from 2D images for textureless industrial metallic parts for a semistructured bin-picking task. The appearance of metallic reflective parts is highly dependent on the camera viewing direction, as well as the distribution of light on the object, making conventional vision-based methods unsuitable for the task. We propose a solution using direct light at a fixed position to the camera, mounted directly on the robot’s gripper, that allows us to take advantage of the reflective properties of the manipulated object. We propose a data-driven approach based on convolutional neural networks (CNN), without the need for a hard-coded geometry of the manipulated object. The solution was modified for an industrial application and extensively tested in a real factory. Our solution uses a cheap 2D camera and allows for a semi-automatic data-gathering process on-site.

Learning robot anomaly recovery skills from multiple time-driven demonstrations

Robots are prone to making anomalies when performing manipulation tasks in unstructured environments, it is often desirable to rapidly adapt the robotic behavior to avoid environmental changes by learning from experts’ demonstrations. We propose a framework for learning robot anomaly recovery skills from time-driven demonstrations based on a Gaussian process regression with prior mean derived by Gaussian mixture regression, named as mean-prior GPR (MP-GPR), which allows an end-user to adjust the anomalous trajectory intuitively by simultaneously considering the variability of the demonstrations and the adaptation of recovery skills. Evaluations are divided into two phases, a benchmarking dataset with robot reaching, pushing, writing, and pressing tasks are first used to verify the path accuracy and variability, and then a real-time robot bin-picking task for evaluating the adaptation of the framework. Our method has a fair comparison with probabilistic-based methods in the field of robot learning from demonstrations, including Gaussian mixture regression, probabilistic movement primitives, and kernelized movement primitives. The results indicate that our proposed method can efficiently encode the variability from multiple demonstrations and rapidly anomaly recovery skills learning by modulating a learned trajectory to safe via-points.

Probabilistic Multi-View Fusion of Active Stereo Depth Maps for Robotic Bin-Picking

The reliable fusion of depth maps from multiple viewpoints has become an important problem in many 3D reconstruction pipelines. In this work, we investigate its impact on robotic bin-picking tasks such as 6D object pose estimation. The performance of object pose estimation relies heavily on the quality of depth data. However, due to the prevalence of shiny surfaces and cluttered scenes, industrial grade depth cameras often fail to sense depth or generate unreliable measurements from a single viewpoint. To this end, we propose a novel probabilistic framework for scene reconstruction in robotic bin-picking. Based on active stereo camera data, we first explicitly estimate the uncertainty of depth measurements for mitigating the adverse effects of both noise and outliers. The uncertainty estimates are then incorporated into a probabilistic model for incrementally updating the scene. To extensively evaluate the traditional fusion approach alongside our own approach, we will release a novel representative dataset with multiple views for each bin and curated parts. Over the entire dataset, we demonstrate that our framework outperforms a traditional fusion approach by a 12.8% reduction in reconstruction error, and 6.1% improvement in detection rate. The dataset will be available at https://www.trailab.utias.utoronto.ca/robi .

Sheet-Based Gripper Featuring Passive Pull-In Functionality for Bin Picking and for Picking Up Thin Flexible Objects

This letter presents a novel versatile parallel gripper equipped with a sheet-embedded belt that enables passive pull-in operations. The belt is wrapped around each fingertip of the gripper, and it passively moves with the opening and closing motions of the gripper. The fingertip is made of plastic, and it is deflected when a grasping force is applied to the fingertip. This deflection enables the belt to roll up with the closing motion of the gripper, thereby passively pulling the grasped workpiece towards the gripper. This pull-in mechanism is effective in performing bin-picking tasks as well as in picking up thin, flexible objects one at a time. The softness of the sheet-embedded belt surface provides high adaptability to the shape of the object; therefore, many kinds of workpieces can be picked up. The effectiveness of the proposed gripper is experimentally demonstrated in this study.

Fast Object Pose Estimation Using Adaptive Threshold for Bin-Picking

Robotic bin-picking is a common process in modern manufacturing, logistics, and warehousing that aims to pick-up known or unknown objects with random poses out of a bin by using a robot-camera system. Rapid and accurate object pose estimation pipelines have become an escalating issue for robot picking in recent years. In this paper, a fast 6-DoF (degrees of freedom) pose estimation pipeline for random bin-picking is proposed in which the pipeline is capable of recognizing different types of objects in various cluttered scenarios and uses an adaptive threshold segment strategy to accelerate estimation and matching for the robot picking task. Particularly, our proposed method can be effectively trained with fewer samples by introducing the geometric properties of objects such as contour, normal distribution, and curvature. An experimental setup is designed with a Kinova 6-Dof robot and an Ensenso industrial 3D camera for evaluating our proposed methods with respect to four different objects. The results indicate that our proposed method achieves a 91.25% average success rate and a 0.265s average estimation time, which sufficiently demonstrates that our approach provides competitive results for fast objects pose estimation and can be applied to robotic random bin-picking tasks.

Addressing perception uncertainty induced failure modes in robotic bin-picking

We present a comprehensive approach to handle perception uncertainty to reduce failure rates in robotic bin-picking. Our focus is on mixed-bins. We identify the main failure modes at various stages of the bin-picking task and present methods to recover from them. If uncertainty in part detection leads to perception failure, then human intervention is invoked. Our approach estimates the confidence in the part match provided by an automated perception system, which is used to detect perception failures. Human intervention is also invoked if uncertainty in estimated part location and orientation leads to a singulation planning failure. We have developed a user interface that enables remote human interventions when necessary. Finally, if uncertainty in part posture in the gripper leads to failure in placing the part with the desired accuracy, sensor-less fine-positioning moves are used to correct the final placement errors. We have developed a fine-positioning planner with a suite of fine-motion strategies that offer different tradeoffs between completion time and postural accuracy at the destination. We report our observations from system characterization experiments with a dual-armed Baxter robot, equipped with a Ensenso three-dimensional camera, to perform bin-picking on mixed-bins.

Automated Bin-Picking with Active Vision

In this research, an automated robotic bin-picking system employing active vision for picking up randomly distributed plumbing parts is presented. This system employs an actively-controlled single eye-in-hand system to observe structured light projected onto a set of plumbing parts in a bin. By using image processing and iterative closest point (ICP) algorithms, a single plumbing part that could possibly be taken from the bin is detected. Specifically, by projecting stationary structured light patterns onto the set of plumbing objects, the features on the surfaces of plumbing parts can be reconstructed by actively moving the eye-in-hand camera while performing visual tracking of those features. An effective 3D segmentation technique is employed to extract the point cloud of a single plumbing part that can possibly be grasped successfully. Once the object point cloud is obtained, one needs to determine the coordinate transformation from the end-effector to the selected plumbing part for grasping motion. With the point cloud matching result based on utilizing the ICP algorithm, the position and orientation of the selected plumbing part can be correctly estimated if the deviation of the object point cloud from the model point cloud is small. The control command can thus be given to the robotic manipulator to accomplish the automated bin-picking task. To effectively expand the allowed deviation of the object point cloud, an approximate pose estimation algorithm is employed before performing the ICP algorithm. The proposed approach can virtually estimate any pose of the plumbing part and has been successfully experimented with an industrial manipulator equipped with eye-in-hand single-camera vision and a LCD projector fixed in the work space demonstrating the feasibility and effectiveness. The proposed automated bin-picking system appears to be cost-effective and have great potentials in industrial factory automation applications.

Autonomous manipulation with a general-purpose simple hand

While complex hands seem to offer generality, simple hands are often more practical. This raises the question: how do generality and simplicity trade off in the design of robot hands? This paper explores the tension between simplicity in hand design and generality in hand function. It raises arguments both for and against simple hands, it considers several familiar examples, and it proposes an approach for autonomous manipulation using a general-purpose but simple hand. We explore the approach in the context of a bin-picking task, focused on grasping, recognition, and localization. The central idea is to use learned knowledge of stable grasp poses as a cue for object recognition and localization. This leads to some novel design criteria, such as minimizing the number of stable grasp poses. Finally, we describe experiments with two prototype hands to perform bin-picking of highlighter markers.

Stable recognition of specular objects by the eigenwindow method

This paper proposes a recognition method for partially occluded objects in bin-picking tasks using principal component analysis. Although it is well known to be effective in recognizing an isolated object, as was shown by Murase and Nayar, the current method cannot be applied to partially occluded objects that are typical in bin-picking tasks. The analysis also requires that the object be centered in an image before recognition. These limitations of eigenspace analysis are due to the fact that the whole appearance of an object is utilized as a template for the analysis. We propose a new method, referred to as the “eigenwindow” method, that stores multiple partial appearances of an object in an eigenspace. Such partial appearances require a large amount of memory space. Three measurements, detectability, uniqueness, and reliability, among windows are developed to eliminate redundant windows and thereby reduce memory requirements. Using a pose-clustering method among windows, the method determines the pose of an object and the object type of itself. We have implemented the method and verified its validity with recognition of multispecularity objects. © 1998 Scripta Technica, Syst Comp Jpn, 29(7): 12–20, 1998

Stable recognition of specular objects by the eigenwindow method

This paper proposes a recognition method for partially occluded objects in bin-picking tasks using principal component analysis. Although it is well known to be effective in recognizing an isolated object, as was shown by Murase and Nayar, the current method cannot be applied to partially occluded objects that are typical in bin-picking tasks. The analysis also requires that the object be centered in an image before recognition. These limitations of eigenspace analysis are due to the fact that the whole appearance of an object is utilized as a template for the analysis. We propose a new method, referred to as the “eigenwindow” method, that stores multiple partial appearances of an object in an eigenspace. Such partial appearances require a large amount of memory space. Three measurements, detectability, uniqueness, and reliability, among windows are developed to eliminate redundant windows and thereby reduce memory requirements. Using a pose-clustering method among windows, the method determines the pose of an object and the object type of itself. We have implemented the method and verified its validity with recognition of multispecularity objects. © 1998 Scripta Technica, Syst Comp Jpn, 29(7): 12–20, 1998

Detectability, uniqueness, and reliability of eigen windows for stable verification of partially occluded objects

This paper describes a method for recognizing partially occluded objects for bin-picking tasks using eigenspace analysis, referred to as the eigen window method, that stores multiple partial appearances of an object in an eigenspace. Such partial appearances require a large amount of memory space. Three measurements, detectability, uniqueness, and reliability, on windows are developed to eliminate redundant windows and thereby reduce memory requirements. Using a pose clustering technique, the method determines the pose of an object and the object type itself. We have implemented the method and verified its validity.

Generating an interpretation tree from a CAD model for 3D-object recognition in bin-picking tasks

This article describes a method to generate 3D-object recognition algorithms from a geometrical model for bin-picking tasks. Given a 3D solid model of an object, we first generate apparent shapes of an object under various viewer directions. Those apparent shapes are then classified into groups (representative attitudes) based on dominant visible faces and other features. Based on the grouping, recognition algorithms are generated in the form of an interpretation tree. The interpretation tree consists of two parts: the first part for classifying a target region in an image into one of the shape groups, and the second part for determining the precise attitude of the object within that group. We have developed a set of rules to find out what appropriate features are to be used in what order to generate an efficient and reliable interpretation tree. Features used in the interpretation tree include inertia of a region, relationship to the neighboring regions, position and orientation of edges, and extended Gaussian images. This method has been applied in a task for bin-picking objects that include both planar and cylindrical surfaces. As sensory data, we have used surface orientations from photometric stereo, depth from binocular stereo using oriented-region matching, and edges from an intensity image.