Real-time grasping strategies using event camera

Xiaoqian Huang,Rajkumar Muthusamy,Abdulla Ayyad,Yahya Zweiri,Dongming Gan,Lakmal Seneviratne,Mohamad Halwani,Dewald Swart

doi:10.1007/s10845-021-01887-9

Abstract

Robotic vision plays a key role for perceiving the environment in grasping applications. However, the conventional framed-based robotic vision, suffering from motion blur and low sampling rate, may not meet the automation needs of evolving industrial requirements. This paper, for the first time, proposes an event-based robotic grasping framework for multiple known and unknown objects in a cluttered scene. With advantages of microsecond-level sampling rate and no motion blur of event camera, the model-based and model-free approaches are developed for known and unknown objects’ grasping respectively. The event-based multi-view approach is used to localize the objects in the scene in the model-based approach, and then point cloud processing is utilized to cluster and register the objects. The proposed model-free approach, on the other hand, utilizes the developed event-based object segmentation, visual servoing and grasp planning to localize, align to, and grasp the targeting object. Using a UR10 robot with an eye-in-hand neuromorphic camera and a Barrett hand gripper, the proposed approaches are experimentally validated with objects of different sizes. Furthermore, it demonstrates robustness and a significant advantage over grasping with a traditional frame-based camera in low-light conditions.

Highlights

Robots equipped with grippers has become increasingly popular and important for grasping tasks in the industrial field, because they provide the industry with the benefit of cutting manufacturing time while improving throughput
Since the object segmentation and visual servoing are accomplished in the camera frame, the position adjustment is executed after visual servoing to compensate the manually measured deviation between the centers of event camera and Barrett hand
A model-based and a model-free approaches for multiple-object grasping in a cluttered scene are developed

Summary

Introduction

Robots equipped with grippers has become increasingly popular and important for grasping tasks in the industrial field, because they provide the industry with the benefit of cutting manufacturing time while improving throughput. The vision-based robotic grasping system can be categorized along various criteria (Kleeberger et al 2020). It can be summarized into analytic and data-driven methods depending on the analysis of the geometric properties of objects (Bohg et al 2013; Sahbani et al 2012). According to whether or not building up the object’s model, Journal of Intelligent Manufacturing (2022) 33:593–615 the vision-based grasping can be divided into model-based and model-free approaches (Zaidi et al 2017; Kleeberger et al 2020). Model-free methods are more flexible for both known and unknown objects by learning geometric parameters of objects based on vision. Lots of standard vision-based robotic grasping systems are explored for many applications, such as garbage sorting (Zhihong et al 2017), construction (Asadi et al 2021) and human interaction (Úbeda et al 2018)

Methods

Results

Conclusion