Pallet detection and localization with RGB image and depth data using deep learning techniques

Abstract

This paper presents a novel approach of pallet identification and localization algorithm (PILA) based on RGB image and depth data. The algorithm is implemented in C++ for real-time running and the RGB and depth data from low-cost RGB-D camera. Deep neural network (DNN) method is applied to detect and locate the pallet in the RGB images. The pallet's point cloud data is correlated with the labeled region of interest (ROI) in the RGB images through RGB-D fusion. The pallet's front-face plane is extracted and the orientation of the pallet is obtained at the same time. The triangle centric points of pallet's front-face could be determined with extracting x and y lines at the edge by the simple geometrical rules. Experimentally, the orientation angle and centric location of the two kinds of pallets are investigated with natural pallet surface without any artificial markings. The results show that the pallet could be located with the 3D localization accuracy of 1cm and the angle resolution of 0.4 degree at the distance of 3m. The end-to-end running time is less than 700 ms from CAN-IO interface and this is a promising solution for autonomous pallet picking instrument and self-driving forklift applications.