Tomato 3D pose detection algorithm based on keypoint detection and point cloud processing

Abstract

Tomatoes are widely grown all over the world and are one of the favourite vegetables of humanity. Tomato harvesting requires a lot of labor. With the aging population and the increasing demand for tomatoes, it is imminent to develop tomato picking robots. In the picking environment with a complex background, a 3D pose of the target is essential. It can provide guidance for robotic arm attitude and obstacle avoidance during picking. In this paper, a tomato pose detection algorithm (TPD) is proposed for 3D pose detection of a single fruit of clustered tomato. The TPD algorithm is divided into two modules: YOLO-lmk model and the point cloud processing module. By analyzing the network structure, optimizing parameters, optimizing loss function, adding attention mechanism and adding keypoint prediction, a YOLO-lmk model with YOLO v5s as the core is proposed to realize tomato bounding box and keypoint detection. The point cloud processing module is composed of point cloud segmentation, voxel downsampling, removing outliers, Euclidean color clustering, RANSAC sphere fitting, and keypoint index. As the experimental results show, YOLO-lmk model bounding box mAP is 92.9%, dlmk is 7.9, FLoating point Operations (FLOPs) is 16.6B, and speed is 0.062 s/sheet. The dlmk represents the Euclidean distance between true keypoints and predicted keypoints. Compared with YOLO v5s, mAP is increased 1.8%, and FLOPs is increased only 0.1B. The point cloud processing module only takes 0.028 s to complete a tomato 3D pose detection, which is fast. The accuracy of the TPD algorithm in detecting tomatoes is 93.4%, and the time cost is 0.09 s for detecting one tomato. The TPD algorithm can provide a theoretical basis for tomato 3D pose detection, and provide a reference for other fruits (pears, citrus, apples) 3D pose detection.