Extracting cow point clouds from multi-view RGB images with an improved YOLACT++ instance segmentation

Abstract

Obtaining shape features from a farm environment is the basis for animal size measurement, body condition scoring, weight prediction, and individual identification. To improve the robustness of point cloud segmentation and fully utilize 2D (Two dimension) and 3D (Three dimension) information for segmenting animal shape data, a novel pixel-level segmentation method of cow point clouds was proposed by fusing real images with high resolution and point clouds. First, the 3D point clouds of cows were generated from the multi-view images by using the Structure from Motion (SfM) algorithm, and the corresponding relationship between the point clouds and the real image was established by the camera interior and exterior parameters. Second, for 2D image segmentation, the improved, single-stage instance segmentation algorithm YOLACT++ (You Only Look At CoefficienTs) was applied to improve the segmentation accuracy of cow body regions, and the average precision of multi-view images was 85.9%. Last, in point cloud segmentation, the image segmentation results of the cows were mapped to 3D point clouds by the correspondence between point clouds and images, and the visual points of different viewpoints were fused to obtain the 3D point cloud of the target cow. The average relative error of truncated areas between the proposed method and manual segmentation of the point cloud was 2.18%, and the change in the truncated area caused by the increase in images was less than 5% with the appropriate number of images (20–40). The results demonstrated the effectiveness of the proposed method in 3D point cloud segmentation of cows and its potential application in animal shape acquisition.