Fusion of RGB, optical flow and skeleton features for the detection of lameness in dairy cows

Abstract

Lameness is common in dairy cows. Methods that used RGB-based images to detect lameness in dairy cows always have low accuracy and poor robustness because of the complex background environment involved. In this study, a lameness detection method for dairy cows based on multiple features, including RGB, optical flow and skeletons, was proposed. The network was divided into three branches according to different inputs: for Branch1 and Branch3, a convolutional neural network (CNN) was used to predict lameness according to the input images and optical flow, and for Branch2, a spatial temporal graph convolutional network (ST-GCN) was used to predict lameness according to the cows’ skeletons. Finally, the weight was adjusted, and the prediction scores of these three branches were fused to complete the lameness detection process. In this research, 680 different videos were used for training and testing. The segmentation ratio of the train set and test set was 6:1. The best ACC was 97.20% when the weight of the RGB, optical flow and skeletons was 1:0.5:0.5. To verify the robustness of the method, the gamma transform was used to adjust the brightness of the image to simulate a change in illumination. Under different illumination settings, the maximum error of the method was 2.65%, which was significantly lower than other methods without skeletons. The results showed that the proposed method was effective for the detection of early-stage lameness, severe lameness, and non-lameness of dairy cows. • Fusion of RGB, optical flow and skeleton features for the detection of lameness in dairy cows. • Cow skeletons are used to increase network attention to cows. • A spatial temporal graph convolutional network (ST-GCN) was used to predict lameness according to the cows' skeletons. • This method is more stable in the antagonism of environmental robustness.