Using a CNN-LSTM for basic behaviors detection of a single dairy cow in a complex environment

Abstract

The basic behaviors of dairy cows (drinking, ruminating, walking, standing and lying) are closely related to their physiological health status. Consequently, intelligent behavior recognition is of significance for the automatic diagnosis and precision farming of dairy cows. Realizing the accurate behaviors classification in complex environments involving low quality surveillance videos, complex illumination and weather changes is a key problem in dairy farming that must be solved. In this study, CNN-LSTM (fusion of convolutional neural network and long short-term memory) an algorithm for recognizing the basic behaviors of a single cow, was proposed. First, the VGG16 trained on ImageNet was used as the network skeleton to extract the feature vector sequence corresponding to each video, so as to avoid the shortcomings of traditional feature engineering which were time-consuming and laborious. Then, these features were input Bi-LSTM (bidirectional long short-term memory) classification model, which could extract semantic information of time series data in two directions, so as to realize accurate recognition of dairy cow’s basic behaviors. To verify the effectiveness of the VGG16 feature extraction network used in this research, 1370 segments of approximately 18 h of videos collected from dairy farm monitoring cameras were tested and compared with those of five different feature extraction networks based on VGG19, ResNet18, ResNet101, MobileNet V2 and DenseNet201. Moreover, the effects of changes in illumination, weather, and wind velocity on behaviors recognition were tested and discussed. The test results indicated that the precision of the proposed algorithm for the recognition of the five behaviors ranged from 0.958 to 0.995, the recall ranged from 0.950 to 0.985, and the specificity ranged from 0.974 to 0.991, while the average precision, recall and specificity were 0.971, 0.965 and 0.983, respectively. The average recognition accuracy of the proposed method was 0.976, which was higher than the methods based on VGG19, ResNet18, ResNet101, MobileNet V2 and DenseNe201 by 0.08 × 10−2, 1.97 × 10−2, 2.19 × 10−2, 2.85 × 10−2 and 2.34 × 10−2, respectively. Furthermore, the influences of illumination, weather and wind speed on the algorithm were discussed. The results showed that the difference of behavior recognition accuracy of this method under the above interference was less than 0.02, indicating that the method is good in stability. The research results showed that it was feasible to use the proposed algorithm to recognize behaviors of a single target dairy cow. This study could not only provide valuable references for the behaviors identification and understanding of multiple target dairy cows based on computer vision in complex environments such as low-quality surveillance video, complex illumination and weather variations, but also contribute to their physiological health assessment and remote diagnosis. The study may be valuable for the dairy cows’ prevention and treatment of health and reproduction problems using the “medical-engineering interdisciplinary” approach.