Mask R-CNN and Centroid Tracking Algorithm to Process UAV Based Thermal-RGB Video for Drylot Cattle Heat Stress Monitoring

Abstract

This study developed and evaluated an algorithm for processing thermal-RGB video feeds captured by an unmanned aerial vehicle (UAV) to automate heat stress monitoring in cattle housed in the drylots. The body surface temperature (BST) of individual cows was used as an indicator of heat stress. UAV data were collected using RGB and thermal infrared imagers, respectively, at 2 and 6.67 cm per pixel spatial resolution in Spring 2023 (dataset-1) and Summer 2024 (dataset-2). Study sites were two commercial drylots in Washington State. The custom algorithms were developed to: (1) detect and localize individual cows using a Mask R-CNN-based instance segmentation model combined with centroid tracking; and (2) extract BST by averaging the thermal-imagery pixels for each of the segmented cows. The algorithm showed higher detection accuracy with RGB images as input (F1 score: 0.89) compared to thermal (F1 score: 0.64). BST extraction with combined RGB and thermal imaging approach required corrections for alignment problems associated with differences in optics, imaging field of view, resolution, and lens properties. Consequently, thermal imaging-only approach was adopted for assessing real-time cow localization and BST estimation. Operating at one frame per second, algorithm successfully detected 72.4% and 81.65% of total cows in video frames from dataset-1 (38 s) and -2 (48 s), respectively. The mean absolute difference between algorithm output and ground truth (BSTGT) was 2.1 °C (dataset-1) and 3.3 °C (dataset-2), demonstrating satisfactory performance. With further refinements, this approach could be a viable tool for real-time heat stress monitoring in large-scale drylot production systems.