Mask R-CNN aided fruit surface temperature monitoring algorithm with edge compute enabled internet of things system for automated apple heat stress management

Abstract

Our prior study focused on development of internet of things (IoT) and edge-compute enabled crop physiology sensing system (CPSS) for apple sunburn monitoring. Edge compute algorithm on CPSS estimated sunburn susceptibility as fruit surface temperature (FST) through pixel-by-pixel multiplication of captured thermal infrared images with segmented fruits binary mask. The segmentation was performed using color-based K means clustering approach. This limited CPSS applicability to monitor sunburn of red colored cultivars only and when fruits develop color, typically late growing season. This is a key research gap as recent weather patterns have shown that sunburn can occur during early growing season when fruits are green to yellow. Therefore, aim of this study was to develop and field evaluate cultivar and color independent mask region-convolution neural network (R-CNN) aided fruit segmentation model and edge compute compatible FST estimation algorithm. Season long field data were collected in 2021 using eight CPSS nodes (three in cv. WA38 [Cosmic crisp] and five in cv. Honeycrisp). Collected data were used to develop and validate mask R-CNN based fruit segmentation model. Developed mask R-CNN based model was able to segment fruits of two apple cultivars and of varying colors with 91.4 % average precision. In orchard evaluations (2022 season), the resulting algorithm ported on CPSS was able to accurately segment (dice similarity coefficient = 0.89) and estimate apple FST with < 0.5 °C error compared to ground truth data. With compute time of about 37 s, data processing time was reduced by 22 % over previous algorithm. High ambient temperature (>35 °C) on a warmer day resulted in multiple throttling errors caused by excessive CPU temperature; however, the CPSS performance was uncompromised in FST estimation. Ambient air temperature did not affect RAM utilization and CPU clock frequency. Overall, developed FST algorithm can potentially be used as input to actuate water-based cooling system.