Developing neural-based hardware and soft ware complex with a mobile application for monitoring apple fruits on tree canopy

Abstract

The paper presents a soft ware and hardware complex with a mobile application based on a neural network, designed to identify apple fruits on tree canopy, to count their number, to determine the quantity of fruits affected by diseases, as well as to estimate the growth rate of apple fruits and, thus, to calculate the total yield during the growing season. The developed soft ware and hardware complex consists of a photo (image) collection unit with client soft ware (a mobile application, a digital camera), a unit for processing the obtained images, which includes a database and a neural network, and a unit for interpretation of the obtained data. A neural network based on VGG-16 and SSD architecture was developed to identify apple fruits on the tree canopy for evaluating apple fruits and distinguishing sound fruits and those affected by disease. Training of the neural network was based on the selected classes of sound red and green apple fruits, and apple fruits affected by diseases – scab, powdery mildew, fruit rot, as well as mechanical damage. The soft ware runs and operates on Ubuntu operating system, a mobile application – on Android. The soft ware package and mobile application are capable of processing incoming photos (images) online, as well as to use previously captured photos. The generated database collects structured information about all field measurements and calculations of the number of apple fruits on the planting rows under study. The experiments conducted on an industrial apple plantation showed that the accuracy of estimating the total number of fruits on the tree canopy compared to the true value was 94.7%, the accuracy of calculating the number of affected fruits was 90.4%. When technical requirements for the server and requirements for images are met, the average recognition rate does not exceed 0.6 seconds per image, the average segmentation rate of the apple fruits from the background does not exceed 0.8 seconds per image, the average speed of analyzing one image and obtaining the recognition result does not exceed 1.5 seconds.