Prediction of the fruit development stage of sweet pepper (Capsicum annum var. annuum) by an ensemble model of convolutional and multilayer perceptron

Abstract

An ensemble model of convolutional neural network (CNN) and multilayer perceptron (MLP) models was developed to detect sweet pepper ( Capsicum annuum var. annuum ) fruits in images and predict their development stages. The plants were grown in four rows in a greenhouse, and images were collected from each row. Plant environment and growth data were collected every minute and month, respectively. The fruit development stage was classified into immature, breaking, and mature stages with a CNN using images. The immature stage was internally divided into four stages with an MLP, so a total of six stages were classified using the CNN–MLP ensemble model. The plant growth and environmental data and the information from the CNN output were used for the MLP input. The average accuracy of the six stages was F1 score = 0.77 and IoU = 0.86. The ensemble model showed acceptable performance in predicting fruit development stages. The CNN-only model could classify the mature and breaking stages well, but the immature stages were not distinguished, while the MLP-only model could hardly classify the fruit stage except the immature stages. The most influential factors in classification were the data obtained from CNN and the plant growth and environment data. The ensemble models could help in appropriate labour allocation and strategic management by detecting individual fruits in images and predicting precise fruit development stages. • Ensemble of CNN and MLP models to predict fruit developmental stage of sweet pepper. • CNN-only and MLP-only models could not predict all six stages. • Ensemble model accurately detected fruits and predicted six developmental stages. • Model could be expanded to assist in labour allocation and strategic management.