Machine vision approach to identifying and grading Strawberries

Abstract

Machine vision systems for quality inspection of processing and production lines are associated with increases in productivity, cost savings, and quality control consistency, having become common place in many industries including agriculture. However, automated strawberry quality control has historically been a challenge due to the delicate nature of the fruit’s flesh, making it more prone to damage, bruising, discolouration, and softening than other fruits and vegetables. Previous efforts have been made to grade strawberries using conveyor systems and cameras, generating good or excellent results. However, these methods do not consider the speed at which berries are packed, or the amount of damage-inducing handling required in order to place the fruit on a process line (automated or not), making these systems unsuitable for real-time production. This thesis outlines the research, design, experimentation, and development of an Strawberry Quality Assurance (SQA) vision system which is capable of grading full punnets of strawberries after they have been packed. Fruit is picked from the field, and packaged into containers, before the additional step of being loaded into the quality control vision system and finally, a heat-seal machine which seals the punnets with tamper-proof plastic film. Using this method, the packing workers do not need to slow down, or perform any new tasks, rather the flow of filled punnets is simply diverted to the additional production line. The cameras acquire four images at a rate of up to two per second in order to capture each punnet from above and below (through the transparent plastic) in visible and infrared wavelengths. Visible (RGB) images are used to assess the berry’s colour (ripeness), and visible foreign object likelihood, whilst the infrared images are used for bruise detection. Once the images have been analysed, the system uses an orthogonally directed pneumatic burst of air to eject the defected punnets from the line so that they can be repackaged and reassessed. The initial version of the system used algorithms involving image processing and colour analysis in order to perform under and over ripe detection with good results of 94.7% and 90.6%, respectively, for the multiple instance category. After redesign and upgrade to strobing LEDs and open-source software, machine learning experiments showed that the SVM and unique neural networks underperformed with the best training accuracy results of 84.11% and 76.30%, respectively. Improved performance was found using Resnet-50 pre-trained network in order to extract a feature vector for each image, which is then evaluated by each of the binary classification models. Area Under ROC (AUROC) curves are greater than 80% for both under ripe and foreign object models, whilst the over ripe class score is lower at 58%. The system has entered it’s fifth season as production-ready having already imaged and assessed 290,888 punnets of which 50,270 were rejected, resulting in 17.28% failure rate in total, helping to ensure the consistent quality assurance of production. The successful implementation of the SQA project provides a platform for continuing work including improving performance of the current classifiers, and investigating the potential of improved region-based bounding box method in order to improve visibility for both operator and developer. Data collection and labelling/annotation are performed regularly in order to re-train networks generating improved results, and to reduce inter-seasonal or inter-cultivar concept shift in the models.