Proposed Machine Vision Research Articles

Automatic Sorting and Grading of Fruits Based on Maturity and Size Using Machine Vision and Artificial Intelligence

This paper introduces a computer vision-based system designed for the automated grading and sorting of agricultural products based on their size and maturity. The proposed machine vision system aims to replace traditional manual methods commonly used for sorting and grading fruits. Manual inspection often struggles to ensure consistency in grading and uniformity in sorting. To address these challenges and enhance the quality of fruit grading, image processing and machine learning algorithms can be employed. Key attributes such as the fruit’s shape, color, and size can be analyzed to enable a non-destructive approach to classification and grading. Automation of these processes becomes feasible when standardized criteria for grading are established. Such systems offer faster operations, save time, and reduce manual labor, making them highly valuable to meet the increasing demand for premium-quality agricultural produce.

Measurement of bending deformation of sheet metal using machine vision

ABSTRACT The aim of this study is to propose a fast and effective method to measure the curvature distribution and bending angle of sheet metal in the bending deformation using machine vision. An open-source computer vision library (OpenCV) was used for image processing. First, the image is calibrated for perspective and lens distortion. After that, the image is subjected to color extraction and binary conversion. The outer geometry of the material was obtained using edge detection. With the proposed method, the bending angle and curvature distribution in the four-point bending and V-bending tests are measured. The proposed machine vision method exhibits errors of 0.55% and 0.12% compared with the 3D scanner measurements for the bending angle in the four-point bend and V-bend tests after springback, respectively. The curvature distribution in bending deformation was also obtained. As an application of the proposed method, the moment – curvature diagram of the material was determined based on the obtained curvature at the center. Additionally, the optimal stroke was calculated for the target bend angle during the V-bending process. The proposed method proved to be an accurate and efficient way to measure bending deformation of sheet metal.

Machine vision-based gait scan method for identifying cognitive impairment in older adults.

Early identification of cognitive impairment in older adults could reduce the burden of age-related disabilities. Gait parameters are associated with and predictive of cognitive decline. Although a variety of sensors and machine learning analysis methods have been used in cognitive studies, a deep optimized machine vision-based method for analyzing gait to identify cognitive decline is needed. This study used a walking footage dataset of 158 adults named West China Hospital Elderly Gait, which was labelled by performance on the Short Portable Mental Status Questionnaire. We proposed a novel recognition network, Deep Optimized GaitPart (DO-GaitPart), based on silhouette and skeleton gait images. Three improvements were applied: short-term temporal template generator (STTG) in the template generation stage to decrease computational cost and minimize loss of temporal information; depth-wise spatial feature extractor (DSFE) to extract both global and local fine-grained spatial features from gait images; and multi-scale temporal aggregation (MTA), a temporal modeling method based on attention mechanism, to improve the distinguishability of gait patterns. An ablation test showed that each component of DO-GaitPart was essential. DO-GaitPart excels in backpack walking scene on CASIA-B dataset, outperforming comparison methods, which were GaitSet, GaitPart, MT3D, 3D Local, TransGait, CSTL, GLN, GaitGL and SMPLGait on Gait3D dataset. The proposed machine vision gait feature identification method achieved a receiver operating characteristic/area under the curve (ROCAUC) of 0.876 (0.852-0.900) on the cognitive state classification task. The proposed method performed well identifying cognitive decline from the gait video datasets, making it a prospective prototype tool in cognitive assessment.

An investigation of methods to improve the accuracy of classical dance steps based on machine vision recognition technology

Abstract Methods that use motion capture data to analyze human behavior are highly interpretable and offer significant advantages in vision-based dance step analysis. In this study, we propose a fusion feature extraction method that describes the rotational information embedded in the skeleton and combines the advantages of single-skeleton feature extraction and Li group feature extraction methods for recognition. The feasibility and validity of the proposed model are verified by conducting recognition experiments on BVH data of classical dance steps performed by subjects. In the ablation experiments, the average accuracy of the classical dance step recognition model based on fused features for step movement recognition is improved by 3.01% and 1.86% compared with the neural network model and the Lie group network model, respectively. It has been proven that adding rotation information to skeletal features can effectively differentiate 3D motion trajectories in similar dance steps. Furthermore, the dance movement trajectories derived from this model are very clear and can be utilized to direct the correct joint point positions throughout the movement. The recognition accuracy of the fusion feature extraction-based recognition method for all seven classical dance basic foot positions is greater than 90%, which reflects the accuracy of the proposed machine vision model in recognizing classical dance steps.

Smoke Detection of Marine Engine Room Based on a Machine Vision Model (CWC-Yolov5s)

According to statistics, about 70% of ship fire accidents occur in the engine room, due to the complex internal structure and various combustible materials. Once a fire occurs, it is difficult to extinguish and significantly impacts the crew’s life and property. Therefore, it is urgent to design a method to detect the fire phenomenon in the engine room in real time. To address this problem, a machine vision model (CWC-YOLOv5s) is proposed, which can identify early fires through smoke detection methods. Firstly, a coordinate attention mechanism is added to the backbone of the baseline model (YOLOv5s) to enhance the perception of image feature information. The loss function of the baseline model is optimized by wise intersection over union, which speeds up the convergence and improves the effect of model checking. Then, the coordconv coordinate convolution layer replaces the standard convolution layer of the baseline model, which enhances the boundary information and improves the model regression accuracy. Finally, the proposed machine vision model is verified by using the ship video system and the laboratory smoke simulation bench. The results show that the proposed model has a detection precision of 91.8% and a recall rate of 88.1%, which are 2.2% and 4.6% higher than those of the baseline model.

Vision-Based System for Black Rubber Roller Surface Inspection

This paper proposes a machine vision system for the surface inspection of black rubber rollers in manufacturing processes. The system aims to enhance the surface quality of the rollers by detecting and classifying defects. A lighting system is installed to highlight surface defects. Two algorithms are proposed for defect detection: a traditional-based method and a deep learning-based method. The former is fast but limited to surface defect detection, while the latter is slower but capable of detecting and classifying defects. The accuracy of the algorithms is verified through experiments, with the traditional-based method achieving near-perfect accuracy of approximately 98% for defect detection, and the deep learning-based method achieving an accuracy of approximately 95.2% for defect detection and 96% for defect classification. The proposed machine vision system can significantly improve the surface inspection of black rubber rollers, thereby ensuring high-quality production.

Hardware Deployable Edge-AI Solution for Prescreening of Oral Tongue Lesions Using TinyML on Embedded Devices

Diagnosing oral cavity cancer (OCC) in its initial stages is an effective way to reduce patient mortality. However, current prescreening solutions are manual and the resultant clinical treatment is not cost effective for the average individual, primarily in developing nations. In this letter, we present an automated and inexpensive prescreening solution utilizing artificial intelligence (AI) deployed on embedded edge devices to detect benign and premalignant superficial oral tongue lesions. The proposed machine vision solution utilizes a clinically annotated photographic dataset of nine types of superficial oral tongue lesions to retrain a MobileNetV2 neural network using transfer learning. In this approach, we also utilized TensorFlow Lite for Microcontrollers to quantize a 32-bit floating point (float32) precision model into an 8-bit integer (int8) model for deployment on power and resource-constrained OpenMV Cam H7 Plus embedded edge device. The quantized int8 model was able to detect the nine tongue lesions with an accuracy of 98.69% on the test set. More than 60% reduction in on-device RAM and flash memory usage were measured for the int8 model when compared to the equivalently performing float32 model for relatively the same inference speed (~1.1 ms) on the target edge device.

The Selecting Optimal Ball-Receiving Body Parts Using Pose Sequence Analysis and Sports Biomechanics

The purpose is to explore a personalized and targeted training mode in football player training. Firstly, this work introduces the principle and advantages of machine vision sensing. Secondly, from the biomechanical point of view, the influence of the acceleration of several joints and the joint angle on the ball receiving effect is analyzed. Furthermore, the football player’s in-game receiving image is collected using machine vision technology, and the pose image data are preprocessed to construct a data set. Then, a new model is constructed and trained using Haar-like feature (HLF) and (Adaptive Boosting (Adaboost). Finally, the recognition model of the football receiving pose is tested, and the recognition effect is compared with the mainstream recognition model. The results show that the recognition parameter of the traditional method based on the Halcon recognition pose system is 5.12 at 20 times and then begins to decline. In contrast, the identification parameters based on the Industrial Robot Vision System Development (IRVSD) platform are much higher than those based on Halcon. It slightly decreases when the training times are 60 and then gradually increases. However, the recognition parameters based on the proposed machine vision have been far higher than those of the two traditional methods and maintained at about 10. This is because the proposed method extracts the foul image features, establishes the pose sequence potential function, and analyzes it in more detail, thus improving the recognition accuracy. The player’s pose recognition model based on HLF and AdaBoost algorithm can identify and evaluate the ball-receiving pose, thus guiding the receiving improvement. The finding shows that the proposed recognition technology can recognize and evaluate the players’ ball-receiving image, providing a new direction for applying artificial intelligence technology in sports.

Deep learning-based postharvest strawberry bruise detection under UV and incandescent light

Bruising is one of the major defects of strawberries. Strawberry bruised areas are often visible as flattened, sunken, and discolored. Postharvest strawberries can be bruised by compression, impact, or vibration forces during harvesting, transportation, and packinghouse operations. These bruises make strawberries more vulnerable to rot or disease, thus shortening their shelf life. Bruises can also cause consumers to abandon purchase decisions. Therefore, inspecting strawberry bruises is an important procedure for commercial strawberry farms to guarantee the high quality of strawberries sent to the market. A novel technique was developed in this study to detect bruises on strawberry images captured by a color camera under incandescent and ultraviolet (UV) light. Mask Region-Based Convolutional Neural Networks (Mask R-CNN), a deep learning method, was utilized for the strawberry bruise detection. Four bruise severities were classified according to the estimated bruise ratio of every single strawberry (Minor Bruise, Light Bruise, Moderate Bruise, and Severe Bruise). The results showed that Mask R-CNN could accurately detect whether a strawberry is bruised or not under both lighting conditions. However, the strawberry bruise severity classification accuracy of the proposed machine vision technique was improved significantly under UV light compared to incandescent light. For whole strawberry bruise detection under UV light, the F1 score for detecting strawberry bruises was 0.99, and the highest F1 score for whole strawberry bruise severity classification was 0.92 for Minor Bruise. This strawberry bruise detection system can assist farmers in detecting bruised strawberries on the processing line or an automated strawberry harvester, improve the quality of strawberries that are sent to the market, and avoid shortening the shelf life of fresh strawberries.

Machine vision system based on a coupled image segmentation algorithm for surface-defect detection of a Si3N4 bearing roller.

Defect detection is a critical way to ensure quality for silicon-nitride-bearing rollers. To improve detection efficiency and precision for silicon-nitride-bearing roller surface defects, in this paper, a novel machine vision system for the detection of its surface defects is designed. This method combines image segmentation and wavelet fusion to extract features from an image. In turn, the features are used in a classifier based on the K-nearest neighbor for defect classification. The optimized image segmentation algorithm that is combined with wavelet fusion is the innovation of the proposed method. It is evaluated using different defect images acquired by the machine vision system. Our experiments show that the proposed machine vision system's precision in anomaly detection of the silicon-nitride-bearing roller surface can achieve 98.5%; further, its classification precision of various defects is greater than 91.5%. It has resulted in a solution for the automatic identification of the silicon-nitride-bearing roller surface defects.

Design of a composite lighting system based on a freeform and a rod lens for machine vision.

Light-emitting diodes (LEDs) have been widely utilized in machine vision lighting systems such as the process monitoring system in the additive manufacturing field, owing to their long life, high illumination efficiency, and controllable dimming. The quality of the lighting system directly affects the efficiency and accuracy of the entire monitoring system. However, existing designs cannot meet the optical efficiency and uniformity requirements at short lighting distances and small inspection areas with mixed multi-spectrum channels. This paper thus proposes a novel, to the best of our knowledge, design method of integrating a freeform surface lens and a square-shaped rod lens. The optical characteristics under different working distances and targeting surface types have been optimized and evaluated. Meanwhile, tolerance analysis has been utilized to demonstrate the feasibility of installation. With the use of the software Tracepro, simulation results showed that the designed composite machine vision lighting system can obtain an optical efficiency of 81.704% and an illuminance uniformity of 95.804% within the inspection area at a distance of 250 mm. Furthermore, verification experiments with a prototype were performed, demonstrating the feasibility of the proposed machine vision lighting system.

Multiplex sensing of complex mixtures by machine vision analysis of TLC-SERS images

Thin layer chromatography in tandem with surface-enhanced Raman scattering (TLC-SERS) has demonstrated tremendous potentials as a new analytical chemistry tool to detect a wide range of substances from real-world samples. However, it still faces significant challenges of multiplex sensing from complex mixtures due to the imperfect separation by TLC and the resulting interference of SERS detection. In this article, we propose a multiplex sensing method of complex mixtures by machine vision analysis of the scanning image of the TLC-SERS results. Briefly, various pure substances in solution and the complex mixture solution are separated by TLC followed by one-dimensional SERS scanning of the entire TLC plate, which generates TLC-SERS images of all target substances along the chromatography path. After that, a machine vision method is employed to extract the template images from the TLC-SERS images of pure substance solutions. Finally, we apply a feature point matching strategy based on the Winner-take-all principle, which matches the template image of each pure substance with the mixture image to confirm the existence and derive the position of each target substance in the TLC plate, respectively. Our experimental results based on the mixture solution of five different substances show that the proposed machine vision analysis is highly selective, sensitive and does not require artificial analysis of the SERS spectra. Therefore, we envision that the proposed machine vision analysis of the TLC-SERS imaging is an objective, accurate, and efficient method for multiplex sensing of trace level of target substances from complex mixtures.

Development of Machine Vision System for Off-Line Inspection of Fine Defects on Glass Screen Surface

The complicated manufacturing process and poor industrial environment inevitably produces unacceptable scratches on glass screen surface. In this study, a machine vision system is developed to automatically detect fine scratch, which has a width smaller than 30 μm, on glass screen surface for the purpose of appearance quality assessment. The imaging hardware of the proposed machine vision system is made up of a high-resolution camera and a wide field lens so that a high-quality image is achieved in a relatively large image area. The captured glass screen images were firstly classified with YOLOv3 to exactly distinguish fine scratches in two other kinds of defects including dusts and stains. Then the U-Net network was applied on the selected scratches to estimate their dimensions approximately. The proposed algorithms combination has a fast image processing speed and it is able to effectively reduce the false segmentation of other defects. The results of the developed machine vision system show a relatively good agreement with the well-established microscopic method. It demonstrates our machine vision method is able to image and quantitative analyze scratches with dimensions between 25 μm to 100 μm.

A cooperative mobile robot and manipulator system (Co-MRMS) for transport and lay-up of fibre plies in modern composite material manufacture

Composite materials are widely used in industry due to their light weight and specific performance. Currently, composite manufacturing mainly relies on manual labour and individual skills, especially in transport and lay-up processes, which are time consuming and prone to errors. As part of a preliminary investigation into the feasibility of deploying autonomous robotics for composite manufacturing, this paper presents a case study that investigates a cooperative mobile robot and manipulator system (Co-MRMS) for material transport and composite lay-up, which mainly comprises a mobile robot, a fixed-base manipulator and a machine vision sub-system. In the proposed system, marker-based and Fourier transform-based machine vision approaches are used to achieve high accuracy capability in localisation and fibre orientation detection respectively. Moreover, a particle-based approach is adopted to model material deformation during manipulation within robotic simulations. As a case study, a vacuum suction-based end-effector model is developed to deal with sagging effects and to quickly evaluate different gripper designs, comprising of an array of multiple suction cups. Comprehensive simulations and physical experiments, conducted with a 6-DOF serial manipulator and a two-wheeled differential drive mobile robot, demonstrate the efficient interaction and high performance of the Co-MRMS for autonomous material transportation, material localisation, fibre orientation detection and grasping of deformable material. Additionally, the experimental results verify that the presented machine vision approach achieves high accuracy in localisation (the root mean square error is 4.04 mm) and fibre orientation detection (the root mean square error is 1.84∘) and enables dealing with uncertainties such as the shape and size of fibre plies.

Integrating Vision System to a Pick and Place Cartesian Robot

Vision aided pick and place cartesian robot is a combination of machine vision system and robotic system. They communicate with each other simultaneously to perform object sorting. In this project, machine vision algorithm for object sorting to solve the problem in failure sorting due to imperfection of images edges and different types of colours is proposed. The image is acquired by a camera and followed by image calibration. Pre-processing of image is performed through these methods, which are HSI colour space transformation, Gaussian filter for image filtering, Otsu’s method for image binarization, and Canny edge detection. LabVIEW edge-based geometric matching is selected for template matching. After the vision application analysed the image, electrical signal will send to robotic arm for object sorting if the acquired image is matched with template image. The proposed machine vision algorithm has yielded an accurate template matching score from 800 to 1000 under different disturbances and conditions. This machine vision algorithm provides more customizable parameters for each methods yet improves the accuracy of template matching.

A Machine Vision System Based on Driving Recorder for Automatic Inspection of Rail Curvature

Because of long distance of railway lines, it is difficult to find an appropriate method to inspect the rail track condition efficiently and accurately. In this article, a machine vision system based on driving recorder and image signal processing is proposed to evaluate the rail curvature automatically. The proposed machine vision system consists of four modules including the video acquisition module, the image extraction module, the image processing module, and the track condition assessment module. Three classic edge detection methods are adopted and compared for rail edge detection. In line with the videos of driving recorder, coordinate systems for train and rail are defined in the Lagrangian space, and the track curvature is estimated using the proposed chord offset method and double measurement method. For evaluating the track condition, an index describing the concordance between the train and track is defined. In the case study, a set of videos from the driving recorders of trains during their in-service operations are analyzed by the proposed technique, and the obtained results are verified by comparison with those obtained by a track geometry inspection vehicle. It is shown that the proposed technique can evaluate the track curvature accurately. Moreover, the influence of the position of deployed driving recorder, the focal length and anti-shake of camera on the accuracy of evaluation results is discussed. It is testified that the proposed technique provides a simple and reliable way to inspect the track curvature.

Fast semantic segmentation method for machine vision inspection based on a fewer-parameters atrous convolution neural network.

Owing to the recent development in deep learning, machine vision has been widely used in intelligent manufacturing equipment in multiple fields, including precision-manufacturing production lines and online product-quality inspection. This study aims at online Machine Vision Inspection, focusing on the method of online semantic segmentation under complex backgrounds. First, the fewer-parameters optimization of the atrous convolution architecture is studied. Atrous spatial pyramid pooling (ASPP) and residual network (ResNet) are selected as the basic architectures of ηseg and ηmain, respectively, which indicate that the improved proportion of the participating input image feature is beneficial for improving the accuracy of feature extraction during the change of the number and dimension of feature maps. Second, this study proposes five modified ResNet residual building blocks, with the main path having a 3 × 3 convolution layer, 2 × 2 skip path, and pooling layer with ls = 2, which can improve the use of image features. Finally, the simulation experiments show that our modified structure can significantly decrease segmentation time Tseg from 719 to 296 ms (decreased by 58.8%), with only a slight decrease in the intersection-over-union from 86.7% to 86.6%. The applicability of the proposed machine vision method was verified through the segmentation recognition of the China Yuan (CNY) for the 2019 version. Compared with the conventional method, the proposed model of semantic segmentation visual detection effectively reduces the detection time while ensuring the detection accuracy and has a significant effect of fewer-parameters optimization. This slows for the possibility of neural network detection on mobile terminals.

Intelligent Machine Vision Model for Defective Product Inspection Based on Machine Learning

Quality control is one of the industrial tasks most susceptible to be improved by implementing technological innovations. As an innovative technology, machine vision enables reliable and fast 24/7 inspections and helps producers to improve the efficiency of manufacturing operations. The accessible data by vision equipment will be used to identify and report defective products, understand the causes of deficiencies and allow rapid and efficient intervention in smart factories. From this perspective, the proposed machine vision model in this paper combines the identification of defective products and the continuous improvement of manufacturing processes by predicting the most suitable parameters of production processes to obtain a defect-free item. The suggested model exploits all generated data by various integrated technologies in the manufacturing chain, thus meeting the requirements of quality management in the context of Industry 4.0, based on predictive analysis to identify patterns in data and suggest corrective actions to ensure product quality. In addition, a comparative study between several machine learning algorithms, both for product classification and process improvement models, is performed in order to evaluate the designed system. The results of this study show that the proposed model largely meets the requirements for the proper implementation of these techniques.

Infrared imaging based machine vision system to determine transient shape of isotherms in submerged arc welding

Submerged arc welding (SAW) is one of the important industrial welding processes for heavy industries which include ship building and steel industries. The transient temperature distribution during the welding process has a significant influence on the heat affected zone width, microstructure formation, residual stresses, distortion, and hence on the fatigue behaviour of the welded structures. To accurately determine the transient temperature distribution, the exact shape of the heat source distribution needs to be considered. Machine vision can be a unique method for determination of such transient temperature distribution. In this paper, a machine vision system is developed to determine the transient shape of isotherm of submerged arc welded plates. It is based on infrared imaging to capture and extract the temperature history of the welding process. The transient temperature distribution is observed to be egg shape. The shape of isotherm is analyzed and validated with the experimental data acquired for a wide range of process parameters. These comparisons showed close agreement. The extracted data from the proposed machine vision system is used as the input information for various investigations, such as maximum temperature, isotherm generation, cooling rate plots and image segmentation studies. The developed method is a contactless method. It not only eliminates the possible errors encountered in contact measurement methods, but also provides a user friendly interface and accurate results in less time, and hence significantly reduces the experimental time. This method demonstrated its successful application for submerged arc welding, and furthermore it can be applied for the estimation of transient temperature field in other manufacturing processes.

Leaf Disease Detection Based on Local Gabor Binary Pattern Histogram Sequence and Neural Network

Agriculture forms the main source of food in India, especially in the southern area. The economy of India directly depends on agriculture plants. But due to some major diseases such as blast, brown spot, and bacterial blight, there is a reduction in plant growth which greatly affects agricultural productivity. The farmers add irrelevant pesticides with their limited knowledge which will degrade the quality of the crop but also degrade the soil quality. In the proposed method Machine Vision techniques based on neural networks are used to detect plant health or diseases indicated by leaf anomaly. Image processing algorithms such as K means clustering is used to segment affected areas. From the segmented images of the plant leaf, features are extracted using Color Coherence Vector (CCV) and Local Gabor Binary Pattern Histogram Sequence (LGBPHS). The extracted features are fed as input to a backpropagation neural network to classify the unhealthy leaf.