Automatic Visual Inspection Research Articles

Attention Pyramid Dilated Region-based Model for Metallurgical Defect Detection

Abstract Defect recognition is the key to realizing automatic visual inspection and quality control in intelligent manufacturing. Deep learning (DL) has been widely applied to locate damaged areas in images, characterize impurities of materials and further analyze the quality of products. However, automatic defect detection by DL in practical industrial applications is still a challenge due to the lack of sufficient datasets and appropriate recognition methods. Here, a novel Attention Pyramid Dilated Region-based Convolutional Neural Network (ADRCNN) is proposed to realize the multi-scale defect recognition and pixel-level instance segmentation of metallographic images. We also collected a dataset containing 900 images of commercial A356 aluminum alloy with different casting defects. The ADRCNN model is evaluated on our dataset with an average detection accuracy of 87.2% and 93.3% on validation and test sets respectively. To address the overfitting problem, a pretraining fine-tuning strategy is implemented by using the pretraining weights from large-scale datasets and temporarily freezing the weights of the backbone network during the training process. The experimental results show that the proposed model can achieve satisfactory defect segmentation in metallographic images, promoting the development of intelligent manufacturing in alloy industries.

Cutting-Edge Methods in Visage Expression Recognition and Ocular Tracking for Analyzing Remote Video Interviews

This paper focuses to augment an in the flesh image and video processor, capacitated with an artificial intelligence (AI) arbitrator that can speculate a job interviewee’s behavioral adeptness according to the facial dictions or expressions. We put forward a sentiment analysis model using machine learning and CNN for the reinforcement of decision-making in the job interview process. This is bought up by histogram of oriented gradients and support vector machine (HOG-SVM) with the addition of convolutional neural network (CNN) recognition in real-time video recorded interviews. The goal of applying this technology, is to develop a method that could automatically decode a candidate’s behavior by his or her facial language (or micro expressions) based on the behavioral ecology view of facial displays (BECV) which is different from the classical perspective of recognizing emotional states. This paper dispenses a trailblazing technique of determining the performance of a candidate in a video interview. To do this, a delineation of the analysis of sentiments and eye tracking technique study is intricated in which the results can be processed on a single screen to select the right person for the hire. Key Words: Artificial Intelligence(AI), Histogram of Oriented Gradients(HOG), Support Vector Machine(SVM), Convolutional Neural Network(CNN), Behavioral Ecology View of Facial Displays, Eye Tracking Technique, AVI(Automatic Visual Inspection).

A deep learning approach for automated PCB defect detection: A comprehensive review

The ever-expanding market for electronic devices has significantly heightened the demand for high-quality printed circuit boards (PCBs). Even minor defects in PCBs can pose substantial safety risks for end-users. This article provides a comprehensive review on deep learning-based approaches for PCB defect detection. Our exploration covers various critical aspects, including the classification of PCB defects, automated vision inspection (AVI) techniques, object detection methodologies, and the widespread adoption of deep learning models. Specifically, we focus on the state-of-the-art approach known as region-based Fully Convolutional Network with Feature Pyramid Networks (FPN-RFCN). Additionally, we discuss effective data augmentation techniques and commonly used evaluation metrics in this domain. This review provides valuable insights for researchers, practitioners, and industry professionals engaged in PCB quality assurance.

FC-YOLO: an aircraft skin defect detection algorithm based on multi-scale collaborative feature fusion

Abstract Aircraft skin defects pose a threat to the safety and airworthiness of the aircraft. The front line of engineering has requirements of high precision and stable defect detection, which cannot be met by existing deep learning methods, due to conflicting information between multi-scale features. Herein, a Fine-Coordinated YOLO (FC-YOLO) algorithm is proposed to detect aircraft skin defects. Firstly, the ELAN-C module with Coordinate & Channel Attention mechanism is applied to the backbone network to enhance multi-scale detection precision. Secondly, the Adaptive-Path Aggregation Network structure is proposed to make features containing more information by adding a shortcut weighted by the Adaptively Spatial Feature Fusion (ASFF) module. The ASFF adaptively allocates the weights of features with different sizes to reduce the inconsistency of features between different levels during feature fusion to improve detection precision. Finally, the SCYLLA-IoU loss function is introduced to calculate the directional loss between the bounding box and the ground truth box to elevate the stability of the training. Experiments are executed with a self-constructed ASD-DET dataset and the public NEU-DET dataset. Results show that the mAP of FC-YOLO is improved by 3.1% and 2.7% compared to that of the original YOLOv7 on the ASD-DET dataset and the NEU-DET dataset. In addition, on the ASD-DET dataset and NEU-DET dataset, the mAP of FC-YOLO was higher than that of YOLOv8, RT-DETR by 1.4%, 1.6% and 2.2%, 3.8%, respectively. By which, it is shown that the proposed FC-YOLO algorithm is promising for the future automatic visual inspection of aircraft skin.

From classification to segmentation with explainable AI: A study on crack detection and growth monitoring

Monitoring surface cracks in infrastructure is crucial for structural health monitoring. Automatic visual inspection offers an effective solution, especially in hard-to-reach areas. Machine learning approaches have proven their effectiveness but typically require large annotated datasets for supervised training. Once a crack is detected, monitoring its severity often demands precise segmentation of the damage. However, pixel-level annotation of images for segmentation is labor-intensive. To mitigate this cost, one can leverage explainable artificial intelligence (XAI) to derive segmentations from the explanations of a classifier, requiring only weak image-level supervision. This paper proposes applying this methodology to segment and monitor surface cracks. We evaluate the performance of various XAI methods and examine how this approach facilitates severity quantification and growth monitoring. Results reveal that while the resulting segmentation masks may exhibit lower quality than those produced by supervised methods, they remain meaningful and enable severity monitoring, thus reducing substantial labeling costs. Code and data available at https://github.com/EPFL-IMOS/crack-explanations.

Hybrid Semiconductor Wafer Inspection Framework via Autonomous Data Annotation

Abstract In smart manufacturing, semiconductors play an indispensable role in collecting, processing, and analyzing data, ultimately enabling more agile and productive operations. Given the foundational importance of wafers, the purity of a wafer is essential to maintain the integrity of the overall semiconductor fabrication. This study proposes a novel automated visual inspection (AVI) framework for scrutinizing semiconductor wafers from scratch, capable of identifying defective wafers and pinpointing the location of defects through autonomous data annotation. Initially, this proposed methodology leveraged a texture analysis method known as gray-level co-occurrence matrix (GLCM) that categorized wafer images—captured via a stroboscopic imaging system—into distinct scenarios for high- and low-resolution wafer images. GLCM approaches further allowed for a complete separation of low-resolution wafer images into defective and normal wafer images, as well as the extraction of defect images from defective low-resolution wafer images, which were used for training a convolutional neural network (CNN) model. Consequently, the CNN model excelled in localizing defects on defective low-resolution wafer images, achieving an F1 score—the harmonic mean of precision and recall metrics—exceeding 90.1%. In high-resolution wafer images, a background subtraction technique represented defects as clusters of white points. The quantity of these white points determined the defectiveness and pinpointed locations of defects on high-resolution wafer images. Lastly, the CNN implementation further enhanced performance, robustness, and consistency irrespective of variations in the ratio of white point clusters. This technique demonstrated accuracy in localizing defects on high-resolution wafer images, yielding an F1 score greater than 99.3%.

Visual coating inspection framework via self-labeling and multi-stage deep learning strategies

AbstractAn instantaneous and precise coating inspection method is imperative to mitigate the risk of flaws, defects, and discrepancies on coated surfaces. While many studies have demonstrated the effectiveness of automated visual inspection (AVI) approaches enhanced by computer vision and deep learning, critical challenges exist for practical applications in the manufacturing domain. Computer vision has proven to be inflexible, demanding sophisticated algorithms for diverse feature extraction. In deep learning, supervised approaches are constrained by the need for annotated datasets, whereas unsupervised methods often result in lower performance. Addressing these challenges, this paper proposes a novel deep learning-based automated visual inspection (AVI) framework designed to minimize the necessity for extensive feature engineering, programming, and manual data annotation in classifying fuel injection nozzles and discerning their coating interfaces from scratch. This proposed framework comprises six integral components: It begins by distinguishing between coated and uncoated nozzles through gray level co-occurrence matrix (GLCM)-based texture analysis and autoencoder (AE)-based classification. This is followed by cropping surface images from uncoated nozzles, and then building an AE model to estimate the coating interface locations on coated nozzles. The next step involves generating autonomously annotated datasets derived from these estimated coating interface locations. Subsequently, a convolutional neural network (CNN)-based detection model is trained to accurately localize the coating interface locations. The final component focuses on enhancing model performance and trustworthiness. This framework demonstrated over 95% accuracy in pinpointing the coating interfaces within the error range of ± 6 pixels and processed at a rate of 7.18 images per second. Additionally, explainable artificial intelligence (XAI) techniques such as t-distributed stochastic neighbor embedding (t-SNE) and the integrated gradient substantiated the reliability of the models.

Swin-Transformer -YOLOv5 for lightweight hot-rolled steel strips surface defect detection algorithm.

An essential industrial application is the examination of surface flaws in hot-rolled steel strips. While automatic visual inspection tools must meet strict real-time performance criteria for inspecting hot-rolled steel strips, their capabilities are constrained by the accuracy and processing speed of the algorithm used to identify defects. To solve the problems of poor detection accuracy, low detection efficiency, and unsuitability of low computing power platforms of the hot-rolled strip surface defect detection algorithm The Swin-Transformer-YOLOv5 model based on the improved one-stage detector is proposed. By employing GhostNet, the model's lightweight design, and guaranteed detection accuracy are both achieved. The C3 module introduces Swin-Transformer to address the issues of cluttered backdrops of defect photos and easily confused defect categories. With the addition of the CoordAttention module, the model's capacity to extract defective features is improved, and its performance keeps getting better. The issue of huge differences in different scales and poor detection of small flaws is resolved by employing BiFPN for feature fusion, and the detector's capacity to adapt to targets of different scales is improved. The experimental results demonstrate that the improved Swin-Transformer-Yolov5 model significantly outperforms the industry-standard target detection algorithms, and the model's mAP value still improves by 8.39% over the original model while reducing the number of parameters, GFLOPs, and weight by 36.6%, 40.0%, and 34.7%, respectively. The model is better suited for use on low-arithmetic platforms as a result.

Deep Learning for Automated Visual Inspection in Manufacturing and Maintenance: A Survey of Open- Access Papers

Quality assessment in industrial applications is often carried out through visual inspection, usually performed or supported by human domain experts. However, the manual visual inspection of processes and products is error-prone and expensive. It is therefore not surprising that the automation of visual inspection in manufacturing and maintenance is heavily researched and discussed. The use of artificial intelligence as an approach to visual inspection in industrial applications has been considered for decades. Recent successes, driven by advances in deep learning, present a possible paradigm shift and have the potential to facilitate automated visual inspection, even under complex environmental conditions. For this reason, we explore the question of to what extent deep learning is already being used in the field of automated visual inspection and which potential improvements to the state of the art could be realized utilizing concepts from academic research. By conducting an extensive review of the openly accessible literature, we provide an overview of proposed and in-use deep-learning models presented in recent years. Our survey consists of 196 open-access publications, of which 31.7% are manufacturing use cases and 68.3% are maintenance use cases. Furthermore, the survey also shows that the majority of the models currently in use are based on convolutional neural networks, the current de facto standard for image classification, object recognition, or object segmentation tasks. Nevertheless, we see the emergence of vision transformer models that seem to outperform convolutional neural networks but require more resources, which also opens up new research opportunities for the future. Another finding is that in 97% of the publications, the authors use supervised learning techniques to train their models. However, with the median dataset size consisting of 2500 samples, deep-learning models cannot be trained from scratch, so it would be beneficial to use other training paradigms, such as self-supervised learning. In addition, we identified a gap of approximately three years between approaches from deep-learning-based computer vision being published and their introduction in industrial visual inspection applications. Based on our findings, we additionally discuss potential future developments in the area of automated visual inspection.

Enhanced YOLOv8 with BiFPN-SimAM for Precise Defect Detection in Miniature Capacitors

In the domain of automatic visual inspection for miniature capacitor quality control, the task of accurately detecting defects presents a formidable challenge. This challenge stems primarily from the small size and limited sample availability of defective micro-capacitors, which leads to issues such as reduced detection accuracy and increased false-negative rates in existing inspection methods. To address these challenges, this paper proposes an innovative approach employing an enhanced ‘you only look once’ version 8 (YOLOv8) architecture specifically tailored for the intricate task of micro-capacitor defect inspection. The merging of the bidirectional feature pyramid network (BiFPN) architecture and the simplified attention module (SimAM), which greatly improves the model’s capacity to recognize fine features and feature representation, is at the heart of this methodology. Furthermore, the model’s capacity for generalization was significantly improved by the addition of the weighted intersection over union (WISE-IOU) loss function. A micro-capacitor surface defect (MCSD) dataset comprising 1358 images representing four distinct types of micro-capacitor defects was constructed. The experimental results showed that our approach achieved 95.8% effectiveness in the mean average precision (mAP) at a threshold of 0.5. This indicates a notable 9.5% enhancement over the original YOLOv8 architecture and underscores the effectiveness of our approach in the automatic visual inspection of miniature capacitors.

Improved STMask R-CNN-based defect detection model for automatic visual inspection of an optics lens.

A lens defect is a common quality issue that has seriously harmed the scattering characteristics and performance of optical elements, reducing the quality consistency of the finished products. Furthermore, the energy hotspots coming from the high-energy laser through diffraction of optical component defects are amplified step by step in multi-level laser conduction, causing serious damage to the optical system. Traditional manual detection mainly relies on experienced workers under a special light source environment with high labor intensity, low efficiency, and accuracy. The common machine vision techniques are incapable of detecting low contrast and complex morphological defects. To address these challenges, a deep learning-based method, named STMask R-CNN, is proposed to detect defects on the surface and inside of a lens in complex environments. A Swin Transformer, which focuses on improving the modeling and representation capability of the features in order to improve the detection performance, is incorporated into the Mask R-CNN in this case. A challenge dataset containing more than 3800 images (18000 defect sample targets) with five different types of optical lens defects was created to verify the proposed approach. According to our experiments, the presented STMask R-CNN reached a precision value of 98.2%, recall value of 97.7%, F1 score of 97.9%, mAP@0.5 value of 98.1%, and FPS value of 24 f/s, which outperformed the SSD, Faster R-CNN, and YOLOv5. The experimental results demonstrated that the proposed STMask R-CNN outperformed other popular methods for multiscale targets, low contrast target detection and nesting, stacking, and intersecting defects sample detection, exhibiting good generalizability and robustness, as well as detection speed to meet mechanical equipment production efficiency requirements. In general, this research offers a favorable deep learning-based method for real-time automatic detection of optical lens defects.

Terminals and Connector Inspection Innovation using Machine Vision

Customer demands for product quality that are increasingly complex and require more accurate inspection are not sufficient if done manually due to high costs and different operator precision. To solve this problem, automatic vision inspection was created to check the product quality of terminal-type electronic components. Machine Vision is robust equipment for examining electrical components, agroindustry, surface mounting technology industry, plastic industry, and glass industry. In this research, the design and application of machine vision are used to inspect terminals and connectors, supporting automotive and electrical equipment components. Machine vision checks three quality functions, namely structural quality, surface quality, and dimensional quality. Optimization of image capture using backlighting and darkfield lighting. Implementing machine vision is carried out by verifying image retrieval's stability using a gage study with the results that there is no measurement bias.

A framework for flexible and reconfigurable vision inspection systems

Reconfiguration activities remain a significant challenge for automated Vision Inspection Systems (VIS), which are characterized by hardware rigidity and time-consuming software programming tasks. This work contributes to overcoming the current gap in VIS reconfigurability by proposing a novel framework based on the design of Flexible Vision Inspection Systems (FVIS), enabling a Reconfiguration Support System (RSS). FVIS is achieved using reprogrammable hardware components that allow for easy setup based on software commands. The RSS facilitates offline software programming by extracting parameters from real images, Computer-Aided Design (CAD) data, and rendered images using Automatic Feature Recognition (AFR). The RSS offers a user-friendly interface that guides non-expert users through the reconfiguration process for new part types, eliminating the need for low-level coding. The proposed framework has been practically validated during a 4-year collaboration with a global leading automotive half shaft manufacturer. A fully automated FVIS and the related RSS have been designed following the proposed framework and are currently implemented in 7 plants of GKN global automotive supplier, checking 60 defect types on thousands of parts per day, covering more than 200 individual part types and 12 part families.

Probability of detection curve for the automatic visual inspection of steel bridges

AbstractThe damage tolerant design philosophy is based on periodical inspections and provides safe bridge operation preventing fatigue cracks from growing up to a critical size. It is possible to optimize the management costs of a bridge throughout its life by designing the inspection frequency, which depends on the capabilities of the inspection method. However, such optimization requires knowledge about the performance of inspections that are envisioned to take place during the use of the bridge. Regular visual inspections is the most frequently applied type of inspection of bridge structures. Recent advances in computer vision technologies provide a strong basis for the development of automatic damage detection systems that can support regular visual inspection, thus increasing the reliability of the inspection. Several automatic crack detection systems have been developed in the past years. However, the performances of such systems have not been evaluated in the way as for traditional non‐destructive inspection methods, i.e. in terms of probability of detection curves and detectability limits. This restricts the applicability of automatic visual inspections for inspection planning and damage tolerant design. This paper proposes an encoder‐decoder neural network for segmentation of cracks on images of steel bridges. A probability of detection curve is calculated for this neural network.

Component Detection for Power Line Inspection Using a Graph-Based Relation Guiding Network

Detecting the components in aerial images is a crucial task in automatic visual inspection for power lines. Currently, deep learning models guided by external knowledge have achieved promising performances compared to directly applying the benchmark detectors. However, the component relationship, as human commonsense knowledge for object reasoning, is rarely investigated in this field. This study presents a graph-based relation guided network for power line component detection, which exploits correlations of regions, images, and categories. The visual relation module is employed to learn region-to-region relationship and enhance the visual features of each proposal that may contain components. Meanwhile, two guidance modules are proposed to capture image-to-region correlation and distinctively facilitate the category classification and position regression, which has not been considered in previous methods. Moreover, the category graphs built in these two modules are able to explore category-to-category dependencies that can further promote the network ability. Experimental results demonstrate that the proposed method can achieve more accurate and reasonable component detection compared to previous methods, which verifies the effectiveness of the proposed model incorporated with relation knowledge.

Deep transfer learning‐based approach for detection of cracks on eggs

AbstractEggs are the most nutrient‐rich and protein‐dense food that is affordable and consumed by most of the population. But cracks on the eggshells can pave the way for microbial contamination of the eggs posing threat to the health of the consumers. Therefore, it is crucial for the egg industries to check for the quality of eggs before reaching consumers to avoid outbreaks. Automatic visual inspection is one of the techniques that has gained popularity in the field of eggshell crack identification because of the advancements in hardware and deep learning. However, due to the limited training data, classifying eggs using eggshell crack images is challenging. Therefore, deep transfer learning was employed in this study using a pre‐trained DenseNet121 architecture in three ways—fine‐tuning (FT), feature extraction, and training deep features on machine learning classifiers. The results revealed that the FT technique performed best among the three approaches with 98.38% accuracy.Practical ApplicationsDeep transfer learning‐based techniques in food quality assessments are seeing a rise in the food industry. Eggshell crack detection is one of the challenges that can be addressed using the transfer learning approach. The purpose of this study was to investigate the effects of different transfer learning approaches in classifying the image patches extracted from egg images and determine the most effective technique to identify cracks on eggs, despite the difficulty of the classification task using a small data set of egg images. These results can promote the application of transfer learning in non‐destructive egg quality evaluation in the egg processing industries.

Applying Machine Learning to the Visual Inspection of Filled Injectable Drug Products.

With machine learning (ML), we see the potential to better harness the intelligence and decision-making abilities of human inspectors performing manual visual inspection (MVI) and apply this to automated visual inspection (AVI) with the inherent improvements in throughput and consistency. This article is intended to capture current experience with this new technology and provides points to consider for successful application to AVI of injectable drug products. The technology is available today for such AVI applications. Machine vision companies have integrated ML as an additional visual inspection tool with minimal upgrades to existing hardware. Studies have demonstrated superior results in defect detection and reduction in false rejects, when compared with conventional inspection tools. ML implementation does not require modifications to current AVI qualification strategies. The utilization of this technology for AVI will accelerate recipe development by use of faster computers rather than by direct human configuration and coding of vision tools. By freezing the model developed with artificial intelligence tools and subjecting it to current validation strategies, assurance of reliable performance in the production environment can be achieved.

6‐2: Potential Failure Detection using Unsupervised Clustering and Anomaly Detection

Automated Visual Inspection(AVI) is an effective way for mobile display product to detect defects which may cause quality problem. The rules of determining defects are based on knowledge for an accumulated engineer's experience. However, it is difficult to detect potential failure that may cause in the production process, because we determine failure of panel based on boundary samples. In this study, we propose an analysis method to find abnormalities by quantifying the potential failure with AVI data using unsupervised learning.

Fault Diagnosis of Train Body Sign Abnormal Pattern with Deep Learning Based Target Detection

<p>With the development of high-speed trains in recent years, security issues have received more attention. Automatic visual inspection of the train operation system for detecting abnormalities has become a fundamental element to guarantee the safety of the train operation. Train body sign patterns like the loss and fracture of signs and lock catch (SLC) on the electrical box cover (EBC) affect the regular operation of the train electrical system. In this paper, to ensure the safe operation of the train, a novel method combining a faster region-based convolutional neural network (Faster R-CNN) and similarity metrics is proposed to detect the abnormality of SLCs on train EBC. First, the positions of body train signs of multiple sizes are located by Faster R-CNN. Then, the regions of interest (ROI) are cut out and resized to the same size as the corresponding template images. Finally, by similarity measures, the status of the train body sign pattern is judged by comparing with the given threshold similarity value between ROIs and the template images. It is worth noting that the combination of Faster R-CNN and cosine similarity renders high accuracy in small target detection and strong robustness in image similarity comparison. The effectiveness of the proposed fault detection method and its superiority over the other types of combined methods are verified by actual experiments on the train of Guangzhou Metro Line 2.</p> <p> </p>

A visual inspection system for detecting end gap and light leakage of the piston ring

A fully automatic visual inspection system for piston ring was proposed in this study. This system can automatically feed the piston ring, and then detect the end gap and the light leakage of the piston ring. Moreover, the piston ring can be classified after detection and the detection data can be saved for further analysis. To achieve this objective, the detection equipment was established for detecting the end gap and the light leakage, and corresponding software was developed based on Labview. The experiment was carried out and the detection performance of the system was evaluated. The result shows that the proposed system has a good accuracy for detecting the piston ring.