Automatic Visual Inspection Research Articles

Multilabel image analysis on Polyethylene Terephthalate bottle images using PETNet Convolution Architecture

Packaging is one of the important aspects of the product. Good packaging can increase the competitiveness of a product. Therefore, to maintain the quality of the packaging of a product, it is necessary to have a visual inspection. Furthermore, an automatic visual inspection can reduce the occurrence of human errors in the manual inspection process. This research will use the convolution network to detect and classify PET (Polyethylene Terephthalate) bottles. The Convolutional Neural Network (CNN) method is one approach that can be used to detect and classify PET bottle packaging. This research was conducted by comparing seven network architecture models, namely VGG-16, Inception V3, MobileNet V2, Xception, Inception ResNet V2, Depthwise Separable Convolution (DSC), and PETNet, which is the architectural model proposed in this study. The results of this study indicate that the PETNet model gives the best results compared to other models, with a test score of 96.04%, by detecting and classifying 461 of 480 images with an average test time of 0.0016 seconds.

A review of the automated timber defect identification approach

Timber quality control is undoubtedly a very laborious process in the secondary wood industry. Manual inspections by operators are prone to human error, thereby resulting in poor timber quality inspections and low production volumes. The automation of this process using an automated vision inspection (AVI) system integrated with artificial intelligence appears to be the most plausible approach due to its ease of use and minimal operating costs. This paper provides an overview of previous works on the automated inspection of timber surface defects as well as various machine learning and deep learning approaches that have been implemented for the identification of timber defects. Contemporary algorithms and techniques used in both machine learning and deep learning are discussed and outlined in this review paper. Furthermore, the paper also highlighted the possible limitation of employing both approaches in the identification of the timber defect along with several future directions that may be further explored.

CANet: Contextual Information and Spatial Attention Based Network for Detecting Small Defects in Manufacturing Industry

Despite the promising development of Automatic Visual Inspection (AVI) in the manufacturing industry, detecting small-sized defects with fewer pixels coverage remains a challenging problem due to its insufficient attention and lack of semantic information. Most exsiting convolutional inspection methods overlook the long-range dependence of context and lack adaptive fusion strategies to exploit heterogeneous features. To address these issues in AVI, this paper proposes a novel contextual information and spatial attention based network (CANet), which consists of two steps, namely CAblock and LaplacianFPN, for effective perception and exploitation of small defect features. Specifically, CAblock extracts semantic information with rich context by encoding spatial long-range dependence and decoding contextual information as channel-specific bias through a Spatial Attention Encoder (SAE) and a Context Block Decoder (CBD), respectively. LaplacianFPN further performs adaptive feature fusion considering both feature consistency and heterogeneity via two parallel branches. As a benchmark, a self-built Engine Surface Defects (ESD) dataset collected in real industry containing 89.70% small defects is constructed. Experimental results show that CANet achieves mAP-50 improvements of 1.5% and 4.3% compared to state-of-the-art methods on NEU-DET and ESD, which demonstrates the effectiveness of the proposed method. The code is now available at https://github.com/xiuqhou/CANet.

Surface Damage Identification for Heritage Site Protection: A Mobile Crowd-sensing Solution Based on Deep Learning

This article addresses the general problem of built heritage protection against both deterioration and loss. To continuously monitor and update the structural health status, a crowd-sensing solution based on powerful and automatic deep learning technique is proposed. The aim of this solution is to get rid of the limitations of manual and visual damage detection methods that are costly and time-consuming. Instead, automatic visual inspection for damage detection on walls is efficiently and effectively performed using an embedded Convolutional Neural Network (CNN). This CNN detects the most frequent types of surface damage on wall photos. The study has been conducted in the Kasbah of Algiers where the four following types of damages have been considered: Efflorescence, spall, crack, and mold. The CNN is designed and trained to be integrated into a mobile application for a participatory crowd-sensing solution. The application should be widely and freely deployed, so any user can take a picture of a suspected damaged wall and get an instant and automatic diagnosis through the embedded CNN. In this context, we have chosen MobileNetV2 with a transfer learning approach. A set of real images have been collected and manually annotated and have been used for training, validation, and test. Extensive experiments have been conducted to assess the efficiency and the effectiveness of the proposed solution, using a 5-fold cross-validation procedure. Obtained results show in particular a mean weighted average precision of 0.868 ± 0.00862 (with a 99% of confidence level) and a mean weighted average recall of 0.84 ± 0.00729 (with a 99% of confidence level). To evaluate the performance of MobileNetV2 as a feature extractor, we conducted a comparative study with other small backbones. Further analysis of CNN activation using Grad-Cam has also been done. Obtained results show that our method remains effective even when using a small network and medium- to low-resolution images. MobileNetV2-based CNN size is smaller, and computational cost better, compared to the other CNNs, with similar performance results. Finally, detected surface damages have also been plotted on a geographic map, giving a global view of their distribution.

An Automatic Visual Inspection of Oil Tanks Exterior Surface Using Unmanned Aerial Vehicle with Image Processing and Cascading Fuzzy Logic Algorithms

This paper presents an automatic visual inspection of exterior surface defects of oil tanks using unmanned aerial vehicles (UAVs) and image processing with two cascading fuzzy logic algorithms. Corrosion is one of the defects that has a serious effect on the safety of the surface of oil and gas tanks. At present, human inspection, and climbing robots inspection are the dominant approach for rust detection in oil and gas tanks. However, there are many shortcomings to this approach, such as taking longer, high cost, and covering less surface area inspection of the tank. The purpose of this research is to detect the rust in oil tanks by localizing visual inspection technology using UAVs, as well as to develop algorithms to distinguish between defects and noise. The study focuses on two basic aspects of oil tank inspection through the images captured by the UAV, namely, the detection of defects and the distinction between defects and noise. For the former, an image processing algorithm was developed to improve or remove noise, adjust the brightness of the captured image, and extract features to identify defects in oil tanks. Meanwhile, for the latter aspect, a cascading fuzzy logic algorithm and threshold algorithm were developed to distinguish between defects and noise levels and reduce their impact through three stages of processing: The first stage of fuzzy logic aims to distinguish between defects and low noise generated by the appearance of objects on the surface of the tank, such as trees or stairs, and reduce their impact. The second stage aims to distinguish between defects and medium noise generated by shadows or the presence of small objects on the surface of the tank and reduce their impact. The third stage of the thresholding algorithm aims to distinguish between defects and high noise generated by sedimentation on the surface of the tank and reduce its impact. The samples were classified based on the output of the third stage of the threshold process into defective or non-defective samples. The proposed algorithms were tested on 180 samples and the results show its superiority in the inspection and detection of defects with an accuracy of 83%.

Hyper Parameter Optimization for Transfer Learning of ShuffleNetV2 with Edge Computing for Casting Defect Detection

A casting defect is an expendable abnormality and the most undesirable thing in the metal casting process. In Casting Defect Detection, deep learning based on Convolution Neural Network (CNN) models has been widely used, but most of these models require a lot of processing power. This work proposes a low-power ShuffleNet V2-based Transfer Learning model for defect identification with low latency, easy upgrading, increased efficiency, and an automatic visual inspection system with edge computing. Initially, various image transformation techniques were used for data augmentation on casting datasets to test the model flexibility in diverse casting. Subsequently, a pre-trained lightweight ShuffleNetV2 model is adapted, and hyperparameters are fine-tuned to optimize the model. The work results in a lightweight, adaptive, and scalable model ideal for resource-constrained edge devices. Finally, the trained model can be used as an edge device on the NVIDIA Jetson Nano-kit to speed up detection. The measures of precision, recall, accuracy, and F1 score were utilized for model evaluation. According to the statistical measures, the model accuracy is 99.58%, precision is 100%, recall is 99%, and the F1-Score is 100 %.

Automatic visual inspection under solar panels in Photovoltaics for linkage to IoT platform

Automatic visual inspection under solar panels in Photovoltaics for linkage to IoT platform

Rain-Like Layer Removal From Hot-Rolled Steel Strip Based on Attentive Dual Residual Generative Adversarial Network

Rain-like layer removal from hot-rolled steel strip surface is proved to be a workable measure for suppressing the false alarms frequently triggered in automated visual inspection (AVI) instrument. This paper extends the scope of “rain-like layer” from dispersed waterdrops to splashing water streaks and tiny white droplets. And a targeted method with both channel-wise and spatial-wise attention, namely attentive dual residual generative adversarial network (ADRGAN), is proposed. Meanwhile, a newly updated steel surface image dataset with typical natures of “rain-like layer” gathered from actual hot-rolling line, Steel_Rain, is opened for the first time. The comparison experimental results between our proposed network and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">eleven</i> prestigious networks show that our ADRGAN-restored images are the closest to the ground-truth images on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">six</i> public datasets, especially on the newly-opened industrial dataset Steel_Rain, it yeilds the best scores of 56.8627 peak signal to noise ratio (PSNR), 0.9980 structural similarity index (SSIM), 0.134 mean-square error (MSE) and 0.006 learned perceptual image patch similarity (LPIPS). In the final verification test, the concept of rain-like layer removal has been proved to perform best in defect inspection, where <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">four</i> traditional defect detection algorithms are involved. And as expected, defect detection methods assisted by ADRGAN yield the minimum false-alarms <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .

MobileNetV2-based Transfer Learning Model with Edge Computing for Automatic Fabric Defect Detection

In textile manufacturing, fabric defect detection is an essential quality control step and a challenging task. Earlier, manual efforts were applied to detect defects in fabric production. Human exhaustion, time consumption, and lack of concentration are the main problems in the manual defect detection process. Machine vision systems based on deep learning play a vital role in the Industrial Internet of things (IIoT) and fully automated production processes. Deep learning centered on Convolution Neural Network (CNN) models have been commonly used in fabric defect detection, but most of these models require high computing resources. This work presents a lightweight MobileNetV2-based Transfer Learning model to assist defect detection with low power consumption, low latency, easy upgrade, more efficiency, and an automatic visual inspection system with edge computing. Firstly, different image transformation techniques were performed as data augmentation on four fabric datasets for the model's adaptability in various fabrics. Secondly, fine-tuning hyperparameters of the MobileNetV2 with transfer learning gives a lightweight, adaptable and scalable model that suits the resource-constrained edge device. Finally, deploy the trained model to the NVIDIA Jetson Nano-kit edge device to make its detection faster. We assessed the model based on its accuracy, sensitivity rate, specificity rate, and F1 measure. The numerical simulation reveals that the model accuracy is 96.52%, precision is 96.52%, recall is 96.75%, and F1-Score is 96.52%.

A Novel Industrial Application of CNN Approach: Real Time Fabric Inspection and Defect Classification on Circular Knitting Machine

Fabric Automatic Visual Inspection (FAVI) system provides reliable performance on fabric defects inspection. This study presents a machine vision system developed to adapt in circular knitting machines where fabric defects can be automatically controlled and detected defects can be classified. The knitted fabric surface are detected during real-time manufacturing. For the classification process, three different transfer learning architectures (ResNet-50, AlexNet, GoogLeNet) have been applied. The five common knitted fabric defects were recognized with the artificial intelligence-based software and classified with an average success rate of 98% using ResNet-50 architecture. The success rates of the trained networks were compared.

Active Compliance Control System for Intelligent Inspection Robot in Power Room

In order to solve the problems of blind spot and time-consuming and laborious monitoring in manual inspection of power information and communication rooms, a kind of active compliant control system for intelligent inspection robots of power information and communication rooms is proposed. This research includes the noncontact detection method of the inspection robot in the power room based on machine vision; the optimization environment of the power room; the control of the inspection robot through the monitoring system terminal; the automatic visual inspection of the machine through the monitoring network transmission; the display of the monitoring terminal and the use of the calibration of the machine; and the edge detection method of the inspection image. The motion model of the inspection robot in the power room is constructed and the compliant control method of the inspection robot is designed. The experimental results show that the ineffective fluctuation range of torque of each joint is the smallest when the inspection robot is disturbed by obstacles in the process of executing signal change instructions, and the average fluctuation range of robot limbs is 0.65%. Conclusion. Based on the research results of this essay, it is proved that the controllability of power signal lamp change can be optimized separately in the future, and the safety of circuit operation can be improved.

Defect Synthesis Using Latent Mapping Adversarial Network for Automated Visual Inspection

In Industry 4.0, internet of things (IoT) technologies are expanding and advanced smart factories are currently being developed. To build an automated visual inspection (AVI) and achieve smartization of steel manufacturing, detecting defects in products in real-time and accurately diagnosing the quality of products are essential elements. As in various manufacturing industries, the steel manufacturing process presents a class imbalance problem for products. For example, fewer defect images are available than normal images. This study developed a new image synthesis methodology for the steel manufacturing industry called a latent mapping adversarial network. Inspired by the style-based generative adversarial network (StyleGAN) structure, we constructed a mapping network for the latent space, which made it possible to compose defect images of various sizes. We discovered the most suitable loss function, and optimized the proposed method in terms of convergence and computational cost. The experimental results demonstrate the competitive performance of the proposed model compared to the traditional models in terms of classification accuracy of 92.42% and F-score of 93.15%. Consequently, the problem of data imbalance is solved, and higher productivity in steel products is expected.

UzADL: Anomaly detection and localization using graph Laplacian matrix-based unsupervised learning method

Visual inspection is an essential quality control process in industrial businesses. It is usually automated due to its tedious procedure. An automated visual inspection (AVI) attempts to detect items with abnormal patterns based on image data. Recent developments in computer vision models, especially the introduction of deep convolutional neural networks, has extensively improved the accuracy and speed of AVI systems. However, supervised learning approaches for AVI necessitate a large number of annotated data, while the unsupervised ones lack accuracy and interpretability as well as require an extensive amount of time for training and inference. Therefore, in this study, we propose an unsupervised learning-based computationally inexpensive, efficient, and interpretable model UzADL for AVI to address the aforementioned problems. This system has three principal stages. First, unlabeled images are annotated using a pseudo-labeling algorithm. Second, the obtained instances are trained during a training process stage. Third, identified abnormal instances’ defective regions are explicitly visualized using an anomaly interpretation technique. Owing to an elaborate unsupervised learning method based on the pseudo-labeling algorithm using graph Laplacian matrix that allows transforming defect detection into a classification task, the proposed system has rapid convergence ability and significantly outperforms existing deep learning-based AVI methods. In the experiments conducted with three real-life fabric material databases NanoTWICE, MVTec anomaly detection (MVTec AD), and DWorld datasets UzADL outperformed other methods in terms of accuracy and speed when assessed using several evaluation metrics.

Nutritional metabolism evaluation and image segmentation of the chicken muscle and internal organs for automatic evisceration.

The chicken is rich in various proteins, fatty acids, polysaccharides, trace elements, and other human essential nutrients that contribute to its high nutritional value. In this study, the expression levels of nutrition-related genes (acetyl-CoA acyltransferase, ACAA) of native chicken breeds were investigated. The level of GgalACAA1-2 transcripts expression in the liver of chicken was significantly higher than that of muscle and heart. Moreover, three protein extracts were isolated from the muscle, heart, and liver tissues from the chicken, and their nutritional function was evaluated in the present study. These protein extracts had excellent DPPH and hydroxyl radical scavenging capacities and exhibited significant superoxide anion scavenging ability. Moreover, the protein extracts of muscle tissue showed an important mouse splenocyte proliferation activity and could be used as an immunomodulator of natural origin. In addition, this report presented an automatic visual inspection of chicken viscera using the active contour algorithms and the image processing method for eviscerating by the parallel robot. The recognition and positioning rate of chicken viscera obtained by the proposed method could reach 96.45%. These methods provided basic data for automated poultry slaughter and segmentation, avoiding unnecessary health risks by a pathogenic microorganism, such as avian influenza, Newcastle disease virus, and coronavirus. Moreover, the internal organs of the chicken could be fully harvested by the image segmentation of automatic evisceration, which also facilitated the processing value of these internal organs as by-products of poultry.

An Extended Modular Processing Pipeline for Event-Based Vision in Automatic Visual Inspection.

Dynamic Vision Sensors differ from conventional cameras in that only intensity changes of individual pixels are perceived and transmitted as an asynchronous stream instead of an entire frame. The technology promises, among other things, high temporal resolution and low latencies and data rates. While such sensors currently enjoy much scientific attention, there are only little publications on practical applications. One field of application that has hardly been considered so far, yet potentially fits well with the sensor principle due to its special properties, is automatic visual inspection. In this paper, we evaluate current state-of-the-art processing algorithms in this new application domain. We further propose an algorithmic approach for the identification of ideal time windows within an event stream for object classification. For the evaluation of our method, we acquire two novel datasets that contain typical visual inspection scenarios, i.e., the inspection of objects on a conveyor belt and during free fall. The success of our algorithmic extension for data processing is demonstrated on the basis of these new datasets by showing that classification accuracy of current algorithms is highly increased. By making our new datasets publicly available, we intend to stimulate further research on application of Dynamic Vision Sensors in machine vision applications.

Bottle Line Detection using Digital Image Processing with Machine Learning

Image processing is often used in industrial applications. The automatic visual inspection of a product before it is packaged and dispatched to the client is one of the most common uses for image processing. Using these types of systems in production facilities helps the avoidance of situations where a faulty or sub-standard product is shipped to the customer. The quantity of the fluid in each bottle differs with the accuracy level of the machine and furthermore due to mistakes done by employee during the underlying arrangement. Inappropriate filling during a bottle leads the makers to dispose of whole bunches of an item resulting in huge capital loss. A Economical mechanized review structure for inline packaging which detects oddities has been examined. The system are ending up being to be assign with checking the degree of the fluid filled inside each bottle as the container goes through the assessment framework introduced round the current packaging line. It will help to keep up the product quality and quantity as well and furthermore maintains the accuracy of the filling. The bit of the space between the outside of the fluid inside the bottle and container will be used to observe the differences like underfilling and overfilling. The proposed work is accomplished using image processing techniques in Machine Learning using python platform. Here we use ANN algorithm. Process of detecting the liquid level in bottle as follow as a group of pictures were given, taken under close to consistent lighting conditions in the manufacturing plant. These arrangement of pictures contains both underfilled and overfilled bottles which the packaging organization expects that you should use to identify certain issues as a component of the filling, covering and naming tasks before the final packing stage. The distance for each dataset must be same. We need to provide threshold value to detect if the bottle is overfilled or underfilled.

Automatic subway tunnel crack detection system based on line scan camera

Automatic visual inspection of cracks on concrete surface has become increasingly important for the health diagnosis of subway tunnel structures to prevent catastrophic failures. This paper designs an on-board image acquisition system using multiple line scan cameras to capture the full-section images of tunnel surface and proposes a robust automatic crack detection method for noisy and complicated tunnel surface images. The detection method consists of three steps. The first step is a crack enhancing process by using morphological image processing and frequency-domain enhancing algorithm. In the second step, a multistage fusion filtering algorithm is designed to filter out background noise and interference from images of water stain and various railway devices on the tunnel surface. The third step is to use an improved seed growth algorithm to segment real cracks from the noisy background. The detection system is tested in the Beijing Metro Line 1, and the results demonstrate that the proposed method outperforms existing methods that are only robust for images of small area with a simple background.

Development of Machine Vision to Increase the Level of Automation in Indonesia Electronic Component Industry

Human involvement in the assembly part manufacturing process is still relatively high. However, automation solutions are not flexible enough to be applied to manufacturing systems. It is essential to evaluate each work activity so that automation can be implemented effectively. We developed an automatic vision inspection using machine vision. The level of automation (LoA) in the company increases, and the impact caused by process failures on manual systems can be eliminated during inspection activities. The automation level increase in the inspection area is described and analyzed using the Hierarchy Task Analysis (HTA). Inspection data process activity and quality data are collected to determine the CCD camera selection, lamp selection, and lens selection. Three quality objectives, such as geometric quality, surface quality, and structural quality, are identified automatically using machine vision. Furthermore, after applying machine vision, an analysis of current LoA conditions and future LoA conditions is carried out. The results showed that the application of machine vision could increase the Level of Automation in the product inspection activity by 81.8%. There is a strong correlation (R = 0.924) between manual measurements carried out by operators and machine vision.

Learning Local Descriptor for Comparing Renders with Real Images

We present a method to train a deep-network-based feature descriptor to calculate discriminative local descriptions from renders and corresponding real images with similar geometry. We are interested in using such descriptors for automatic industrial visual inspection whereby the inspection camera has been coarsely localized with respect to a relatively large mechanical assembly and presence of certain components needs to be checked compared to the reference computer-aided design model (CAD). We aim to perform the task by comparing the real inspection image with the render of textureless 3D CAD using the learned descriptors. The descriptor was trained to capture geometric features while staying invariant to image domain. Patch pairs for training the descriptor were extracted in a semisupervised manner from a small data set of 100 pairs of real images and corresponding renders that were manually finely registered starting from a relatively coarse localization of the inspection camera. Due to the small size of the training data set, the descriptor network was initialized with weights from classification training on ImageNet. A two-step training is proposed for addressing the problem of domain adaptation. The first, “bootstrapping”, is a classification training to obtain good initial weights for second training step, triplet-loss training, that provides weights for extracting the discriminative features comparable using l2 distance. The descriptor was tested for comparing renders and real images through two approaches: finding local correspondences between the images through nearest neighbor matching and transforming the images into Bag of Visual Words (BoVW) histograms. We observed that learning a robust cross-domain descriptor is feasible, even with a small data set, and such features might be of interest for CAD-based inspection of mechanical assemblies, and related applications such as tracking or finely registered augmented reality. To the best of our knowledge, this is the first work that reports learning local descriptors for comparing renders with real inspection images.

Efficient deep feature extraction and classification for identifying defective photovoltaic module cells in Electroluminescence images

Electroluminescence (EL) imaging has become the standard test procedure for defect detection throughout the production, installation and operation stages of solar modules. Using this test, defects such as micro cracks, broken cells, and finger interruptions on photovoltaic modules could be easily detected and potential power loss issues could be effectively addressed. Although EL test is a very powerful inspection method, interpreting the EL images could be quite challenging due to the inhomogeneous background and complex defect patterns. Therefore, evaluating the damaged cells and determining the defect severity require expertise, and could be time consuming to apply these processes manually for each cell. Hence, the automatic visual inspection of photovoltaic cells is very important. In this study, a novel automatic defect detection and classification framework for solar cells is proposed. In the proposed Deep Feature-Based (DFB) method, the image features extracted through deep neural networks are classified with machine learning methods such as support vector machines, K-Nearest Neighborhood, Decision Tree, Random Forest and Naive Bayes. Thus, classical machine learning and deep learning techniques are used together. In order to combine the features taken from different deep network architectures in various combinations, the minimum Redundancy Maximum Relevance (mRMR) algorithm is employed for the feature selection. In this way, the dimensions of the feature vectors are reduced and the classification performance is increased with fewer features. With the determination of the best features extracted from different layers of deep neural networks, state-of-the-art results were obtained for both 4-class and 2-class datasets. Moreover, a Lightweight Convolutional Neural Network (L-CNN) architecture has been proposed and trained from scratch, and the results are compared with previous works. As a result, the highest scores are obtained using DFB method with Support Vector Machines (SVM) and classification scores of 90.57% and 94.52% were obtained for the dataset with 4 - class and 2 - class, respectively. The proposed DFB-SVM models outperformed other studies using the same dataset. The results showed that the proposed framework can detect PV cell defects with high accuracy.