Visual Inspection Of Defects Research Articles

Visual defect inspection of touch screens using multi-angle filtering in curvelet domain

Touch screens are widely used in smartphones and tablets. These screens exhibit a pattern of directional, regular lines on their surface. The intricate texture of this background, which quickly causes interference, poses a significant challenge in detecting surface defects. Surface defects can be mainly classified into two types: linear and planar. Existing methods cannot effectively detect both types of defects. This study proposes a curvelet transform-based multi-angle filtering method. It can effectively attenuate regular patterns from panel images with textural backgrounds and preserve fine linear and planar defects in the reconstructed image. Curvelet transform is a multi-scale directional transformation that can capture the curved edges of objects well. The filtered curvelet coefficients are then reconstructed into the spatial domain and binarized using a threshold based on the interval estimation skill. The results of the trial show that the suggested approach can precisely locate and identify defects in touch panels. The rate of defect detection (1-β) stands at 93.33 %. The rate of defect misjudgment (α) is at a low of 1.26 %. The correct classification rate (CR) is impressively high at 98.69 %, indicating that the proposed method provides fine-grained segmentation results over existing methods for detecting surface defects on touch panels.

Non-contact Testing of Large Concrete Structures for Voids and Delaminations

Concrete is the most used construction material for civil engineering structures, as it is reliable, cost-efficient, and long lasting. Infrastructure made of reinforced concrete can be used for many decades, provided that maintenance is rigorous. The necessary periodic checks are typically manual tasks, including visual inspection for defects on the outside and hammering to find internal voids and imminent spalling. In this a paper, a non-contact system for automated inspection is presented. Two parallel lasers are used with a scanning head on a mobile platform to cover large areas. The first laser emits a high energy pulse, igniting plasma on the surface and subsequently inducing a shock wave into the structure. The second laser measures the resulting vibration. Signal analysis of individual measurement points give a first indication about the presence of faults. Unsupervised learning and visual computing are used to find and narrow down damaged areas in the scanning data with higher probability. After systematic testing in the laboratory, measurements were conducted recently in two tunnels with concrete linings. This paper provides an overview of the developed hardware and signal analysis techniques. Furthermore, the recent verification results are presented for the first time. In future, the proposed method can be combined with geometric scans and automated optical inspection of the surface to offer a holistic inspection approach and relieve the authorities that are responsible for operation and maintenance of the growing number of aging infrastructures.

Augmented Hybrid Learning for Visual Defect Inspection in Real-World Hydrogen Storage Manufacturing Scenarios

Augmented Hybrid Learning for Visual Defect Inspection in Real-World Hydrogen Storage Manufacturing Scenarios

Multi-position industrial defect inspection using self-training siamese networks with mix strategies

Structural defects account for a large proportion of defects, and acquiring large batches of high-quality labels is labor-intensive and time-consuming for industrial visual defect inspection tasks. This paper addresses the above problem by exploiting sufficient unlabeled samples, and aims to achieve superior model performance with some labeled data by using self-training methods that incorporate positional information. Specifically, this paper proposes a novel self-training architecture, MixSiam, which uses a Multi-Position-based Mix strategy (MPMix) and Siamese network structure for defect classification. Furthermore, considering the prediction noise problem in unlabeled data during training, we propose a progressive MPMix (MPMix+) strategy to reduce the negative impacts of noise on model training. Finally, we validate the effectiveness of our architecture on industrial datasets. For example, our method achieves 71.40% and 87.01% accuracy on the SMT (Surface Mounting Technology) dataset and MBH (Motor Brush Holder) dataset with only 100 labeled samples, which are 2.40% and 5.86% higher than the state-of-the-art FixMatch method, respectively. Compared with the supervised algorithm with 3,600 labels, our method achieves comparable accuracy on the SMT and MBH datasets, respectively, while saving 2/3 the amount of labeled data. In conclusion, MixSiam effectively utilizes unlabeled industrial data and improves model accuracy with fewer labeled samples, thus reducing the burden of data annotation in industrial production.

Explainable attention-based fused convolutional neural network (XAFCNN) for tire defect detection: an industrial case study

Ensuring tire quality is crucial in the manufacturing industry, particularly for race cars, where defective tires present a significant safety risk. Visual inspection for defects in tires is crucial; however, identifying defects in complex, textured tires has been proven to be a challenging task. This paper tackles this challenge by introducing XAFCNN, an Explainable Attention-based Fused Convolutional Neural Network for tire defect detection. XAFCNN’s novel architecture, including a Special Attention Module (SAM) and custom CNN structure, coupled with Grad-CAM visualization, prevents overfitting, enhances local feature mapping, enables detection of small defects, and proffers valuable insights into the model’s reasoning, enabling confident interpretation of its predictions. The model was trained on a dataset from a leading global tire manufacturer, including 38,710 x-ray images of defective tires and 83,985 defect-free tire images, covering 15 defect types and 50 design patterns. The results demonstrate the model’s exceptional performance compared to literature, achieving a recall rate of 86.85%, a precision of 98.5%, an F1 score of 92.31%, and an overall accuracy of 95.40%. This research, with its substantial dataset and high-performing model, advances automated tire defect detection, satisfying the industry’s need for accurate and reliable inspections, ultimately enhancing human safety.

Adaptive explainable artificial intelligence for visual defect inspection.

Explainable Artificial Intelligence promises to deliver means so that humans better understand the rationale behind a particular machine learning model. In the image domain, such information is frequently conveyed through heat maps. Along the same line, information regarding defect detection for unsupervised methods applied to images can be conveyed through anomaly maps. Nevertheless, heat maps or anomaly maps can convey inaccurate information (artifacts), or their perceptions may differ across different persons. Therefore, the user experience could be enhanced by collecting human feedback and creating predictive models on how these could be recolored to bridge the gap between the original heat maps and anomaly maps created with explainability techniques and the output expected by humans. We envision this work as relevant in at least two scenarios. First, enhance anomaly and heat maps when conveying information regarding machine vision models deployed in production to remove information deemed unnecessary by the user but systematically present through the explainability technique due to underlying model issues (artifacts). Second, adapt anomaly and heat maps based on users’ perceptual needs and preferences.

Application of Task-Aligned Model Based on Defect Detection

In recent years, with the rise of the automation wave, reducing manual judgment, especially in defect detection in factories, has become crucial. The automation of image recognition has emerged as a significant challenge. However, the problem of how to effectively improve the classification of defect detection and the accuracy of the mean average precision (mAP) is a continuous process of improvement and has evolved from the original visual inspection of defects to the present deep learning detection system. This paper presents an application of deep learning, and the task-aligned approach is firstly used on metal defects, and the anchor and bounding box of objects and categories are continuously optimized by mutual correction. We used the task-aligned one-stage object detection (TOOD) model, then improved and optimized it, followed by deformable ConvNets v2 (DCNv2) to adjust the deformable convolution, and finally used soft efficient non-maximum suppression (Soft-NMS) to optimize intersection over union (IoU) and adjust the IoU threshold and many other experiments. In the Northeastern University surface defect detection dataset (NEU-DET) for surface defect detection, mAP increased from 75.4% to 77.9%, a 2.5% increase in mAP, and mAP was also improved compared to existing advanced models, which has potential for future use.

Automated Visual Inspection of Defects in Transparent Display Layers Using Light-Field 3-D Imaging

Since a display panel comprises multiple layered components, defects may occur within different layers through manufacturing processes. Traditional visual inspection systems with a 2D camera cannot identify the occurrence location among layers. Several 3D imaging technologies, such as CT, TSOM, and MRI, suffer from slow performance and a large form factor. In this work, we propose a novel visual inspection method to detect defects on a display panel using light-field 3D imaging. Without powering the target display panel, we first acquire the high-resolution depth information of defects located inside the transparent layers. We then convert the depth information to the object coordinate system to estimate the physical locations of defects. We automatically classify the types of defects and their layer locations along the depth axis in multiple transparent layers of the display panel. Lastly, our experimental results validate that our method can successfully detect and classify various display defects.

Application of Mobile Mapping System to a Cable-Stayed Bridge in Thailand.

Infrastructures must be inspected regularly to ensure serviceability and public safety. In the case of the Thailand expressway, 200 km of an elevated structure must be inspected once a year. Thailand expressway is an elevated reinforced concrete structure. Visual inspection for defects and structural movements such as excessive deflections, transverse movements, or settlements is a cumbersome process. Therefore, a mobile mapping 3D laser scanning (MLS) which is a high-resolution 3D laser scanner (Trimble MX-8) equipped on a vehicle, was introduced. Scanning was performed on live traffic on the expressway. From MLS, both the structure geometry and pavement point cloud data were obtained. A good agreement between elevations of the Rama XI bridge in Bangkok measured by point cloud data using MLS and by a real-time kinematic survey was obtained. The effect of mesh size on the output by MLS was investigated. It was found that a mesh size of 10 cm reduced the computational effort by 75% when compared to a mesh size of 5 cm. However, the International Roughness Index was reduced by 5%. International Roughness Index (IRI) estimated by MLS was close to the IRI values measured by the profilometer. However, a significant overestimation in the case of rutting depth was observed.

Streaming Machine Learning and Online Active Learning for Automated Visual Inspection.

Quality control is a crucial activity performed by manufacturing companies to verify product conformance to the requirements and specifications. Standardized quality control ensures that all the products are evaluated under the same criteria. The decreased cost of sensors and connectivity enabled an increasing digitalization of manufacturing and provided greater data availability. Such data availability has spurred the development of artificial intelligence models, which allow higher degrees of automation and reduced bias when inspecting the products. Furthermore, the increased inspection speed reduces overall costs and time required for defect inspection. In this research, we compare five streaming machine learning algorithms applied to visual defect inspection with real-world data provided by Philips Consumer Lifestyle BV. Furthermore, we compare them in a streaming active learning context, which reduces the data labeling effort in a real-world context. Our results show that active learning reduces the data labeling effort by almost 15% on average for the worst-case while keeping an acceptable classification performance. The use of machine learning models for automated visual inspection is expected to speed up the quality inspection up to 40%.

Inspecting and classifying physical failures in MEMS substrates during fabrication using computer vision

Visual inspection for defects has been predominantly used in all realms of the manufacturing sector. MEMS substrates and circuits face multiple challenges during the fabrication phase due to the highly reduced dimensions and the intricacy of structures developed over the substrate. The key to obtaining smooth microstructure manufacturing is to ensure that the substrate is flawless i.e., without physical defects. The manufacturing process of the metallic wafers does include a quality assessment section, however, that involves a simple dye-based comparison and rejection of the chip in case there is a defect detected in the chip. In the current manufacturing scenario, the defects are not analyzed further than this considering the complexity involved in such an analysis and the usefulness of the same. The proposed algorithm uses lightweight image processing and deep learning models using convolutional layers to detect and classify the defects in both semiconductor and polymer MEMS substrates. An average cross-validation accuracy of 99.2% and high precision and recall rates of over 99% were obtained using the proposed methodology for the wafer map defect identification. The defects classified can be used by the manufacturer to extract parts of the wafer/substrate that are still usable or even utilize the defects in different devices for novel use cases.

Development of a User-Centric Bridge Visual Defect Quality Control Assisted Mobile Application: A Case of Thailand’s Department of Highways

Digital innovations have changed the way many industries operate, but the construction industry has been slow to adopt these technologies. However, challenges such as low productivity, project overruns, labor shortages, and inefficient performance management have motivated Thailand’s Department of Highways to adopt digital innovations to build a competitive advantage. Because this industry requires a large work force, obstacles to collaboration can result in ineffective project management. We aimed to improve collaboration on bridge inspections that typically requires the involvement of many people, personal judgement, and extensive travel to survey bridges across the country. One major challenge is to standardize human judgement. To address these challenges, we developed a user-centric bridge visual defect quality control mobile application to improve collaboration and assist field technicians to conduct visual defect inspection. Our results can be used as a case study for other construction firms to embrace digital transformation technologies. This research also demonstrates the new-product development process using the new technology in known markets innovation development and technology acceptance model. We offer several recommendations for future research, including other infrastructure applications.

Improving Manufacturing Applications of Machine Learning by Understanding Defect Classification and the Critical Error Threshold

Machine learning (ML) is unlocking patterns and insight into data to provide financial value and knowledge for organizations. Use of machine learning in manufacturing environments is increasing, yet sometimes these applications fail to produce meaningful results. A critical review of how defects are classified is needed to appropriately apply machine learning in a production foundry and other manufacturing processes. Four elements associated with defect classification are proposed: Binary Acceptance Specifications, Stochastic Formation of Defects, Secondary Process Variation, and Visual Defect Inspection. These four elements create data space overlap, which influences the bias associated with training supervised machine learning algorithms. If this influence is significant enough, the predicted error of the model exceeds a critical error threshold (CET). There is no financial motivation to implement the ML model in the manufacturing environment if its error is greater than the CET. The goal is to bring awareness to these four elements, define the critical error threshold, and offer guidance and future study recommendations on data collection and machine learning that will increase the success of ML within manufacturing.

Efficient Cell Segmentation from Electroluminescent Images of Single-Crystalline Silicon Photovoltaic Modules and Cell-Based Defect Identification Using Deep Learning with Pseudo-Colorization.

Solar cells may possess defects during the manufacturing process in photovoltaic (PV) industries. To precisely evaluate the effectiveness of solar PV modules, manufacturing defects are required to be identified. Conventional defect inspection in industries mainly depends on manual defect inspection by highly skilled inspectors, which may still give inconsistent, subjective identification results. In order to automatize the visual defect inspection process, an automatic cell segmentation technique and a convolutional neural network (CNN)-based defect detection system with pseudo-colorization of defects is designed in this paper. High-resolution Electroluminescence (EL) images of single-crystalline silicon (sc-Si) solar PV modules are used in our study for the detection of defects and their quality inspection. Firstly, an automatic cell segmentation methodology is developed to extract cells from an EL image. Secondly, defect detection can be actualized by CNN-based defect detector and can be visualized with pseudo-colors. We used contour tracing to accurately localize the panel region and a probabilistic Hough transform to identify gridlines and busbars on the extracted panel region for cell segmentation. A cell-based defect identification system was developed using state-of-the-art deep learning in CNNs. The detected defects are imposed with pseudo-colors for enhancing defect visualization using K-means clustering. Our automatic cell segmentation methodology can segment cells from an EL image in about s. The average segmentation errors along the x-direction and y-direction are only pixels and pixels, respectively. The defect detection approach on segmented cells achieves accuracy. Along with defect detection, the defect regions on a cell are furnished with pseudo-colors to enhance the visualization.

Deep learning-based visual defect-inspection system for reinforced concrete bridge substructure: a case of Thailand’s department of highways

Deep learning-based visual defect-inspection system for reinforced concrete bridge substructure: a case of Thailand’s department of highways

Smooth Surface Defect Detection by Deep Learning Based on Wrapped Phase Map

Object detection is widely used in the industry for visual inspection of defects. But it fails to fully reflect the defects on the smooth surface due to the difficulty in illuminating, and the high performance of defect detection tends to be a challenge. In this paper, we proposed a novel defect detection method for laptop panels. On the one hand, we utilized the wrapped phase map obtained by phase-shift reflection fringes for defect detection and used the DenseFuse network to fuse the wrapped phase map with the fringe modulation map to optimize the dataset. On the other hand, we proposed an improved deep learning-based network Vovecwnet. The experiments indicated that our image-based multi-information fusion system could achieve excellent performance in defect detection on the smooth surface (e.g., laptop panels surface), with high accuracy of 97.5% and a few reference time 0.0762s on GTX 2080ti GPU. Thus, our proposed method outperformed and tackled the challenges encountered in the defect detection of smooth surfaces.

Unsupervised Pre-Training of Imbalanced Data for Identification of Wafer Map Defect Patterns

Visual defect inspection and classification are significant steps of most manufacturing processes in the semiconductor and electronics industries. Known and unknown defects on wafer maps tend to cluster, and these spatial patterns provide valuable process information for supporting manufacturing in determining the root causes of abnormal processes. In previous studies, data augmentation-based deep learning (DL) techniques were most commonly used for the identification of wafer map defect patterns (WMDP). Data augmentation is an effective technique for improving the accuracy of modern image classifiers. However, current data augmentation implementations were manually designed for the WMDP problem. In this study, we propose a DL-based method with automatic data augmentation for the WMDP task. Basically, it focuses on learning effective discriminative features, from wafer maps, through a deep network structure. The network consists of a convolution-based variational autoencoder (CVAE) sequentially. First, we pre-trained the CVAE on large training data in an unsupervised manner. Second, we fine-tuned the encoder of the CVAE, which was followed by a neural network (NN) classifier, in a supervised manner. Additionally, we describe a simple procedure for automatically searching for improved data augmentation policies. The policy mainly consists of five image processing functions: rotation, flipping, shifting, shearing range, and zooming. The effectiveness of the proposed method was demonstrated through experimental results obtained from a simulation dataset and a real-world wafer map dataset (WM-811K). This study provides guidance for the application of deep learning in semiconductor manufacturing processes to improve product quality and yield.

Examination of image data augmentation in CNN for the purpose of visual defect inspection process

Examination of image data augmentation in CNN for the purpose of visual defect inspection process

TruingDet: Towards high-quality visual automatic defect inspection for mental surface

Visual surface defect detection, which aims to obtain the locations of defects and classify each defect into the corresponding category in a given image, is a critical task in an actual production process. Nowadays, more and more methods have made excellent progress in visual defect inspection. However, there still exist three tough challenges where these methods cannot handle well: large defect shape change, large-scale variation, and high-quality defect localization. In this paper, a Convolutional Neural Networks (CNN) based visual defect detection framework is proposed, which elegantly mitigated these three problems by introducing three well-designed components including deformable convolution module, balanced feature pyramid module and cascade head module. First, the feature maps contained with defect shape information are adaptively extracted by Resnet/ResneXt network with the deformable convolution operator. Then the balanced feature pyramid module is attached to the feature extraction module to obtain information-fused multilayer feature maps. Finally, the cascade head is applied to refine the predicted bounding box to achieve high-quality defect localization. Under the COCO evaluation metrics, our method significantly obtains 45.2 mAP with a large margin (4.9 AP) compared with Faster RCNN baseline.

Visual Defect Inspection for Deep-Aperture Components With Coarse-to-Fine Contour Extraction

This paper investigates automatic quality inspection for the components with a small diameter and deep aperture. An automatic pick-and-place system is constructed, which employs an endoscope to achieve better image quality aiming at the characteristics of the component. A coarse-to-fine contour extraction algorithm with four steps is presented to inspect the component’s quality. First, approximate locations of the targets are estimated using faster region-based convolutional neural networks (faster RCNN). Second, the corresponding edge image is obtained by using the multiscale probability boundary (mPb) detector. Third, edge enhancement is performed, which is based on the Brownian motion model. Fourth, the corresponding contours are finely extracted by edge grouping. A shape analyzing algorithm is utilized to classify the components based on the extracted contours. Comparison experiments fully demonstrate the superiority of the proposed inspection method over existing methods. Meanwhile, successful inspection results on challenging real-world image data prove that the system is of practical significance to industrial applications.