Automatic Surface Defect Research Articles

Dual attention deep learning network for automatic steel surface defect segmentation

AbstractA dual attention deep learning network is developed to classify three types of steel defects, locate their positions, and depict their shapes on the steel surface in an automatic and accurate manner. The novel pixel‐level detection algorithm called DAN‐DeepLabv3+ integrates a dual attention module into the DeepLabv3+ framework in pursue of more precise segmentation results. For one thing, the dual parallel attention module helps to explicitly model rich contextual dependencies over local feature representations in the spatial and channel dimensions. For another, the popular DeepLabv3+ in an encoder‐decoder architecture is useful in capturing multi‐scale contextual information and sharp object boundaries. The DAN‐DeepLabv3+ is applied to an available dataset containing 6666 images, where three types of steel defects are taken by high‐frequency cameras and have been annotated manually. Experimental results show that, compared with other deep learning models, DAN‐DeepLabv3+ based on the Xception backbone exhibits the best segmentation performance under the mean intersection over union (IoU) of 89.95% and the frequency‐weighted IoU of 97.34%. Besides, the F1‐score for the three kinds of defects can reach 86.90%, 99.20%, and 92.81%. From the comparative study, it has been found that the adoption of the dual attention module and DeepLabv3+ contributes to boosting the segmentation performance. The significance of the proposed hybrid model lies in the enhancement in accurately detecting single or multiple steel defects, which has proven to outperform other classical methods.

Automatic Rail Surface Defects Inspection Based on Mask R-CNN

Rail surface defects have negative impacts on riding comfort and track safety, and could even lead to accidents. Based on the safety database (2020) of the Federal Railroad Administration (FRA), rail surface defects have been among the main factors causing derailments. During the past decades, there have been many efforts to detect such rail surface defects. However, the applications of earlier methods are limited by the high requirements of specialized equipment and personnel training. To date, rail surface defect inspection is still a very labor-intensive and time-consuming process, which hardly satisfies the field maintenance expectations. Therefore, a cost-effective and user-friendly automatic system that can inspect the rail surface defects with high accuracy is urgently needed. To address this issue, this study proposes a computer vision-based instance segmentation framework for rail surface defect inspection. A rail surface database including 1,040 images (260 source images and 780 augmented images) has been built. The classic instance segmentation model, Mask R-CNN, has been re-trained and fine-tuned for inspecting rail surface defects with the customized dataset. The influences of different backbones and learning rates are investigated and discussed. Experimental results indicate the ResNet101 backbone reaches better inspection capability. With a learning rate of 0.005, the re-trained Mask R-CNN model can achieve the best performance on the bounding box and mask predictions. Sixteen images are used to test the inspection performance of the fine-tuned model. The results are promising and indicate potential field applications in the future.

Automatic surface defect segmentation for hot-rolled steel strip using depth-wise separable U-shape network

Accurate and efficient image segmentation can contribute to improving the recognition rate of surface defects for hot-rolled steel strips. However, due to its variances in shape, position, defect type and fuzzy boundary, surface defect segmentation is a challenging task. To address this issue, a depth-wise separable U-shape network (DSUNet) is proposed. In order to reduce the computation complexity and accelerate the segmentation performance, depth-wise separable convolution is employed to replace the traditional convolutional layer. In addition, a multi-scale module is proposed to extract multi-scale context and improve the segmentation accuracy. The experimental results indicate that the accuracy and dice of DSUNet reach 95.42% and 80.8%, respectively, and the DSUNet can segment 38.5 images per second, which suggests that the DSUNet can precisely segment surface defects for hot-rolled steel strip with high efficiency.

Intelligent recognition of surface defects of parts by Resnet

Nowadays, Automatic metal surface defect recognition is an important research direction in the field of surface defect recognition, and more convolution neural network algorithms are applied in this field. However, with the deepening of network layers, network degradation will occur. We propose a ResNet method for classifying metal surface defects. After experimental testing, we use ResNet34 to build an identification network. After training 300 epoch of the network using the NEU surface defect data set, the convergence of the network is very good. The accuracy of test set verification is 93.67% and higher than that of other surface defect recognition algorithms. Also, we can deepen the number of layers ResNet the network without worrying about network degradation.

Automatic defect detection of metro tunnel surfaces using a vision-based inspection system

Due to the impact of the surrounding environment changes, train-induced vibration, and human interference, damage to metro tunnel surfaces frequently occurs. Therefore, accidents caused by the tunnel surface damage may happen at any time, since the lack of adequate and efficient maintenance. To our knowledge, effective maintenance heavily depends on the all-round and accurate defect inspection, which is a challenging task, due to the harsh environment (e.g., insufficient illumination, the limited time window for inspection, etc.). To address these problems, we design an automatic Metro Tunnel Surface Inspection System (MTSIS) for the efficient and accurate defect detection, which covers the design of hardware and software parts. For the hardware component, we devise a data collection system to capture tunnel surface images with high resolution at high speed. For the software part, we present a tunnel surface image pre-processing approach and a defect detection method to recognize defects with high accuracy. The image pre-processing approach includes image contrast enhancement and image stitching in a coarse-to-fine manner, which are employed to improve the quality of raw images and to avoid repeating detection for overlapped regions of the captured tunnel images respectively. To achieve automatic tunnel surface defect detection with high precision, we propose a multi-layer feature fusion network, based on the Faster Region-based Convolutional Neural Network (Faster RCNN). Our image pre-processing and the defect detection methods also promising performance in terms of recall and precision, which is demonstrated through a series of practical experimental results. Moreover, our MTSIS has been successfully applied on several metro lines.

Automatic surface defects classification of Kinnow mandarins using combination of multi‐feature fusion techniques

AbstractAnalyzing the quality of biological products is a difficult task, as their physicochemical properties such as size, shape, color, and texture change over time. Among many quality attributes, automatic classification of defects is a challenging work due to similarity or diversity of defects in terms of shape, color, and texture within intra and inter cultivars. In this work, an image analysis and machine learning‐based method to identify and classify four defects of Kinnow mandarins is proposed. A simple and fast adaptive thresholding technique was used to segment the defects. Defects discriminatory abilities of three prevalent texture descriptors namely local binary patterns, gray level co‐occurrence matrix (GLCM), and gray level run length matrix (GLRLM) were explored. In order to measure the effectiveness of color models in food analysis techniques, texture features were extracted on individual and combined color channels of three popular color models, that is, RGB, HSV, and CIELAB. Two machine‐learning techniques: random forest (RF) and artificial neural networks (ANNs) classifiers were trained with extracted features to predict the defects. The highest accuracies of 93.5 and 89.3% and average accuracies of 88.95 and 80.67% were achieved by ANN and RF classifiers, respectively, for the feature set {GLCM, GLRLM} on {H,S,V} color set.Practical applicationsKinnow mandarins' cultivation area occupies around 50% of the total citrus farming regions in India. Currently, the expert laborers manually carry out the grading of Kinnow fruits based on the existence of external defects. This process is labor‐intensive, inefficient, and time‐consuming. However, now this labor‐intensive task can be replaced with automatic fruit classification machines based on computer vision (CV) and machine learning (ML) technologies. Presently, there is a scarcity of studies as well as algorithms to automate the process of Kinnow fruit defects classification. The developed CV‐ and ML‐based algorithm is capable to identify and discriminate four types of external defects pertaining to Kinnow mandarins. The classification accuracy and other performance measures achieved on the developed algorithm make it ideal for real‐time online Kinnow defects classification systems.

Deep Learning-Based Generic Automatic Surface Defect Inspection (ASDI) With Pixelwise Segmentation

Automatic surface defect inspection (ASDI) is a crucial and challenging problem in industry because it affects the quality and efficiency of production greatly. Deep learning-based methods achieve promising improvements for surface detection, but they rely on a massive training data set that is impractical in industry. In this study, we propose a generic method that works effectively even on the small size training data set. Especially, we introduce a ResMask generative adversarial network (GAN) framework that is a residual GAN to expand the insufficient defect data sets. Meanwhile, the existing inspection data sets are so much easier to detect than the data sets in the real industrial scenarios that new industrial surface defect data sets containing more diverse and challenging images are established. Then, a coarse-to-fine module (CFM) that consists of a coarse detection subnetwork and a fine segmentation subnetwork is proposed for the needs of fast detection for high-resolution images. In the coarse detection stage, spatial pyramid pooling (SPP) is utilized to increase the receptive field of the network, reduce the false detection rate, and determine the approximate location of defects. In the fine segmentation stage, the receptive field of the network is enlarged by atrous SPP (ASPP), and skip links that incorporate low-level with high-level features achieve pixelwise precision on defect segmentation. Finally, our algorithm has achieved state-of-the-art results in DAGM, HR, and WB data sets (0.859, 0.761, and 0.805, respectively) according to MIoU. It achieves an average processing time of 44.4 ms on the test images with a resolution of <inline-formula> <tex-math notation="LaTeX">$512 \,\,\times \,\, 512$ </tex-math></inline-formula> and 131.6 ms for <inline-formula> <tex-math notation="LaTeX">$2048 \times 2048$ </tex-math></inline-formula> images. On the DAGM data set, the detection accuracy of mean intersection over union (MIoU) reaches 0.859.

Automatic and Efficient Metallic Surface Defect Detection Based on Key Pixel Point Locations

Surface defect detection aims to accurately recognize and distinguish types of defects and plays a key role in many applications. However, most of the recent studies focus on specific scenario detection and do not fairly consider the balance between the speed and accuracy. In the paper, we propose a key pixel points location-oriented method to identify multiscale defects, with several important properties: 1) A real-time template matching-based model is designed to speed up the process by introducing the Gaussian operator; 2) An improved Hough-based model is used to achieve a higher detection precision by deep mining both incremental properties and parallel properties; and 3) An adaptive filtering-based image preprocessing method is proposed to eliminate the interference of multiple types of clutters and noises. In the experiments, a mean average rate of 96% was achieved to detect and classify four types of common defects and the average time was reduced to 0.149 s. Furthermore, we fully evaluate the proposed method on two public datasets collected in real production lines and compare the results with other state-of-the-art methods. The results show that the proposed method achieved better balanced performance in many real application scenarios.

Method for automatic railway track surface defect classification and evaluation using a laser‐based 3D model

Inspection of physical surface defects is a significant concern in many industrial areas. In railway systems, this process mainly includes the detection and classification of defects in rails and wheels, for which laser-based optical inspection technologies have gradually been applied in the form of 2D profile measurement, benefiting from its high precision and robustness to surface conditions. However, defect classification and evaluation after the initial detection works still rely heavily on human inspectors to make maintenance suggestions. The linear nature of rails makes it possible to increase the dimension of rail measurement data from 2D to 3D by aligning 2D profiles along the rail, from which more comprehensive diagnosis information becomes available. In combination with appropriate artificial intelligence algorithms, this approach can potentially replace human-dominated defect classification and evaluation work. This study presents a 3D model-based railway track surface defect classification and evaluation method. A set of geometrical features are extracted from the 3D model of track surface defects to describe a distinguishable pattern for each category of defect. Multi-class classifiers are then tested and have shown promising results on a group of artificial track surface defects, giving a systemic solution for 3D model-based automatic track surface defect inspection.

SurfNetv2: An Improved Real-Time SurfNet and Its Applications to Defect Recognition of Calcium Silicate Boards.

This paper presents an improved Convolutional Neural Network (CNN) architecture to recognize surface defects of the Calcium Silicate Board (CSB) using visual image information based on a deep learning approach. The proposed CNN architecture is inspired by the existing SurfNet architecture and is named SurfNetv2, which comprises a feature extraction module and a surface defect recognition module. The output of the system is the recognized defect category on the surface of the CSB. In the collection of the training dataset, we manually captured the defect images presented on the surface of the CSB samples. Then, we divided these defect images into four categories, which are crash, dirty, uneven, and normal. In the training stage, the proposed SurfNetv2 is trained through an end-to-end supervised learning method, so that the CNN model learns how to recognize surface defects of the CSB only through the RGB image information. Experimental results show that the proposed SurfNetv2 outperforms five state-of-the-art methods and achieves a high recognition accuracy of 99.90% and 99.75% in our private CSB dataset and the public Northeastern University (NEU) dataset, respectively. Moreover, the proposed SurfNetv2 model achieves a real-time computing speed of about 199.38 fps when processing images with a resolution of 128 × 128 pixels. Therefore, the proposed CNN model has great potential for real-time automatic surface defect recognition applications.

Automated surface feature detection using fringe projection: An autoregressive modeling-based approach

An automatic surface defect detection algorithm is proposed in a fringe projection profilometry setup. An exponential phase field associated with the fringe projected surface image is analyzed using two-dimensional auto-regressive (AR) model. The AR model coefficients capture the local fringe frequency information. The variation in the fringe frequency from the defect-free to defective surface region is utilized as a signature for defect identification and localization. The fringe frequency threshold required for the image segmentation into defective and defect free regions is derived from the mean frequency computed over the same image which obviates the need for the reference image. In order to enhance the computation efficiency of the algorithm the image is divided into small patches and analysis is performed patch-wise. The simulation and experimental results demonstrate the advantages of the proposed phase based surface evaluation method over the commonly used intensity based methods.

Automatic Metallic Surface Defect Detection and Recognition with Convolutional Neural Networks

Automatic metallic surface defect inspection has received increased attention in relation to the quality control of industrial products. Metallic defect detection is usually performed against complex industrial scenarios, presenting an interesting but challenging problem. Traditional methods are based on image processing or shallow machine learning techniques, but these can only detect defects under specific detection conditions, such as obvious defect contours with strong contrast and low noise, at certain scales, or under specific illumination conditions. This paper discusses the automatic detection of metallic defects with a twofold procedure that accurately localizes and classifies defects appearing in input images captured from real industrial environments. A novel cascaded autoencoder (CASAE) architecture is designed for segmenting and localizing defects. The cascading network transforms the input defect image into a pixel-wise prediction mask based on semantic segmentation. The defect regions of segmented results are classified into their specific classes via a compact convolutional neural network (CNN). Metallic defects under various conditions can be successfully detected using an industrial dataset. The experimental results demonstrate that this method meets the robustness and accuracy requirements for metallic defect detection. Meanwhile, it can also be extended to other detection applications.

Defect Detection of Mobile Phone Surface Based on Convolution Neural Network

Automatic surface defect detection of mobile phone in large scale needs to process high resolution images and handle various defects while achieving high accuracy rate. This study proposes a defect detection method based on convolution neural network (CNN). Firstly, the original surface image of mobile phone is obtained using industrial linear array camera. Secondly, the obtained images are automatically segmented into specified sizes by the proposed preprocessing step. Moreover, we design the CNN on basis of GoogLeNet network, which greatly reduces the number of parameters without compromising prediction rate. At last the designed CNN are trained and tested. The trained CNN can be combined with a sliding window technique to detect any ROI with size larger than 256×256 resolutions in the original images. The experimental results show that the defect detection rate of the designed CNN can achieve as high as 99.5%.

Surface Defect Inspection for Vehicle Heat Exchangers Based on LBP and SVM

To tackle the problems appeared in traditional surface defect inspection for vehicle heat exchangers which is mainly depended on manual execution, machine vision has been introduced due to its unparalleled characteristics of on-line, fast, high precision and non-contact. In this paper, an automatic surface defect detection algorithm employing machine vision technology has been presented. Treating the surface defect detection as a texture discrimination problem, the proposed method first uses an LBP (Local Binary Pattern) descriptor to represent the image of the heat exchanger, which has efficiently alleviated the effect of uneven illumination. Then the whole texture image has been partitioned into several non-overlapped 32×32 blocks and each block is subject to classification by an SVM (Support Vector Machine) to judge if it is a defect area. The experiments have been performed and the results have demonstrated the effectiveness and the efficiency of the proposed method.

Automatic surface defect detection for mobile phone screen glass based on machine vision

Defect detection using machine vision technology plays an important role in the manufacturing process of mobile phone screen glass (MPSG). This study proposes an improved detection algorithm for MPSG defect recognition and segmentation. Considering the problem of MPSG image misalignment caused by vibrations in the mobile stages, a contour-based registration (CR) method is used to generate the template image used to align the MPSG images. Based on this registration result, the combination of subtraction and projection (CSP) is used to identify defects on the MPSG image, which can eliminate the influence of fluctuation in ambient illumination. To segment the defects with a fuzzy grey boundary from a noisy MPSG image, an improved fuzzy c-means cluster (IFCM) algorithm is developed in this study. A defect detection system is developed, and the proposed algorithms are validated using a number of experimental tests on MPSG images. The testing results demonstrate that the approach proposed in this study can effectively detect various defects on MPSG and that it has better performance than other methods.

Vision-based automatic detection of steel surface defects in the cold rolling process: considering the influence of industrial liquids and surface textures

In the cold rolling process, automatic surface defect detection could be considered as an important technique to monitor product quality, and so far, substantial kinds of surface defects have been already successfully inspected. However, the interference of industrial liquids and produced surface textures has been rarely studied yet. To address this problem, the vision-based automatic detection method is proposed in this paper containing three main parts: (i) region extraction, (ii) defect detection, and (iii) industrial liquid quantification. Four sets of experiments are designed to verify the feasibility, accuracy, and stability of the proposed method. The results indicate that the proposed method can detect most cracks and scratches with the accuracy of more than 91 %. Also, the proposed method could accurately quantify the industrial liquids including their numbers (with the average accuracy of more than 90 %) and their sizes (with the accuracy of more than 91 %); it is therefore expected to be promising because it could be utilized to track the online running status of the rolling system and provide rolling system maintenance suggestions.

Research on Identification of Dust Particles on COF

Chip On Film(COF) is the key component of electronic products, and is different from Printed Circuit Board(PCB). The properties of high flexibility, thin thickness, lightweight and high wiring density make it difficult to inspect COF, especially dust particles interference. Dust particles are similar to defects, and it is hard to identify dust particles from defects, so dust particles interference of quality test is the difficulty of automatic surface defect detection. In this paper, a new method to identify dust particles is discussed from abnormal area called junction points detection and machine learning method Support Vector Machine(SVM) according to the characteristics of dust particles. As a result, a 94.8% correct rate of dust particles images identification has been achieved with the method.

A cost-effective and automatic surface defect inspection system for hot-rolled flat steel

Defective steel brings economic and commercial reputation losses to the hot-strip manufacturers, and one of the main difficulties in using machine-vision-based technique for steel surface inspection is time taken to process the massive images suffering from uneven illumination. This paper develops a modular and cost-effective AOI system for hot-rolled flat steel in real time. Firstly, a detailed system topology is constructed according to the design goals covering the vast majority of steel mills, lighting setup and typical defect patterns are presented as well. Secondly, the image enhancement method is designed to overcome the uneven-lighting, over- or under-exposure. Thirdly, the defect detection algorithm is developed based on variance, entropy and average gradient derived from non-overlapping 32×32 pixel blocks of steel surface images. Fourthly, the proposed algorithms are implemented on FPGA in parallel to improve the inspection speed. Finally, 18,071 contiguous images (4096×1024 pixel) acquired from 7 defective steel rolls have been inspected by the realized AOI system to evaluate the performance. The experimental results show that the proposed method is speedy and effective enough for real applications in the hot-rolled steel manufacturing, with 92.11% average accuracy while 5.54% false-negative rate. HighlightsDetailed system topology is constructed according to the design goals covering the vast majority of steel mills.A new dynamic image enhancement method is designed to overcome the uneven-lighting, over- or under-exposure.A defect detection algorithm with adaptive thresholding mechanism is developed based on image block variance, entropy and average gradient.The inspection speed of the proposed algorithms running on FPGA is evaluated.Testing procedures and evaluation results from the applied hot-rolling mill are discussed.

An Intelligent and Confident System for Automatic Surface Defect Quantification in 3D

Automatic surface defect inspection within mass production of high-precision components is growing in demand and requires better measurement and automated analysis systems. Many manufacturing industries may reject manufactured parts that exhibit even minor defects, because a defect might result in an operational failure at a later stage. Defect quantification (depth, area and volume) is a key element in quality assurance in order to determine the pass or failure criterion of manufactured parts. Existing human visual analysis of surface defects is qualitative and subjective to varying interpretation. Non-contact and three dimensional (3D) analyses should provide a robust and systematic quantitative approach for defect analysis. Various 3D measuring instruments generate point cloud data as an output, although they work on different physical principles. Instrument’s native software processing of point cloud data is often subject to issues of repeatability and may be non-traceable causing significant concern with data confidence.This work reports the development of novel traceable surface defect artefacts produced using the Rockwell hardness test equipment on flat metal plate, and the development of a novel, traceable, repeatable, mathematical solution for automatic defect detection and quantification in 3D. Moreover, in order to build-up the confidence in automatic defect analysis system and generated data, mathematical simulated defect artefacts (soft-artefact) have been created. This is then extended to a surface defect on a piston crown that is measured and quantified using a parallel optical coherence tomography instrument integrated with 6 axis robot. The results show that surface defect quantification using implemented solution is efficient, robust and more repeatable than current alternative approaches.

Review of vision-based steel surface inspection systems

Steel is the material of choice for a large number and very diverse industrial applications. Surface qualities along with other properties are the most important quality parameters, particularly for flat-rolled steel products. Traditional manual surface inspection procedures are awfully inadequate to ensure guaranteed quality-free surface. To ensure stringent requirements of customers, automated vision-based steel surface inspection techniques have been found to be very effective and popular during the last two decades. Considering its importance, this paper attempts to make the first formal review of state-of-art of vision-based defect detection and classification of steel surfaces as they are produced from steel mills. It is observed that majority of research work has been undertaken for cold steel strip surfaces which is most sensitive to customers' requirements. Work on surface defect detection of hot strips and bars/rods has also shown signs of increase during the last 10 years. The review covers overall aspects of automatic steel surface defect detection and classification systems using vision-based techniques. Attentions have also been drawn to reported success rates along with issues related to real-time operational aspects.