Visual Inspection Applications Research Articles

Deep Learning for Automated Visual Inspection in Manufacturing and Maintenance: A Survey of Open- Access Papers

Quality assessment in industrial applications is often carried out through visual inspection, usually performed or supported by human domain experts. However, the manual visual inspection of processes and products is error-prone and expensive. It is therefore not surprising that the automation of visual inspection in manufacturing and maintenance is heavily researched and discussed. The use of artificial intelligence as an approach to visual inspection in industrial applications has been considered for decades. Recent successes, driven by advances in deep learning, present a possible paradigm shift and have the potential to facilitate automated visual inspection, even under complex environmental conditions. For this reason, we explore the question of to what extent deep learning is already being used in the field of automated visual inspection and which potential improvements to the state of the art could be realized utilizing concepts from academic research. By conducting an extensive review of the openly accessible literature, we provide an overview of proposed and in-use deep-learning models presented in recent years. Our survey consists of 196 open-access publications, of which 31.7% are manufacturing use cases and 68.3% are maintenance use cases. Furthermore, the survey also shows that the majority of the models currently in use are based on convolutional neural networks, the current de facto standard for image classification, object recognition, or object segmentation tasks. Nevertheless, we see the emergence of vision transformer models that seem to outperform convolutional neural networks but require more resources, which also opens up new research opportunities for the future. Another finding is that in 97% of the publications, the authors use supervised learning techniques to train their models. However, with the median dataset size consisting of 2500 samples, deep-learning models cannot be trained from scratch, so it would be beneficial to use other training paradigms, such as self-supervised learning. In addition, we identified a gap of approximately three years between approaches from deep-learning-based computer vision being published and their introduction in industrial visual inspection applications. Based on our findings, we additionally discuss potential future developments in the area of automated visual inspection.

Towards Workflows for the Use of AI Foundation Models in Visual Inspection Applications

AbstractThe latest successes in AI have been largely driven by a paradigm known as Foundation Models (FMs), large Neural Networks pretrained on massive datasets that thereby acquire impressive transfer learning capabilities to adapt to new tasks. The emerging properties of FMs have unlocked novel tantalizing applications for instance enabling the generation of fluent text and realistic images from text descriptions. The impact of FMs on technical domains like civil engineering is however still in its infancy, owing to a gap between research development and application use cases. This paper aims to help bridge this gap and promote adoption among technical practitioners, specifically in visual inspection applications for civil engineering. For that we analyze the requirements in terms of data availability making particular use cases amenable to the pretraining/fine‐tuning paradigm of FMs, i.e. situations where labeled data is scarce or costly, but unlabeled data is abundant. We then illustrate proof‐of‐concepts workflows using FMs, in visual inspection applications. We hope that our contribution will mark the start of conversations between AI researchers and civil engineers on the potential of FMs to accelerate workflows supporting vision tasks for maintenance inspections and decisions.

Unsupervised defect segmentation with pose priors

AbstractSingle shot, semantic bounding box detectors, trained in a supervised manner are popular in computer vision‐aided visual inspections. These methods have several key limitations: (1) bounding boxes capture too much background, especially when images experience perspective transformation; (2) insufficient domain‐specific data and cost to label; and (3) redundant or incorrect detection results on videos or multi‐frame data; where it is a nontrivial task to select the best detection and check for outliers. Recent developments in commercial augmented reality and robotic hardware can be leveraged to support inspection tasks. A common capability of the previous is the ability to obtain image sequences and camera poses. In this work, the authors leverage pose information as “prior” to address the limitations of existing supervised learned, single‐shot, semantic detectors for the application of visual inspection. The authors propose an unsupervised semantic segmentation method (USP), based on unsupervised learning for image segmentation inspired by differentiable feature clustering coupled with a novel outlier rejection and stochastic consensus mechanism for mask refinement. USP was experimentally validated for a spalling quantification task using a mixed reality headset (Microsoft HoloLens 2). Also, a sensitivity study was conducted to evaluate the performance of USP under environmental or operational variations.

Research on Defect Inspection Technology for Bump Height in Wafer-Level Packaging Based on the Triangulation Method

With the advent of the era of big data and the vigorous development of consumer electronics, the demand for higher-speed processing capacity for gigantic amounts of data is increasing; this requires finer and far more numerous connections between dies as an essential component for connection and electronic communication between layers. The coplanarity of the height of micro bumps is crucial to ensure the reliability of the chip. The triangulation method has been widely used in various types of visual inspection applications over the past several years due to its high measurement accuracy and fast detection speed. In this paper, the application of triangulation technology is analyzed and used in micro bump height measurement. The measurement system proposed consists of a delicate design, a linearly arranged light projection module and a high-quality imaging module. Along the light stripe, multiple-height data of micro bumps are collected in a frame, and the full-field measurement is realized as the wafer is driven by the high-precision motional stage, following the optimal measurement route planned previously. In the scanning process, the images of the light strip projected on the top and bottom of the micro bumps are simultaneously acquired by the sophisticated micro vision system, and then the heights of micro bumps are calculated from a simple formula based on the geometry of the specimen and the system configuration. The experimental results demonstrate that the measuring deviation of the micro bump height has an accuracy of 1.5 μm.

CASPPNet: a chained atrous spatial pyramid pooling network for steel defect detection

Automatic machine vision-based defect detection has been successfully applied to many industrial visual inspection applications. However, automatic steel surface defect detection is still a challenging task due to diverse defect categories, low-contrast between defect and complex texture background. To address these challenges, a chained atrous spatial pyramid pooling network (CASPPNet) is proposed for steel surface defect detection. In CASPPNet, chained atrous spatial pyramid pooling is designed to enlarge receptive field and obtain enrich semantic information. An improved global attention feature fusion module is introduced to achieve feature interaction and salience. Moreover, residual boundary refinement block is introduced to get more complete defect boundary. Comparative experimental results verify that our method is superior to the state-of-the-art segmentation methods on public accessible SD-saliency-900 datasets and can meet the requirement of real-time online detection (the detection efficiency is at over 47 FPS on a single GPU).

Multi-Uav Trajectory Planning for 3d Visual Inspection of Complex Structures

This paper presents a new trajectory planning algorithm for 3D autonomous UAV volume coverage and visual inspection. The algorithm is an extension of a state-of-the-art Heat Equation Driven Area Coverage (HEDAC) multi-agent area coverage algorithm for 3D domains. With a given target exploration density field, the algorithm designs a potential field and directs UAVs to the regions of higher potential, i.e., higher values of remaining density. Collisions between the agents and agents with domain boundaries are prevented by implementing the distance field and correcting the agent's directional vector when the distance threshold is reached. A unit cube test case is considered to evaluate this trajectory planning strategy for volume coverage. For visual inspection applications, the algorithm is supplemented with camera direction control. A field containing the nearest distance from any point in the domain to the structure surface is designed. The gradient of this field is calculated to obtain the camera orientation throughout the trajectory. Three different test cases of varying complexities are considered to validate the proposed method for visual inspection. The simplest scenario is a synthetic portal-like structure inspected using three UAVs. The other two inspection scenarios are based on realistic structures where UAVs are commonly utilized: a wind turbine and a bridge. When deployed to a wind turbine inspection, two simulated UAVs traversing smooth spiral trajectories have successfully explored the entire turbine structure while cameras are directed to the curved surfaces of the turbine's blades. In the bridge test case an efficacious visual inspection of a complex structure is demonstrated by employing a single UAV and five UAVs. The proposed methodology is successful, flexible and applicable in real-world UAV inspection tasks.

A two-stage CNN for automated tire defect inspection in radiographic image

Visual inspection plays a crucial role during the manufacturing of tires which are essential for safe driving. Due to complicated anisotropic multi-texture background and ambiguous defect in it, automated tire defect detection is facing huge challenges and high costs. In this study, a novel two-stage convolutional neural network (CNN) is proposed for tire inspection by combining an optimized YOLOv3 and improved pyramid scene parsing network. Comparative experiments are conducted with the-state-of-the-art to validate the effectiveness and superior performance of the proposed method. The proposed two-stage CNN method achieves an average precision of 91.39%, the defect semantic segmentation achieves a mean intersection over union of 87.86%. The average detection time for a tire is 1.158 s such that the method can be effectively implemented into the industrial workflow. It can also be easily applied to different visual inspection applications.

Efficient Binarized Convolutional Layers for Visual Inspection Applications on Resource-Limited FPGAs and ASICs

There has been a recent surge in publications related to binarized neural networks (BNNs), which use binary values to represent both the weights and activations in deep neural networks (DNNs). Due to the bitwise nature of BNNs, there have been many efforts to implement BNNs on ASICs and FPGAs. While BNNs are excellent candidates for these kinds of resource-limited systems, most implementations still require very large FPGAs or CPU-FPGA co-processing systems. Our work focuses on reducing the computational cost of BNNs even further, making them more efficient to implement on FPGAs. We target embedded visual inspection tasks, like quality inspection sorting on manufactured parts and agricultural produce sorting. We propose a new binarized convolutional layer, called the neural jet features layer, that learns well-known classic computer vision kernels that are efficient to calculate as a group. We show that on visual inspection tasks, neural jet features perform comparably to standard BNN convolutional layers while using less computational resources. We also show that neural jet features tend to be more stable than BNN convolution layers when training small models.

Automatic optical inspection platform for real-time surface defects detection on plane optical components based on semantic segmentation.

The tendency to increase the accuracy and quality of optical parts inspection can be observed all over the world. The imperfection of manufacturing techniques can cause different defects on the optical component surface, making surface defects inspection a crucial part of the manufacturing of optical components. Currently, the inspection of lenses, filters, mirrors, and other optical components is performed by human inspectors. However, human-based inspections are time-consuming, subjective, and incompatible with a highly efficient high-quality digital workflow. Moreover, they cannot meet the complex criteria of ISO 10110-7 for the quality pass and fail optical element samples. To meet the high demand for high-quality products, intelligent visual inspection systems are being used in many manufacturing processes. Automated surface imperfection detection based on machine learning has become a fascinating and promising area of research, with a great direct impact on different visual inspection applications. In this paper, an optical inspection platform combining parallel deep learning-based image-processing approaches with a high-resolution optomechanical module was developed to detect surface defects on optical plane components. The system involves the mechanical modules, the illumination and imaging modules, and the machine vision algorithm. Dark-field images were acquired using a 2448×2048-pixel line-scanning CMOS camera with 3.45 µm per-pixel resolution. Convolutional neural networks and semantic segmentation were used for a machine vision algorithm to detect and classify defects on captured images of optical bandpass filters. The experimental results on different bandpass filter samples have shown the best performance compared to traditional methods by reaching an impressive detection speed of 0.07 s per image and an overall detection pixel accuracy of 0.923.

Evaluation of Visual Inspection of Cervix with Acetic Acid and Liquid Based in Cervical Cancer Screening with Cervical Biopsy.

Cervical cancer is the second most common cancer among women in developing countries. Cervical cancer generally develops slowly over a period of 10-15 years. Incidence and mortality related to cervical cancer both have declining in developed countries because of effective screening programs through Papanicolaou smear. Therefore, cervical cancer can be prevented through implementation of different methods of screening programs like visual inspection of cervix with application of acetic acid visual inspection with acetic acid, liquid based cytology and human papilloma virus deoxyribonucleic acid. The purpose of this study is to compare the efficacy of visual inspection with Acetic Acid with liquid based cytology in cervical cancer screening taking cervical biopsy as a gold standard. The study was conducted at Paropakar Maternity and Women's Hospital, Kathmandu. One hundred forty four patients underwent visual inspection with acetic acid and liquid based cytology test followed by biopsy for confirmation of the lesion, when required. Data were obtained and statistically analyzed. Out of 144 screened patients, 62 (43.05%) were positive in visual inspection with acetic acid test. Eighteen (12.5%) cases were positive in liquid based cytology. Thirteen women were positive with both tests. Thirty-nine cases underwent histopathological examination including 13 cases who were positive in both tests. The sensitivity, specificity, positive predictive value and negative predictive value for visual inspection with acetic acid was 81.25%, 65.22%, 61.90% and 83.33%, whereas for liquid based cytology it was 100%, 91.30%, 88.89% and 94.87% respectively. Liquid based cytology was more efficacious to diagnose atypical cells with higher sensitivity and specificity in comparison to that of visual Inspection with Acetic Acid test.

Visual Inspection Application of Substation Based on Internet of Things

After the Internet, the Internet of Things has become another innovative technology in recent years. In recent years, the Internet of Things technology has been applied to various industries, such as agricultural production, intelligent transportation, industrial production, etc., which also includes visual inspection of substations operating system. In the implementation of the Internet of Things visual inspection application system of the substation, the wireless data collection of the substation is a key part of the entire system. The system needs to use advanced wireless sensing technology, remote data transmission technology, Internet technology and other key technologies of the Internet of Things. Since the patrol inspection of substations in the domestic power system is still dominated by manual patrol inspection, due to the high intensity of the patrol inspection and the harsh operating environment, the work efficiency is insufficient. This article starts from the needs of the Internet of Things- based substation visual inspection application, integrates autonomous robots, sensor fusion technology, visual recognition technology and GPS positioning technology, builds a 24/7, all-round autonomous inspection visualization system, and initially achieves substation autonomy Patrol function. Secondly, it analyzes the principles of the autonomous robot inspection and visualization system, expounds infrared information collection and alarm functions, and analyzes a series of functions from information visualization and recognition to background operations. The research results show that the autonomous robot inspection visualization system can improve the inspection efficiency of the power station by more than 85%, and effectively reduce the strength and risk of workers.

Views of veterinarians and meat inspectors concerning the practical application of visual meat inspection on domestic pigs in Finland

The post-mortem inspection of domestic pigs within the European Union was revised in 2014, primarily to include visual meat inspection of each carcase and offal. Palpations and incisions were removed from routine meat inspection procedures, as they are mostly used to detect pathological lesions caused by organisms irrelevant for public health, and instead can cause cross-contamination of carcases with foodborne pathogens. However, examination of all external surfaces of the carcase and organs, declaration of patho-physiological lesions as unfit for human consumption, and possibility for minimal handling of carcases and offals were held in place. In addition, the European Food Safety Authority suggested that palpation and incisions should be performed outside the slaughter line, but this was not incorporated in the revised legislation. We surveyed in 2014 the opinions of meat inspectors and veterinarians using an online questionnaire to determine what practical measures are required for the visual meat inspection procedure and when meat inspection staff consider additional palpations and incisions necessary. Based on the survey, turning the carcase and organs or technical arrangements such as mirrors were seen necessary to view all external surfaces. In addition, the pluck set cannot be trimmed on the side line. Local lesions, such as abscesses and lesions in the lymph nodes, signs of systemic infection and lymphoma, were the major lesions requiring additional post-mortem meat inspection procedures. Meat inspection personnel raised concerns on the poor quality of food chain information and export requirements demanding palpations and incisions. The efficient use of visual meat inspection requires legislation to better support the implementation and application of it, changes in the slaughter line layout and a possibility to classify incoming pig batches based on their risk.

Efficient Evolutionary Learning Algorithm for Real-Time Embedded Vision Applications

This paper reports the development of an efficient evolutionary learning algorithm designed specifically for real-time embedded visual inspection applications. The proposed evolutionary learning algorithm constructs image features as a series of image transforms for image classification and is suitable for resource-limited systems. This algorithm requires only a small number of images and time for training. It does not depend on handcrafted features or manual tuning of parameters and is generalized to be versatile for visual inspection applications. This allows the system to be configured on the fly for different applications and by an operator without extensive experience. An embedded vision system, equipped with an ARM processor running Linux, is capable of performing at roughly one hundred 640 × 480 frames per second which is more than adequate for real-time visual inspection applications. As example applications, three image datasets were created to test the performance of this algorithm. The first dataset was used to demonstrate the suitability of the algorithm for visual inspection automation applications. This experiment combined two applications to make it a more challenging test. One application was for separating fertilized and unfertilized eggs. The other one was for detecting two common defects on the eggshell. Two other datasets were created for road condition classification and pavement quality evaluation. The proposed algorithm was 100% for fertilized egg detection and 98.6% for eggshell quality inspection for a combined 99.1% accuracy. It had an accuracy of 92% for the road condition classification and 100% for pavement quality evaluation.

Incremental learning of concept drift in Multiple Instance Learning for industrial visual inspection

Most Multiple Instance Learning (MIL) algorithms are designed with the assumption that the target concept is stationary in time, i.e. it is drawn from a stationary unknown distribution. However, in real industrial applications, like automatic visual inspection where defects may evolve, MIL has to deal with changing target concepts whose statistical characteristics may vary over time. Despite this fact, there is little discussion about how to learn from data in non-stationary environments (or data with concept drift) using multiple instance learners. In this work, an incremental MIL algorithm is proposed in order to learn non-stationary and recurrent target concepts in industrial visual inspection applications. Experiments on both synthetic and real-world datasets are conducted to test the performance of the proposed approach. Real-world datasets come from the automatic visual inspection task in industry. The experimental results show that the proposed approach is able to handle changing target concepts over time.

On the Application of Automated Machine Vision for Leather Defect Inspection and Grading: A Survey

Reliably and effectively detecting and classifying leather surface defects is of great importance to tanneries and industries that use leather as a major raw material such as leather footwear and handbag manufacturers. This paper presents a detailed and methodical review of the leather surface defects, their effects on leather quality grading and automated visual inspection methods for leather defect inspection. A detailed review of inspection methods based on leather defect detection using image analysis methods is presented, which are usually classified as heuristic or basic machine learning based methods. Due to the recent success of deep learning methods in various related fields, various architectures of deep learning are discussed that are tailored to image classification, detection, and segmentation. In general, visual inspection applications, where recent CNN architectures are classified, compared, and a detailed review is subsequently presented on the role of deep learning methods in leather defect detection. Finally, research guidelines are presented to fellow researchers regarding data augmentation, leather quality quantification, and simultaneous defect inspection methods, which need to be investigated in the future to make progress in this crucial area of research.

Accuracy and Reliability of Visual Inspection and Smartphone Applications for Measuring Finger Range of Motion.

Measurement of finger range of motion is critical in clinical settings, especially for outcome analysis, clinical decision making, and rehabilitation/disability assessment. Although goniometer measurement is clinically considered the gold standard, its accuracy compared with the true radiographic measurements of the joint angles remains questionable. The authors compared 3 smartphone applications and visual inspection measurements of the finger joints with the radiographic measurements and determined interrater reliability for these measurement tools. A finger was held in place using an aluminum-alloy splint, and a fluoroscopic image was acquired by a mini C-arm. An independent observer measured each joint flexion angle of the fluoroscopic image using a universal handheld goniometer, and this was used as the reference. Finger joint flexion angles were then independently measured by 3 observers using 3 different smartphone applications. In addition, visual inspection was used to estimate the flexion angles of finger joints. The results of this study suggest that all 3 smartphone measurement tools, as well as visual inspection, agree and correlate well with the reference fluoroscopic image measurement. Average differences between the fluoroscopic image measurements with the measured angles using the tools studied ranged from 9.4° to 12.2°. The mean correlation coefficients for each smartphone application exceeded 0.7. Overall interrater reliabilities were similar, with the interclass correlation coefficient being greater than 0.9 for all of the measurement tools. These data suggest that new smartphone applications hold promise for providing accurate and reliable measures of range of motion. [Orthopedics. 2018; 41(2):e217-e221.].

Polygon guarding with orientation

The art gallery problem is a classical sensor placement problem that asks for the minimum number of guards required to see every point in an environment. The standard formulation does not take into account self-occlusions caused by a person or an object within the environment. Obtaining good views of an object from all orientations despite self-occlusions is an important requirement for surveillance and visual inspection applications. We study the art gallery problem under a constraint, termed △-guarding, that ensures that all sides of any convex object are always visible in spite of self-occlusion.Our contributions in this paper are three-fold: We first prove that Ω(n) guards are always necessary for △-guarding the interior of a simple polygon having n vertices. Second, we present a O(log⁡copt) factor approximation algorithm for △-guarding polygons with or without holes, when the guards are restricted to vertices of the polygon. Here, copt is the optimal number of guards. Third, we study the problem of △-guarding a set of line segments connecting points on the boundary of the polygon. This is motivated by applications where an object or person of interest can only move along certain paths in the polygon. We present a constant factor approximation algorithm for this problem – one of the few such results for art gallery problems.

Multicriteria Assessment and Prioritization of Hospital Renewal Needs

AbstractAccurate performance assessment of building systems and subsystems (e.g., roof, boiler, windows, etc.) is essential to properly prioritize the subsystems that compete for capital renewal funding. Typically, the performance assessment of buildings has been based on physical condition alone, which is not sufficient for specialized facilities such as hospitals. This paper introduces a practical multicriteria performance assessment framework for hospitals with four unique features: (1) it divides the hospital into three zones with different characteristics and importance: clinical, nursing, and support; (2) it assesses the performance of hospital subsystems, considering four key performance indicators: condition, level of service, sustainability, and risk; (3) it assesses the service quality (temperature, lighting, noise, etc.) within various indoor spaces and accordingly determines the level of service of related subsystems; and (4) it incorporates a simple all-on-site visual inspection application t...

Efficient and accurate image alignment using TSK-type neuro-fuzzy network with data-mining-based evolutionary learning algorithm

Abstract Image alignment is considered a key problem in visual inspection applications. The main concerns for such tasks are fast image alignment with subpixel accuracy. About this, neural network-based approaches are very popular in visual inspection because of their high accuracy and efficiency of aligning images. However, such methods are difficult to identify the structure and parameters of neural network. In this study, a Takagi-Sugeno-Kang-type neuro-fuzzy network (NFN) with data-mining-based evolutionary learning algorithm (DMELA) is proposed. Compared with traditional learning algorithms, DMELA combines the self-organization algorithm (SOA), data-mining selection method (DMSM), and regularized least square (RLS) method to not only determine a suitable number of fuzzy rules, but also automatically tune the parameters of NFN. Experimental results are shown to demonstrate superior performance of the DMELA constructed image alignment system over other typical learning algorithms and existing alignment systems. Such system is useful to develop accurate and efficient image alignment systems.

Real-time image segmentation for visual inspection of pharmaceutical tablets

Visual appearance is an important quality factor of pharmaceutical tablets. Due to the vast quantities of produced tablets and high-quality requirements, pharmaceutical companies are interested in employing automated systems for real-time visual tablet inspection with the speeds of up to 100 tablets per second. Such systems require reliable tablet manipulation, illumination, image acquisition, tablet image analysis, classification, and sorting system. Tablet image segmentation, in which each tablet image is partitioned into the tablet region and background, is the first and very important step in tablet image analysis. It this paper, we propose a novel real-time segmentation method for grey-level images that is based on border tracking. The proposed method was designed to be accurate, robust, and computationally undemanding. The performances of the method were objectively assessed on a large number of simulated and real tablet images. The obtained results indicated high reliability, accuracy, and speed. The 100% reliability was obtained for segmentation of real images of pharmaceutical tablets, while the segmentation times were no more than 1.5 ms or 15% of the whole time available for tablet image analysis. As such, the proposed method proved feasible for real-time visual quality inspection of pharmaceutical tablets. Based on just a few assumptions that are usually fulfilled, the method may be a valuable segmentation tool for many other visual quality inspection applications.