Industrial Inspection Tasks Research Articles

Design and Analysis of an Adaptive Pipeline Detection and Correction Mechanism

This paper presents a novel adaptive pipeline probe detection and correction mechanism designed to address the challenge of detection interference caused by the movement of wall-climbing robots, particularly in complex environments such as water-cooled walls. The mechanism ensures that the detection probe can accurately detect individual pipelines even when the robot deviates from its intended path. To achieve this, the system incorporates a self-adaptive deviation correction mechanism that maintains consistent detection performance without requiring adjustments to the robot's spatial position. The design includes a variable stiffness analysis of the buffer spring within the correction mechanism, which is optimized to minimize the impact of the robot's movement on the detection components. By carefully selecting the spring's size and stiffness parameters, the mechanism reduces vibration and enhances the stability and reliability of pipeline detection under offset conditions. In addition to maintaining detection accuracy, the system also supports automatic marking of pipelines that exhibit quality issues, ensuring that any detected defects are easily traceable. This adaptive mechanism not only improves detection efficiency but also enhances the overall operational stability of wall-climbing robots in industrial inspection tasks. The results demonstrate the mechanism's effectiveness in mitigating the challenges posed by uneven friction and time delays in the control system, making it a significant contribution to the field of robotic inspection systems.

Multi-position industrial defect inspection using self-training siamese networks with mix strategies

Structural defects account for a large proportion of defects, and acquiring large batches of high-quality labels is labor-intensive and time-consuming for industrial visual defect inspection tasks. This paper addresses the above problem by exploiting sufficient unlabeled samples, and aims to achieve superior model performance with some labeled data by using self-training methods that incorporate positional information. Specifically, this paper proposes a novel self-training architecture, MixSiam, which uses a Multi-Position-based Mix strategy (MPMix) and Siamese network structure for defect classification. Furthermore, considering the prediction noise problem in unlabeled data during training, we propose a progressive MPMix (MPMix+) strategy to reduce the negative impacts of noise on model training. Finally, we validate the effectiveness of our architecture on industrial datasets. For example, our method achieves 71.40% and 87.01% accuracy on the SMT (Surface Mounting Technology) dataset and MBH (Motor Brush Holder) dataset with only 100 labeled samples, which are 2.40% and 5.86% higher than the state-of-the-art FixMatch method, respectively. Compared with the supervised algorithm with 3,600 labels, our method achieves comparable accuracy on the SMT and MBH datasets, respectively, while saving 2/3 the amount of labeled data. In conclusion, MixSiam effectively utilizes unlabeled industrial data and improves model accuracy with fewer labeled samples, thus reducing the burden of data annotation in industrial production.

Focus Estimation Methods for Use in Industrial SFF Imaging Systems

A Shape-From-Focus (SFF) is a three-dimensional imaging method based on focus information. It is not yet widely used for in-line industrial inspection or measurement tasks. The main reasons are the time it takes to capture a 3D image of the inspected product and the presence of interference affecting image quality. This paper compares operators for estimating focus in source images in the scope of their use in constructing an industrial 3D scanner. Interferences were introduced by using additional illuminators and changing the acquisition parameters. The use of industrial-grade cameras, industrial-grade illuminators, and electrically controlled optics are discussed. A novel approach in the research is using an electrically tunable lens to move the position of the image plane during 3D image acquisition. The research was conducted on various surfaces found on typical industrial products. The research showed which focus estimation operators can be applied to SFF imaging within the range of interference considered. It was also confirmed that using the centre of gravity method for scene reconstruction allows for an increase in resolution compared to the maximum method.

A novel AR recoloring technique to enhance operator performance on inspection tasks in Industry 4.0 environments.

Over the past few years, the manufacturing industry has increasingly embraced Augmented Reality (AR) for inspecting real products, yet faces challenges in visualization modalities. In fact, AR content presentation significantly impacts user performance, especially when virtual object colors lack real-world context. Additionally, the lack of studies in this area compounds uncertainty about visualization effects on user performance in inspection tasks. This study introduces a novel AR recoloring technique to enhance user performance during industrial assembly inspection tasks. This technique automatically recolors virtual components based on their physical counterparts, improving distinctiveness. Experimental comparisons with AR experts and representative users, using objective and subjective metrics, demonstrate the proposed AR recoloring technique enhances task performance and reduces mental burden during inspection activities. This innovative approach outperforms established methods like CAD and random modes, showcasing its potential for advancing AR applications in manufacturing, particularly in the inspection of products.

A comparative study of Augmented Reality rendering techniques for industrial assembly inspection

In the manufacturing industry, Augmented Reality (AR) has shown significant potential in enhancing operators’ capabilities while performing inspection and assembly activities. However, the augmented visualization of virtual models on physical components can present challenges and potential misunderstandings, as the visualization mode greatly influences the perception of components and the amount of information received. This study investigates the impact of rendering techniques on user performance during industrial assembly inspection tasks. In particular, a set of rendering techniques, suitable for industrial AR applications, have been selected and implemented through a dedicated AR tool. The selected AR rendering techniques have been compared, in terms of qualitative and quantitative metrics, in order to test their efficiency and effectiveness for industrial assembly inspection activities. 17 domain experts and 33 representative users have been involved in the experimental study which outcomes reveal that the rendering techniques play a significant role in assisting operators for identifying design discrepancies in industrial products. Furthermore, a correlation has been observed between the two AR rendering techniques best suited to support industrial inspection activities and the types of assembly errors detected.

Lean back or lean in? Exploring social loafing in human–robot teams

Introduction: Thanks to technological advances, robots are now being used for a wide range of tasks in the workplace. They are often introduced as team partners to assist workers. This teaming is typically associated with positive effects on work performance and outcomes. However, little is known about whether typical performance-reducing effects that occur in human teams also occur in human–robot teams. For example, it is not clear whether social loafing, defined as reduced individual effort on a task performed in a team compared to a task performed alone, can also occur in human–robot teams.Methods: We investigated this question in an experimental study in which participants worked on an industrial defect inspection task that required them to search for manufacturing defects on circuit boards. One group of participants worked on the task alone, while the other group worked with a robot team partner, receiving boards that had already been inspected by the robot. The robot was quite reliable and marked defects on the boards before handing them over to the human. However, it missed 5 defects. The dependent behavioural measures of interest were effort, operationalised as inspection time and area inspected on the board, and defect detection performance. In addition, subjects rated their subjective effort, performance, and perceived responsibility for the task.Results: Participants in both groups inspected almost the entire board surface, took their time searching, and rated their subjective effort as high. However, participants working in a team with the robot found on average 3.3 defects. People working alone found significantly more defects on these 5 occasions–an average of 4.2.Discussion: This suggests that participants may have searched the boards less attentively when working with a robot team partner. The participants in our study seemed to have maintained the motor effort to search the boards, but it appears that the search was carried out with less mental effort and less attention to the information being sampled. Changes in mental effort are much harder to measure, but need to be minimised to ensure good performance.

Sparse random neural networks for online anomaly detection on sensor nodes

Whether it is used for predictive maintenance, intrusion detection or surveillance, on-device anomaly detection is a very valuable functionality in sensor and Internet-of-things (IoT) systems. In this paper, we introduce a novel anomaly detection technique based on sparse, random neural networks. The sparsity in the model allows for a very efficient implementation on embedded or resource constrained hardware. Our approach supports continuous online learning where the model is deployed to the sensor device without any prior training. As new data becomes available, the model is updated and becomes better at detecting anomalies. We experimentally validate our approach on several default benchmark data sets in the visual domain as well as on industrial quality inspection and predictive maintenance tasks. We show that our approach achieves a very favorable trade-off between computational cost and anomaly detection accuracy.

DHT: Dynamic Vision Transformer Using Hybrid Window Attention for Industrial Defect Images Classification

Industrial defect detection is gaining importance in the control of industrial product quality. Highly accurate and efficient defect detection with complex and variable industrial defect types is therefore an interesting but challenging problem. Vision transformers have been highly successful in a variety of computer vision tasks, due to their ability to capture global information in images. Nevertheless, simply capturing global information is problematic. On the one hand, because they are incapable of inductive bias as Convolutional Neural Network (CNN), transformers will have difficulty focusing on local features of defects in industrial defect image inspection tasks. On the other hand, using global computation leads to excessive memory and computational cost. To mitigate these issues, we propose a new vision transformer architecture which contains Hybrid Window Attention (HWA) and Dynamic Token Normalization (DTN). HWA, which combines pooling attention and window attention, makes the computational complexity reduced to improve efficiency. DTN enables transformers to focus on both the global information and the local features of defects, thus providing improved accuracy of industrial surface defect detection. Extensive experiments demonstrate that our Dynamic Vision Transformer (DHT) achieves 96.8% and 98.5% classification accuracy on the NEU dataset and the DAGM dataset, respectively, with a low computational complexity.

A fully actuated aerial manipulator system for industrial contact inspection applications

PurposeThis paper aims to present a fully actuated aerial manipulator (AM) with a robust motion/force hybrid controller for conducting contact-typed inspection tasks in industrial plants.Design/methodology/approachAn AM is designed based on a hexarotor with tilted rotors and a rigidly attached end effector. By tilting the rotors, the position and attitude of the AM can be controlled independently, and the AM can actively exert forces on industrial facilities through the rigidly attached end effector. A motion/force hybrid controller is proposed to perform contact-typed inspection tasks. The contact-typed inspection task is divided into the approach phase and the contact phase. In the approach phase, the AM automatically approaches the contact surface. In the contact phase, a motion/force hybrid controller is used for contact-typed inspection. Finally, a disturbance observer (DOB) is used to estimate external disturbances and used as feedforward compensation.FindingsThe proposed AM can slowly approach the contact surface without significant impact in the contact phase. It can realize constant force control in the direction normal to the contact surface in the contact phase, whereas the motion of the remaining directions can be controlled by the operator. The use of the DOB ensures the robustness of the AM in the presence of external wind disturbances.Originality/valueA fully actuated AM system with a robust motion/force hybrid controller is proposed. The effectiveness of the proposed AM system for conducting contact-typed industrial inspection tasks is validated by practical experiments.

An effective industrial defect classification method under the few-shot setting via two-stream training

Visual surface defect inspection provides an important tool for product quality assessment in a wide range of industrial applications. In recent years, diverse convolutional neural network (CNN) models have been developed for the high-accuracy image classification of natural objects. However, it remains a challenging task to develop deep-learning-based approaches for surface defect inspection of industrial products. The main challenge is that surface defects of industrial products occur in low probability, therefore it is impractical to build a large image dataset containing different types of defects to train/fine-tune deep CNN models to achieve satisfactory performance when applied to industrial inspection tasks. To overcome the above-mentioned problem, we build a light-weight defect classification model based on the pre-trained SqueezeNet architecture and present three effective techniques to achieve high-accuracy defect classification using very few collected defective training samples. First, we experimentally demonstrate that it is reasonable to freeze shallower convolutional layers and only fine-tune deeper layers in the last convolutional stage when there are not enough training samples. Second, we integrate batch-size independent Group Normalization (GN) into SqueezeNet to compute stable statistics based on a limited number of defective samples for training the classification model. Third, we add an auxiliary task to perform the similarity comparison of two defective samples during the training phase, significantly increasing the size of the training data and generating more distinctive defect-specific representations. The experimental results on the NEU-CLS dataset and USB-FS dataset verify the effectiveness of the proposed method. Our method can obtain accurate defect classification results when very few defective training samples are provided, i.e., about 97.69% and 82.92% average classification accuracy on the NEU-CLS and USB-FS datasets, respectively, under the 5-shot setting (5 images per defect category).

Heterogeneous robots coordination for industrial plant inspection and evaluation at World Robot Summit 2020

This paper describes a plant inspection robot team consisting of three types of robots that won the World Robot Summit 2020 plant disaster prevention challenge. There is a social demand to use robots for industrial plant inspection tasks to decrease the workload and risks of human operators. In this study, the authors developed a plant inspection system using a tracked vehicle, a mecanum-wheeled vehicle, and an unmanned aerial vehicle (UAV). The roles between work-related tasks and visual inspection tasks are divided by two operators, resulting in efficient operation. Experimental results show that the tracked vehicle can conduct challenging tasks such as climbing up the stairs and large-valve operations. The results also showed that the visual inspection task was performed 58% faster by the mecanum vehicle in a narrow passage and 48% faster by the UAV on the stairs, as compared with the tracked vehicle in the same situations. The analysis of the competition results showed that we obtained points in 57% (160/280) of the work-related tasks and 85% (170/200) of the visual inspection tasks as well as 92% (110/120) of the emergency response tasks in the final round.

New at the short end of the spectrum

AbstractFor many industrial inspection tasks, wavelengths outside of visible light are more meaningful than the light spectrum perceivable by humans. New and powerful short‐wave infrared (SWIR) and ultraviolet (UV) industrial cameras as well as polarization cameras are now available for such applications.

Deep Learning for improving the efficiency of dimensional measurement workflows with high-resolution X-ray computed tomography

High-resolution X-ray computed tomography (CT) instruments, also known as three-dimensional (3D) X-ray microscopes, can be adapted for dimensional metrology applications such as geometric dimensioning and tolerancing of metallic components. However, CT scanning times can be prohibitively high for industrial measurement inspection tasks owing to the poor contrast from X-ray attenuation in Ferrous metals, especially if the measurement of spatial resolutions under 5 µm are required. This paper describes a software-defined approach to dramatically reducing total exposure time (or scanning time) while maintaining resolution loss within 2 micrometers as compared to the baseline scans acquired over 6 hours. Here, we combine two deep learning (DL) codes in our surface extraction workflow to compensate for lower signal-to-noise ratio in short exposure data (acquired with lower number of projections): (1) a surface determination (post-reconstruction) , and (2) a denoising algorithm (pre-reconstruction). Training data was acquired from a scan of an 8-hole automotive fuel injector (sample 1) with a 165 µm nominal diameter per hole. For testing the accuracy of the workflow, a separate scan of a 6-hole side-mount injector (sample 2) was acquired. For both samples, the acquired X-ray projections (or radiographs) were binned down to 10X such as to simulate faster scans. For training and testing workflows, the full exposure scans (baseline) were used as target and the shorter exposure scans as inputs to the deep learning models. To determine loss of surface accuracy from the baseline case, a metric is formulated (in micrometers) and the trends are reported for when the total measurement time was reduced by up to 10X (up to 0.6 hours, using only 360 projections). We report that scan times can be reduced by over 10X while retaining the limiting the resolution loss to under 1 micrometer.

Deep Learning for Unsupervised Anomaly Localization in Industrial Images: A Survey

Currently, deep learning-based visual inspection has been highly successful with the help of supervised learning methods. However, in real industrial scenarios, the scarcity of defect samples, the cost of annotation, and the lack of a priori knowledge of defects may render supervised-based methods ineffective. In recent years, unsupervised anomaly localization algorithms have become more widely used in industrial inspection tasks. This paper aims to help researchers in this field by comprehensively surveying recent achievements in unsupervised anomaly localization in industrial images using deep learning. The survey reviews more than 120 significant publications covering different aspects of anomaly localization, mainly covering various concepts, challenges, taxonomies, benchmark datasets, and quantitative performance comparisons of the methods reviewed. In reviewing the achievements to date, this paper provides detailed predictions and analysis of several future research directions. This review provides detailed technical information for researchers interested in industrial anomaly localization and who wish to apply it to the localization of anomalies in other fields.

Defect detection of injection molding products on small datasets using transfer learning

Appearance defect detection of products is a demanding procedure in the manufacturing process. Existing appearance defect inspection mainly relies on manual visual inspection, which is neither efficient nor accurate enough to ensure the manufacturing quality. Thus, automatic defect inspection has become an urgent demand. A critical problem hindering extensive applications of automatic defect inspection is that there are only limited defective samples to develop classification algorithms, leading to inadequate accuracy or robustness to meet industrial requirements. This paper proposed a knowledge reuse strategy to train convolutional neural network (CNN) models to improve defect inspection accuracy and robustness. By introducing model-based transfer learning and data augmentation, the knowledge from other vision tasks is transferred to industrial defect inspection tasks, resulting in high accuracy with limited training samples. Experimental results on an injection molding product showed that the detection accuracy was improved to about 99% when only 200 images per category were available. In comparison, conventional CNN models and the support vector machine method could achieve an average accuracy of only about 88.70% and 86.90%, respectively. The proposed method was also robust enough in detecting complicated defects which had many diversified appearances. The visualization method also proved that the performance improvement of the proposed method was because the model accurately extracted the discriminative features of the defective regions in the input images. The proposed method is meaningful for automatic defect detection in the manufacturing process.

Stable Autonomous Spiral Stair Climbing of Tracked Vehicles Using Wall Reaction Force

In this letter, an autonomous spiral stair climbing method for tracked vehicles using the reaction force from side walls has been proposed. Spiral stairs are one of the most difficult terrains for tracked vehicles because of their asymmetrical ground shape and small turning radius. Tracked vehicles are expected to be used in industrial plant inspection tasks, where robots should navigate on multiple floors by ascending the stairs. Spiral or curved stairs are installed as part of inspection passages for cylindrical facilities, such as boilers, chimneys, or large tanks. Previously, the authors have experimentally demonstrated that the wall-following motion is effective for stabilizing and accelerating spiral stair climbing. However, the complete automation of climbing motion or the analysis of why the same motion is generated even if a disturbance exists in the initial entry angle to the wall should be investigated. In this study, the authors developed an autonomous spiral stair climbing method using the wall reaction force and clarified the applicable limitations of this method using a geometrical model. Autonomous spiral stair climbing is realized by attaching passive wheels on its collision point and automating the motions of main tracks and sub-tracks. The geometrical model shows the expected trajectory of the robot on the spiral stairs, which suggests that the robot' s rotation radius converges to a specific value; this is experimentally confirmed by measuring the robot's motion. The wall-following motion of robots is equivalent to human inspectors grasping handrails while climbing stairs. Through collisions with surrounding objects, motion is stabilized and certainty is guaranteed.

Efficient neural network compression via transfer learning for machine vision inspection

Several practical difficulties arise when trying to apply deep learning to image-based industrial inspection tasks: training datasets are difficult to obtain, each image must be inspected in milliseconds, and defects must be detected with 99% or greater accuracy. In this paper we show how, for image-based industrial inspection tasks, transfer learning can be leveraged to address these challenges. Whereas transfer learning is known to work well only when the source and target domain images are similar, we show that using ImageNet—whose images differ significantly from our target industrial domain—as the source domain, and performing transfer learning, works remarkably well. For one benchmark problem involving 5,520 training images, the resulting transfer-learned network achieves 99.90% accuracy, compared to only a 70.87% accuracy achieved by the same network trained from scratch. Further analysis reveals that the transfer-learned network produces a considerably more sparse and disentangled representation compared to the trained-from-scratch network. The sparsity can be exploited to compress the transfer-learned network up to 1/128 the original number of convolution filters with only a 0.48% drop in accuracy, compared to a drop of nearly 5% when compressing a trained-from-scratch network. Our findings are validated by extensive systematic experiments and empirical analysis.

Deeper and Mixed Supervision for Salient Object Detection in Automated Surface Inspection

In recent years, researches in the field of salient object detection have been widely made in many industrial visual inspection tasks. Automated surface inspection (ASI) can be regarded as one of the most challenging tasks in computer vision because of its high cost of data acquisition, serious imbalance of test samples, and high real-time requirement. Inspired by the requirements of industrial ASI and the methods of salient object detection (SOD), a task mode of defect type classification plus defect area segmentation and a novel deeper and mixed supervision network (DMS) architecture is proposed. The backbone network ResNeXt-101 was pretrained on ImageNet. Firstly, we extract five multiscale feature maps from backbone and concatenate them layer by layer. In addition, to obtain the classification prediction and saliency maps in one stage, the image-level and pixel-level ground truth is trained in a same side output network. Supervision signal is imposed on each side layer to realize deeper and mixed training for the network. Furthermore, the DMS network is equipped with residual refinement mechanism to refine the saliency maps of input images. We evaluate the DMS network on 4 open access ASI datasets and compare it with other 20 methods, which indicates that mixed supervision can significantly improve the accuracy of saliency segmentation. Experiment results show that the proposed method can achieve the state-of-the-art performance.

Aerial Manipulator With Rolling Base for Inspection of Pipe Arrays

This paper considers the inspection by contact of long arrays of pipe structures in hard-to-reach places, typical of chemical plants or oil and gas industries, presenting the design of a hybrid rolling-aerial platform capable of landing and moving along the pipes without wasting energy in the propellers during the inspection. The presented robot overcomes the limitation in terms of operation time and positioning accuracy in the application of flying robots to industrial inspection and maintenance tasks. The robot consists of a hexa-rotor platform integrating a rolling base with velocity and direction control, and a 5-DOF (degree of freedom) robotic arm supported by a 1-DOF linear guide system that facilitates the deployment of the arm in the array of pipes to inspect their contour once the platform has landed. Given a set of points to be inspected in different arrays of pipes, the path of the multirotor and the rolling platform is planned with a hybrid RRT* (Rapidly-exploring Random Tree) based algorithm that minimizes the energy consumption. The performance of the system is evaluated in an illustrative outdoor scenario with two arrays of pipes, using a laser tracking system to measure the position of the robot from the ground control station.

Defect Detection and Classification by Training a Generic Convolutional Neural Network Encoder

Non-destructive Testing (NDT) often involves analysing images to identify (rare) defects. We propose a method for locating and classifying abnormalities using Convolutional Neural Networks (CNNs). A particular problem is that it is often difficult to get large numbers of examples of images of defects, making training a classifier challenging. To address this problem we generate large numbers of synthetic images by combining real defects with different backgrounds. These images are used to train a U-Net style network to perform defect detection at the pixel level. We also demonstrate that the encoder of the network produces features which can be applied to the defect classification task at the image level. Both the defect detection and classification modules are tested on multiple small data sets. Our results show that these modules are able to fulfil the industrial component inspection task at the pixel level (locating defect regions) and image level (identifying if an image contains a defect).