Real-time Inspection Research Articles

Inspection Algorithm of Welding Bead Based on Image Projection

The shear reinforcement of dual-anchorage (SRD) is used to enhance the safety of reinforced concrete structures in construction sites. In SRD, welding is used to create shear reinforcement, and after production, a quality inspection of the welding bead is required. Since the welding bead of SRD is inspected for quality by measuring both horizontal and vertical lengths, it is necessary to obtain this information for quality inspection. However, it is difficult to inspect the quality of welding beads using existing methods based on segmentation, due to the similarity in texture between the welding bead and the base material, as well as discoloration around the welded area after welding. In this paper, we propose an algorithm that detects the welding bead using an image projection algorithm for pixels and classifies the quality of the welding bead. This algorithm detects the position of welding beads using the brightness values of an image. The proposed algorithm reduces the amount of computation time by first specifying the region of interest and then performing the analysis. Results from experiments reveal that the algorithm accurately classifies welding beads into good or bad classes by obtaining all brightness values in the vertical and horizontal directions in the SRD image. Furthermore, comparison tests with conventional algorithms demonstrate that the classification accuracy of the proposed algorithm is the highest. The proposed algorithm will be helpful in the real-time welding bead inspection field where fast and accurate inspection is crucial.

Thermal inspection of photovoltaic modules with deep convolutional neural networks on edge devices in AUV

Solar photovoltaic (PV), a green energy harvesting system, is growing worldwide rapidly. It is a friendly environmental energy system. During operation, anomalies could appear in the PV modules, which reduce energy harvesting efficiency, shorten the lifetime and lead to an increase in the amount of heavy metals being released into the environment. In this research, a remote sensing method is proposed for the fast and efficient detection of anomalies in photovoltaic (PV) systems. An infrared radiation (IR) camera mounted on flying vehicles (e.g., drone) to capture IR images of solar panels. Then, convolutional neural networks (CNN) are developed to detect abnormal cells in the PV systems. The CNN model are then quantized and implemented on edge devices for real-time detection of anomalies. We evaluated these models on 11 types of anomalies in PV modules collected from 826 solar panels worldwide. Moreover, the classification decision of each layer of the CNN model can be visualized by the Gradient-weighted Class Activation Mapping method to reinforce the classification output. In addition, we implemented the models on an edge device (Jetson Nano) to evaluate real-time inspection capabilities. The proposed model is efficient compared to the other previous models, which takes only 1.04 ms to progress one image on average while still yields a high accuracy of 85.35% when running on the Jetson Nano board.

Development and Assessment of a Field-Programmable Gate Array (FPGA)-Based Image Processing (FIP) System for Agricultural Field Monitoring Applications

Field imagery is an effective way to capture the state of the entire field; yet, current field inspection approaches, when accounting for image resolution and processing speed, using existent imaging systems, do not always enable real-time field inspection. This project involves the innovation of novel technologies by using an FPGA-based image processing (FIP) device that eliminates the technical limitations of the current agricultural imaging services available in the market and will lead to the development of a market-ready service solution. The FIP prototype developed in this study was tested in both a laboratory and outdoor environment by using a digital single-lens reflex (DSLR) camera and web camera, respectively, as the reference system. The FIP system had a high accuracy with a Lin’s concordance correlation coefficient of 0.99 and 0.91 for the DLSR and web camera reference system, respectively. The proposed technology has the potential to provide on-the-spot decisions, which in turn, will improve the compatibility and sustainability of different land-based systems.

Flow chamber staining modality for real-time inspection of dynamic phenotypes in multiple histological stains.

Traditional histological stains, such as hematoxylin-eosin (HE), special stains, and immunofluorescence (IF), have defined myriads of cellular phenotypes and tissue structures in a separate stained section. However, the precise connection of information conveyed by the various stains in the same section, which may be important for diagnosis, is absent. Here, we present a new staining modality-Flow chamber stain, which complies with the current staining workflow but possesses newly additional features non-seen in conventional stains, allowing for (1) quickly switching staining modes between destain and restain for multiplex staining in one single section from routinely histological preparation, (2) real-time inspecting and digitally capturing each specific stained phenotype, and (3) efficiently synthesizing graphs containing the tissue multiple-stained components at site-specific regions. Comparisons of its stains with those by the conventional staining fashions using the microscopic images of mouse tissues (lung, heart, liver, kidney, esophagus, and brain), involving stains of HE, Periodic acid-Schiff, Sirius red, and IF for Human IgG, and mouse CD45, hemoglobin, and CD31, showed no major discordance. Repetitive experiments testing on targeted areas of stained sections confirmed the method is reliable with accuracy and high reproducibility. Using the technique, the targets of IF were easily localized and seen structurally in HE- or special-stained sections, and the unknown or suspected components or structures in HE-stained sections were further determined in histological special stains or IF. By the technique, staining processing was videoed and made a backup for off-site pathologists, which facilitates tele-consultation or -education in current digital pathology. Mistakes, which might occur during the staining process, can be immediately found and amended accordingly. With the technique, a single section can provide much more information than the traditional stained counterpart. The staining mode bears great potential to become a common supplementary tool for traditional histopathology.

Real-time rubber quality model based on CNN-LSTM deep learning theory

Predicting the quality of a rubber compound is a necessity for intelligent mixing. However, the conventional quality prediction based on the power curve has low precision and significant dispersion, making it challenging to satisfy the need for real-time rubber mixing quality inspection. In this research, the vibration signals were collected for the first time as an input feature of the mixing quality prediction model, with carbon black dispersion is utilized as a quality index. On the basis of the theory of deep learning, the online quality prediction model of mixing was constructed using a variety of featured extraction methods and neural network structures, and the models were compared and tested. After 5 experiments, the average root mean square error (RMSE) of convolutional neural network-long short-term memory network (CNN-LSTM) model is 0.1160, which is 21.46% higher than that of LSTM model after time-domain featured extraction, demonstrating the efficacy and superiority of CNN-LSTM end-to-end model. This study is essential for the progression and breakthrough of the real-time mixing quality optimization issue.

Automated coplanarity inspection of BGA solder balls by structured light

The coplanarity of BGA (Ball Grid Array) solder balls is critical for reliable connectivity in semi-conductor units. However, existing ball height inspection techniques require high-cost equipment and skilled operators, which are not feasible for using in real-time manufacturing process. In this paper, a fast measurement method of coplanarity inspection of solder balls is developed based on structured light technique. The hardware of the system includes a camera and a projector. The digital-light-processing (DLP) projector emits programmed image patterns and the digital complementary-metal-oxide-semiconductor (CMOS) camera captures distorted image patterns on solder balls. The image processing software is developed to compare the original and distorted image patterns to reconstruct the three-dimensional (3D) solder balls. The method has been calibrated extensively and exhibits accuracy within 5 μm mean squared error compared to ground truth values determined by X-ray computed tomography. The proposed method achieves reliable, in-line ball height measurement and could be potentially used for real-time in-line coplanarity inspection of BGA chip during manufacturing.

A survey of real-time surface defect inspection methods based on deep learning

A survey of real-time surface defect inspection methods based on deep learning

Nut Geometry Inspection Using Improved Hough Line and Circle Methods.

Nuts are the cornerstone of human industrial construction, especially A-grade nuts that can only be used in power plants, precision instruments, aircraft, and rockets. However, the traditional nuts inspection method is to manually operate the measuring instrument for conducting an inspection, so the quality of the A-grade nut cannot be guaranteed. In this work, a machine vision-based inspection system was proposed, which performs a real-time geometric inspection of the nuts before and after tapping on the production line. In order to automatically screen out A-Grade nuts on the production line, there are 7 inspections within this proposed nut inspection system. The measurements of parallel, opposite side length, straightness, radius, roundness, concentricity, and eccentricity were proposed. To shorten the overall detection time regarding nut production, the program needed to be accurate and uncomplicated. By modifying the Hough line and Hough circle, the algorithm became faster and more suitable for nut detection. The optimized Hough line and Hough circle can be used for all measures in the testing process.

In-process phased array ultrasonic weld pool monitoring

In recent years, there have been increasing economic and industrial drivers for the development of real-time non-destructive evaluation directly at the point of manufacture. Real-time inspection and monitoring of welding processes can help to reduce fabrication costs by detecting defects as they occur, enabling more efficient and cost-effective builds. This paper shows, for the first time, the use of phased array ultrasonics to monitor and analyse the molten weld pool during deposition of multi-pass gas tungsten arc welds. The received ultrasonic signals are shown to contain information related to key physical transitions occurring within the welding process, namely the melting and solidification of the weldment. Furthermore, the technique used here is shown to be effective for determining weld quality in real-time with significant signal changes occurring when defects such as Lack of Root Penetration are present. The accurate focusing and steering capabilities offered by phased arrays are used to successfully isolate the molten weld pool from the surrounding solidified weldment during deposition of multiple layers of a multi-pass weld.

HAIS: Highways Automated-Inspection System

A smart city is a trending concept describing a new generation of cities operated intelligently with minimal human intervention. It promotes energy sustainability, minimal environmental impact, and better governance. In transportation, the remote highway infrastructure monitoring will enhance the driver’s safety, continuously report road conditions, and identify potential hazardous incidents such as accidents, floods, or snow storms. In addition, it facilitates the integration of future cuttingedge technologies such as self-driving vehicles. This paper presents a general introduction to a smart monitoring system for automated real-time road condition inspection. The proposed solution includes hardware devices/nodes and software applications for data processing, road condition inspection using hybrid algorithms based on digital signal processing, and artificial intelligence technologies. The proposed system has an interactive web interface for real-time data sharing, infrastructure monitoring, visualization, and management of inspection reports which can improve the maintenance process.

A Novel Real-Time Autonomous Crack Inspection System Based on Unmanned Aerial Vehicles.

Traditional methods on crack inspection for large infrastructures require a number of structural health inspection devices and instruments. They usually use the signal changes caused by physical deformations from cracks to detect the cracks, which is time-consuming and cost-ineffective. In this work, we propose a novel real-time crack inspection system based on unmanned aerial vehicles for real-world applications. The proposed system successfully detects and classifies various types of cracks. It can accurately find the crack positions in the world coordinate system. Our detector is based on an improved YOLOv4 with an attention module, which produces 90.02% mean average precision (mAP) and outperforms the YOLOv4-original by 5.23% in terms of mAP. The proposed system is low-cost and lightweight. Moreover, it is not restricted by navigation trajectories. The experimental results demonstrate the robustness and effectiveness of our system in real-world crack inspection tasks.

A Monte Carlo simulation for Moving Light Source in Intracavity PDT.

We developed a simulation method for modeling the light fluence delivery in intracavity Photodynamic Therapy (icav-PDT) for pleural lung cancer using a moving light source. Due to the large surface area of the pleural lung cavity, the light source needs to be moved to deliver a uniform dose around the entire cavity. While multiple fixed detectors are used for dosimetry at a few locations, an accurate simulation of light fluence and fluence rate is still needed for the rest of the cavity. We extended an existing Monte Carlo (MC) based light propagation solver to support moving light sources by densely sampling the continuous light source trajectory and assigning the proper number of photon packages launched along the way. The performance of Simphotek GPU CUDA-based implementation of the method - PEDSy-MC - has been demonstrated on a life-size lung-shaped phantom, custom printed for testing icav-PDT navigation system at the Perlman School of Medicine (PSM) - calculations completed under a minute (for some cases) and within minutes have been achieved. We demonstrate results within a 5% error of the analytic solution for multiple detectors in the phantom. PEDSy-MC is accompanied by a dose-cavity visualization tool that allows real-time inspection of dose values of the treated cavity in 2D and 3D, which will be expanded to ongoing clinical trials at PSM. PSM has developed a technology to measure 8-detectors in a pleural cavity phantom using Photofrin-mediated PDT that has been used during validation.

PDT Light Fluence Phantom Modeling of the Human Pleural Cavity: A Proof-of-Concept Pre-Clinical Study.

We have developed a novel scanning protocol for a life-sized human phantom model using handheld three-dimensional (3D) surface acquisition devices. This technology will be utilized to develop light fluence modeling of the internal pleural cavity space during Photodynamic Therapy (PDT) of malignant mesothelioma. The external aspect of the chest cavity phantom was prefabricated of a hardened synthetic polymer resembling ordinary human anatomy (pleural cavity space) and the internal aspect remained hollow without any characterizations. Both surfaces were layered with non-reflective adhesive paper to create non-uniformed surface topographies. These surface characteristics were established in randomized X-Y-Z coordinates ranging in dimensions from 1-15mm. This protocol utilized the handheld Occipital Scanner and the MEDIT i700. The Occipital device required a minimum scanner-to-surface distance of 24cm and the MEDIT device 1cm respectively. The external and internal aspects of the phantom model were successfully scanned acquiring digital measurements in actual value and converted into a digital image file. The initial surface rendering was acquired by the Occipital device and applied with proprietary software to guide the MEDIT device to fill voided areas. This protocol is accompanied by a visualization tool that allows for real-time inspection of surface acquisition in 2D and 3D. This scanning protocol can be utilized to scan the pleural cavity for real-time guidance for light fluence modeling during PDT, which will be expanded to ongoing clinical trials.

An Algorithm for Real-Time Aluminum Profile Surface Defects Detection Based on Lightweight Network Structure

Surface defects, which often occur during the production of aluminum profiles, can directly affect the quality of aluminum profiles, and should be monitored in real time. This paper proposes an effective, lightweight detection method for aluminum profiles to realize real-time surface defect detection with ensured detection accuracy. Based on the YOLOv5s framework, a lightweight network model is designed by adding the attention mechanism and depth-separable convolution for the detection of aluminum. The lightweight network model improves the limitations of the YOLOv5s framework regarding to its detection accuracy and detection speed. The backbone network GCANet is built based on the Ghost module, in which the Attention mechanism module is embedded in the AC3Ghost module. A compression of the backbone network is achieved, and more channel information is focused on. The model size is further reduced by compressing the Neck network using a deep separable convolution. The experimental results show that, compared to YOLOv5s, the proposed method improves the mAP by 1.76%, reduces the model size by 52.08%, and increases the detection speed by a factor of two. Furthermore, the detection speed can reach 17.4 FPS on Nvidia Jeston Nano’s edge test, which achieves real-time detection. It also provides the possibility of embedding devices for real-time industrial inspection.

A deep context learning based PCB defect detection model with anomalous trend alarming system

The quality of a printed circuit board (PCB) is paramount towards ensuring proper functionality of electronic products. To achieve the required quality standards, substantial research and development efforts were invested to automate PCB inspection for defect detection, primarily using computer vision techniques. Despite these advancements, the accuracy of such techniques is often susceptible towards varying board and component size. Efforts to increase its accuracy especially for small or tiny defects on a PCB often lead to a tradeoff with reduced real-time performance, which in turn limits its applicability in the manufacturing industry. Hence, this paper puts forward an enhanced deep learning network which addresses the difficulty in inferring tiny or varying defects on a PCB in real-time. Our proposed enhancements consist of i) A novel multi-scale feature pyramid network to enhance tiny defect detection through context information inclusion; and ii) A refined complete intersection over union loss function to precisely encapsulate tiny defects. Experimental results on a publicly available PCB defects dataset demonstrate that our model achieves 99.17% mean-average precision, while maintaining real-time inferencing speed at 90 frames per second. In addition, we introduce three trend detection algorithms which alert an operator when abnormal development of defect characteristics is detected. Each algorithm is responsible for localizing defect buildups, increasing defect size and increasing defect occurrences, respectively. As a whole, the proposed model is capable of performing accurate and reliable real-time PCB inspection with the aid of an automated alert capability. The dataset and trained models are available at: https://github.com/JiaLim98/YOLO-PCB.

Automatic Detection and Imaging of Rivet Hole Defects for Aircraft Structures With Optimized Sensor Array Using Eddy Current Method and Image Analysis

Fatigue cracking in riveted lap joints on aircraft could cause severe safety consequences in flight. The prior inspection for the integrity of the second layer of aircraft structure with rivet removed is necessary due to the priority of safety. Methods such as radiographic and ultrasonic testing are widely applied but require relatively expensive equipment, mechanical scanning or professional operators. Eddy current method can be adopted for non-destructive testing (NDT) inspection, which is safe and low-cost. In this paper, a hand-held eddy current (EC) sensor array was optimised based on sensitivity map analysis. The EC sensor array was used with a multi-channel EM instrument, which was capable of inspecting defects around the fastener locations covered by the head of rivets. A simulated sample of riveted joint with defects was examined with the eddy current sensing system and the collected data was processed by Hough Circle Transformation (HCT) and texture analysis. The analysis results could detect and locate fatigue cracks on the sample. The presented eddy current sensing system in this paper achieved an automatic, real-time, portable, convenient and accurate inspection method for defects in riveted joints.

Study on IoT and Cloud-based Smart Living Room Ventilation System with Real-Time Remote Control and Inspection

Currently, fire zone smoke control systems are installed with other systems, such as indoor air control, lighting, and communication, in large scale buildings, such as exhibition halls, marts, and multiplex malls, making it difficult to access and identify the required system. Additionally, most fire zone smoke control equipment are installed in a high position and most of the places are covered the ceiling with finishing material after installation, so inspection and maintenance are difficult and dangerous. By developing a smart fire zone smoke control system, an authorized person can remotely inspect and control it via the internet. Additionally, the real-time self-testing function enables constant inspection and confirmation. Furthermore, the smart living room ventilation system is installed in a duct dedicated to smoke control or combined with air conditioning and ventilation to ensure reliable smoke control performance. By remotely controlling the smart living room ventilation system according to the fire site, it is possible to delay the smoke descent time to the minimum and secure a safe evacuation route.

Risk-based asset life management for steam turbines and generators

ABSTRACT To ensure a good return on investment, it is important for power plant operators to set up an asset management program that will ensure reliable operation over the commercial life of the plant. General Electric (GE) has been applying a variety of asset management technologies to improve power plant life and performance for our customers. These methods consider various inputs including equipment design knowledge; asset condition assessed from real-time monitoring and inspection data; and lessons learned from repairing, operating and maintaining thousands of assets globally. In this paper, a subset of that information is applied to develop a methodology to perform risk-based health and life assessments for steam turbines and generators. The method has been used on several steam power plants around the world, to help operators of ageing steam plants make the right decisions related to end-of-life management or life extension.

An augmented reality maintenance assistant with real-time quality inspection on handheld mobile devices

Advances of industry 4.0 are enabling augmented reality (AR) devices to be deployed across the manufacturing sector to enhance worker engagement and performance. This evolution of AR in the workspace is driving changes in how workers on the factory floor interact with the enterprise. AR is evolving to provide work instructions for assembly and maintenance. Recent demonstrations show the capability to allow for quality inspection to be integrally evaluated as part of the AR workflow. Previous integrations of quality include the ability for manual assessment to be logged into the enterprise as part of completing work instructions; however, this is largely based on the perception of the user to determine if they accurately performed the AR instructions. This study is designed to demonstrate the ability of an enterprise-level AR maintenance experience to go beyond manual inspection by embedding real-time quality assessment into the work instructions. Essentially, the goal is to enable the enterprise to perform a self-assessment of the quality of the work performed by maintenance personnel. Furthermore, the AR instructions enable additional instruction to empower maintenance personnel to correct their work prior to moving on to other tasks. The novelty of this work is focused on deployment on a mobile device with integrated vision inspection using the same device that is being used to deploy the AR work instructions. This work enumerates several essential qualities of mobile quality inspection tools and outlines some of the challenges associated with the development of such a system. Finally, results from testing the system demonstrate that handheld mobile devices can be used to capture inspection images while simultaneously deploying AR work instructions. Analysis of the image correction, timing for vision process of the quality check, and overall performance of the integrated system are presented.

Optimization of Computational Resources for Real-Time Product Quality Assessment Using Deep Learning and Multiple High Frame Rate Camera Sensors

Human eyes generally perform product defect inspection in Indonesian industrial production lines; resulting in low efficiency and a high margin of error due to eye tiredness. Automated quality assessment systems for mass production can utilize deep learning connected to cameras for more efficient defect detection. However, employing deep learning on multiple high frame rate cameras (HFRC) causes the need for much computation and decreases deep learning performance, especially in the real-time inspection of moving objects. This paper proposes optimizing computational resources for real-time product quality assessment on moving cylindrical shell objects using deep learning with multiple HFRC Sensors. Two application frameworks embedded with several deep learning models were compared and tested to produce robust and powerful applications to assess the quality of production results on rotating objects. Based on the experiment results using three HFRC Sensors, a web-based application with tensorflow.js framework outperformed desktop applications in computation. Moreover, MobileNet v1 delivers the highest performance compared to other models. This result reveals an opportunity for a web-based application as a lightweight framework for quality assessment using multiple HFRC and deep learning.