Inspection Environment Research Articles

Design and Implementation of UAVs for Bird’s Nest Inspection on Transmission Lines Based on Deep Learning

In recent years, unmanned aerial vehicles (UAV) have been increasingly used in power line inspections. Birds often nest on transmission line towers, which threatens safe power line operation. The existing research on bird’s nest inspection using UAVs mainly stays at the level of image postprocessing detection, which has poor real-time performance and cannot obtain timely bird’s nest detection results. Considering the above shortcomings, we designed a power inspection UAV system based on deep learning technology for autonomous flight, positioning and photography, real-time bird nest detection, and result export. In this research, 2000 bird’s nest images in the actual power inspection environment were shot and collected to create the dataset. The parameter optimization and test comparison for bird’s nest detection are based on the three target detection models of YOLOv3, YOLOv5-s, and YOLOX-s. A YOLOv5-s bird’s nest detection model optimized for bird’s nest real-time detection is proposed, and it is deployed to the onboard computer for real-time detection and verification during flight. The DJI M300 RTK UAV was used to conduct a test flight in a natural power inspection environment. The test results show that the mAP of the UAV system designed in this paper for bird’s nest detection is 92.1%, and the real-time detection frame rate is 33.9 FPS. Compared with the previous research results, this paper proposes a new practice of using drones for bird’s nest detection, dramatically improving the real-time accuracy of bird’s nest detection. The UAV system can efficiently complete the task of bird’s nest detection in the process of electric power inspection, which can significantly reduce manpower consumption in the power inspection process.

Dual-View Stereovision-Guided Automatic Inspection System for Overhead Transmission Line Corridor

Overhead transmission line corridor detection is important to ensure the safety of power facilities. Frequent and uncertain changes in the transmission line corridor environment requires an efficient and autonomous UAV inspection system, whereas the existing UAV-based inspection systems has some shortcomings in control model and ground clearance measurement. For one thing, the existing manual control model has the risk of striking power lines because it is difficult for manipulators to judge the distance between the UAV fuselage and power lines accurately. For another, the ground clearance methods based on UAV usually depend on LiDAR (Light Detection and Ranging) or single-view visual repeat scanning, with which it is difficult to balance efficiency and accuracy. Aiming at addressing the challenging issues above, a novel UAV inspection system is developed, which can sense 3D information of transmission line corridor by the cooperation of the dual-view stereovision module and an advanced embedded NVIDIA platform. In addition, a series of advanced algorithms are embedded in the system to realize autonomous control of UAVs and ground clearance measurement. Firstly, an edge-assisted power line detection method is proposed to locate the power line accurately. Then, 3D reconstruction of the power line is achieved based on binocular vision, and the target flight points are generated in the world coordinate system one-by-one to guide the UAVs movement along power lines autonomously. In order to correctly detect whether the ground clearances are in the range of safety, we propose an aerial image classification based on a light-weight semantic segmentation network to provide auxiliary information categories of ground objects. Then, the 3D points of ground objects are reconstructed according to the matching points set obtained by an efficient feature matching method, and concatenated with 3D points of power lines. Finally, the ground clearance can be measured and detected according to the generated 3D points of the transmission line corridor. Tests on both corresponding datasets and practical 220-kV transmission line corridors are conducted. The experimental results of different modules reveal that our proposed system can be applied in practical inspection environments and has good performance.

Infrared nondestructive detection of defective visible-reflective solder joints in solenoid coil connectors

Nondestructive infrared detection of solder defects in solenoid coil connectors was investigated. A detailed experimental setup and a meticulous and repeatable experimental procedure were developed. The experimental methods were firmly based on the application of heat transfer science. Time averaged excess temperature ([Formula: see text]) was the sole discrimination parameter, and a difference larger than 2°C between good and defective solders was conservatively considered for good discrimination. Analysis of variance determined the minimum differences in [Formula: see text] between good and defective solder joints. The effects in discriminability of heating time ( th), time integration interval length ( ti), region of interest (ROI) geometry, and solder radiative properties were investigated. A rectangular ROI tightly enclosing the good solder was found as the best discrimination ROI. Matte black paint increasing radiative emission of solder joints significantly reduced discriminability. The defects were discriminated from the good solder for a rectangular ROI, th = 78 s, and ti = 20 s. Moreover, an excellent total inspection time of 34 s per solenoid coil is obtained by considering a well-controlled inspection environment during manufacturing, severe defects detection, and a possible 1°C difference for discrimination. Several aspects of the novel inspection procedure developed can greatly benefit the thermal imaging scientific community.

Research on Small Sample Data-Driven Inspection Technology of UAV for Transmission Line Insulator Defect Detection

Insulators are important safety devices on high-voltage transmission lines. An insulator inspection system based on UAVs is widely used. Insulator defect detection is performed against two main engineering problems: 1. The scarcity of defect images, which leads to a network overfitting problem. 2. The small object detection, which is caused by the long aerial photography distance, and the low resolution of the insulator defect area pictures. In this study, firstly, the super-resolution reconstruction method is used to augment the dataset, which can not only solve the overfitting problem but also enrich the image texture features and pixel values of defect areas. Secondly, in the process of insulator defect detection, a two-stage cascading method is used. In the first stage, the rotated object detection algorithm is used to realize the object location of insulator strings, and then images of the identified insulators are cropped to reduce the proportion of the background area in defect images. In the second stage, YOLO v5 is used for the detection of insulator caps that are missing defects. The method proposed shows good detection effect on the self-built training set which contains only 85 images captured from real inspection environments. The method has practical industrial application value.

UAV system with trinocular vision for external obstacle detection of transmission lines.

External obstacle detection is a significant task in transmission line inspection and is related to the safe operation of the power transmission grid. In recent years, unmanned aerial vehicles (UAVs) equipped with different devices have been widely used for transmission line inspection. However, because of the complex environment of transmission lines and weak power line textures in the obtained images, most existing methods and systems cannot meet the requirements for real-time and high-accuracy external obstacle detection of transmission lines. In this paper, a novel, to the best of our knowledge, UAV system integrated trinocular vision technology with remote sensing is developed to achieve better external obstacle detection of transmission lines in real time, which is composed of a DJ-Innovations (DJI) UAV equipped with a global positioning system (GPS), angle sensors, trinocular vision including three visible cameras with the same parameters, and a small processor with a pre-implanted software algorithm. In this paper, a new method for external obstacle detection of transmission lines is proposed to satisfy the requirements for real-time and high-accuracy practical inspection applications. First, the original trinocular images need to be rectified. Then, the rectified trinocular images are adopted to achieve three-dimensional reconstruction of power lines. Finally, based on trinocular vision, bag of feature, and GPS, the clearance distance measurement, obstacle classification, and obstacle location are realized. Experimental tests on 220 kV transmission lines reveal that our proposed system can be applied in practical inspection environments and has good performance.

Research on the Robot’s Intelligent Inspection, Its Target Detection Method

Intelligent inspection robot has the characteristics of programmable, and can be applied to various inspection environments. In general, studies show the present trend is to replace humans with an inspection robot thereby reducing the risks and improving the inspection efficiency. The intelligent inspection robot is based on intelligent technology and has programmability. In this paper, based on the research of intelligent inspection robot technology, we analyze inspection techniques, their algorithms, functions, characteristics and other important parameters. The research mainly focuses on two things: the target detection, methods and improved Adaboost algorithm to improve the accuracy of target detection; the Camshift algorithm which is improved to complete tracking design, timely data acquisition, timely problem discovery and timely solution. The target detection and target tracking are studied and their algorithms are analyzed. We present that a tracking algorithm based on improved Camshift deals with the problems which exist in traditional Camshift algorithm. In addition, we present Meanshift algorithm improves the Camshift algorithm for the whole-process tracking, automation and intelligence level, and efficient tracking. Next, combined with relevant technologies and techniques, the algorithm is improved to complete the target detection design and tracking design, and to solve the problems of inaccurate target detection and untimely detection.

Comparison of optimized Markov Decision Process using Dynamic Programming and Temporal Differencing – A reinforcement learning approach

Reinforcement learning is one of the promising approaches for operations research problems. The incoming inspection process in any manufacturing plant aims to control quality, reduce manufacturing costs, eliminate scrap, and process failure downtimes due to non-conforming raw materials. Prediction of the raw material acceptance rate can regulate the raw material supplier selection and improve the manufacturing process by filtering out non-conformities. This paper presents a Markov model developed to estimate the probability of the raw material being accepted or rejected in an incoming inspection environment. The proposed forecasting model is further optimized for efficiency using the two reinforcement learning algorithms (dynamic programming and temporal differencing). The results of the two optimized models are compared, and the findings are discussed.

AI-Enabled Robotic NDE for Structural Damage Assessment and Repair

The aim of this paper is to develop the concept and a prototype of an intelligent mobile robotic platform that is integrated with nondestructive evaluation (NDE) capabilities for autonomous live inspection and repair. In many industrial environments, such as the application of power plant boiler inspection, human inspectors often have to perform hazardous and challenging tasks. There is a significant chance of injury, considering the confined spaces and limited visibility of the inspection environment and hazards such as pressurization and improper water levels. In order to provide a solution to eliminate these dangers, the concept of a new robotic system was developed and prototyped that is capable of autonomously sweeping the region to be inspected. The robot design contains systematic integration of components from robotics, NDE, and artificial intelligence (AI). A magnetic track system is used to navigate over the vertical steel structures required for examination. While moving across the inspection area, the robot uses an NDE sensor to acquire data for inspection and repair. This paper presents a design of a portable NDE scanning system based on eddy current array probes, which can be customized and installed on various mobile robot platforms. Machine learning methods are applied for semantic segmentation that will simultaneously localize and recognize defects without the need of human intervention. Experiments were conducted that show the NDE and repair capabilities of the system. Improvements in human safety and structural damage prevention, as well as lowering the overall costs of maintenance, are possible through the implementation of this robotic NDE system.

OBJECT TRACKING CONTROL USING A GIMBAL MECHANISM

Abstract. This work describes a control solution for real time object tracking in images acquired for a RPAS on an object inspection environment. This, controlling a 3-axis gimbal mechanism to control a camera orientation embedded to a RPAS, using its image processed for feedback. The objective of control is to maintain the target of interest at the center of the image plane. The proposed solution uses a YOLOv3 object detection model in order to detect the target object and determine, thru rotation matrices, the new desired angles to converge the object’s position to the center of the image. To compare results of the proposed control, a linear control was tuned using a linear PI algorithm. Simulation and practice experiments successfully tracked the desired object in real time using YOLOv3 in both control approaches presented.

Analytical Modeling of the Interaction Between Soft Balloon-Like Actuators and Soft Tubular Environment for Gastrointestinal Inspection.

Accessing tubular environment is critical in medicine. For example, gastrointestinal tract related cancers are the leading causes of cancer deaths globally. To diagnose and treat these cancers, clinicians need accessing the gastrointestinal tract, for example, colon and small intestine, which are soft biological tubes. Soft balloon assisted locomotion is one of the promising methods for accessing bio-duct. It has been widely used in enteroscopy and other medical devices. However, the interaction between the balloon and the soft tube is seldom studied, such as the interaction pressure and the anchoring force. In this work, we present the first modeling of the interaction between soft balloon actuators and soft tubular environment. The free inflation model of soft balloon actuators was first presented. Then a constrained inflation model of the soft balloon in a soft tube was established. Finally, the anchoring force model between the soft balloon and the soft tube was developed. On average, the mean error of the predictions in these three models is 0.228 kPa (or 3.14%), 0.56 kPa (or 7.8%), and 0.22 N (or 14.7%), respectively. In the future, these models could be used for guiding balloon-actuator designs by minimizing the interaction pressure while maintaining sufficient anchoring force during the locomotion in soft tubes.

Deep learning smartphone application for real‐time detection of defects in buildings

Condition assessment and health monitoring (CAHM) of built assets requires effective and continuous monitoring of any changes to the material and/or geometric properties of the assets in order to detect any early signs of defects or damage and act on time. Most of the traditional CAHM techniques, however, depend on manual labour despite that, in some cases, the inspection environment can be unsafe and could lead to low efficiency or misjudgement of the severity of the defect. In recent years, computer vision techniques have been proposed as an automated alternative to the traditional CAHM techniques as methods for extracting and analysing feature-related information from asset images and videos. Such methods have proven to be robust and effective solutions, complementary to current time-consuming and unreliable manual observational practices. This work is concerned with the development of a deep learning-based smartphone app, which allows real-time detection of four types of defects in buildings, namely, cracks, mould, stain and paint deterioration. Since smartphones are widely available and equipped with high-resolution cameras, this application can offer a practical, low-cost solution for condition assessment procedures of built assets. The obtained results are promising and support the feasibility and effectiveness of the approach to identify and classify various types of building defects.

Development of gantry radial arm robotic system for corrosion inspection on wall of reactor tank TRIGA

This paper focuses on the design upgrade of existing scanner of gantry radial arm robotic system for corrosion inspection on wall reactor of a TRIGA tank. The scanner design will be used for under water inspection environment verification. This inspection can be performed by using ultrasonic C-scan that will be designed in this project. This gantry radial arm system is a modular design that allows upgrading, easy dimensional matching and, specifications related to kinematic and technique. The gantry radial arm system is expected to be user-friendly. SolidWorks software was used to design the prototype of the gantry radial arm. Most of the components are made from Aluminium which is much sturdy and low cost. Meanwhile for the rod holder system, spring mechanism was used to prevent rod from bending when it moves down into the water. The test results show that the system designed is safe to be used, as it has been proved by simulation and calculation.

Real-Time Detection and Spatial Localization of Insulators for UAV Inspection Based on Binocular Stereo Vision

Unmanned aerial vehicles (UAVs) have become important tools for power transmission line inspection. Cameras installed on the platforms can efficiently obtain aerial images containing information about power equipment. However, most of the existing inspection systems cannot perform automatic real-time detection of transmission line components. In this paper, an automatic transmission line inspection system incorporating UAV remote sensing with binocular visual perception technology is developed to accurately detect and locate power equipment in real time. The system consists of a UAV module, embedded industrial computer, binocular visual perception module, and control and observation module. Insulators, which are key components in power transmission lines as well as fault-prone components, are selected as the detection targets. Insulator detection and spatial localization in aerial images with cluttered backgrounds are interesting but challenging tasks for an automatic transmission line inspection system. A two-stage strategy is proposed to achieve precise identification of insulators. First, candidate insulator regions are obtained based on RGB-D saliency detection. Then, the skeleton structure of candidate insulator regions is extracted. We implement a structure search to realize the final accurate detection of insulators. On the basis of insulator detection results, we further propose a real-time object spatial localization method that combines binocular stereo vision and a global positioning system (GPS). The longitude, latitude, and height of insulators are obtained through coordinate conversion based on the UAV’s real-time flight data and equipment parameters. Experiment results in the actual inspection environment (220 kV power transmission line) show that the presented system meets the requirement of robustness and accuracy of insulator detection and spatial localization in practical engineering.

Analysis of Flaw Detection Sensitivity of Phased Array Ultrasonics in Austenitic Steel Welds According to Inspection Conditions.

The anisotropy and inhomogeneity exhibited by austenitic steel in welds poses a challenge to nondestructive testing employing ultrasonic waves, which is predominantly utilized for the inspection of welds in power plants. In this study, we assess the reliability of phased array ultrasonic testing (PAUT) by analyzing the flaw detection sensitivity of ultrasonic beams in anisotropic welds, based on the inspection conditions. First, we simulated the sectorial scan technique, frequently employed for the inspection of actual welds, while taking into account the ultrasonic wave mode, frequency, and shape and position of a flaw. Subsequently, we analyzed the flaw sensitivity by comparing A-scan signals and S-scan results. The sensitivity analysis results confirmed the detection of all flaws by considering at least two inspection methods based on the shape and position of the flaw. Furthermore, we verified our model by performing an experiment under the same conditions as the simulation and found that the results were in agreement. Hence, we find that the simulation modeling technique proposed in this study can be utilized to develop suitable inspection conditions, according to the flaw characteristics or inspection environment.

Localization Method Based on Image Processing for Autonomous Driving of Mobile Robot in the Linear Infrastructure

In recent years, the deterioration of infrastructure facilities such as bridges has become a problem. Precautionary measures such as visual inspection and repair by humans are in place as countermeasures for aging; however, there are issues with cost and safety in such inspections. If inspection by robots becomes possible, both these aspects will be improved, which will significantly contribute to the maintenance of infrastructure facilities. In this paper, we propose a complex image processing technique to specify the location of feature points as coordinates through smartphone cameras to obtain the location information of feature points needed for positioning BIREM-IV-P developed to support bridge inspection. The corners located in the bridge inspection environment are used as feature points, and the corners are specified using Harris corner detection, which is a conventional corner detection method, to obtain the position of the feature points. In addition, to compensate for the shortcomings of Harris corner detection, a line segment in the image is detected using the Hough transform, and the intersection points of the line segments are recognized as corners. By combining the results of the two detection methods in this manner, the target feature points can be accurately specified. Then, the position of the feature points of the specified image coordinate system can be changed to the world coordinate system. As a result, it was possible to detect the location of the target feature points in a three-dimensional coordinate system.

Edge-based cover recognition and tracking method for an AR-aided aircraft inspection system

Cabin hatch cover plays an important role for a typical aircraft to ensure the aircraft quality and flight safety. In the test flight inspection and routine maintenance of an aircraft, one crucial step is to open cabin hatch covers one by one for detailed inspection. Authorized operators suffer from difficulties to recognize different covers with similar shapes and different inspection requirements. In this paper, an edge-based cover recognition and tracking method is proposed to recognize different hatch covers with similar shapes. First, a fast edge feature is proposed to describe image contours with simple geometric constraints. Second, based on the edge feature, a novel cover descriptor, consisting of shape and position description vectors, is presented to recognize those different covers with similar shapes. Third, on the basis of recognized cover landmarks, a direct visual odometry–based camera tracking method is presented to improve the robustness of cover recognition. The experiments are implemented in a piece of simplified mockup of aircraft cabin skin, and the results show that the proposed method has good practicability and real-time property. Meanwhile, the tracking accuracy is also good enough in the augmented reality inspection environment.

Study on the Design of Comprehensive Test Environment for the Performance of Cable Tunnel Patrol Robot

The cable tunnel patrol robot has gradually attracted the attention of cable operation and maintenance units due to its high inspection efficiency and strong work safety[1]. In order to test the functions of cable tunnel patrol robot such as autonomous inspection, defect identification and fire fighting, a targeted functional test environment for cable tunnel patrol robot was designed, including autonomous inspection environment and defect detection environment. The test environment can not only be used to test the performance of cable tunnel patrol robot, but also provide an important reference for the construction of autonomous inspection environment for cable tunnel robot.

New imaging algorithm for material damage localisation based on impedance measurements under noise influence

Abstract The electro-mechanical impedance (EMI) measurements have been extensively studied in recent years to provide a reliable diagnosis of aerospace infrastructures. Existing imaging algorithms for EMI-based damage localisation have been proposed for controlled inspection environments or under the sole influence of temperature variations. However, the presence of signal noise may alter impedance signals and limit the use of the EMI method in real operating scenarios. Based on this issue, this short communication proposes a novel EMI probabilistic imaging algorithm for damage localisation under noisy inspections. Furthermore, as another advantage compared to traditional techniques, which do not perform a noise compensation, the proposed application does not require high computational cost since it does not require licensed software or the calculation of acoustic parameters. Experimental results on an aluminium plate-like structure revealed that the new algorithm proved to be adequate to image the location of damage under noise influence as opposed to traditional approaches.

Multisensor Fusion for Magnetic Flux Leakage Defect Characterization Under Information Incompletion

Magnetic flux leakage (MFL) inspection robots are widely used in the pipeline inspection industry to obtain MFL information from multisensor sources. Multisensor fusion is required for precise defect characterization. In this article, a multisensor fusion framework is presented to fuse and enhance MFL information for defect characterization. First, a space-vector-based information fusion method is proposed for better reflecting the spatial distribution of real magnetic field, where the direction and amplitude information of magnetic field are considered simultaneously. Second, considering the information incompletion caused by unstable inspection environment and abnormal device, the improved conditional generative adversarial nets are designed to enhance MFL information. These networks can effectively reconstruct multisensor information by refining the generator loss function, so that the enhanced MFL information can be better applied for defect characterization. Experimental results are compared with state of the art in detail, which highlight the superiority of the proposed multisensor fusion method in MFL inspection.

Detection of Power Line Insulator Defects Using Aerial Images Analyzed With Convolutional Neural Networks

As the failure of power line insulators leads to the failure of power transmission systems, an insulator inspection system based on an aerial platform is widely used. Insulator defect detection is performed against complex backgrounds in aerial images, presenting an interesting but challenging problem. Traditional methods, based on handcrafted features or shallow-learning techniques, can only localize insulators and detect faults under specific detection conditions, such as when sufficient prior knowledge is available, with low background interference, at certain object scales, or under specific illumination conditions. This paper discusses the automatic detection of insulator defects using aerial images, accurately localizing insulator defects appearing in input images captured from real inspection environments. We propose a novel deep convolutional neural network (CNN) cascading architecture for performing localization and detecting defects in insulators. The cascading network uses a CNN based on a region proposal network to transform defect inspection into a two-level object detection problem. To address the scarcity of defect images in a real inspection environment, a data augmentation method is also proposed that includes four operations: 1) affine transformation; 2) insulator segmentation and background fusion; 3) Gaussian blur; and 4) brightness transformation. Defect detection precision and recall of the proposed method are 0.91 and 0.96 using a standard insulator dataset, and insulator defects under various conditions can be successfully detected. Experimental results demonstrate that this method meets the robustness and accuracy requirements for insulator defect detection.