Vision-based Method Research Articles

Vision-Based Damage Detection Method Using Multi-Scale Local Information Entropy and Data Fusion

Low-spatial-resolution measurements from contact sensors and excessive measurement noise have impeded the implementation of vibration-based damage detection. To tackle these challenges, we propose a novel vision-based damage detection method combining multi-scale signal analysis theory and data fusion algorithm. For high-spatial-resolution vibration measurements, phase-based optical flow estimation algorithm is adopted to deploy virtual sensors on the structure, yielding reliable mode shapes. We then introduce the concept of entropy into damage detection. A novel damage index, defined in Gaussian multi-scale space and named multi-scale local information entropy (MS-LIE), is proposed. The MS-LIE integrates the multi-scale analysis component and the entropy analysis component, addressing both the issue of detection sensitivity and noise immunity, thereby showcasing enhanced performance. Moreover, a data fusion technique for multi-scale damage information is developed to further mitigate the noise-induced uncertainty and pinpoint damage locations. A series of numerical and experimental scenarios are designed to validate the method, and the results indicate that the proposed method accurately detects single and multiple damages in noisy environments, obviating the need for baseline data as a reference.

Automated vision-based bolt loosening detection and localization for operating hydraulic turbine rotors

Turbine rotors are susceptible to bolt loosening and detachment due to factors such as vibration, stress concentration, and material degradation during the prolonged operation of hydropower units. These issues can result in significant safety incidents and economic losses. This article presents a novel vision-based automatic detection method for identifying loose bolts in turbine rotors by analyzing the angle between marking lines on the top surfaces of studs and nuts. The proposed method integrates enhanced you only look once version 8 (YOLOv8) for object detection, DeepSORT for multitarget tracking, DeepLabV3+ for image segmentation, and Theil-Sen estimation for line fitting. This approach effectively locates loose bolts on the rotor and determines the loosening angle. Experiments were conducted during the commissioning of a large hydroelectric unit, yielding results that demonstrate an average loosening angle calculation error 0.633°. With a loosening threshold established at 2°, the method achieves a detection accuracy of 98.88%. The proposed method does not depend on the historical states of the bolts and provides precise identification of bolt loosening angle up to 180°, thereby making it suitable for application to operating turbine rotors.

Motion adaptive vision-based vibration measurement and modal identification for the roof masts of a tall building

On May 18, 2021, occupants in Saige Plaza Building felt significant building motions together with its roof masts caught in obvious vibrations (May 18 vibration event), which were recorded by a surveillance camera installed on the roof of the building. This study aims to investigate the vibration characteristics of masts by processing the video data using computer vision technique. A motion adaptive vision-based vibration measurement method (M-DAVIM) is firstly proposed, focusing on dealing with the adverse motion effects of the camera itself on measuring dynamic displacements. The M-DAVIM incorporates time-domain and frequency domain correction procedures to reduce noise caused by camera self-vibration, and utilizes a CNN-based object tracking method to search objects that blurred by camera shaking. Indoor periodic vibration tests and field tests of photovoltaic panels demonstrated that M-DAVIM outperforms previous vision-based methods in accurately measuring displacements under unfavorable conditions, such as target rotation, motion blurring, and background interference. The proposed M-DAVIM was then applied to measure the dynamic displacements of the roof masts of Saige Building and identify their modal parameters (frequencies and damping ratios) based on the limited video data. A vibration component of 7.60 Hz was identified as the camera self-vibration and was effectively corrected by the M-DAVIM method. Based on finite element analysis and given the wind condition, the twin-mast might mainly experience the vortex-induced vibration at 2.12 Hz with two masts vibrating synchronously in-plane along opposite directions during May 18 to 22, 2021. This study demonstrates the robustness and effectiveness of the M-DAVIM and shows its potential application for long-term field monitoring of large-scale structures under severe outdoor environments.

Dual-arm shaping of soft objects in 3D based on visual servoing and online FEM simulations

In this work, we propose a vision-based and finite element method (FEM)-based controller to automate the 3D shaping of soft objects with dual-arm robots. Our controller relies on a data-based approach to learn how the robot’s actions result in object deformations, while also running FEM-based simulations to infer the shape of the whole body. These model-based simulations are used to generate initial shape data, allowing to extract visual features through a principal component analysis and thus estimate the interaction matrix of the object–robot system. In contrast with most existing shape servoing controllers, our new model-based approach continuously predicts the object deformations produced by the robot, which are then compared to the visually observed deformation feedback. This iterative process enables to correct the deformed mesh model before updating the interaction matrix. To validate this new control methodology, we present a detailed experimental study with a dual-arm robot and different soft objects, which showcases the performance of our automatic shaping framework.

Efficient UAV localization using combined autoencoder and SIFT

In GNSS-denied environments, accurate Unmanned Aerial (UAV) localization faces significant challenges. This paper introduces a vision-based localization method combining autoencoder and SIFT algorithms, referred to as AE+SIFT. The method compresses high-resolution map images into low-dimensional vectors, which are stored in a database for efficient retrieval. During the localization process, UAV images are encoded and matched with the database, followed by SIFT and homography projection for precise positioning. The AE+SIFT approach enhances localization accuracy, achieving an average coordinate error of 3.94 meters relative to the ground truth. Notably, when UAV images are misaligned with reference images, our method outperforms the existing AE method in terms of accuracy.

Monocular Vision-Based 3D Reconstruction Method for Parts in Opto-Mechanical Assembly

Abstract In automated assembly, accurately capturing multi-dimensional parameters of parts is essential. Traditional methods include teaching methods or laser-based approaches, with the former relying on experience for pre-calibration and the latter being costly. This paper proposes a low-cost, monocular vision-based method for 3D reconstruction in opto-mechanical assembly. Taking the laser crystal component, including the laser gain medium and its heat sink, in a solid-state laser as an example, a monocular camera is firstly mounted on a robot and captures images of parts from specific angles. Due to the reflective properties of metal surfaces, some image information is lost. To address this problem, a 2D gamma function-based adaptive illumination correction algorithm is then proposed. Next, the improved SIFT algorithm is used to extract and match feature points from the collected images. Subsequently, the SFM approach generates a sparse point cloud, which is then densified using the CMVS/PMVS algorithm. Finally, the Poisson surface reconstruction algorithm is used to complete the 3D reconstruction of the laser crystal heat sinks. Experimental results indicate that the adaptive correction algorithm for non-uniform illumination images can extract more feature points and improve the number of matches. The reconstructed 3D model has high accuracy, providing important technical support for the next step in achieving automated opto-mechanical assembly.

Low-cost 3D vision-based triangulation system for ultrasonic probe positioning

Abstract Introduction: In ultrasonic imaging, such as echocardiography, accurately positioning the probe in relation to the patient’s body or an external coordinate system is typically done manually. However, when developing speckle-tracking methods for echocardiology, ensuring consistency in probe positioning is essential for reliable data interpretation. To address this challenge, we present a vision-based system and method for probe positioning in this study. Materials and Methods: Our system comprises two cameras, a calibration frame with eight markers of known coordinates in the frames’ local coordinate system, and a probe holder with four markers. The calibration process involves image segmentation via region growing and extraction of the camera projection matrices. Subsequently, our positioning method also utilises marker segmentation, followed by estimating the markers’ positions using triangulation. Results: To evaluate the system’s performance, we conducted tests using a validation plate with five coplanar circular markers. The distances between each pair of points were calculated, and their errors compared to the true distances were found to be within a maximum of 0.7 mm. This level of accuracy is comparable to ultrasonic imaging resolution and thus deemed sufficient for the intended purpose. Conclusion: For those interested in replicating or modifying our methods, the supplementary material includes the complete design of the calibration frame and the Matlab code.

Sand gradation detection method based on local sampling

Sand gradation is an important reference factor for concrete proportion design, which has a significant impact on the strength and workability of concrete. The efficiency of using the traditional vibrating sieving method to measure the gradation is low, and the machine vision-based sand gradation inspection method can realize the automation of sand gradation inspection, which can significantly improve the inspection efficiency and reduce the inspection cost. The particle size of construction sand is usually between 0.075 and 4.75 mm, and to capture images of sand below 0.15 mm requires a high-resolution camera that can only maintain a small object distance and a small shooting field of view, so it is difficult for the image acquisition system to capture images of all the sand particles, and this is a challenging problem in sand grading inspection. In order to solve this problem, this study proposes a sand gradation detection method based on localized sampling, and develops a hardware system for localized sampling and a software system for gradation detection. The hardware system uses a flexible vibrating disk to uniformly disperse the sand particles, realizing random local sampling of sand particles, and calculating the grain size gradation of the test samples based on the local data of multiple local sampling. The experimental results show that the local sampling gradation and the sample gradation have a similar distribution, and the local gradation detection error of multiple sampling is smaller. This study provides an effective method for the automated detection of sand grain size gradation between 0.075 and 4.75 mm, which can significantly increase the frequency of construction sand gradation detection, improve the quality control level of construction sand, and improve the quality of construction.

A Machine Vision-Based Measurement Method for the Concentricity of Automotive Brake Piston Components

The concentricity error of automotive brake piston components critically affects the stability and reliability of the brake system. Traditional contact-based concentricity measurement methods are inefficient. In order to address the issue of low detection efficiency, this paper proposes a non-contact concentricity measurement method based on the combination of machine vision and image processing technology. In this approach, an industrial camera is employed to capture images of the measured workpiece’s end face from the top of the spring. The edge contours are extracted through the implementation of image preprocessing algorithms, which are then followed by the calculation of the outer circle center and the fitting of the inner circle center. Finally, the concentricity error is calculated based on the coordinates of the inner and outer circle centers. The experimental results demonstrate that, in comparison to a coordinate measuring machine (CMM), this method exhibits a maximum error of only 0.0393 mm and an average measurement time of 3.9 s. This technology markedly enhances the efficiency of measurement and fulfills the industry’s requirement for automated inspection. The experiments confirmed the feasibility and effectiveness of this method in practical engineering applications, providing reliable technical support for the online inspection of automotive brake piston components. Moreover, this methodology can be extended to assess concentricity in other complex stepped shaft parts.

Machine Vision-Based Real-Time Monitoring of Bridge Incremental Launching Method.

With the wide application of the incremental launching method in bridges, the demand for real-time monitoring of launching displacement during bridge incremental launching construction has emerged. In this paper, we propose a machine vision-based real-time monitoring method for the forward displacement and lateral offset of bridge incremental launching in which the linear shape of the bottom surface of the girder is a straight line. The method designs a kind of cross target, and realizes efficient detection, recognition, and tracking of multiple targets during the dynamic process of beam incremental launching by training a YOLOv5 target detection model and a DeepSORT multi-target tracking model. Then, based on the convex packet detection and K-means clustering algorithm, the pixel coordinates of the center point of each target are calculated, and the position change of the beam is monitored according to the change in the center-point coordinates of the targets. The feasibility and effectiveness of the proposed method are verified by comparing the accuracy of the total station and the method through laboratory simulation tests and on-site real-bridge testing.

Deep learning models for vision-based occupancy detection in high occupancy buildings

Accurate occupancy information is crucial for enhancing energy efficiency and reducing carbon emissions in buildings. However, the inherent unpredictability of occupants introduces uncertainties in energy analysis and control strategy development. To address these challenges, this study proposes a vision-based method employing state-of-the-art deep learning models to capture real-time occupancy profiles in crowded indoor spaces. Utilising a self-collected image dataset, various deep learning models, including Single Shot MultiBox Detector (SSD), Faster Region-based Convolutional Neural Networks (Faster R-CNN), and different versions of You Only Look Once (YOLO) were trained and evaluated. An experiment was conducted in a lecture room equipped with cameras and environmental sensors to evaluate the performance of each model in terms of precision, computational efficiency, and adaptability to varying occupancy levels during a lecture session. The session included varying occupancy conditions: entering (barely occupied), during the lecture (typical occupancy), and leaving the room (again barely occupied). Among the models tested, YOLOv8x exhibited the best performance in terms of accuracy, while SSD lagged notably. The impact on the detection performance of various locations of camera setups was also explored. Energy simulations revealed that deep learning-based model generated occupancy profiles significantly deviated from conventional “fixed” occupancy profiles, resulting in a 13.45 % variation in predicted heating energy demand. However, compared to the ground truth, these profiles showed minimal variation (up to 6.72 %) for the Faster R-CNN and YOLO models, highlighting their accuracy and robustness. Additionally, although the deep learning-based occupancy profiles generally overpredicted the recorded data, the CO2 concentration trends they predicted aligned closely with the recorded data, unlike the “fixed” occupancy profiles. The findings underscore the importance of realistic occupancy profiles for reliable energy predictions in buildings and demonstrate the potential of the proposed vision-based method for advancing occupancy detection and building energy management.

A Machine Vision-Based Fiber Profile Image Recognition Method for Alignment of FBG Inscribing

A Machine Vision-Based Fiber Profile Image Recognition Method for Alignment of FBG Inscribing

An advanced machine vision-based method for abnormal detection of transverse vibrations in ship propulsion shafting

The working environment of ship propulsion shafting is harsh and the force condition is complex, which often produces all-directional vibration. Its working condition will directly affect the navigation performance of the ship. To overcome the limitations of complicated installation route, tedious maintenance process and high cost of traditional contact vibration sensors, an approach of transverse vibration identification model based on machine vision was proposed to realize multi-point vibration displacement sensing and anomaly analysis of shafting. The displacement information of the video signal is extracted by the displacement sensing strip labeling method, and the abnormal state of the ship propulsion shafting is analyzed by the dynamic kernel principal component analysis (DKPCA) algorithm. The experimental results show that the approach can accurately detect the continuous vibration displacement of shafting in the range of 180 r/min, and can work normally under two abnormal conditions: sudden external excitation and continuous uneven external excitation. In addition, this approach can quickly and accurately monitor the motion state of shafting, and realize the perception and recognition of abnormal vibration state of shafting. The research and application of this approach in ship shafting vibration monitoring is of great significance to the development of unmanned and intelligent ships.

Detecting GNSS spoofing and Re-localization on UAV based on imagery matching

Abstract Due to rapid advancements in global navigation satellite system (GNSS), computer, and microelectronics technologies, there is a growing popularity and widespread promotion of high-performance, cost-effective intelligent UAVs across various applications. Recently, GNSS spoofing attacks have emerged as a significant obstacle hindering the long-term development of UAVs. UAVs rely heavily on unprotected GNSS signals for navigation, making them highly vulnerable to spoofing. This paper presents an algorithm that employs image matching for detecting GNSS spoofing and re-localization on UAVs using a deep learning methodology. This method functions autonomously solely reliance on camera and does not require alterations to the antenna configuration. Utilizing a camera-equipped UAV, we evaluate the likeness between real-time aerial photographs and satellite imagery by leveraging the position information provided by UAV. By identifying disparities between images taken by a spoofing affected drone and authentic ones, the spoofing can be identified using deep neural network models. Upon detecting spoofing, this paper presents a vision-based re-localization method for UAVs. Experimental results demonstrate an approximately 88.4% accuracy of our model in detecting GNSS spoofing attacks within 100 ms and 88% success rate of re-localization with the accuracy of less than 15 m. Our algorithm exclusively depends on publicly accessible satellite imagery, offering an intelligent and efficient approach for detecting UAV GNSS spoofing and re-localization in GNSS denied environment.

Computer vision-based intelligent detection method for the residual capability of energy dissipators in flexible protection systems

A residual capability intelligent detection method based on computer vision is proposed to address the issues of low efficiency, poor accuracy, and high danger in manual measurement of energy dissipators in flexible protection systems. The proposed method first establishes a binary semantic segmentation dataset for energy dissipators and trains a salient object detection deep neural network to segment the energy dissipator binary map; Then, it uses morphological image processing and contour detection to calculate the residual capability automatically. U2-Net, U2-Netp, and BASENet were trained and compared by a dataset with 500 ring-type energy dissipator images. The proposed method was validated through a quasi-static tensile test and a full-scale impact test. Compared with the most accurate integration calculation method, the error of the proposed method does not exceed 3 %, and the efficiency is improved by about 25 times compared to the most commonly used manual detection method.

Computer vision-based substructure isolation method for localized damage identification

In recent years, vision-based structural damage identification techniques have garnered significant attention due to their simplicity and cost-effectiveness. Nevertheless, dynamic response-based vision methods face challenges related to the limitations of the field of view (FOV), i.e., the extent of the FOV is often sacrificed in order to ensure sufficient monitoring accuracy, which reduces the amount of data available for structural health monitoring (SHM). This study presents a solution to this problem by employing the substructure isolation method (SIM), which focuses on local vibrations of the structure and enables local damage identification by isolating the area of interest. Additionally, a method combining kernelized correlation filter (KCF) with sub-pixel template matching is used to extract vibration information recorded by visual sensor. This strategy balances both the efficiency and accuracy of displacement extraction. In numerical simulations, the performance of the SIM based on acceleration response and displacement response is compared, demonstrating the potential compatibility of visual sensors with the SIM. Finally, the effectiveness of the proposed vision-based local damage identification method is validated through vibration testing of a shear frame model.

A Vision-Based Simultaneous Calibration Method for Dual-Robot Collaborative System

A Vision-Based Simultaneous Calibration Method for Dual-Robot Collaborative System

Research on weak appearance defect detection method for engine piston rings based on 3D vision

Abstract As a critical engine component, the appearance quality of piston rings directly affects engine performance and lifespan. Subtle defects like minor chipping, cracks, and scratches can significantly impact engine safety and durability. Traditional 2D image processing and AI-based methods struggle to detect these defects, being prone to interference and resulting in high false positive and negative rates. This paper introduces a 3D vision-based detection method, which captures detailed surface morphology to improve defect detection accuracy. Experimental results demonstrate that this approach effectively reduces false positives and negatives, significantly enhancing detection accuracy.

An estimation method for vision-based autonomous landing system for fixed wing aircraft

In this paper, we propose a vision-based estimation method for autonomous landing of fixed-wing aircraft for Advanced Air Mobility. The concept of autonomous flight with minimal human intervention has gained significant interest with a particular focus on autonomous landing. The goal is to overcome the limitations of existing instrument landing systems and reduce reliance on GNSS during aircraft landing. The proposed system leverages the following techniques for high-performance and real-time operations in various real-world environments and during all stages of landing. A deep learning segmentation model was directly learned, created, and used for robust runway recognition performance against environmental changes around the runway. Algorithms were designed to adapt to different flight stages considering the varying information offered by image data when the aircraft is in mid-air and on the ground. The relative lateral position from the aircraft to the runway was estimated using bird’s-eye-view conversion and inverse projection techniques instead of conventional perspective-n-point methods for reducing the recognition error occurring from the conversion between 2D image and 3D world. Finally, the mixed-precision technique was used for the segmentation model to improve inference speed and ensure real-time operation in real-world deployment. Estimation stability and reliability were improved by using a Kalman Filter to cope with sensor uncertainties caused by the real-world flight environment. Consequently, we show that the average error in the lateral position estimation by the proposed method is comparable to the accuracy of a single GNSS and demonstrate successful autonomous landing of our test aircraft with a set of flight tests.

Vision-based multi-view reconstruction for high-precision part positioning in industrial robot machining

Traditional industrial robot machining typically involves offline programming of machining trajectories, with part positioning accomplished through robotic tools. However, this method has limitations, particularly when dealing with complex workpieces and dynamic environments. To tackle this problem, this paper introduces a vision-based multi-view reconstruction approach for part positioning in industrial robot machining. Multiple cameras capture multi-view images of the part, reconstruct its 3D point cloud, and match it with the CAD model of the part to achieve high-precision positioning. This method effectively translates offline programmed robot trajectories to real-world scenarios, improving the flexibility and adaptability of robot machining. Experimental results confirm the high accuracy and robustness of the proposed vision-based reconstruction method, achieving a relative error of less than 2 mm for 85.7% of points, with an overall root mean square error of 1.09 mm.