Quasi-autonomous bolt-loosening detection method using vision-based deep learning and image processing

Abstract

Abstract In this study, a quasi-autonomous vision-based method is newly proposed for detecting loosened bolts in critical connections. The main idea of the approach is to estimate the rotational angles of bolts from the connection images by integrating deep learning technology with image processing techniques. Firstly, a regional convolutional neural network (RCNN)-based deep learning algorithm is developed to automatically detect and crop plausible bolts in the connection image. Also, the Hough line transform (HLT)-based image processing algorithm is designed to automatically estimate the bolt angles from the cropped bolt images. Secondly, the proposed vision-based approach is validated for bolt-loosening detection in a lab-scale girder connection using images captured by a smartphone camera. The accuracy of the RCNN-based bolt detector and the HLT-based bolt angle estimator are examined under different levels of perspective distortion and shooting distance. Finally, the practicality of the proposed vision-based method is verified on a real-scale girder bridge connection containing numerous bolts. The images of the connection are captured by an unmanned aerial vehicle and transferred to a computer where a quasi-autonomous bolt-loosening detection process is performed via the proposed algorithm. The experimental results demonstrate potentials of the proposed approach for quasi real-time bolt-loosening monitoring of large bolted connections. The results show that the perspective angle should not go beyond 40 degrees to ensure the accuracy of the detection results.