Sewer defect instance segmentation, localization, and 3D reconstruction for sewer floating capsule robots

Abstract

Most automated sewer inspection tasks are based on closed-circuit television (CCTV) methods and focus on image classification or object detection but fail to obtain information on fine-grained sewer defects. Targeting the sewer defect detection from a video and performing the sewer inspection in a lightweight, low-cost, and practical manner, this study develops a sewer detection framework based on computer vision technology for the sewer floating capsule robot. The proposed framework performs three main sub-tasks, including instance segmentation, real-time sewer inspection device localization, and real three-dimensional (3D) model reconstruction, to realize sewer defect detection. An improved mask regional convolutional neural network (Mask RCNN), which integrates split attention module and balanced L1 loss module, is proposed for robust feature representation. In addition, an improved data augmentation method developed according to the sewer defect instance segmentation tasks is introduced. The 3D reconstruction and real-time localization of a sewer scene are achieved using the structure-from-motion techniques for the sewer floating capsule robot. Extensive experiments are conducted, and the experimental results show that the proposed method is effective and robust. The average precision of instance segmentation at the intersection of the union value of 0.5 is 92.7%, and the maximum 3D model measurement error is 1 m. However, video sequence information and multi-sensor fusion, combining the inertial measurement unit and vision technique, could be studied in the future to achieve better generalization and robust results.