Chapter 10 - Mixed reality-assisted smart bridge inspection for future smart cities

Abstract

Smart infrastructures aim more efficient and accurate methods of routine inspection for long-term monitoring of the infrastructure to make smarter decision on maintenance and rehabilitation. Although some recent technologies (i.e., robotic techniques) that are currently in practice can collect objective, quantified data, the inspector’s own expertise is still critical in many instances. Yet, these technologies are designed to replace human expertise, or are ineffective in terms of saving time and labor. This chapter investigates a new methodology for structural inspections with the help of mixed reality technology and real-time machine learning to accelerate certain tasks of the inspector such as detection, measurement, and assessment of defects, and easy accessibility to defect locations. A functional, real-time machine learning system that can be ideally deployed in mixed reality devices and headsets which can be used by inspectors during their routine concrete infrastructure inspection is introduced. The deep learning models to be employed in the AI system can localize a concrete defect in real time and further analyze it by performing pixel wise segmentation while running on a mobile device architecture. First, a sufficiently large database of concrete defect images is gathered from various sources including publicly available crack and spalling datasets, real-world images taken during bridge inspections, and the public images from the internet search results. For defect localization, various state-of-the-art deep learning model architectures are investigated based on their memory allocation, inference speed, and flexibility to deploy different deep learning platforms. YoloV5s model was found to be the optimal model architecture for concrete defect localization to be deployed in the mixed reality system. For defect quantification, several segmentation architectures with three different classification backbones are trained on the collected image dataset with segmentation labels. Based on the model evaluation results, the PSPNet with EfficientNet-b0 backbone is found to be the best performing model in terms of inference speed and accuracy. The selected models for defect localization and quantification are deployed to the mixed reality platform and image tracking libraries are configured in the platform environment, and accurate distance estimation is accomplished using a calibration process. Lastly, a methodology for condition assessment of concrete defects using the mixed reality system is discussed. The proposed methodology can locate and track the defects using the mixed reality platform, which can eventually be transferred to cloud data and potentially used for remote assessments or updating a digital twins or BIMs.