Distilling Knowledge from a Transformer-Based Crack Segmentation Model to a Light-Weighted Symmetry Model with Mixed Loss Function for Portable Crack Detection Equipment

Abstract

The detection of cracks is extremely important for maintenance of concrete structures. Deep learning-based segmentation models have achieved high accuracy in crack segmentation. However, mainstream crack segmentation models have very high computational complexity, and therefore cannot be used in portable crack detection equipment. To address this problem, a knowledge distilling structure is designed by us. In this structure, a large teacher model named TBUNet is proposed to transfer crack knowledge to a student model with symmetry structure named ULNet. In the TBUNet, stacked transformer modules are used to capture dependency relationships between different crack positions in feature maps and achieve contextual awareness. In the ULNet, only a tiny U-Net with light-weighted parameters is used to maintain very low computational complexity. In addition, a mixed loss function is designed to ensure detail and global features extracted by the teacher model are consistent with those of the student model. Our designed experiments demonstrate that the ULNet can achieve accuracies of 96.2%, 87.6%, and 75.3%, and recall of 97.1%, 88.5%, and 76.2% on the Cracktree200, CRACK500, and MICrack datasets, respectively, which is 4–6% higher than most crack segmentation models. However, the ULNet only has a model size of 1 M, which is suitable for use in portable crack detection equipment.