Regular patrol inspection of concrete dams can detect cracks at an early stage. However, conventional crack segmentation models based on deep learning (DL) are difficult to be deployed in resource-constrained mobile devices due to the large number of parameters. This paper describes a lightweight semantic segmentation model, termed as CrackTrNet, for images of concrete dam cracks. CrackTrNet is a hybrid U-shaped model based on convolutional neural network (CNN) and Vision Transformer. The CNN is adopted to extract low-level visual features and the Transformer focuses on learning the global contextual information. The results demonstrate that its segmentation accuracy can reach 97.60%, while the model size is only 34.86 MB, which is 66.12%–87.85% lower than that of current mainstream DL-based models. To make the model more practical, a crack inspection mobile application (APP) is developed using Android Studio. The integration of lightweight CrackTrNet and APP can effectively assist the intelligent inspection of dam cracks to ensure structural safety.