Self-built masonry structures are commonly used as houses in rural areas of the central plains of China. These structures pose significant safety risks due to the low level of self-building and non-standard designs. The primary type of damage to the walls of masonry houses is cracks, while the decorative layers of the walls may also experience peeling or cracking. The shapes, sizes, and surface characteristics of these damages are complex and unique to each type. To achieve fast identification of surface damage on masonry house walls, this paper proposes a cascade detection model that combines a multilevel cascade classifier with a parallel sub-segmentation network. The VGG16 neural network is used as the backbone for a multi-level series classifier. The model is trained by using a damage image set of the wall decorative layers in rural masonry houses. The classification of background and damage, as well as the classification of cracks and peels, are sequentially completed. Then, the parallel sub-segmentation model uses EfficientNet-B7 as the encoder and combines it with the U-Net framework skeleton to perform pixel-level segmentation of peeling and cracks. Finally, the output of parallel sub-segmentation networks is superimposed and fused to generate a complete image containing segmentation information of peeling and cracks. The cascade form network structure adopted in this model significantly reduces the training difficulty and enhances the model accuracy. Compared with Segformer and Deeplabv3+ network, the average IoU of the proposed method for on-site masonry wall images can be increased by 0.29 and 0.08, respectively.