The existing building structures are mainly composed of concrete and masonry structures. This paper proposes a Fractal Theory-based Method (FTM) designed to efficiently detect surface cracks in both concrete and masonry building structures, addressing limitations in existing crack recognition techniques, such as weak adaptability to complex building surfaces and requiring a large amount of computational and human resource. A novel Soft Box-Counting (SBC) algorithm is proposed to calculate texture roughness information from multi-channel and multi-scale crack images, effectively mitigating the influence of non-feature information on crack identification results. The SBC algorithm is integrated into neural networks through the fractal embedding mechanism, enhancing feature analysis through visualization experiments. The resulting FTM model integrates texture roughness information from crack images and employs fractal theory to accurately classify crack regions while minimizing the similarity between crack and non-crack regions. In comparison to existing models, FTM demonstrates superior identification performance at the same computational level on benchmark datasets for surface cracks in both concrete and masonry building structures. The FTM model exhibits enhanced identification universality and higher resource utilization efficiency.