To solve the problems of poor recognition of apparent concrete damage and the difficulty of efficient quantitative characterization of the damage parameters, this study proposes a damage detection method that integrates the target localization algorithm and the quantitative characterization algorithm of the damage based on computer vision. This paper proposes a YOLO v5-RS network combining Res2Net and SimAM to solve the problem of indistinctive apparent damage feature. The damage is segmented by image processing, and the crack and crush damage quantification method are proposed. The results show that compared with YOLO v5, the mAP and F1 of YOLO V5-RS are improved by 8.4% and 0.07, respectively, which indicates that the recognition of apparent structural damage has high robustness. To verify the recognition method’s generalization ability and the characterization method’s accuracy, four cases of different scale are introduced in this paper.