The damage degree of a reinforced concrete structure is closely related to the generation and expansion of cracks. However, the traditional damage assessment methods of reinforced concrete structures have defects, including low efficiency of crack detection, low accuracy of crack extraction, and dependence on the experience of inspectors to evaluate the damage of structures. Because of the above problems, this paper proposes a damage assessment method for concrete members combining the U-Net convolutional neural network and crack fractal features. Firstly, the collected test crack images are input into U-Net for segmenting and extracting the output cracks. The damage to the concrete structure is then classified into four empirical levels according to the damage index (DI). Subsequently, a linear regression equation is constructed between the fractal dimension (D) of the cracks and the damage index (DI) of the reinforced concrete members. The damage assessment is then performed by predicting the damage index using linear regression. The method was subsequently employed to predict the damage level of a reinforced concrete shear wall–beam combination specimen, which was then compared with the actual damage level. The results demonstrate that the damage assessment method for concrete members proposed in this study is capable of effectively identifying the damage degree of the concrete members, indicating that the method is both robust and generalizable.