Damage identification in post-earthquake reinforced concrete (RC) structures based on semantic segmentation has been recognized as a promising approach for rapid and non-contact damage localization and quantification. In damage segmentation tasks, damage regions are often set against complex backgrounds, featuring irregular geometric boundaries and intricate textures, posing significant challenges to model segmentation performance. Additionally, the absence of public datasets exacerbates these challenges, hindering advancements in this field. In this paper, a pyramid Haar wavelet downsampling attention UNet (PHA-UNet) semantic segmentation network is proposed, and a database containing 1400 images of damaged RC components (PEDRC-Dataset) with pixel-level annotations is established. In the proposed PHA-UNet, attention mechanisms, multiscale feature fusion, Haar wavelet downsampling, and transfer learning are introduced to address above challenges. Finally, the proposed PHA-UNet is compared with four existing image segmentation architectures on both the Cityspace and the PEDRC-Dataset.