• RCA with high CMC value, complex shape and surface leads to worse interfacial adhesion failure. • GA-BP artificial neural network presented considerable capacity to predict the interfacial damage behavior. • The cement mortar content exhibited the greatest influence on the adhesion property of asphalt mortar-RCA interface. Asphalt-aggregate interface’s adhesion properties commonly affect the damage initiation and evolution within asphalt concrete materials, related to pavement durability and quality. The scope of this research was to investigate the influence of Recycled Concrete Aggregate (RCA) features on asphalt mortar-aggregate interface adhesion. Firstly, a three-dimensional reconstruction model of RCA was carried out using X-ray CT tomography and digital image processing. In this regard, five feature indicators, namely cement mortar content, sphericity, flat and elongated ratio, angularity, and surface texture, were proposed. Based on a bilinear cohesive zone model, the interface damage behavior of asphalt mortar-RCA was investigated by using a uniaxial compression simulation. Finally, a GA-BP artificial neural network was conducted to predict and quantify the effect of each feature indicator of RCA on interface adhesion. The results showed that when RCA had lower cement mortar content, higher sphericity value, and smoother surface, the asphalt mortar-RCA system was less prone to interface adhesion failure. The 5-14-1 GA-BP artificial neural network proposed in this study showed very good performance in predicting the interfacial dissipation damage energy with a mean-squared error value of 3.52 × 10 −4 for testing dataset. The cement mortar content parameter exhibited a remarkable influence on the interface adhesion property, and its global contribution to the interfacial dissipation damage energy (0.3486) was more than twice that of the surface texture parameter (0.1316). In future studies, the performance characteristics of cement mortar can be further investigated, thereby proposing RCA’s performance optimization technology.