In this paper, the applicability of the elastic recovery (resilience) experiment for asphalt-rubber (AR) binders has been quantitatively assessed. The mechanical model, based on the viscoelastic constitutive relation and particle inclusion theory, was developed. The interfacial detachment between crumb rubber (CR) particles and asphalt caused by stress concentration was analyzed with Weibull statistical equations. Based on the road roughness excitation, the vehicle-road coupling vibration model was established to analyze the impact of vehicle loading on road surface deformation. AR binders with different CR particle sizes were assessed using scanning electron microscope (SEM) imaging and prepared for testing the elastic recovery (resilience). The results showed that the greater internal stress caused by the longer stretch length of AR binders in the elastic recovery experiment was ten times higher than that obtained from the resilience experiment, leading to the interfacial detachment between asphalt and the CR particles. Hence, the elastic property of some of the CR particles with high modulus was not reflected, resulting in the test values being lower than actual values. With the reduction of CR particle size, the interfacial detachment was improved in the elastic recovery experiment due to intense material interchange and the enhancement of interfacial bond strength. The millimeter-scale compression deformation of the AR binder in the resilience experiment was closer to the actual deformation of the road surface. The experimental time of resilience (120 min) has been reported less than that for elastic recovery (200 min–230 min). This study shows that the resilience experiment has a significant advantage in assessing the elastic property of the AR binder.