To further study the fatigue damage evolution law of asphalt mixture in service period, the accelerated pavement testing (APT) of graded gravel base asphalt pavement and in-situ pavement was conducted. The full-scale test road of graded gravel base asphalt pavement was built, and the mechanical response sensors and temperature sensors were embedded. The fatigue damage test of the full-scale test road was conducted at 15°C through the accelerated loading test (ALT) equipment. The mechanical responses under the repeated wheel loadings were captured by the strain sensor at the bottom of the asphalt layer, and the deflections of the pavement under different cycles of wheel load were tested by Falling Weight Deflectometer (FWD). The fatigue damage model of asphalt mixture was established, and the accuracy of the model was verified by the accelerated loading fatigue test of in-situ pavement. The results show that asphalt layer bears an alternating effect of tensile and compressive strain, and when the wheel load is reduced or the driving speed increases, the tensile strain at the bottom of the asphalt layer decreases. As an evaluation index of fatigue damage of asphalt layer, the residual modulus ratio can accurately characterize the cumulative degree of fatigue damage. The modulus of asphalt layer shows an inverse S-type curve with the repeated wheel loadings. Based on the quantitative analysis of the whole process of fatigue damage of asphalt layer, a nonlinear damage evolution model of modulus attenuation of asphalt mixture was established, which would provide a reference for the study of pavement service performance attenuation law in different climatic environments and structural compositions in the future.