Asphalt pavements are inevitably affected by coupled effects of dry-wet/freeze-thaw cycles, UV radiation, temperature, and loading during service, resulting in degradation of their road performance and reduced service life. Therefore, this study used crumb rubber (CR) and polyphosphoric acid (PPA) as modifiers to prepare CR/PPA composite modified asphalt mixtures for testing. To investigate the mechanical properties of the composite modified asphalt mixtures under the action of multi-factor coupling. Meanwhile, the performance analysis of matrix asphalt and SBS modified asphalt mixtures were carried out as a comparison. Density and water penetration tests, splitting tests, beam bending tests, and four-point bending fatigue tests reveal the evolution of asphalt mixture road performance. Finally, regression fitting was performed on the test data to establish the mechanical damage model of each type of asphalt mixture. The results show that with the increase of wet-dry/freeze-thaw cycles and the increase of UV radiation time, the void ratio and water permeability coefficient increase and the splitting strength, bending and tensile strain, and fatigue life decrease for all types of asphalt mixtures. When the number of wet-dry/freeze-thaw cycles reached 8, the mechanical properties of the asphalt mixture decreased significantly, while the changes leveled off after the number of cycles reached 15. When the UV radiation time is less than 4 months, the mechanical properties of the asphalt mixture decay more slowly. However, with a further increase in radiation time, the mechanical properties decay rapidly. The water-saturated specimens have a more significant effect on the mechanical damage of each type of asphalt mixture. Compared with the other two mixtures, the composite modified asphalt mixtures have superior mechanical properties and fatigue life. The mixture performance damage model based on the least squares Levenberg-Marquardt method (L-M method) was able to simulate the experimental data well, with a minimum correlation coefficient R2 of 0.9285. In summary, the road performance of all types of asphalt mixtures under the coupling of multiple factors has deteriorated, while the composite modified asphalt mixtures have better anti-damage characteristics, with good prospects for promotion.