Fatigue damage can develop within asphalt pavement due to the continuous impact of driving loads and natural elements. Understanding the process of asphalt mixtures’ fatigue damage is crucial for guiding the design, maintenance, and repair of asphalt pavement. This research aims to establish a model that characterizes the mixtures’ modulus decay under a three-dimensional (3-D) stress state. Firstly, asphalt mixes were subjected to direct tensile (DT), indirect tensile (IDT), unconfined compressive (UC) strength and fatigue tests, and the resulting data were analyzed. Then, modulus decay models under DT, IDT, and UC conditions were established, and the modulus decay patterns under the three loading modes were compared and analyzed. Finally, using the fatigue stress strength ratio Δ (a fatigue resistance index for asphalt mixtures that takes into account the impacts of stress state and loading rate), a unified characterization model for asphalt mixes’ modulus decay under a 3-D stress state was created. According to the study’s findings, asphalt mixes’ modulus decay during fatigue damage exhibits obvious nonlinear characteristics. While the asphalt mixes’ modulus decay law with various loading modalities is similar under the same conditions, the decay rate may differ. Essentially, the speed of the modulus decay of a certain asphalt mixture primarily depends on the value of Δ during service. A larger Δ indicates a faster modulus decay. This study offers a theoretical foundation for the conversion from material fatigue damage to structural fatigue damage, which is vital for enhancing the asphalt pavements’ construction quality and longevity.