Rock asphalt which is a natural asphalt, along with other modifiers, can be used to improve the performance of asphalt binder from crude oil. In this research, a systematic study was conducted to investigate the fatigue failure evolution behaviors of rock asphalt composite modified asphalt (RACMA). The fatigue performance of the modified asphalt subjected to rolling thin film oven (RTFO) and pressurized aging vessel (PAV) was explored to determine and assess various fatigue life indices. The applicability of various fatigue life indexes (Nf50, Nfm, Nδ, NR, and Np20) which are determined according to five different fatigue failure criteria were evaluated through statistical analysis and geometric position method. The results indicated that the Nf50 and Nfm in the stress-controlled mode and the Nf50 and NR in the strain-controlled mode of time sweep test exhibited comparable fatigue lives, respectively. The index Nf50 was recommended to estimate the fatigue performance in both stress and strain-controlled modes of time sweep test. Further, it was found that the optimal dosage of rock asphalt (RA) and styrene-butadiene-styrene (SBS) was 5% and 2% respectively in the composite modified asphalt (RA/SBS) in stress-controlled mode, while the content of RA and crumb rubber (CR) in the composite modified asphalt (RA/CR) was 5% and 18% respectively in strain-controlled mode to achieve maximum fatigue lives. The reduction of RA content and the increment of CR content can significantly improve the fatigue resistance of the RA/CR and extend its fatigue life by >5 times. The RACMA generally exhibits good fatigue resistance after RTFO aging.