The first-order second-moment checking point method was introduced to judge the instability probability and evaluate the stability of the TCURM. The frequency-response stability calculation and the reliability results were compared, and a frequency-response stability and reliability analysis method was proposed. Taking the Zengziyan W12# unstable rock mass in Nanchuan, Chongqing, China, as an example, the calculation shows that the dynamic indexes and geological indexes decrease as the stiffness of the deterioration area decreases. According to the statistical data of the laboratory test and the field investigation, reliability theory is used to evaluate the stability of the TCURM, and the failure probabilities are 80.3% and 96.27% under natural and saturated conditions, which correspond to states of poor stability and instability, respectively. The reliability evaluation results are consistent with the conclusion of the frequency-response stability analysis. The new method can provide a theoretical basis for developing the dynamic monitoring and early warning indicators of the TCURM and disaster prevention and mitigation in mountainous areas.