The safe operation of high rockfill dams is an important issue related to human life and property safety. However, the stability analysis of high core rockfill dams with complex spatial morphology is mostly confined to two-dimensional, and appropriate stability analysis methods are scarce. Therefore, it is necessary to develop the three-dimensional stability analysis method for high core rockfill dams and study the spatial stability mechanism after impoundment. Firstly, a finite element limit equilibrium framework suitable for stability analysis of high rockfill dams is introduced, and the horn-shaped three-dimensional failure mechanism is presented for optimization. Then, according to the statistical results of the actual core rockfill dam above 100 m, 15 high dam models with actual Duncan-Chang parameters between 100 m and 300 m are established to study the three-dimensional instability mechanism. Meanwhile, grey relation analysis (GRA) is used for quantitative analysis of stability sensitive factors. Finally, the stability analysis of the Quxue core rockfill dam (173.2 m) was completed within the established stability framework. The results show that the combined failure mechanism with the horn-shaped is suitable for dam stability analysis, and with the increase of height and slope ratio of dam, the most dangerous sliding surface evolves to shallow layer. In addition, GRA results show that slope ratio is the most sensitive parameter compared to dam height and material parameters. The practical engineering stability results of Quxue show that the research results can provide a reliable reference for the stability of the high core rockfill dam.