Wheel polygon wear has emerged as a critical issue in the development of high-speed trains in China, but the mechanism underlying its formation remains unknown. The formation mechanism and its effect factors of wheel polygonalization are investigated in depth in this paper using theoretical analysis and numerical simulation. A longitudinal self-excited vibration and vertical modal vibration coupling model of wheel-rail is established, considering the comprehensive effects of wheelset flexibility, vehicle spring mass, and foundation under rail. The critical condition of the system Hopf bifurcation is determined by theoretical analysis, and the longitudinal vibration characteristics of the wheel are studied using numerical simulation. Furthermore, the influence factors on wheel polygonalization are investigated. The research method makes a useful attempt to unify the relevant viewpoints of resonance theory and self-excited vibration theory, and the research results have theoretical guiding significance and engineering reference value for the prediction and suppression of wheel polygonal wear.