This paper aims to create a friction coupling finite element model of the wheelset–track system, taking into account various curve radii and corresponding wheel/rail contact states. This model will serve as a tool to analyze the underlying causes of curve squeal phenomena. Furthermore, we conducted an investigation into the impact of rail and wheel wear on system stability. This paper utilized a complex eigenvalue analysis (CEA) method to accurately calculate and predict vibration instabilities in the system. The results show that the smaller the radius of the curve, the more prone to it is lateral creep between the rail and the leading wheelset, so that the system is more prone to self-excited frictional vibrations, ultimately resulting in the generation of curve squeal. In addition, the wear of the wheel tread/rail head increases the instability of the system. When the train passes through a section with rail corrugation, rail corrugation suppresses the occurrence of curve squeal.