PurposeUnder the high-speed operating conditions, the effects of wheelset elastic deformation on the wheel rail dynamic forces will become more notable compared to the low-speed condition. In order to meet different analysis requirements and selecting appropriate models to analyzing the wheel rail interaction, it is crucial to understand the influence of wheelset flexibility on the wheel-rail dynamics under different speeds and track excitations condition.Design/methodology/approachThe wheel rail contact points solving method and vehicle dynamics equations considering wheelset flexibility in the trajectory body coordinate system were investigated in this paper. As for the wheel-rail contact forces, which is a particular force element in vehicle multibody system, a method for calculating the Jacobian matrix of the wheel-rail contact force is proposed to better couple the wheel-rail contact force calculation with the vehicle dynamics response calculation. Based on the flexible wheelset modeling approach in this paper, two vehicle dynamic models considering the wheelset as both elastic and rigid bodies are established, two kinds of track excitations, namely normal measured track irregularities and short-wave irregularities are used, wheel-rail geometric contact characteristic and wheel-rail contact forces in both time and frequency domains are compared with the two models in order to study the influence of flexible wheelset rotation effect on wheel rail contact force.FindingsUnder normal track irregularity excitations, the amplitudes of vertical, longitudinal and lateral forces computed by the flexible wheelset model are smaller than those of the rigid wheelset model, and the virtual penetration and equivalent contact patch are also slightly smaller. For the flexible wheelset model, the wheel rail longitudinal and lateral creepages will also decrease. The higher the vehicle speed, the larger the differences in wheel-rail forces computed by the flexible and rigid wheelset model. Under track short-wave irregularity excitations, the vertical force amplitude computed by the flexible wheelset is also smaller than that of the rigid wheelset. However, unlike the excitation case of measured track irregularity, under short-wave excitations, for the speed within the range of 200 to 350 km/h, the difference in the amplitude of the vertical force between the flexible and rigid wheelset models gradually decreases as the speed increase. This is partly due to the contribution of wheelset’s elastic vibration under short-wave excitations. For low-frequency wheel-rail force analysis problems at speeds of 350 km/h and above, as well as high-frequency wheel-rail interaction analysis problems under various speed conditions, the flexible wheelset model will give results agrees better with the reality.Originality/valueThis study provides reference for the modeling method of the flexible wheelset and the coupling method of wheel-rail contact force to the vehicle multibody dynamics system. Furthermore, by comparative research, the influence of wheelset flexibility and rotation on wheel-rail dynamic behavior are obtained, which is useful to the application scope of rigid and flexible wheelset models.