The rubber joints mounted on swing arms provide the main part of the yaw stiffness for wheelsets and thus have a significant influence on the dynamic behaviour of railway vehicles. However, their nonlinear dynamic properties concerning the frequency, amplitude, and temperature were not fully considered in previous studies, most of which adopt simple models such as the Kelvin–Voigt model to represent rubber components. This study aims to investigate the radial nonlinear properties of swing arm rubber joints under various conditions and assess their effects on the dynamic performance of high-speed trains. A nonlinear rubber spring model which combines the fractional derivative Zener model with Berg’s friction model is established. To achieve a better fit to the measurements of a swing arm rubber joint, an optimisation-based method is employed to identify the model parameters. The viscous and frictional effects of the rubber joint are compared at different frequencies, amplitudes, and temperatures, by determining the predominant element of the nonlinear model. Additionally, the nonlinear rubber spring model is integrated into a high-speed vehicle dynamic model to investigate the effects of the nonlinear properties of swing arm rubber joints on the vehicle dynamic behaviour through the MATLAB/SIMPACK co-simulation method.