Cement and emulsified asphalt (CA) mortar is a critical viscoelastic component of ballastless track systems, exhibiting pronounced temperature sensitivity. However, accurately predicting the mechanical responses of CA mortar due to temperature fluctuations remains a challenge. This work proposes a novel variable-order fractional Maxwell (VOFM) model to investigate the viscoelastic properties of CA mortar under temperature variations. The VOFM model is numerically solved using a time-varying coefficient ladder (TVL) model. The VOFM model is validated by comparing with both numerical integration methods and experimental data. The basic properties of the VOFM model, the viscoelastic responses of CA mortar under varying temperatures and alternating stress are systematically investigated. The results show that the mechanical properties of VOFM model agree with the FM model when time-dependent parameters of the VOFM model correspond to that of the FM model. Moreover, high temperatures have a promoting effect on the deformation of CA mortar under both alternating and constant stress conditions. The peak amplitude of CA mortar strain under practical varying temperature and stress loading correlates with the trend of temperature changes.