Accurately simulating the vibration response of the axle-box bearing with a localized defect on the roller can guide the proposal of several health condition estimation methods. However, there is still a lack of discussion and analysis on the detailed motion process of the roller defect passing through the inner/outer race. For filling this gap, a roller defect model considering time-varying impact is proposed and applied to the vehicle–track coupling system considering axle-box bearings. In addition to the Hertz contact force due to the relative displacement between the roller and race, time-varying impact excitation is designed according to the law of the conservation of energy. At first, an onsite test and an experimental test are implemented to validate the accuracy of the roller defect model and the vehicle–track coupling model, respectively. Then, the effect of the roller defect on the vibration characteristics of the components in the vehicle system is investigated. Finally, the influence of the defect size and vehicle speed on the vibration response of the axle-box bearing is studied, and the change rule of the time-varying impact is also researched with the defect size. Besides, the axle-box vibration response is simulated under variable vehicle speeds, and order tracking technique is applied to analyze the simulated result.