When an earthquake causes the wheel-rail detachment of a high-speed train, the derailment evaluation criteria based on wheel-rail force faces failure. At this time, the lateral relative displacement of the wheel-rail (LRDWR) is more intuitive and continuous to assess the running safety of a high-speed train. In the past, many experimental studies have been conducted on measuring the LRDWR of trains. However, there is still a lack of effective measurement methods to solve the problem of LRDWR measurement in complex texture and high-speed dynamic scenes. Therefore, a lateral relative wheel-rail displacement measurement method based on machine vision and the improved Hourglass network (LRWRDVH) is proposed in this paper to determine the LRDWR by predicting the pixel coordinates of a few wheel-rail keypoints. The scaled-down bridge running test of the high-speed train is designed for validation, and then the LRDWR measurement of high-speed train under earthquake is carried out using the LRWRDVH method to evaluate the running safety of high-speed train. The test results show that the LRWRDVH method is effective and accurate and can be further used to study the running safety of high-speed trains under earthquakes.