This paper presents an analytical method for the instability analysis of an axially compressed rail modeled as an infinite beam resting on a viscoelastic foundation and subjected to a moving vehicle modeled as a single-axle mass-spring-damper system. The vibration equations of the rail and vehicle are first derived. By using Laplace and Fourier transformations, the expressions of the Laplace images of the vibrations of the rail-vehicle interaction are obtained. Then the D-decomposition technique is employed to analyze the instability of the vibration system. It is shown that instability occurs for lower masses as the compression axial force increases. It also is shown that the critical mass in the case of the moving vehicle problem is always less than that in the case of the simple moving mass problem. The steady-state response of the rail is found to depend on the total vertical force acting on the moving vehicle, but it is independent of the masses and other parameters of the moving vehicle. Instability of the rail is possible, although the foundation is overdamped.