The unequal-height pier configurations commonly exist in bridge engineering for the adaptation of the varying topography. However, this makes the stiffness of unequal-height piers fluctuating due to the pier's height difference. As piers are the main seismic-resisting components under earthquakes, the irregular stiffness distribution induced by unequal-height pier configurations are unfavorable to the seismic performance of bridge structures. Especially, the high-speed railway (HSR) bridges, which are designed to satisfy strict requirements of structural stability, integrity and smoothness allowing for high velocity, high comfort, high security, are more likely to be adversely affected by the unequal-height pier configuration. Additionally, previous study showed that the track structure that built on the HSR bridge girders is prone to be seismically damaged. Based on this, this paper studies the influence of the unequal-height pier configurations on the seismic displacement responses of HSR bridge-track system, and investigates the damage mechanism. The seismic performance of HSR bridges with different pier configurations are also evaluated from the perspective of probability. Piers with 8 m, 16 m and 24 m are utilized to constitute the symmetric and asymmetric layouts of equal-height and unequal-height pier configurations. Six numerical models of bridge-track system (BTS) are established based on the test data and previous researches. The analysis on seismic responses of six BTS models shows that seismic displacement responses of moveable bearings, sliding layer and shear alveolar rises with the increase of adjacent pier's height difference; It is found that unequal-height piers are more likely to produce large nonuniform displacement responses, which further causes larger displacement responses of components above piers. The moveable bearings are the least fragile in the equal-height pier configuration. However, the sliding layer and the shear alveolar are the most vulnerable with considerable displacement responses in the layout of 8 m and 24 m piers, and become less fragile in the unequal-height pier configuration of 8 m and 16 m piers. In terms of structural safety, the equal-height pier configuration has the least adverse influence on the seismic performance of HSR bridges.