In order to investigate the mechanism of the frequency-dependent viscoelasticity of the rail pad on the acoustic radiation characteristics of a box girder viaduct, this study establishes a high-order model of its dynamic parameters to reveal the frequency-varying viscoelasticity of the rail pad, and establishes a vehicle–track–viaduct vertical coupling model. Finally, the acoustic radiation characteristics of a box girder viaduct are analyzed by combining the finite element method and the boundary element theory. The results show that the S-stiffness and D-stiffness of the rail pad increase with the increase in frequency, and the frequency sensitivity of the S-stiffness is greater than that of the D-stiffness. The high-order characterization model of the dynamic parameters of the rail pad has a good fitting effect. The main influence frequency band of the frequency variable viscoelasticity of the rail pad on the wheel–rail force and the equivalent discrete spring force of the sliding layer is 30–90 Hz, resulting in the shift of the dominant frequency to a high frequency by 4 Hz. We consider that the frequency-varying viscoelasticity of the rail pad will cause the dominant frequency of the acoustic pressure level of the field point to shift to a high frequency of 4–6 Hz, which has the greatest influence on the sound pressure level of each field point at the Peak Frequency Point of Insertion Loss (PFPIL), and the influence degree is consistent, resulting in the maximum value of the total sound pressure level of the surface field increasing by 4.1 dB. Without considering the frequency-varying viscoelasticity of the rail pad, the sound pressure level of each field point at 20–53 Hz will be overestimated and the sound pressure level of each field point in the 53–100 Hz frequency band will be underestimated. The panel sound power level contribution coefficient of the box girder is obviously different at different frequency points.