This paper proposes a spectral element method-based analysis procedure to study further the low-frequency responses of the ground and building induced by high-speed railways. Based on an existing scale model building test in Portugal, the properties of the track–ground submodel are calibrated to determine the moving perturbation on the ground surface. In modeling the soil–structure coupled subsystem, multi-objective optimization is employed to estimate the equivalent structural properties to mitigate computational inefficiency when using continuum elements. The accuracy of the numerical prediction is compared with the experimental records at several monitoring points in terms of ground and floor slab vertical vibrations induced by two types of high-speed trains. In addition to the analysis of possible factors related to systematic discrepancies, special attention is paid to investigating the influence of (a) the presence of frame structure as well as (b) the vehicle properties on ground motion in this particular case.