Research on the SSI effect on flat sites has yielded many valuable conclusions. However, current research on the impacts of various special local terrains on structural dynamics remains limited. For mountainous areas, it is common to construct houses in a multi-step, climbing, and laterally staggered architectural form that follows the mountain terrain. Only through the analysis of the combined action of the upper and lower parts can the seismic performance of this type of structural form be better revealed; considering the influence of SSI effects will be closer to the actual seismic effects. Therefore, to identify the damage factors of the mid-story isolated stilted structures under earthquakes and provide optimized design plans for the structures, six models are established considering three slopes and two types of foundations based on the engineering case in Chongqing, China. Through the elastic-plastic time-history analysis under earthquakes in the down and transverse-slope directions, concludes, compared with not considering SSI, the seismic response of the mid-story isolated stilted structures considering SSI in mountainous areas is amplified. With the increase of the mountain slope, the seismic response of the structures considering SSI increases, and the amplification coefficients are between 1–1.8. The amplification coefficients of the structures without SSI are concentrated around 1, which is less influenced by the slope. The damage to the stilted isolated layer is mainly concentrated in the column and the beam end, and the maximum seismic response appears in the short columns. The foundation soil stress increases with the increase of the mountain slope.