Monitoring and predicting seismic responses of a civil structure can be used to assess its behavior under dynamic loading and to determine its structural health condition. In practice, the employed number of sensors is generally limited by the cost and functionality issues. This paper develops a practical solution of four-stage procedures, focusing on prediction of seismic displacement responses at all building floors using acceleration measurements at the optimized sensor locations. In this paper, a novel multi-scale attention-based recurrent neural network is proposed. In particular, the attention mechanisms in the network effectively focuses on more relevant input data among bidirectional ground accelerations and multivariate acceleration responses. The seismic response data for training the developed neural network is generated by performing nonlinear time historical analysis of a three-dimensional finite element model. A floor displacement warping loss is designed to numerically measure the discrepancy between the prediction and the ground truth. A case study is performed using the numerical and real-world data of a high-rise building to systematically evaluate the prediction performance of the proposed methodology. Results demonstrate that the proposed method outperforms the compared state-of-the-art methods in terms of prediction accuracy and reliability.