Early and accurate damage evaluation after earthquakes is critical for planning an efficient and timely emergency response. State-of-the-art rapid evaluation techniques of structural damage include the use of fragility or vulnerability curves. However, fragility-based damage functions may vary significantly, depending on the ground motion characteristics, soil conditions, and structural geometric properties. A novel stacked long short-term memory (LSTM) network with overlapping data was developed in this study to overcome this issue. The ground motion time histories are divided into several stacks and feed to the LSTM network, and the data are overlapped with the preceding stack to link each stack. The stacked LSTM reduces the temporal dimension by stacking and generating new features, and shortens the time required for training. The proposed network significantly reduces the training time required (approximately 97%) and enhances the test accuracies (80%–95%) as the number of stacks increases. OpenSees is utilized for the creation of the numerical model of ductile frames (using concentrated plasticity modeling approach) and nonductile frames (using distributed plasticity modeling approach). Although these structures have different response mechanisms, the proposed LSTM network shows the diversity in predicting the earthquake-induced damage with a high degree of accuracy (80%–95%). The performance of the proposed model on different types of structures (nonductile and ductile building frames and a non-ductile bridge) with the same network shows the flexibility of the model.