In the field of seismic response assessment, simplified procedures have long been pursued to circumvent the rigorous, state-of-the-art nonlinear response history analysis (NRHA) and incremental dynamic analysis (IDA) that typically require tedious computations. To this end, this study investigates artificial neural networks (ANN) as prediction models to bypass IDA and quickly and reliably determine the structural drift responses of buildings, without employing surrogate models or analysis techniques that lower the quality of the estimates. Three designs of such prediction models are identified in terms of their scope, implementation and corresponding database, commonly employed in research, while identifying common pitfalls in their design. The investigations involve ten steel-frame multi-story structures, considering cyclic and in-cycle material deterioration and P-Delta phenomena. More than 17-thousand recorded ground motions (GM) are employed to perform IDA on these frame-building models, resulting in a unique database of responses ranging from linear to collapse, with more than 3-million NRHA. Based on this database, a prediction model is developed for each building, capable of predicting the IDA under a GM excitation not included in the database. Additionally, another prediction model is developed that can predict the IDA of a building under a GM excitation, both of which are not included in the database. These investigations show that ANN are excellent tools to circumvent IDA and capture the record-to-record uncertainty by rapidly and reliably predicting structural drifts ranging from linear responses to the collapse limit state, while building-to-building predictions are additionally feasible if the employed database is appropriate.