This paper proposes the use of the deterministic-stochastic subspace identification (DSI) method, an input-output parametric linear system identification method, for characterization of nonlinear dynamic structural systems based on their time-varying amplitude-dependent instantaneous (i.e., based on short time-windows) modal parameters. Performance of the DSI method for estimation of instantaneous modal parameters of nonlinear systems is investigated using numerical as well as experimental data. In this study, DSI is used for extracting instantaneous modal parameters of single degree-of-freedom (SDOF) as well as 7-DOF systems with different hysteretic material behavior. Nonlinear responses of the SDOF and 7-DOF systems are simulated due to different seismic excitations using the OpenSees structural analysis software. Modal identification results are compared with those obtained using wavelet transform and the exact values. Effects of four input factors are studied on the variability of identified instantaneous modal parameters: (1) type of material nonlinearity, (2) level of nonlinearity, (3) input excitation, and (4) length of data windows used in the identification. The accuracy of the identified instantaneous modal parameters is evaluated along the response time history while varying the above mentioned input factors. Overall, DSI outperforms the wavelet transform for short-time/instantaneous modal identification of nonlinear structural systems and provides reasonably accurate results especially when the material hysteretic behavior is smooth such as the considered Giuffré-Menegotto-Pinto hysteretic model. Finally, DSI has been applied for short-time modal identification of a full-scale seven-story reinforced concrete shear wall structure based on its measured response to different seismic base excitations on a shake table. The identified instantaneous natural frequencies of the first vibration mode can accurately track the variation in the structure's effective stiffness along its response.