A mixed dual frame-wall resilient system, composed of a high-strength moment-resisting steel frame (HS-MRF) and self-centering steel plate shear walls (SC-SPSWs), was proposed to address the issue of frame expansion. To study the steady-state dynamic responses of the structural system, an analytical solution of a single-degree-of-freedom nonlinear oscillator under harmonic excitation was solved using the method of averaging. The steady-state responses of HS-MRF, self-centering frame and steel plate shear wall that are represented by the bilinear, self-centering and pinching hysteresis, respectively, were studied individually, followed by a full system study. A stability analysis was also carried out to evaluate the behaviour of the singular points. The results show that the hysteretic damping in the bilinear system can significantly reduce the resonance, and the responses are always stable. Jump phenomenon occurs in the self-centering system, and increasing the post-yield stiffness can reduce the nonlinearity, thus decreasing the resonance and unstable region. Unbounded resonance responses appear in the pinching system under large excitation but can change to jump behaviour under small excitation. A left-unbounded unstable region was also found when the unloading force is small. In the combined-hysteresis system, increasing the portion of bilinear hysteresis has negligible effects on the resonance but can reduce the nonlinearity under lower excitation intensities. However, under higher intensities, increasing bilinear hysteresis can significantly decrease peak responses and reduce both the jump and unbounded phenomena. The unstable regions also strongly depend on the ratio of hysteretic models. These findings shed light on the inherent dynamic properties of the dual frame-wall system and suggest that a well-design can achieve a favourable seismic performance.