The weak story failure often occurs in steel frame structures and the collapse risk of the frames will increase because of concentrated plastic deformation in the soft story. The work presented in this paper aims to avoid the soft story mechanisms and the improve self-centering capacity of steel frames by employing the innovative rocking truss. Compared with the previous rocking system, it is more flexible in application and suitable for retrofit of existing buildings. The theoretical approach to quantifying the stiffness demand of the rocking truss was proposed and presented two critical parameters (equivalent bending stiffness ratio η and the rotation stiffness ratio λ) for the design of the rocking truss. The 1/3 scaled steel frame with rocking truss (SFRT) was tested and theoretical analysis results were validated by test results. The finite element model of SFRT was built following the boundary conditions of the test specimen. The numerical results are consistent with the experimental observations. The nonlinear static and dynamic analysis results of the SFRT were compared with those of the same steel frame (SF) with a bracing system. It was found that the rocking truss can reduce the maximum and residual drifts of the main frame as expected, and the plastic development of the RBS at the beam end in SFRT is less than that in SF with a bracing system. The parametric analysis of the rocking connections with different values of prestressing force was carried out. The results confirm that the rotational stiffness and load-bearing capacity of the rocking connection can be accurately evaluated by the theoretical force-displacement curve. The effect of the equivalent bending stiffness ratio η and the rotation stiffness ratio λ on the lateral deformation pattern of structures was discussed by theoretical analysis, and the design procedure of the rocking truss is recommended to satisfy the demand of uniform inter-story drift and self-centering capacity.