This paper investigates the seismic performance enhancement of steel frame buildings using a novel hybrid self-centering braces (HSBs) under extremely rare earthquake events. The hybrid self-centering brace consists of shape memory alloy (SMA) cables and viscoelastic (VE) dampers. A prototype bracing system is designed and fabricated to explore its basic mechanical behavior and working mechanism under cyclic loading, with a focus on its failure modes under large deformation loading condition. A multi-material mechanical model is developed to capture the mechanical behavior and failure of the HSB. Furthermore, five steel frame buildings with different parameterized HSBs are designed and modeled in OpenSees. Nonlinear dynamic analyses and incremental dynamic analyses are conducted on the five case-study frames using 44 far-field ground motions. The risk-based seismic performances of steel buildings with HSB are evaluated to assess the performance of HSB during extremely rare seismic events. The results show that the hybrid self-centering brace exhibits excellent self-centering and energy dissipation capabilities with the maximum equivalent viscous damping ratio reaching 9.4 %. Even under large deformations, VE dampers continue to work effectively after the failure of SMA cables, demonstrating remarkable redundancy. Numerical simulations further reveal that the redundancy of HSB can improve the structural seismic resilience in terms of inter-story drift ratio, residual drift, and floor absolute acceleration. The higher the redundancy of HSB in the case-study frames, the smaller the seismic response and mean annual frequency of exceedance of the engineering demand parameters, thereby indicating a significant improvement in seismic performance.