Finite element model updating is significant in damage detection, parameter identification and structural control, yet practical applications to large-scale structure reveal a major defect in computational efficiency. In this paper, an improved substructure-based response sensitivity method is presented to accelerate the convergence of model updating. The equivalent mode is derived to relate higher modes to lower modes. The motion equation is reduced and simplified using the equivalent mode and a small number of lower modes in each substructure. The structural response and response sensitivity are then computed efficiently from the small-sized motion equation. The inertial effect of the discarded higher modes is compensated by deriving an equivalent modal mass matrix related to mass matrix and equivalent mode in the reduced mass matrix. This compensation helps ensure the accuracy of structural response and response sensitivity, avoiding retaining numerous lower modes. The substructure-based response sensitivity method is later developed to conduct the finite element model updating of large-scale structures. The accuracy and efficiency of the proposed method are finally tested by two numerical simulations and one experimental case.