Recent experiments show that super-elasticity degeneration occurs during the cyclic deformation of polycrystalline NiTi SMAs and the extent of such degeneration depends strongly on the grain size of the polycrystalline aggregate. Thus, in this work, a micromechanical model is constructed to describe the grain size dependent super-elasticity degeneration of polycrystalline NiTi SMAs. The polycrystalline aggregate is modeled as a composite material, i.e., each grain-interior (GI) phase is assumed to be a spherical inclusion embedded in a matrix of grain-boundary (GB) phase. For GI phase, two inelastic deformation processes, i.e., martensite transformation and transformation induced plasticity, and their interaction are considered, simultaneously; for the GB phase, its plastic deformation is described by a unified visco-plastic model. To describe the interaction between the inclusion and matrix and calculate the macroscopic overall performances of the modeled composite material, a modified incremental Mori–Tanaka's homogenization method is proposed by introducing the Eshelby's tensor of a spherical inclusion embedded in a finite spherical domain and the affine tangent modulus and affine strain increment. Numerical algorithm is developed to implement the proposed micromechanical model. A new and simple affine linearization method is proposed to linearize the constitutive equations of GI and GB phases. The proposed micromechanical model is verified by comparing the simulated results with the corresponding experimental ones. It is shown that the grain size dependent super-elasticity degeneration of NiTi SMAs can be well captured. Moreover, the calculated results indicate that when the average grain size is large (>160 nm), the super-elasticity degeneration originates from the interaction between the martensite transformation and dislocation slipping in GI phase; while, when the average grain size is small (<160 nm), the super-elasticity degeneration is dominated by the plastic deformation of GB phase and the stress redistribution between GI and GB phase.