In this work, we developed a series of anisotropic hot-deformed Nd-Fe-B magnets with ultrafine grain sizes varying from 125 nm to 234 nm in the lateral direction. Although grain refinement is known to increase the coercivity of magnets, all samples exhibited identical record high coercivity of 2 T. Microstructural analysis showed similar intergranular phase (IGP) composition and crystallographic texture, but an increased number of coarse and isotropic equiaxed grains in the flake boundary regions of the magnets with larger grain size. Although this could explain significant differences in the initial magnetization curves and magnetic domain patterns, the experimental data alone were not enough to identify the main factors limiting the coercivity. To elucidate these, a micromagnetic simulation was performed using a realistic 1:1 scale model. We showed that grain refinement below 200 nm has a limited potential for coercivity enhancement of hot-deformed Nd-Fe-B magnets unless the exchange interaction is weakened by reducing the magnetization of the IGP. Negative contributions to the coercivity from texture deterioration, coarse grains, and magnetostatic interaction were quantified for different values of IGP magnetization. A clear direction on how to further improve the coercivity of hot-deformed magnets was provided.