In order to clarify the relationship among grain size, annealing twins and the shape memory effect in Fe–Mn–Si based shape memory alloys, the Fe–21.63Mn–5.60Si–9.32Cr–5.38Ni (weight %) alloy with a grain size ranging from 48.9 μm–253.6 μm was obtained by adjusting the heating temperature or heating time after 20% cold-rolling. The densities of grain boundaries and annealing twins increase with a decrease in grain size, whereas the volume fraction and width of stress-induced ε martensite after 9% deformation at Ms + 10 K decrease. This result indicates that grain refinement raises the constraint effects of grain boundaries and annealing twins upon martensitic transformation. In this case, the ability to suppress the plastic deformation and facilitate the stress-induced ε martensite transformation deteriorates after grain refinement owing to the enhancement of the constraint effects. It is demonstrated by the result that the difference at Ms + 10 K between the critical stress for plastic yielding and that for inducing martensitic transformation is smaller for the specimen with a grain size of 48.9 μm than for the specimen with a grain size of 253.6 μm. Therefore, the shape memory effect declined by decreasing the grain size.