In this paper, the grain size effect on martensite transformation and its mechanical properties were investigated in two high-carbon steels. The results show that grain refinement can induce a phase transformation of high carbon martensite substructure from twin to dislocations. When grains are refined to smaller than 4 µm, no twin could be observed, instead, full dislocation substructure is obtained. And the martensite substructure transition results in an enhancement in mechanical properties, an ultrahigh tensile strength of 2.28 GPa and a significant elongation of 6.8% are obtained in the 0.61 wt%C steel. A model based on the relationship between dislocation slip stress (τS) and twin shear stress (τT) with grain size (d) is developed to explain the above phase transformation phenomenon. The τT is proportional to d−1 and τS is proportional to d−1/2, both of which increase with reducing grain size, but τT is more sensitive to grain size. Therefore, a critical grain size (dc) is obtained, at which the stresses for twining and for dislocation gliding are equal, lower than this value (i.e. grain is finer), the stress for twin is higher than that for dislocation gliding, so dislocation slip becomes main deformation process in martensite transformation. The theoretically calculated dc is about 2.7–7.6 µm, which is in good agreement with the experimental results. These findings may provide a new way to design ultra-high strength and high ductility steels.