The use of a rapid heating method to achieve heterogeneity of Mn in medium-manganese steel and improve its comprehensive performance has been widely studied and these techniques have been widely applied. However, the heating rate (from α to γ) has not received sufficient attention with respect to its microstructure-evolution mechanism. In this study, the effect of heating rate on the microstructure evolution and hardness of heterogeneous medium-manganese steel was investigated by using X-ray diffraction (XRD), scanning electron microscopy (SEM), electron backscatter diffraction (EBSD), transmission electron microscopy (TEM) and DICTRA simulation. The results showed that the Mn distribution was heterogeneous in the initial microstructure of pearlite due to strong partitioning of Mn between ferrite and cementite. At low heating rates (<10 °C/s), the heterogeneity of Mn distribution was diminished to some extent due to the long-distance diffusion of Mn in high-temperature austenite. Contrastingly, at high heating rates, the initial heterogeneity of the Mn element could be largely preserved due to insufficient diffusion of Mn, which resulted in more ghost pearlite (GP: pearlite-like microstructure with film martensite/RA). Moreover, the high heating rate not only refines the prior austenite grain but also increases the total RA content, which is mainly composed of additional film RA. As the heating rate increases, the hardness gradually increases from 628.1 HV to 663.3 HV, due to grain refinement and increased dislocation density. Dynamic simulations have also demonstrated a strong correlation between this interesting microstructure and the non-equilibrium diffusion of Mn.