One of the core studies on low stacking fault energy (SFE) high-manganese steel is to maintain excellent damping capacity while achieving high strength and toughness. The austenite grain size (AGS) in high-Mn steel affects its SFE and even the strength-plasticity and damping capacity of the steel, but the detailed mechanism related to martensitic transformation has not been clarified. In this study, Fe-17Mn high-Mn damping steel is used as the material to obtain austenite grains of different sizes by controlling the cold rolling and annealing processes. By using in-situ tensile testing, damping capacity testing, and various microscopic analytical techniques, the influence of AGS on the thermal/deformation induced martensitic transformation, work hardening behavior, mechanical properties and damping capacity of the steel is mainly studied. The results indicate that the refinement of austenite grain increases the SFE of high-Mn damping steel and reduces the amount of thermal induced ε-martensite (εth) and stacking faults (SFs). The deformation induced martensitic transformations (DIMTs) in high-Mn damping steel during tensile deformation include γ→εD, εth→α′ and εD→α′ transformation. The deformation induced ε-martensitic transformation is more helpful to improve the work hardening rate than deformation induced α′-martensitic transformation. Due to the concentration of strain partitioning at grain boundaries, twin boundaries and phase boundaries, refinement of austenite grain can promote deformation induced ε-martensitic transformation and improve the steel strength. However, excessively small austenite grains can suppress deformation induced α′-martensitic transformation leading to a significant decrease in elongation. Under the synergetic effects of grain refinement strengthening and multiple martensitic transformations, when the average AGS is about 4.0 μm, the steel has high strength and high plasticity, with a tensile strength of 919 MPa, a yield strength of 518 MPa and an elongation of 46.4 %. The product of tensile strength and elongation is up to 42.6 GPa·%. At low or high strain amplitudes, the damping capacity of this steel shows an opposite trend with the refinement of AGS. The damping capacity of the steel gradually increases with grain refinement at low strain amplitudes of 0.01 %–0.065 %, mainly due to the reduction of the content of weak pinning points. At high strain amplitudes of 0.065 %–0.1 %, the damping capacity of the steel gradually decreases for the reduction of damping sources with the refinement of grain size.