In this work, we studied the effect of initial grain size on the hot compression characteristics of super-304H austenitic stainless steel in a range of strain rates (0.001–1 s−1) at a fixed temperature of 1223 K. Analysis of the flow curves reveals that the flow stress is inversely proportional to the average sub-grain diameter in both coarse and fine-grained specimens at higher strain (≥0.5) levels. Further, the fine-grained specimen following deformation at a low strain rate (0.001 s−1) reveals the occurrence of both continuous dynamic recrystallization (CDRX) and discontinuous dynamic recrystallization (DDRX) mechanisms. In this condition, the CDRX is characterized by a progressive increase of boundary misorientation, while the DDRX is characterized by bulging and strain-induced boundary migration. In contrast, the CDRX characterized by the formation of microbands is majorly responsible for the grain refinement in the fine-grained specimen at higher strain rates (∼1 s−1) and in the coarse-grained specimen at all strain rates (0.001–1 s−1). The superposition of different DRX mechanisms leads to significant variations in grain refinement kinetics with strain rates and initial grain sizes. The findings of this investigation provide a foundation for the accurate control of the microstructures in the studied alloy with different initial grain sizes during the hot working at various strain rates.