In this work, the effect of isothermal tempering time and temperature on microstructure and hardness of 3Cr5Mo2SiVN hot-work die steel was studied. The results indicated that with the increase of isothermal tempering time and temperature, the hardness significantly decreased, which corresponded to the microstructural recovery including the transformation of high angle grain boundaries (HAGBs), the precipitation and coarsening of precipitates, the annihilation and rearrangement of dislocations, the formation and coarsening of sub-grains, and the coarsening and coalescence of laths. The driving force of grain boundary transformation was provided by the dislocation- and heat-induced boundary migration. The increased density of HAGBs was induced by the coalescence of low-angle lath boundaries and low-angle sub-boundaries that continuously absorbed dislocations. In addition, the formation and coarsening of sub-grains were the results of dislocation rearrangement with dislocation cells acting as the core. Moreover, the V-rich M(C, N), Fe-rich M3C, (Fe, Cr)-rich M7C3 and M23C6 formed by the desolvation of alloying elements exhibited an obvious coarsening, decreasing the pinning effect of dislocations and then resulting in the hardness loss. This study offered a systematically revelation into the softening mechanism of hot-work die steel.