Si-modified Fe–Cr–Ni austenitic stainless steel is a promising candidate structural material in lead-cooled fast reactor systems and high-temperature microstructural stability is the challenge for developing this kind of materials. In the present research, the microstructure and corresponding impact toughness evolutions of a 3.6 wt%Si austenitic stainless steel (3.6Si-ASS) aged between 510 and 650 °C up to 5000 h were investigated. It was found that the microstructure showed nearly no change within 2000 h at 510 °C. Subsequently, the G-phase (Cr3Ni2Si) began to form at grain boundaries after 3000 h. This made the matrix near grain boundaries Ni-depleted and promoted the formation of ferrite. Finally, a mixed microstructure consisting of G-phase and ferrite was formed after 5000 h. The appearance of ferrite caused a slight decrease in impact toughness. This evolution was further accelerated during aging at 550 °C, and the impact toughness showed a significant decrease when ferrite appeared. Interestingly, the G-phase and ferrite were suppressed, replaced by intergranular M6C ((Cr, Mo)3Ni2SiC) carbide and intragranular (Cr, Si, Mo)-rich χ-phase during aging at 600 and 650 °C. Herein, M6C carbides preferred to form at grain boundaries, making intergranular cracking easier and reducing the toughness. When aged at 650 °C, the higher diffusion rates of Cr, Ni, Si and Mo increase grain boundaries coverage of M6C and also the number density, diameter, and area fraction of the χ-phase.