A widely used approach for extending the fatigue life of engineering steels is through the introduction of compressive residual stress (CRS). The thermal stabilities of the introduced residual stress may determine their effectiveness in high-temperature application, which had been extensively studied by ex-situ technology in the past two decades. Here, the in-situ heating techniques of synchrotron XRD, XRD and TEM were used to study the evolution of CRS and corresponding microstructural evolution in a gradient nanostructured (GNS) austenitic stainless steel. As the temperature increased upto 500 °C, the magnitude of CRS decreased continuously by 63% when the phase composed of 100% strain-induced α'-martensite in the GNS layer, while it was decreased by 23% in the dual-phases composed of 60% α' and 40% γ. This result was attributed to the fact that the CRS introduced by the α'-martensite belongs to the type III micro-stress and was stored in the dislocations, which shows a high recovery trend upon heating. However, the CRS in the dual-phase structures is the type II intergranular stress, which is relatively stable upon heating. The current work suggested the importance of α'-martensite on the stabilizing of CRS, which may guide the materials design for high-temperature application.