The 304L stainless steel is a major component of residual heat removal circuits of pressurized water reactors (PWRs). The main purpose of this study is to understand the risk of thermal fatigue damage resulting from the machining of the 304L steel pipes inner surface (pre-hardening gradient, residual stresses and scratches), at the scale of the microstructure. This work is based on previous results obtained for pipe specimens thanks to a macroscopic elasto-visco-plastic model. Applied to the pipe specimens, this modelling showed that a thermal loading with temperature gradient, induced a cyclic non-linear biaxial loading at the inner surface of the pipe. In this paper, a polycrystal plasticity model, implemented in a Finite Element (FE) code, is adapted to cyclic loading. An elementary volume (3D aggregate), representing the inner surface and sub-surface of the 304L steel tube, is built from successive polishings and orientation mappings thanks to an Electron Back Scattering Diffraction method. At the grain scale, the polycrystal model is used as a “numerical microscope” to compute the local mechanical fields. Different fatigue criteria are tested to determine their sensitivity to surface properties (roughness, residual stress and pre-hardening) and to the microstructure of the material (crystallographic orientation and grain size). Pre-hardening leads to a lower and more homogeneous distribution of local strain amplitudes in the aggregate, but slightly higher stresses when compared to initial material without hardening. By contrast, surface roughness leads to large localized strain and stress fields in grains located at the bottom of scratches. To determine the surface micro-structural “hot spots” features and to test the sensitivity of different surface conditions, three different fatigue criteria (Manson-Coffin, Fatemi–Socie and Dissipated Energy criteria) have been computed. We point out that the pre-hardening may have a complex effect on fatigue resistance, since it reduces local plastic strain amplitudes, but increases local stresses. Moreover, the pre-hardening has a positive effect on fatigue since it delays damage initiation. By contrast, the surface roughness leads to a negative effect. However, we have shown that the three different fatigue criteria do not deliver similar quantitative predictions. Relevant criteria for high cycle fatigue, such as stress based criteria, are not considered in this paper, since the thermal loading used for computation is large enough to reduce cyclic plastic strain straining within all grains of 304L pipe inner surface for midlife of experiments.