To understand intuitively the residual stress state on the rail surface after abrasive belt rail grinding (ABRG), the influences of grinding process parameters on residual stress were experimentally investigated on the ABRG test bench. Tensile residual stress was found in the grinding direction, while the residual stress in the radial direction was maintained mainly in the form of compressive stress. To investigate the mechanisms of the influencing factors during the forming process of residual stress, a 3D finite element model (FEM) of grain scratching based on thermo-mechanical coupling method was developed. Effects of contact surface friction, grain’s tip radius, grain’s protrusion depth, and grain’s rake angle on residual stress distribution in rail sub-layer were revealed, respectively. In addition, the FEM simulation of residual stress involving adjacent grains scratching was carried out, in which the variation of the residual stress field between the on scratching and the subsequent scratching was observed and discussed. Finally, the suggestions for the design of the last grinding unit and grinding process parameter selection were given based on the findings from the experiment and simulation.