Three grinding wheels were developed via regulating the abrasive grit size (F10, F16, F30), and their performances are evaluated by a self-designed grinding device. The surface integrity of grinded rails and material removal mechanism were discussed. The results illustrate that the abrasive grit size has a significant effect on the residual stress, surface topography, oxidation and crystal structure of grinded rail surfaces. With the decrease of abrasive grit size, the grinding mechanism gradually changes from rubbing and ploughing to cutting operation. Better surface integrity can be acquired using finer abrasives because of the lower roughness, lower magnitude of residual stress and thinner white etching layers (WELs). And the WELs with nanocrystalline structure could yield the pre-fatigue and abridge the RCF life of rail.