We conducted a study on the surface compound modification of shot peening and pure carbon DLC coating to simultaneously meet the requirements of wear resistance and fatigue resistance of spline structure. The effects of surface compound modification were investigated on the surface morphology, residual stress profile, microstructure, and nano-indentation hardness of 16Cr3NiWMovNbE gear steel, and conducted a comparative study on fatigue performance. The results show that the surface compound modification inherits the surface morphology and compressive residual stress gradient of shot peening, while the surface residual stress is slightly smaller than that of shot peening. In addition, surface compound modification still reflects the characteristics of high hardness and high fracture resistance of DLC coatings. Under the bending load based on spline tooth root, compared to the original specimen, the fatigue life after shot peening, pure carbon DLC coating, and surface compound modification is increased by 3.68, 2.35, and 3.36 respectively. Although the compound modified surface still maintains the shot peening morphology with a increasing surface roughness and stress concentration coefficient, the 100 μm-depth compressive residual stress profile and the subgrain refinement layer introduced, as well as the hard surface layer with good load-bearing capacity, have played the role of fatigue strengthening.