With the increasingly stringent requirements on energy saving and environmental-friendly for internal combustion (IC) engine, new concepts like novel combustion mode and surface texturing technology have emerged. In order to have deep understanding about the application effects of surface texturing into IC engine with new combustion mode, a mixed lubrication calculation model considering surface textures on cylinder liner together with different combustion modes has been built in the present work based on a four-cylinder diesel engine. The numerical research was conducted on the change of the surface tribological performance of textured cylinder liner under different combustion modes. The simulation results indicate that compared with the conventional combustion (CC) mode, the combustion phase of premixed charge combustion ignition (PCCI) mode and retarded injection (RI) mode varies and their in-cylinder peak pressure is different at the beginning of power stroke due to different injection strategies applied, which lead to significant differences between piston ring radial working loads. Near TDC, the mixed lubrication duration of PCCI and RI mode increases slightly in comparison to that of CC mode, and these three boundary lubrication durations are almost same. However, the thickness of lubricant film between friction pair of three modes is quite different, resulting in their different tribological properties. Among these parameters, the dimensionless total friction force (DTFF) and the dimensionless friction work of PCCI mode increase by 82.37% and 18.41% at most, respectively, while those of RI mode decrease by 36.50% and 12.45% at most, respectively, compared with the same parameters of CC mode. Therefore, the variations in tribological property of the textured cylinder liner–piston ring (CLPR) with different combustion modes should not be ignored.