In mixed lubrication, the interplay of lubricant flows, solid asperity contact, and material wear between tooth surfaces creates complex and unpredictable contact states on tooth surface. To comprehensively understand the interaction between the lubrication and wear characteristics of the rough tooth surfaces of helical gears, this study established a mixed lubrication sliding wear calculation model for helical gears based on the mixed elastohydrodynamic lubrication model and Archard’s model. Specifically, the study aimed to examine the effects of surface topography features on average film thickness, contact area ratio, and accumulated wear at the meshing point. The findings demonstrated that the texture and power spectral density distributions of a non-Gaussian reconstructed surface closely resembled those of the actual ground surface. Furthermore, for non-Gaussian rough surfaces, a larger wavelength ratio enhanced microwedge motion, which increased film thickness and reduced wear. Additionally, a negatively skewed surface demonstrated better lubrication performance compared to both positively skewed and Gaussian surfaces. This improved performance is evident in the smaller contact area ratio and lower accumulated wear value of the negatively skewed surface.