Abstract In order to detect the gear tooth surface wear fault, this paper presents a new fault diagnosis method based on Symlets wavelet family multi-structure element difference morphological denoising and frequency slice wavelet transform (FSWT). Besides considering the gear backlash, time-varying mesh stiffness, gear error and bearing longitudinal response, and low frequency excitation caused by the torque fluctuation, random disturbance of damping gear ratio, gear backlash, excitation frequency, and meshing stiffness are also considered. Dynamics equations of a three degrees of freedom spur gear transmission system with tooth surface wear fault are established according to Newton’s laws. The 4–5 order variable step Runge–Kutta method has been used for solving the equations to get the vibration signal of the system. Then, the proposed method is applied to extract the wear fault signal, which verifies the feasibility and effectiveness of the proposed method.