Fault diagnosis of rolling bearings can be a serious challenge, as rolling bearings often work under complex conditions and their vibration signals are typically nonlinear and nonstationary. This paper proposes a novel approach to diagnosing faults of rolling bearings based on variational mode decomposition (VMD) and genetic algorithm-optimized wavelet threshold denoising. First, VMD was used to decompose the vibration signals of faulty rolling bearings into a series of band-limited intrinsic mode functions (BLIMFs). During the decomposition, the parameters of VMD were selected by Kullback–Leibler (K–L) divergence. Then, the effective BLIMFs were determined by the analysis of their correlation coefficients and variance contributions. Finally, genetic algorithm-optimized wavelet threshold denoising was proposed to optimize the selection of important parameters, and the optimized threshold function used not only ensures the continuity of the threshold function but also avoids the fixed deviation of the soft threshold. The validity and superiority of the proposed approach were verified by theoretical calculations, numerical simulations and application studies. The results indicate that the proposed approach is promising in fault diagnosis of rotary machinery.