It is critical to detect hidden, periodically impulsive signatures caused by tooth defects in a gearbox. A hybrid demodulation method for detecting tooth defects has been developed in this work based on the variational mode decomposition algorithm combined with modulation intensity distribution. An original multi-component signal is first non-recursively decomposed into a number of band-limited mono-components with specific sparsity properties in the spectral domain using variational mode decomposition. The hidden meaningful cyclostationary features can be clearly identified in the bi-frequency domain via the modulation intensity distribution (MID) technique. Moreover, the reduced frequency aliasing effect of variational mode decomposition is evaluated as well, which is very useful for separating noise and harmonic components in the original signal. The influences of the spectral coherence density and the spectral correlation density of the modulation intensity distribution on the demodulation were also investigated. The effectiveness and noise robustness of the proposed method have been well-verified using a simulated signal compared with the empirical mode decomposition algorithm associated with modulation intensity distribution. The proposed technique is then applied to detect four different defects in a multi-stage gearbox. The results demonstrated that the demodulated numerical information and pigmentation directly illustrated in the bi-frequency plot of the modulation intensity distribution can be successfully used to quantitatively differentiate the four gear defects.