Abstract Security and reliability are important issues that must be paid attention to during the operation of rotating machinery. If defects can be found in the early stage, there will be enough time to take maintenance measures and realize the stable operation of equipment. However, the presence of noise, shaft rotation signals, gear meshing signals, and other interfering factors often obfuscate fault signals, rendering the early detection of defects an arduous undertaking. Against this backdrop, this study presents an advanced approach for early defect detection, integrating the virtues of variational mode decomposition (VMD) and multiscale singular value decomposition (MSVD). Initially, a novel evaluation index is constructed by combining envelope entropy and envelope spectrum sparsity. Based on this a method is proposed to adaptively determine the critical parameters of VMD, enabling the adaptive decomposition of vibration signals into a series of modal components. The optimal sensitive components are then discerned utilizing the characteristic frequency intensity coefficient index. Subsequently, to address the limitations of single VMD methods in effectively suppressing low-frequency noise, the MSVD method is proposed for effective noise reduction, which reconstructs the signal after SVD of the signal within each segment through the operation of successive signal segmentation. Ultimately, envelope spectrum analysis is conducted on the reconstructed signal, facilitating the precise extraction of fault characteristic frequency information and enabling early fault identification. The efficacy of this novel methodology is evaluated through simulations and actual vibration signals, successfully discerning early faults afflicting rotating machinery.