In the aspect of gearbox fault diagnosis, the periodic pulse signal containing fault characteristics is often overwhelmed with other strong interference components, which brings a great challenge for gearbox fault detection and status identification. To address these issues, this paper develops a novel resonance-based sparse signal decomposition (RSSD) fault diagnosis method combined with support vector machine (SVM). Based on the key of the decomposition parameters in the resonant sparse decomposition method, the influence of atom search optimization (ASO) on the quality factor in the resonance-based sparse signal decomposition method is primarily studied. The vibration signal of the gearbox was decomposed by resonance sparse decomposition with the optimal quality factor, from which the high and low-resonance components were obtained. Then, the power spectrum entropy, singular spectrum entropy, and time-domain energy entropy of the low-resonance signal were calculated. Finally, the pattern recognition of gearbox fault was completed with a designed cross-validation SVM pattern recognition model. The numerical simulation and gearbox fault experiments demonstrate that the presented method can achieve great recognition accuracies and effect capacities against interference components involved in the gearbox vibration signal.