Atomic decomposition can represent arbitrary signals in an overcomplete dictionary sparsely and adaptively, and it can match the local structure of signals very well. Therefore, it possesses advantages over traditional basis-expansion-based signal analysis methods, in extracting characteristic waveforms from complicated mechanical vibration signals. Periodic impulses characterize damaged gear vibration. In order to extract the transient features of gear vibration, atomic decomposition methods, including method of frames (MOF), best orthogonal basis (BOB), matching pursuit (MP) and basis pursuit (BP), are used in the analysis of vibration signals from both healthy and faulty gearboxes. With a compound dictionary specially designed to match the local structure of signals, the meshing frequency and its harmonics, impulses and transient phenomena of the damaged gear vibration signals are extracted simultaneously. Furthermore, from the time–frequency plots of atomic decomposition, the gear tooth damage is recognized easily according to the periodic impulses. By comparing with traditional time–frequency analysis methods, e.g. short time Fourier transform and continuous wavelet transform, it is found that atomic decomposition is more effective in simultaneously extracting the impulses and harmonic components of damaged gear vibration signals.