A gear’s vibration signal consists of multiple components, so it is therefore difficult to accurately extract the transient components of gear faults. Currently, sparse representation is capable of separating fault components from multicomponent noisy vibration signals. However, sparse representation methods still suffer problems with poor computational efficiency and the underestimation of amplitude. To tackle these challenges, this paper proposes a nonconvex regularized sparse representation in a tight frame. The tunable Q-factor wavelet transform (TQWT) is proposed as a sparse dictionary, which can portray the waveform characteristics of the gear’s vibration signal. TQWT satisfies the tight-frame condition, hence it can efficiently reduce the amount of calculations. The minimax concave function is used as the penalty function since it stands out from various penalty functions with the ability to maintain amplitude. The simulation and experimental analysis show that this method has a shorter operation time and a better ability to maintain the amplitude.