Feature extraction is one of the core problems in condition monitoring and fault diagnosis of mechanical equipment. In this study, an OMP-NWE method of feature extraction is proposed, aiming at the problem of low accuracy of existing feature extraction method. The OMP-NWE method integrates the strengths of orthogonal matching pursuit (OMP) algorithm with the benefits of nonparametric waveform estimation (NWE). Signal feature extraction model is constructed by design of filter bank and adaptive template signal. Then the vibration signal is linearly decomposed into a set of best matching waveform, which solves the problem that the basis function must be chosen in advance in OMP algorithm. The OMP-NWE method is applied to the feature extraction of the simulation and experimental vibration signal of rolling bearing, compared with the traditional OMP algorithm. Results show that the SNR of the extracted feature signal using OMP-NWE method increased by 19.22 % compared with that using the OMP method, which illustrates that OMP-NWE method has a higher accuracy in the feature extraction of unknown complex vibration signals. This work provides a new idea and a successful example for the feature extraction of vibration signal in the condition monitoring and fault diagnosis of mechanical equipment.