Abstract Accurate extraction of abnormal communication signal features in the network is the basis to ensure the completion of network communication. Therefore, a method of feature extraction of abnormal communication signal in network based on nonlinear technology is proposed. In this method, wavelet transform is adopted to decompose the abnormal network communication signals in the high- and low-frequency bands. According to the distribution characteristics of noise and signal in the frequency band, the corresponding parameters are selected for phase space reconstruction and nonlinear dimension reduction of local tangent space mainstream shape recognition algorithm, and the decomposition coefficients of wavelet packet after noise reduction are reconstructed to realize the nonlinear noise reduction of abnormal signal; the denoised abnormal communication signal in network is mapped to the high-dimensional feature space. The principal component is analyzed in accordance with the nonlinear function in the mapped feature space, and the nonlinear function is solved by self-organizing neural network to output the principal component extraction result. According to test results, this method has a significant signal noise reduction effect, results are more than 92% for different abnormal communication signals, and the features of abnormal signals are accurately extracted.