5G network is a heterogeneous large-scale network. Cognitive Radio (CR) technology can be used to realize selection based on the communication time, communication resources, and communication requirement so as to improve the system performance of the whole communication system. Cognitive Radio (CR) system based on wavelet packet owns better flexibility and bandwidth efficiency. The optimal wavelet packet filter optimization algorithm is proposed in the paper for guaranteeing un-licensed user’s (un-LU) data transmission rate and optimizing the performance of system. The intelligent search algorithm is used to obtain the optimal wavelet filter. The simulation results show that the Intercarrier Interference (ICI) and bit error rate (BER) performance of the new optimal wavelet filter algorithm without sacrificing any un-LU’s subcarriers is better than other three masking subcarriers algorithms.