In this paper, we study the application of frequency response masking (FRM) technology in polyphase discrete Fourier transform (DFT) filter bank (FB). Due to the inability of FRM methods with conventional structure to simultaneously meet the needs of sharp transition band polyphase DFT FB to improve the signal-to-interference ratio of channelized signals and reduce the complexity of FBs, an improved conjugate frequency modulation frequency response masking (CFM-FRM) method is proposed. The proposed CFM-FRM method solves the problem by frequency modulation, and consumes less multipliers by taking advantage of the orthogonality between sub-filters in CFM-FRM FB. The efficient realization of CFM-FRM based polyphase DFT FB is derived and the multi-size configuration method is illustrated via coefficient decimation. We analyze the adaptability of related FRM works to polyphase DFT FB. The simulation and analysis show that the CFM-FRM method is applicable to polyphase DFT FB, the CFM-FRM based polyphase DFT FB has less multipliers than related methods in other works, and the performances of multi-size FBs meet the requirements.