To promote filter bank multicarrier (FBMC) in future communication, a novel design method of nonlinear phase FIR filter (NLPFF) is proposed under the principle of constrained minimax (CMM), by which the designed filter will act as the FBMC prototype filter. In this method, the filter coefficient symmetry characteristic (FCSC) is exploited in replace of the conventional phase response error (PRE), which simplifies the establishment of phase constraint and benefits the construction of direct magnitude constraint. Consequently, the proposed constraints related to FCSC outperform the conventional constraints addressing complex frequency response errors, since the former yields a wider range of controllable PRE as well as the dramatically reduced constraint scale. Moreover, in order to restrict the sampling inter-symbol interference, the Nyquist condition (NYQ) is constrained in the frequency domain, which circumvents the initial nonconvex problem. Simulations confirm the effectiveness of the proposed method, where the FCSC can be sufficiently relaxed to improve the stopband attenuation, making this nonlinear-phase design more flexible than previous designs of FBMC prototype filter. Besides, the bit error rate test demonstrates that the designed NLPFFs are superior to the conventional PHYDYAS filter, therefore, the proposed design can certainly produce high quality prototype filter for FBMC system.