Design of SR-NYQ prototype filter in an FBMC system

Abstract

As a candidate for next generation mobile communication, filter bank multicarrier (FBMC) has several advantages over orthogonal frequency division multiplexing (OFDM), such as larger side-lobe attenuation and no requirements of strict orthogonality and synchronization, in which the prototype filter for subcarrier shaping plays an important role. Currently, the commonly used prototype filters include the PHYDYAS filter and the IOTA filter, but in the corresponding design of both, there are few controllable parameters and the filters flexibility is insufficient. Therefore, a constrained optimization method to directly design the FBMC filter is proposed herein. In this method, the error energy in the stopband is considered the optimization target, which is aimed at reducing the side-lobe leakage of the FBMC waveform. Furthermore, the constraints on the Nyquist (NYQ) condition are implemented in the frequency domain via approximation, through which the control of inter-symbol interference can be achieved. These NYQ constraints contain linear matrix inequalities and a linear inequality for the cut-off frequency point, which circumvent the nonconvex problem caused by the convolution calculation in original constraints. After solving the constrained optimization via semi-definite programming, the designed FIR filter with linear phase can effectively improve stopband attenuation characteristics when the NYQ condition is suitably relaxed, thus improving the frequency selectivity of the FBMC waveform. The bit error rate (BER) simulations indicate that the optimized filters have a certain superiority over the PHYDYAS and IOTA filters. In addition, using the proposed method, the designed filter with shorter length can also result in a BER comparable to that of the PHYDYAS filter, with a multiplication complexity is lower than that of the PHYDYAS filter.