Optical filters are crucial elements in optical communication networks. However, they seriously affect the signal quality, especially in the concatenation condition. In this paper, we study and simulate the signal penalties induced by five types of filters, including Butterworth, Gaussian, Bessel, Fiber Bragg Grating (FBG) and Fabry–Perot (F–P) filters, in order to optimize the optical network performance. Signal penalties, including both filter concatenation effect and filter induced in-band and out-band crosstalk, are analyzed by eye opening penalty (EOP) and Q-penalty. Simulation results show that the Butterworth filter performs best among these four types of filters. Total Q-penalty induced by a Butterworth filter-based demultiplexer/multiplexer pair, considering both filter concatenation effect and crosstalk, is lower than 0.5dB, when the filter bandwidth is in the range of 42–46GHz. Simulations done in a 100Gbps PM-DQPSK optical network indicate that the permitted cascaded number is 14 and 10 with respect to the case of filters aligned and misaligned, respectively. Penalties induced by laser frequency shift are investigated and the performance of the 3rd-order Butterworth filter is compared to the 4th-order superGaussian filter. Finally, discussions of optical filter performances, including insertion loss, reconfigurability and programmability, are presented.