Non-orthogonal wavelength division multiplexing (WDM) channels are crucial in achieving significantly higher capacity and spectral efficiency than conventional schemes. Additionally, conventional schemes face challenges in electrical processing of wide-bandwidth signals. To suppress crosstalk in non-orthogonal WDM channels, the use of cascaded bandpass filters with suitable transmissivity and bandwidth is investigated for effective channel separation of large-capacity and multiplexed optical signals. This study employs the bandwidth ratio as a parameter, which is defined as the ratio of the channel spacing to the signal bandwidth, to quantify the extent of signal overlapping, and uses optical bandpass filters to filter non-orthogonal signals with a bandwidth ratio ranging from 1.0 to 0.75. The non-overlapping portion is set as the transmission bandwidth of the filter, and the crosstalk is maximally reduced with two-stage filters using Nyquist, super Gaussian, and Butterworth shaped filters. According to the analysis, the combination of Gaussian N = 1 and Gaussian N = 5 exhibits the lowest error vector magnitude (EVM) performance within a bandwidth ratio of 0.8 to 1.0. Simulations showed that a two-stage OBPF with a combination of Gaussian N = 1 and Gaussian N = 5 provides the best EVM performance at a bandwidth ratio between 0.85 to 1.0 within the combination of filter shapes and their orders we used. Additionally, the study includes an analytical investigation of the impact of amplified-spontaneous-emission noise associated with the Erbium-doped fiber amplifier (EDFA) and frequency resolution tolerance of the LCOS filter. The results demonstrate that cascaded EDFAs can be achieved down to OSNR of 10 dB at a minimum resolution of 4 GHz. Finally, the optimal filter shape is analyzed for different bandwidth ratios. The investigation shows that the optimal filter shape is independent of the function used and remains consistent across all bandwidth ratios. The tolerance of the filter is primarily affected by the −0.5 dB transmittance bandwidth between subchannels, which varies depending on the bandwidth ratio. Furthermore, a high tolerance is observed in the −20 dB transmittance bandwidth.