Joint Linear and Nonlinear Equalization Based on Cascaded ANN-MLSE With a Modified Loss Function for PAM-4 Optical Transmission

Abstract

High-speed optical transmission systems suffer from severe inter-symbol interferences (ISIs) and nonlinear distortions due to the limited bandwidth and the nonlinear response of optical and electrical components at the transceivers. Previous work has shown the benefits of applying maximum-likelihood sequence estimation (MLSE) to deal with the ISI and using the neural network-based equalizer (NNE) to mitigate strong nonlinearities. However, most of the NNE has a quasi-hard output as the training and validation loss continue to drop to an over-equalized state in which the quasi-hard output sequence loses the symbol-wise soft information of the signal and fails to obtain the BER performance gain from MLSE. In this work, we investigate the cascaded NNE and MLSE equalization scheme to deal with linear and nonlinear impairments. To solve the over-equalized problem, we propose a modified loss function (MLF) assisting NNE to achieve soft-out without entering into the over-equalized state. With the soft output after NNE, joint linear and nonlinear equalization can be realized by cascading the NNE with MLSE. Both numerical simulations and experimental demonstrations confirm the improved performance of our proposed method. With joint linear and nonlinear equalization, improved back-to-back (BtB) and fiber transmission performances of the 220-Gbps PAM-4 signal are demonstrated.