Recurrent neural networks achieving MLSE performance for optical channel equalization.

Abstract

We explore recurrent and feedforward neural networks to mitigate severe inter-symbol interference (ISI) caused by bandlimited channels, such as high speed optical communications systems pushing the frequency response of transmitter components. We propose a novel deep bidirectional long short-term memory (BiLSTM) architecture that strongly emphasizes dependencies in data sequences. For the first time, we demonstrate via simulation that for QPSK transmission the deep BiLSTM achieves the optimal bit error rate performance of a maximum likelihood sequence estimator (MLSE) with perfect channel knowledge. We assess performance for a variety of channels exhibiting ISI, including an optical channel at 100 Gbaud operation using a 35 GHz silicon photonic (SiP) modulator. We show how the neural network performance deteriorates with increasing modulation order and ISI severity. While no longer achieving MLSE performance, the deep BiLSTM greatly outperforms linear equalization in these cases. More importantly, the neural network requires no channel state information, while its performance is comparable to conventional equalizers with perfect channel knowledge.