In order to reduce the power consumption of digital signal processing (DSP) in a coherent optical communication system, a low complexity equalization scheme in DSP flow of a 400 Gb/s DP-16QAM system has been proposed. This scheme is based on Fermat number transform (FNT), which sequentially performs static equalization (SE) and dynamic equalization (DE) in the transform domain. For different distances, the proposed scheme finds the optimal solution under the condition that transform length and data bit width are mutually restricted under different transmission distances while achieving low complexity and optimal performance. The experimental results show that the adopted transform-domain equalization (TrDE) scheme has much lower computational complexity than the traditional frequency-domain equalization (FDE) and time-domain equalization (TDE) nearly without any performance loss. In the 80, 160, and 240 km scenarios, the number of multiplier is reduced by more than 72%, and the advantage becomes more obvious as the transmission capacity increases.