Nonlinear phase noise (known as Gordon–Mollenauer phase noise) results from power fluctuations arising from amplified spontaneous-emission noise of the inline amplifier which is converted into phase fluctuations due to the self-phase modulation effect. It may severely impair the phase of optical signal, and would further degrade the performance of the long-haul optical transmission systems. The case can be compensated partially by mid-span optical phase conjugation(OPC). In this paper, an effective k-nearest neighbor (KNN) detector have been introduced to further improve the performance of OPC system by mitigating non-Gaussian impairments caused by nonlinear phase noise. The effectiveness is investigated in a 112-Gb/s 16QAM optical phase conjugation system with 800-km dispersion-managed link and 1280-km dispersion-shifted-fiber link. The results show that the effective and optimal KNN detector can enhance the overall performance of OPC system, even the asymmetric system.