Due to their ability to boost transmission rate and reach by compensating for Kerr nonlinearity-induced distortions, optical compensation techniques are interesting for broadband wavelength division multiplexing. This is due to the fact that these techniques have been shown to increase transmission rate. Yet, in the majority of instances, they call for more devices to be integrated into the network. This can be challenging for systems that are already in place, and it can be even more challenging for transmissions that are not repeated. The linear and nonlinear compensation system for the Dense Wave Division Multiplexing (DWDM) transmission system has been created and researched in this work. Along the optical link, a new method known as optical phase conjugation (OPC) is used as part of the process of improving the performance of communication systems. This method is used to improve the performance of 896 Gb/s sixteen channels dual polarization signalling of 50 GHz channel spacing over 800 km with three scenarios that use mid and multiple OPC. The results of the simulation indicate that multiple OPC with backward Raman amplifier method on the received signal is effective for nonlinearity compensation when compared with the mid OPC module in terms of enhanced Q-factor improvement, BER performance, EVM, and maximum transmission link length.