The long-distance high data rate transmission is accompanied by many transmission defects, which reduce the performance of optical communication systems. Therefore, due to the need to increase data capacity for the end user, signal regeneration techniques were used to mitigate these non-linear defects in the electric or optical field. Several techniques used to modify the optical signal, which is the possible solution to reduce the limit of non-linear transmission defects, especially before converting the signal from the optical field to the electric field. Since most of the techniques used to reduce the attenuation in the optical signal do not have the ability to reduce the error rate to reach a bit error rate (BER) less than 10-10 with a transmission capacity less than -10dBm. In this paper, the performance evaluation of DPSK has been validated for its adequate use in transmission systems. Nevertheless, the performance of the DPSK optical system was assessed to rely on a different coding format such as: Return to Zero (RZ) & Non-Return to Zero (NRZ-DPSK). The system is designed to withstand high data rate on long-distance optical transmission systems. The system investigated was simulated with OptiSystemTM 13.0 commercial software. The investigated system demonstrated the ability to mitigate nonlinear impairments from the noisy optical DPSK system with high bit-error rate (BER) improvement at low power transmission. Optical DPSK demonstrates the ability to send high-rate data transmission systems for use in the next generation of optical networks.