This paper investigates the performances of differential phase shift keying (DPSK) and quadrature phase shift keying (QPSK) for 400 channels super dense system for the data rate of 160, 170, 180 and 200[Formula: see text]Gbps, respectively. Evaluation has also been investigated in terms of bit error rate (BER), quality factor and received output power with varying input power. Final analysis has reflected that DPSK modulation has shown the acceptable performance in terms of better communication with minor effect of nonlinearity and dispersion using RAMAN-EDFA hybrid optical amplifier (HOA) for the distance of 250[Formula: see text]km.