Wavelength division multiplexing (WDM) is one of the emerging technologies for data transmission in 5G access network assembly between the antenna unit (AU) and the distribution unit (DU). Here, in this work, we evaluate and analyze a dense wavelength division multiplexing (DWDM) — passive optical networks (PONs) employing erbium–ytterbium-doped fiber amplifier (EYDFA) [Formula: see text] Raman hybrid optical amplifier (HOA) using two modulation schemes — modified duobinary return to zero (MDRZ) and differential phase shift keying (DPSK) for alternate channels, where the channel count has been varied from 8 to 40. The system has been analyzed over the wavelength span of 1550–1558.8[Formula: see text]nm with a channel spacing, input power and data rate of 25[Formula: see text]GHz, 0[Formula: see text]dB and 40[Formula: see text]Gb/s, respectively. An average gain, gain ripple and optical signal-to-noise ratio (OSNR) of 20.87, 0.95 and 30[Formula: see text]dB, respectively have been observed in the case of HOA along with the achievement of Q-factor of up to 9 at the receiving end of the system. The DPSK-modulated channels result in higher values of gain and OSNR along with lower gain ripple as compared to MDRZ channels making the proposed system a good option for 5G access network applications.