Abstract Gain flatness of erbium-doped fiber amplifier (EDFA) is an important aspect to support high speed multimedia applications for smart devices, cloud computing, big data analysis and high definition television (HDTV) as it achieves equal power for all dense wavelength division multiplexing (DWDM) channels. In this paper, we compare 32 and 64 channels 10 Gbps DWDM system using hybrid EDFA+ Ytterbium-doped silica fiber amplifier (YDFA) in C + L band for 0.4, 0.8 and 1.6 nm wavelength spacing along with FBG for gain flattening. The system performance has been analyzed in terms of optical signal to noise ratio and spectral gain fluctuation observed from the optical power spectrums. It is observed that YDFA although extends the amplification window of EDFA from 1530 to 1610 nm (C to L band), but gain fluctuates over the operating wavelength range. Gain flatness is enormously improved (5–10 dB) by adding FBG as a gain flattening filter to the hybrid (EDFA+YDFA) amplifier. It is found that as the channel spacing decreases (from 1.6 to 0.4 nm) and the number of channels increases (from 32 to 64), the gain reduces due to inter-channel crosstalk and four wave-mixing effect.