Abstract This study created a 13 Gb/s transmission speed per 300 channel dense wavelength division multiplexing system based on hybrid optical fiber amplifiers and filter techniques to enhance signal gain. The filter techniques used here included a hybrid optical amplifier, a gain flattening filter (GFF), a Fabry–Perot filter, a fiber Bragg grating (FBG), and glass composition technique. The techniques were combined to optimize signal gain as follows: the study began with an erbium-doped fiber amplifier (EDFA), added a Raman amplifier, added GFF, added fiber Bragg grating, added glass composition, and finally achieved a hybrid-modified model after adding Fabry–Perot filter. The signal gain was optimized with the hybrid-modified model via the different operating parameters of the EDFA. The proposed model also showed a better performance (30–40 dB greater) than the previous model (Mahran O, Aly MH. Performance characteristics of dual-pumped hybrid EDFA/Raman optical amplifier. OSA – Applied Optics Jan 2016;55:22–6) after changing the values of the signal wavelength, the EDFA, and the length of the Raman amplifier.