In this paper, we study nonlinear effects in the optical communication fibers such as four wave mixing (FWM) in an integration of dense wavelength division multiplexing (DWDM) with radio over fiber (ROF). We used an optical double side band (ODSB) configuration with direct and external modulation using dual drive Mach–Zehnder modulator. A 16-DWDM-ROF link at different data rates is used for information transmission in single mode fiber. Also, the performance analysis of the proposed system under FWM effect with different parameters such as data rate, channel spacing, input power, fiber length and optical amplifier gain is presented. The results of the proposed system are evaluated based on bit error rate (BER) and maximum quality factor (Q). Results show that the external DSB modulation format has better performance rather than the direct modulation. Generally, for 10 dB input power, 100 GHz channel spacing, 10 Gbps bit rate and 20 km link, the Q is about 20 and BER is less than 10−84 for external ODSB modulation format. Also it has been shown that with increasing data transfer rates, output results such as optical signal-to-noise ratio, BER and Q are deteriorated under influence of FWM-effect.