Objectives: This study aims to enhance and optimize a modern communication system for high data rate transmission using Dense Wavelength Division Multiplexing (DWDM). The objectives include employing a 64-channel DWDM system, implementing different data speeds, and utilizing a dispersion compensation method. Methods: To achieve the objectives, we simulate and analyze the effects of Dispersion Compensation Fiber (DCF) in a DWDM system with 64 channels. The system employs Non-Return-to-Zero (NRZ) modulation format at varied bit rates and multiple energy levels. We propose a method for achieving data rates of 10 to 40 Gbps using NRZ modulation and Erbium-Doped Fiber Amplifier (EDFA) over a single-mode fiber transmission distance of 40 to 160 km, along with a dispersion compensation fiber of 8 to 32 km (DCF). The performance of the developed model is evaluated based on Quality Factor, Bit Error Rate (BER), Eye Height, and Threshold. These metrics are measured at two input energy levels from an optical power source, covering a communication capacity ranging from 0.625 Tbps to 2.5 Tbps. Findings: Through the simulations and analyses, we uncover the impact of dispersion and demonstrate the effectiveness of the proposed method in compensating for dispersion effects. Performance analysis of two different input transmitter power levels, -10 dBm is more efficient compared to 0 dBm input transmitter power in terms of bit error rate and quality factor. Novelty: This study presents a novel approach to improving modern communication systems by utilizing DWDM with a dispersion compensation method. The use of a 64-channel DWDM system, combined with the proposed NRZ modulation technique and dispersion compensation fiber, provides an efficient solution for achieving high data rates over long transmission distances while minimizing the effects of dispersion. The findings contribute to the advancement of communication systems for high data rate transmission. Keywords Dispersion effect, Bit Error Rate, Q-factor, Optisystem software, Erbium doped fiber amplifier