Abstract In this paper, the performance analysis of the WDM (wavelength division multiplexing) system on the optical fiber transmission link is proposed. High data transmission is possible by implementing a WDM optical communication system using different modulation formats. Firstly, the WDM optical communication system is designed on an Optiwave simulation tool and distortion in the optical fiber by “eye pattern analysis” is analyzed. Secondly, the optical SNR of the proposed WDM optical communication system is studied. Further, the bit-error rate (BER) plot is studied and optimization techniques are applied to enhance the proposed system for long-haul communication. Furthermore, the BER plots are analyzed by varying the length of the fiber and applying different coding techniques like NRZ (non-return zero), RZ (return to zero), QAM (quadrature amplitude modulation) etc. using the Optiwave simulation tool. The results obtained in the Optiwave simulation tool are validated by theoretical implementation in the MATLAB tool.