ABSTRACT This work explains the theoretical and mathematical investigations of the transmission performance of an optical Single Sideband (SSB) modulation technique generated by a Mach Zehnder Modulator (MZM) with a 90° and 120° phase shift using hybrid coupler (Modified SSB) in addition to the Vestigial Sideband technique (VSB). The problem addressed in this paper is the effect of non-linearity of the transmitter system and the chromatic dispersion in single mode fiber due to the presence of second- and higher order harmonics. The non-linearity of the transmitter generates second and higher order harmonics, which affects the system performance leading to reduction in the data rate and increased distortion. The transmission performance of SSB, M-SSB and VSB is evaluated based on the extinction ratio of the MZM, different transmission lengths of the fiber, optical signal modulation index and the receiver sensitivity. A mathematical model has been developed for all the systems representing the modulation techniques till the second order. The baseband modulation consists of an RF signal of frequency 18 GHz and an Orthogonal Frequency Division Multiplexing (OFDM) signal with 4-QAM modulation technique converted to optical signal, using the MZM and transmitted over a SMF of distance of 75 Km. Our results show that the M-SSB technique proves to be more immune toward transmitter nonlinearity and chromatic dispersion as compared to its counterparts. In addition to this, it improves the receiver sensitivity by 6.56 dBm. The results generated by software simulation are in tandem with the mathematical model.