Abstract This paper presents the high capacity single-tone Radio over Fiber (RoF) system utilizing advanced modulation technique in conjunction with optimized dispersion compensation scheme to support the high data rate transmission upto 120 Gbps. The proposed system incorporates the Integrated-Alternate Mark Inversion (I-AMI) modulation and Symmetrical-Symmetrical-Post (SSP) compensation technique to enhance the spectral efficiency as well as to reduce the chromatic dispersion effects in long-haul RoF link. The proposed system i.e., I-AMI with SSP has been examined at 60, 80, 100 and 120 Gbps bit rate for various transmission distance. The simulation results obtained in terms of Q-factor, Bit Error Rate (BER) and eye diagram reveal that the proposed system provides wider eye-opening, low BER and high Q-factor as compared to the existing system. The proposed I-AMI with SSP provides the Q-factor of 30.47 whereas the I-AMI with PSP and AMI with SSP provides the Q-factor of 18.77 and 18.40 respectively at 80 Gbps bit rate over a transmission distance of 800 km.