Abstract Radio over Fiber (RoF) is one of the solutions to the “last mile” because of its bandwidth advantage offered by fiber and the mobility and flexibility of wireless systems. In this paper, the transmission performance of DSB, SSB and OCS with three modulation formats is established, and a new single-sideband modulation single-mode optical fiber transmission model is proposed, and simulated analysis is done to the system. In this paper, 40 GHz millimeter-wave signal was generated by external modulation, and the received signal is received by coherent demodulation. In this paper, Matlab and VPI are used for combine programming simulation, DCF method was used to compensate the chromatic dispersion. Through the analysis of dispersion performance, Q value and bit error rate, we can see that the modified SSB modulation has the best transmission performance for 120 km links.