Abstract A new modulation scheme, that can increase the channel capacity of optical fiber link, is proposed in this paper. It is implemented by using a non-orthogonal frequency shift keying (FSK), quadrature amplitude modulation (QAM), and sparse sinusoidal sampling based on a compressed sensing. The proposed technique can allow us to have the improved transmission capacity because frequency spacing among digitally sample RF subcarriers becomes much narrower than the channel bandwidth by the non-orthogonality and the total signal length can be reduced by using the sparse sinusoidal sampling. For experimental verification, a 64-non-orthogonal frequency quadrature amplitude modulation (NoFQAM) signal is generated using non-orthogonal 4-FSK and 16-QAM. The total length of 64-NoFQAM signal is reduced up to 60% at the bit error rate (BER) of 4 × 10−5 by the sparse sinusoidal sampling. The BER of 2 × 10−5 was measured at a frequency spacing between digitally sampled RF subcarriers corresponding to 10% of channel bandwidth (500 MHz). The 7.5-Gb/s transmission of 64-NoFQAM signal with the channel bandwidth of 500 MHz was demonstrated successfully by using 50-km and 300-km optical fiber link.