Abstract Coded modulation (CM), which combines channel codes with high-dimensional modulation schemes, is an attractive technique to improve the spectral efficiency in high-speed optical transmission systems. Non-binary low-density parity check (NB LDPC) codes can provide larger coding gains with smaller codeword lengths compared to their binary counterpart, in addition to reducing the latency of the system. The majority of works based on NB LDPC CM focus on bit-interleaved CM (BICM) and the widely used additive white Gaussian noise (AWGN) channel model. However, the Gaussian channel model is not a realistic model for long-haul intensity-modulated direct-detection (IM/DD) optical systems using erbium-doped fiber amplifiers. In this paper, it is mathematically proved that noise in the received signals after photodetection can be more accurately represented using chi-squared distribution. The efficiency of CM depends on the mapping between coded symbols and the constellation signal points of the modulation scheme. Hence, a symbol interleaved CM with subcarrier modulation for NB LDPC codes based on the proposed channel model is presented in this paper to improve the bit-error rate (BER) performance. The simulation results show that the proposed CM for NB LDPC codes with M-ary quadrature amplitude modulation schemes provides better performance than BICM. The analysis of the error performance of the proposed system gives a coding gain of 1 dB at a BER of 10−7 compared to that of the AWGN channel model. The performance improvement of the system is also evaluated in terms of spectral efficiency and link distance.