With the development of the Internet and information technology, optical fiber communication systems need to meet people’s information demand for large capacity and high speed. High-order phase modulation and channel multiplexing can improve the capacity and data rate of optical fiber communication systems, but they also bring the problem of bit error. To improve the transmission quality and reliability of optical fiber communication systems, forward error correction (FEC) coding techniques are commonly used, which serve as the fundamental approach to enhance the quality and reliability of fiber optic communication systems, ensuring that the received data remain accurate and reliable. The FEC in optical fiber communication systems is divided into three generations. The first generation FEC is mainly hard decision codewords, represented as RS code. The second generation FEC is mainly cascaded code, which stands for interleaved cascaded code. The third generation of FEC mainly refers to soft decision codes, which are represented as low-density parity-check (LDPC) codes. As a kind of FEC, LDPC codes stand out as pivotal contributors in the field of optical communication and have gained remarkable attention due to exceptional error correction performance and low decoding complexity. Based on IEEE802.16e standard, LDPC code with specific code length and rate is compiled and simulated in MATLAB and VPItransmissionMaker 10.1 and successfully incorporated into polarization multiplexed differential quadrature phase shift keying (PM-DQPSK) coherent optical transmission system. The simulation results indicate that the bit error rate (BER) can be reduced to 10−3 when the optical signal-to-noise ratio (OSNR) reaches 14.2 dB, and the BER experiences a reduction by nearly three orders of magnitude when the OSNR is 17.2 dB. These findings underscore the efficacy of LDPC codes in significantly improving the performance of optical communication systems, particularly in scenarios demanding robust error correction capabilities. This study provides valuable, significant results regarding the potential of LDPC codes for enhancing the reliability of optical transmission in real-world applications.