Quadrature Amplitude Modulation Formats Research Articles

Across-dimensional optical constellation de-aggregations from QAMs to PAMs in optical transparent networks

In this paper, an across-dimensional optical constellation de-aggregation scheme is proposed and verified. This de-aggregation scheme builds flexible optical connections between core and edge networks and fills the lack of the conversions between quadrature amplitude modulation (QAM) and pulse amplitude modulation (PAM) formats. 10G Baud quadrature phase shift keying/9QAM/16QAM-to-2×on off keying (OOK)/PAM3/PAM4 constellation de-aggregations are achieved. The enabled constellation manipulations contain polarization-assisted quadrature de-multiplexing and bi-directional vector moving. The key research status of QAM/PAM de-aggregations and adopted theoretical derivations are analyzed and expressed in detail. The signal spectrum, constellations, eye diagrams, transfer characteristics curves, error vector magnitudes and bit error ratios (BERs) during the de-aggregation process are calculated to measure the system performances. With the input optical signal-to-noise ratio (OSNR) of 10 dB/15 dB/20 dB, the de-aggregated OOK/PAM3/PAM4 signals show the receiver OSNR of 19.1 dB/21.9 dB/31.2 dB, respectively, at the BER of 10−3. The proposed scheme presents excellent format scalability and has potential applications in flexible all-optical connections, across-dimensional optical transmissions, and optical transparent networks.

Physical Realizations of Multidimensional Voronoi Constellations in Optical Communication Systems

Multidimensional geometric shaping has been shown to outperform uniform quadrature amplitude modulation (QAM) in optical communication systems but the complexity of symbol decision and bit mapping can often be significant as dimensionality increases. In this paper, a low-complexity geometric shaping method based on multidimensional lattices is investigated both in experiments and simulations. The modulation formats designed based on this method are called Voronoi constellations (VCs) and we study them in 8, 16, and 32 dimensions. We obtain transmission reach improvements of up to 22 and 70% for VCs compared to 4QAM and 16QAM, respectively, in nonlinear long-haul fiber transmission. Moreover, we compare different physical realizations of multidimensional VCs over wavelengths, polarizations, and time slots in both the Gaussian and nonlinear fiber channels. We demonstrate that different physical realizations perform similarly in the fiber-optic back-to-back channel. However, in long-haul transmission systems, spreading the dimensions over time slots can increase the transmission reach up to 4% compared to wavelengths and polarizations. Furthermore, the mutual information and generalized mutual information are estimated and compared to QAM formats at the same spectral efficiencies.

Single-carrier 800-Gb/s self-homodyne coherent transmission of DP-16QAM, DP-32QAM, and DP-64QAM with uncooled DFB laser.

Self-homodyne detection (SHD) is a promising approach to realize high-capacity short-reach optical transmission systems with low cost and low power consumption. We experimentally demonstrate single-carrier net 800-Gb/s SHD transmission with low-cost ∼MHz linewidth distributed feedback (DFB) laser over 2 km, 10 km, 25 km, and 40 km single-mode fiber (SMF) using three different quadrature amplitude modulation (QAM) formats, including 80-Gbaud dual-polarization (DP) 64QAM, 100-Gbaud DP-32QAM, and 120-Gbaud DP-16QAM. Among them, net 800-Gb/s DP-64QAM SHD transmission over 25 km SMF using an uncooled DFB laser with a linewidth of 2.6 MHz is experimentally verified. The detailed experimental performance evaluation of net 800Gb/s SHD system is performed, in which various configurations are considered, such as different laser linewidths, three QAM formats, and different transmission distances. DFB lasers with linewidths of 1 MHz and 2.6 MHz lead to negligible penalty when compared to the same SHD system but using an external cavity laser (ECL) with a linewidth of 26kHz in back-to-back (BTB) case. 80-Gbaud DP-64QAM obtains the highest optical signal-to-noise ratio (OSNR) requirement and the highest bit-error rate (BER) floor but the best tolerance of chromatic dispersion (CD). 120-Gbaud DP-16QAM achieves the lowest OSNR requirement and the lowest BER floor but the worst tolerance of CD. The detailed experimental investigation is conducive to promote the practical application of SHD in different short-reach scenarios.

Low-complexity amplitude-division irregular QAM formats for short-reach unamplified coherent optical systems.

A low-complexity scheme is proposed to realize irregular uniform quadrature amplitude modulation (QAM) formats with Gray mapping, which are named amplitude-division irregular QAM (AD-Ir-QAM) formats. Compared to conventional probabilistic shaping (PS) with the Maxwell-Boltzmann (MB) distribution (PS-MB), irregular QAM formats show a smaller peak-to-average power (PAPR) and achieve a better performance in systems with the peak-power constraint. Compared with irregular QAM formats realized by PS (PS-Ir-QAM), AD-Ir-QAM formats realize a more flexible rate adaptation and have a lower implementation complexity. Experimental results obtained in an unamplified coherent optical system show that, at a generalized mutual information (GMI) of 4.5 bits/2D-symbol, AD-Ir-100QAM achieves gains of 2.1 and 0.5 dB in the power budget over PS-MB-100QAM and PS-Ir-100QAM, respectively.

Orthant-symmetric four-dimensional geometric shaping for fiber-optic channels via a nonlinear interference model.

Maximizing the data throughput for optical fiber communication via signal shaping has usually been regarded as challenging due to the nonlinear interference and implementation/optimization complexity. To overcome these challenges, in this paper, we propose an efficient four-dimensional (4D) geometric shaping (GS) approach to design 4D 512-ary and 1024-ary modulation formats by maximizing the generalized mutual information (GMI) using a 4D nonlinear interference (NLI) model, which makes these modulation formats more nonlinear-tolerant. In addition, we propose and evaluate a fast and low-complexity orthant-symmetry based modulation optimization algorithm via neural networks, which allows to improve the optimization speed and GMI performance for both linear and nonlinear fiber transmission systems. The optimized modulation formats with spectral efficiencies of 9 and 10 bit/4D-sym demonstrate a GMI improvement of up to 1.35 dB compared with their quadrature amplitude modulation (QAM) counterparts in additive white Gaussian noise (AWGN) channel. Numerical simulations of optical transmission over two types of fibers show that the 4D NLI model-learned modulation formats could extend the transmission reach by up to 34% and 12% with respect to the QAM formats and the AWGN-learned 4D modulation formats, respectively. Results of effective signal-to-noise ratio are also presented, which confirm that the extra gains in optical fiber channel come from the enhanced SNR by reducing the modulation-dependent NLI.

Receive-diversity-aided power-fading compensation for C-band IM/DD OFDM systems.

A receive-diversity-aided power-fading compensation (RDA-PFC) scheme is proposed and demonstrated to eliminate the chromatic dispersion (CD)-induced power fading for C-band double-sideband (DSB) intensity modulation and direct detection (IM/DD) orthogonal frequency division multiplexing (OFDM) systems. By combining the responses before and after a dispersive element using a maximal-ratio combining (MRC) algorithm, the CD-induced power fading dips within the signal bandwidth of around 50 GHz can be effectively compensated for, which results in an up to 17.6-dB signal-to-noise ratio (SNR) improvement for the fading subcarriers after transmission over 10 km of standard single-mode fiber (SSMF). Using the 16 quadrature amplitude modulation (QAM) format, a diversity receiver with the proposed RDA-PFC scheme can support 170.6-Gbit/s OFDM signal transmission over a 10-km SSMF and reduces the bit error rate (BER) by more than an order of magnitude compared with a conventional receiver. Moreover, 208.1-Gbit/s adaptive bit and power loading OFDM signal transmission over a 10-km SSMF is realized by the proposed RDA-PFC scheme, which improves the capacity by 15.3% compared with the case without RDA-PFC at a BER of 3.8 × 10-3. The proposed RDA-PFC scheme shows great potential in CD-induced power-fading compensation for high-speed IM/DD OFDM systems.

Practical Considerations in Direct Detection Under Tukey Signalling

The deliberate introduction of controlled intersymbol interference (ISI) in Tukey signalling enables the recovery of signal amplitude and (in part) signal phase under direct detection, giving rise to significant data rate improvements compared to intensity modulation with direct detection (IMDD). The use of an integrate-and-dump detector makes precise waveform shaping unnecessary, thereby equipping the scheme with a high degree of robustness to nonlinear signal distortions introduced by practical modulators. Signal sequences drawn from star quadrature amplitude modulation (SQAM) formats admit an efficient trellis description that facilitates codebook design and low-complexity near maximum-likelihood sequence detection in the presence of both shot noise and thermal noise. Under the practical (though suboptimal) allocation of a 50% duty cycle between ISI-free and ISI-present signalling segments, at a symbol rate of 50 Gbaud and a launch power of -10 dBm the Tukey scheme has a maximum theoretically achievable throughput of 200 Gb/s with an (8,4)-SQAM constellation, while an IMDD scheme achieves about 145 Gb/s using PAM-8. Note that the two mentioned constellations have the same number of magnitude levels and the difference in throughput is resulting from exploiting phase information under using a complex-valued signal constellation.

Power-Efficient Voronoi Constellations for Fiber-Optic Communication Systems

Voronoi constellations (VCs) are considered as an effective geometric shaping method due to their high power efficiencies and low complexity. In this paper, the performance of 16- and 32-dimensional VCs with a variety of spectral efficiencies transmitted in the nonlinear fiber channel are investigated. Both single-channel and wavelength-division multiplexing systems are considered for the transmission of the VCs, as well as different signal processing schemes, including chromatic dispersion compensation and digital backpropagation. Multiple performance metrics including the uncoded bit error rate, mutual information (MI), and generalized mutual information (GMI) of VCs are evaluated. Compared with quadrature amplitude modulation (QAM) formats, the VCs provide 1.0–2.4 dB launch power gains, up to 0.50 bits/symbol/dimension-pair MI gains, up to around 30% potential reach increase at the same MI, and up to 0.30 bits/symbol/dimension-pair GMI gains in a limited launch power range. The observed performance gains over QAM are found higher than in the back-to-back case. Moreover, a general GMI estimation method for very large constellations using importance sampling is proposed for the first time.

High-security constellation shaped self-homodyne coherent system with 4-D joint encryption.

In recent years, the self-homodyne coherent (SHC) system and the constellation shaping (CS) technique have drawn considerable attention due to their abilities to further improve the transmission capacity for various scenarios. From the security point of view, the CS technique and the SHC infrastructure also provide new dimensions for encryption. We propose a high-security and reliable SHC system based on the CS technique and the digital chaos. With a four-dimensional hyperchaotic system, chaotic sequences are generated and used for the exclusive or operation, chaotic constant composition distribution matching, phase disturbance, and optical-layer time-delay disturbance. Moreover, 64-ary circular quadrature amplitude modulation (64CQAM) format is adopted for transmission due to its advantages of sensitivity to phase noise, immunity to conventional digital signal processing, and ability of time-mismatch masking, which is verified by simulation in a SHC system. Last, we conduct an experimental verification in a 20GBaud probabilistically shaped 64CQAM SHC system. Consequently, with a large-linewidth laser source, optical-layer security can be protected by time mismatches of tens of picoseconds. And the digital-layer security is protected by an enormous key space of 10127. The proposed scheme can provide reliable real-time encryption for the optical fiber transmission, serving as a potential candidate for the future high-capacity inter/intra-datacenter security interconnect.

Joint Estimation of Linear and Nonlinear Coherent Optical Fiber Signal-to-Noise Ratio

This letter proposes an estimator based on the neural network (NN) to jointly estimate the linear and nonlinear signal-to-noise ratios. The proposed NN-based estimator utilizes new input features based on the entropy extracted from the received signal. Moreover, the computational complexity of the proposed estimator is analyzed. The dataset utilized for training and testing is constructed from dual-polarization 16-ary quadrature amplitude modulation format over different system configurations of the standard single-mode fiber, such as launch power, transmission distances, and the number of wavelength division multiplexed channels. Numerical results reveal the superiority of the proposed NN-based estimator in terms of accuracy and computational complexity compared to the existing NN-based estimators in the literature.

Optical Generation and Transmission of mmWave Signals in 5G ERA: Experimental Evaluation Paradigm

We demonstrate the generation, of a mmWave signal via the injection of an optical frequency comb (OFC) into an integrated tunable dual distributed Bragg reflector (DBR) laser as well as the fiber transmission and the processing of this signal by an optical beamforming network (OBFN). The dual DBR laser is based on a hybrid indium phosphide (InP) – polymer photonic integrated circuit (PIC). Two different cases have been examined in which the microwave signal is centered around 39 GHz and 60 GHz respectively, carrying quadrature amplitude modulation (QAM) formats at 0.5 Gbaud. In this proof-of-concept scenario, the OBFN consists of two optical paths, where the relative true time delay is induced by an optical delay line (ODL). Extensive comparison between the back-to-back (B2B) case and scenarios with transmission over 25 km of standard single-mode fiber (SSMF) has been made using the error-vector magnitude (EVM) and the bit-error ratio (BER) as evaluation criteria. In all cases, error-free transmission was suggested for all QPSK signals, whereas a worst-case EVM of 11.8% was observed for 16-QAM transmission, successfully showcasing the concept’s potential. The generated microwave signal’s frequency can be set arbitrarily high, provided that high-speed photodetection equipment is available for the detection and down-conversion of the signal. Extension to higher antenna elements (AEs) numbers is straight-forward, relying only on the number of available photodetectors.

Air-gap Fabry-Pérot cavity filtered 30 nm broadband electro-optic frequency combs for high-order coherent communications.

Broadband electro-optic (EO) frequency combs, which have flexible and high repetition frequencies, are prospective light sources for dense-wavelength-division-multiplexed coherent optical communications. In most cases, nonlinear spectral broadening and amplification procedures are needed to achieve broadband and high-power EO frequency combs. This leads to a low optical carrier-to-noise ratio (OCNR) for comb lines, limiting the transmission capacity. Here, we propose to use an air-gap Fabry-Pérot (FP) cavity to improve the OCNR for all the comb lines covering a 30 nm broadband spectrum. A 12 dB OCNR (0.1 nm bandwidth) improvement is obtained experimentally via using an FP cavity with ∼790 MHz bandwidth. We apply a 150-channel filtered EO comb with 25 GHz channel spacing and load 20 GBaud signals on each comb line to demonstrate the effect of OCNR improvement. The 137/150 channels have a bit error rate below the threshold of soft-decision forward error correction when using the 128 quadrature amplitude modulation (QAM) format. However, none of these channels can support this modulation format without cavity filtering. We also investigate dispersion tolerance and the long-term stability when using an air-gap FP cavity, highlighting its advantages. Our results show a practical solution to boost the transmission capacity when applying broadband EO combs in optical communications.

Effect of erbium-doped fiber amplifier loss compensation on 5G new radio millimeter-wave seamless transmission over analog fiber and free space optical fronthaul at 60 GHz

We demonstrate a fifth generation (5G) new radio (NR) signal transmitted by an analog optical and seamless antenna wireless connection at the frequency band of 60 GHz to exploit a high-frequency unlicensed frequency range. An optical frequency doubling technique, using two Mach–Zehnder modulators operating in a carrier suppressed and linear regime, respectively, was adopted to obtain the desired millimeter wave frequency at the photodetector’s output. The proposed system was tested with the 5G NR signals with a maximum bandwidth of 200 MHz and 64 quadrature amplitude modulation format. It was shown that the signal transmitted through the optical fiber and free space optical link with 1 m long seamless antenna transmission at 62 GHz was capable of meeting the signal quality requirements in terms of error vector magnitude. Moreover, the system phase noise performance showed an almost negligible difference between the various system configurations.

Waveform Design for High-Order QAM Faster-Than-Nyquist Transmission in the Presence of Phase Noise

The state-of-the-art radio-frequency (RF) devices limit the deployment of extremely high-order quadrature amplitude modulation (QAM) formats (e.g., 16384-QAM) to meet the high-capacity demand on microwave backhaul links. This paper turns to faster-than-Nyquist (FTN) transmission using lower-order constellations and lower-cost RF devices as a solution to the demand. To realize low-complexity interference cancellation, we pre-equalize the FTN-induced inter-symbol interference at the transmitter by using Tomlinson-Harashima precoding (THP), while at the receiver suppressing the phase noise (PHN) generated by the RF local oscillators with pilot symbol assisted approaches. However, the THP may distort the pilots, which degrades the performance of PHN compensation. To resolve this problem, we propose two pilot designs that are distortion-free to precisely estimate the PHN samples. Moreover, we derive a closed-form expression of the symbol detection signal-to-noise ratio (SNR), in terms of the THP-FTN waveform parameters. With the SNR expression, a waveform optimization procedure is developed to maximize the SNR and enhance the achievable FTN capacity. The proposed scheme is validated in the simulated platform of 4096-QAM microwave link. The results demonstrate that the FTN signaling achieves the system capacity equivalent to that of the 16384-QAM Nyquist signaling with an SNR gain of 5.8 dB.

Impact of Laser Phase Noise on Self-Coherent Transceivers Employing High-Order QAM Formats

Recent studies have shown that the self-coherent systems, including the bi-directional, the Stokes-vector-modulation direct-detection (SVM-DD), and the Kramers-Kronig (KK) schemes, have the potential to reduce the cost of short-reach networks. One of the most attractive features of the self-coherent systems is that they may eliminate the necessity of a narrow-linewidth laser, which is mandatory in the conventional full coherent systems employing high-order quadrature-amplitude-modulation (QAM) formats. On the other hand, it is also recognized that the laser phase noise may have a significant impact on the system performance if there exists a large length mismatch between the signal and the CW tone in the transmission paths. In this paper, we analyze the impact of laser phase noise on the self-coherent systems numerically and experimentally by considering the bi-directional system as an example case. As a result, we show that the performance of the self-coherent system can be described efficiently by using a limited number of normalized parameters. We then reveal the existence of a distinctive threshold on the path length mismatch that influences the bit-error-rate (BER) performance; the penalty increases in a stepwise manner as the length mismatch exceeds the threshold and converges to a constant value. Based on the results, we suggest the feasibility of employing high-order QAM formats to transmit >800 Gb/s signals per polarization using a 10-MHz-linewidth laser. Finally, proof-of-concept experimental results are presented to verify the theoretical and numerical analyses.

The effects of the cross-entropy stopping criterion and quadrature amplitude modulation on iterative turbo decoding performance

One of the most often-used stopping criteria is the cross-entropy stopping criterion (CESC). The CESC can stop turbo decoder iterations early by calculating mutual information improvements while maintaining bit error rate (BER) performance. Most research on iterative turbo decoding stopping criteria has utilised low-modulation methods, such as binary phase-shift keying. However, a high-speed network requires high modulation to transfer data at high speeds. Hence, a high modulation technique needs to be integrated into the CESC to match its speed. Therefore, the present paper investigated and analysed the effects of the CESC and quadrature amplitude modulation (QAM) on iterative turbo decoding. Three thresholds were simulated and tested under four situations: different code rates, different QAM formats, different code generators, and different frame sizes. The results revealed that in most situations, the use of CESC is suitable only when the signal-to-noise ratio (SNR) is high. This is because the CESC significantly reduces the average iteration number (AIN) while maintaining the BER. The CESC can terminate early at a high SNR and save more than 40% AIN compared with the fixed stopping criterion. Meanwhile, at a low SNR, the CESC fails to terminate early, which results in maximum AIN.

True Time Delay Optical Beamforming Network Based on Hybrid Inp-Silicon Nitride Integration

We demonstrate a broadband and continuously tunable 1×4 optical beamforming network (OBFN), based on the hybrid integration of indium phosphide (InP) components in the silicon nitride (Si <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sub> N <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sub> ) platform. The photonic integrated circuit (PIC) comprises a hybrid InP-Si <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sub> N <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sub> external cavity laser, a pair of InP phase modulators, a Si <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sub> N <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sub> optical single-sideband full carrier (SSBFC) filter followed by four tunable optical true time delay lines (OTTDLs), and four InP photodetectors. Each OTTDL consists of eight cascaded thermo-optical micro-ring resonators (MRRs) that impose tunable true time delay on the propagating optical signals. The OBFN-PIC is designed to facilitate the steering of a microwave signal with carrier frequency up to 40 GHz over a continuous set of beam angles. We evaluate the performance of the OBFN-PIC to handle and process microwave signals, measuring the link gain, the noise figure (NF), and the spurious-free dynamic range (SFDR) parameters. Moreover, we assess its beamforming capabilities assuming that the OBFN-PIC is part of a wireless system operating in the downlink direction and feeds a multi-element antenna array. Using microwave signals at 5 and 10 GHz with quadrature amplitude modulation (QAM) formats at 500 Mbaud, we evaluate the performance of the OBFN-PIC under various configurations. We show that error-free performance can be achieved at both operating frequencies and for all the investigated beam angles ranging from 45° to 135°, thus validating its potential for high-quality beamforming performance.

Bayesian Optimization for Nonlinear System Identification and Pre-Distortion in Cognitive Transmitters

We present a digital signal processing (DSP) scheme that performs hyperparameter tuning (HT) via Bayesian optimization (BO) to autonomously optimize memory tap distribution of Volterra series and adapt parameters used in the synthetization of a digital pre-distortion (DPD) filter for optical transmitters. Besides providing a time-efficient technique, this work demonstrates that the self-adaptation of DPD hyperparameters to correct the component-induced nonlinear distortions as different driver amplifier (DA) gains, symbol rates and modulation formats are used, leads to an improvement in transmitter performance. The scheme has been validated in back-to-back (b2b) experiments using dual-polarization (DP) 64 and 256 quadrature amplitude modulation (QAM) formats, and symbol rates of 64 and 80 GBd. For DP-64QAM at 64 GBd, it is shown that the DPD scheme reduces the required optical signal-to-noise ratio (OSNR) at a bit error ratio of 10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-2</sup> by 0.9 dB and 0.6 dB with respect to linear DPD and a heuristic nonlinear DPD approach, respectively. Moreover, we show that the proposed approach also reduces filter complexity by 75% in conjunction with the use of memory polynomials (MP), while achieving a similar performance to Volterra pre-distortion filters.

Bidirectional OFDM-MMWOF transport system based on mixed QAM modulation format using dual mode colorless laser diode and RSOA for next generation 5-G based network

A hybrid bidirectional orthogonal frequency division multiplexing–millimeter wave over fiber (OFDM-MMWOF) transport system based on mixed quadrature amplitude modulation (QAM) format using dual-mode colorless laser diode and reflective semiconductor optical amplifier for next-generation 5G based networks is proposed and experimentally demonstrated. Here, an orthogonal quadruple frequency MMW is generated via optical heterodyne scheme. In case of downlink transmission, 10 Gbps baseband signal is transmitted over 80-km single-mode fiber (SMF) plus 500-m free space optical (FSO) distance and for the uplink, 5 Gbps baseband signal is transmitted over 80-km SMF plus 100-m FSO distance. For wireless transmission, 12.5 Gbps/60 GHz downlink MMW signal is generated and transmitted over 80-km SMF as well as 100-m wireless distance. 6.5 Gbps/30 GHz uplink MMW signal is communicated over 80-km SMF plus 50-m wireless distance. OFDM scheme with mixed QAM modulation technique enhances our spectral efficiency and simultaneously reduces inter symbol interference, inter-carrier interference, dispersion, etc. for long reach communication. The feasibility and reliability of our system is established by achieving low bit error rate (10-3), good error vector magnitude (<10.5% for 32 QAM and < 12.5% for 16 QAM), appropriate signal to noise ratio, and clear constellation diagram.

Fast‐adaptive weighted constellation designs based on hybrid constellation shaping for long‐haul optical fiber communication systems

AbstractA novel fast‐adaptive weighted hybrid‐shaped M‐ary quadrature amplitude modulation (HS‐MQAM) format is proposed in this paper. By introducing a fast‐adaptive weighting factor array related to geometric shaping and probabilistic shaping, the non‐linear impairment is mitigated and the bit error rate (BER) and mutual information (MI) performances are further improved. The simulation results show that the proposed HS‐61QAM outperforms both geometric‐shaped 61 quadrature amplitude modulation (GS‐61QAM) and regular 64QAM, and has more obvious advantages in the long‐haul optical transmission system. For a 120‐Gb/s optical communication system over 10 × 75 km fiber transmission link, compared with GS‐61QAM and regular 64QAM, the proposed approach can achieve about 4.2 and 8 dB OSNR gain at the BER of 10−2, and 0.21 and 0.55 bits/symbol MI gain at the OSNR of 22 dB.