AbstractMost coded modulation schemes, such as Ungerboeck's trellis coded modulation (TCM), are based on equiprobabilities of the signal space points. These schemes improve the bit error rate (BER) performance if all signals have the same norm. For signals with different norm, unequiprobable schemes result in much more improved of BER performance and larger coding gain than those of equiprobable ones. Consequently, a new convolutional coded modulation system utilizing symbol unequiprobability is proposed. Calderbank and Ozarow proposed another system using unequiprobable signaling [2]. In their system a shaping code is employed to save the average signal power without decrease of the coding gain. In the system proposed herein, to save the average signal power and to obtain more coding gain, a convolutional encoder designed in the real or complex number field is introduced to control the signal probabilities directly.An optimum distribution of signaling probability is derived such that the minimum free Euclidean distance can be maximized. Then the convolutional encoder which can achieve the optimum distribution of signal points is derived theoretically. BER performance of the proposed system using such a convolutional encoder is evaluated by computer simulations.Finally, it is shown that for the 16‐QAM system, the proposed scheme achieves about 1.0‐dB improvement in coding gain over Ungerboeck's TCM.