ABSTRACT A new composite model called shadowed Beaulieu-Xie (BX) is used in terahertz and millimeter-wave wireless communication systems. This model characterizes an environment with an arbitrary number of non-line-of-sight and line-of-sight signals. In this paper, an analysis of the average bit error rate (ABER) is done. The analysis is carried out for different coherent and non-coherent modulation techniques. The mathematical expressions of ABER are obtained using the probability density function of the output signal to noise ratio of the shadowed BX fading channel. A new closed-form ABER expression for gray-coded rectangular quadrature amplitude modulation over the shadowed BX channel is also evaluated. The effects of fading and shadowing parameters on the modulation techniques are analyzed and compared.