AbstractThe coordinated multipoint (CoMP) is a promising set of techniques that deals with inter‐cell interference differently than the conventional interference mitigation schemes. This unique approach can effectively mitigate or even exploit the interference, turning it into a useful signal. The CoMP was first introduced in Release‐11 by Third Generation Partnership Project (3GPP) for the fourth generation (4G) long term evolution‐advanced (LTE‐A) systems, enhanced in subsequent releases and recently incorporated as one of the fifth generation (5G) component technologies. In this article, we propose two techniques aiming to improve the performance of existing CoMP schemes. First, we examine the impact of constraint backhaul on the cell coordination and propose an optimal CoMP mechanism that adaptively switches between joint transmission (JT) and coordinated beamforming (CBF) CoMP schemes taking into consideration the average sum rate and the available backhaul capacity. Next, we present a backhaul overhead reduction scheme for CoMP enabled networks, which extends the performance gains of adaptive CoMP scheme. Simulation results suggest that proposed adaptive scheme outclasses conventional CoMP schemes and offers a spectrally efficient solution with increased average user throughput and sum rate.