SummaryIn this paper, the efficiency of various multiple‐input multiple‐output (MIMO) detectors was analyzed from the perspective of highly correlated channels, where MIMO systems have a lack of performance, besides in some cases, an increasing complexity. Considering this hard but a useful scenario, various MIMO detection schemes were accurately evaluated concerning complexity and bit error rate performance. Specifically, successive interference cancellation, lattice reduction, and the combination of them were associated with conventional linear MIMO detection techniques. To demonstrate effectiveness, a wide range of the number of antennas and modulation formats have been considered aiming to verify the potential of such MIMO detection techniques according to their performance‐complexity trade‐off. We have also studied the correlation effect when both transmit and receiver sides are equipped with uniform linear array and uniform planar array antenna configurations. The performance of different detectors is carefully compared when both antenna array configurations are deployed considering a different number of antennas and modulation order, especially under near‐massive MIMO condition. We have also discussed the relationship between the array factor and the bit error rate performance of both antenna array structures.