The performance, complexity and effectiveness of various massive MIMO (M-MIMO) detectors are analyzed operating under highly spatial correlated uniform planar arrays (UPA) channels. In such context, M-MIMO systems present a considerable performance degradation and also, in some cases, an increased complexity. Considering this challenging, but realistic practical scenario, various sub-optimal M-MIMO detection structures are evaluated in terms of complexity and bit error rate (BER) performance trade-off. Specifically, the successive interference cancellation, lattice reduction (LR) and likelihood ascent search (LAS) schemes, as well as a hybrid version combining such structures with conventional linear MIMO detection techniques are comparatively investigated, aiming to improve performance. Hence, to provide a comprehensive analysis, we consider the number of antennas varying in a wide range (from conventional to massive MIMO condition), as well as low and high modulation orders, aiming to verify the potential of each MIMO detection technique according to its performance–complexity trade-off. We have also studied the correlation effect when both transmit and receiver sides are equipped with UPA antenna configurations. The BER performance is verified under different conditions, varying the array configurations, combining the detection techniques, increasing the number of antennas and/or the modulation order, especially aiming a near M-MIMO condition, i.e. up to $$64\times 64$$ and $$121\times 121$$ antennas has been considered. The aggregated LAS technique has demonstrated good performance in scenarios with high number of antennas, while LR and OSIC operates better in high correlated arrangements.