Recent studies have shown that existing detection algorithms are not suitable for high-order quadrature amplitude modulation (QAM) in massive MIMO system. In this paper, with an equivalent objective function by QR decomposition and further relaxation on the constrains, the authors develop an improved semidefinite further relaxation detector (SFRD), which is proved to be convex and has solutions within polynomial complexity time. Using the detection result from the proposed SFRD as the initial vector, they propose a novel semidefinite further relaxation on the likelihood ascent search (SFRLAS) detection algorithm. It has been shown through their studies that the proposed SFRLAS scheme can effectively approach the optimum bit error rate from the maximum-likelihood detection algorithm for systems with high-order QAM and large-scale antennas, however, with a lower computational complexity. The spectral efficiency converges to the theoretical value at a much lower required average received signal-to-noise ratio. It is an effective method for high-order QAM signal detection in massive MIMO system.