For common nonlinear QR decomposition detection algorithm in MIMO system, this paper proposes reliability constraints QR algorithm (RC-QR algorithm) with low complexity. The reliability of soft estimates is judged by shadow area constraint method. Besides, constellation points are introduced as the candidate points. Then the optimal candidate point is chosen from multiple candidate points for feedback. Relative to conventional QR DE composition algorithm, the RC-QR algorithm can significantly improve system interference and greatly reduce error propagation in decision feedback by increasing the algorithm complexity only by a small amount. The simulation results show that the performance of our algorithm improves very obviously when the antenna configuration is 4 × 4 or 6 × 6. When bit error rate is 5 × 1,000, 6 dB signal to noise ratio can be saved. Meanwhile, RC-QR algorithm can flexibly balance bit error rate detection performance and operation complexity through controlling constraint threshold and the quantity of candidate points.