On ‘decorrelation’ in solving integer least-squares problems for ambiguity determination

Abstract

This paper intends to shed light on the decorrelation or reduction process in solving integer least squares (ILS) problems for ambiguity determination. We show what this process should try to achieve to make the widely used discrete search process fast and explain why neither decreasing correlation coefficients of real least squares (RLS) estimates of the ambiguities nor decreasing the condition number of the covariance matrix of the RLS estimate of the ambiguity vector should be an objective of the reduction process. The new understanding leads to a new reduction algorithm, which avoids some unnecessary size reductions in the Lenstra-Lenstra-Lovász (LLL) reduction and still has good numerical stability. Numerical experiments show that the new reduction algorithm is faster than LAMBDA’s reduction algorithm and MLAMBDA’s reduction algorithm (to less extent) and is usually more numerically stable than MLAMBDA’s reduction algorithm and LAMBDA’s reduction algorithm (to less extent).