Influence of errors in coordinate transformation due to uncertainties of the design matrix

Abstract

Coordinate transformation between reference systems is a habitual task in geodetic sciences. Several transformation models and adjustment of observations algorithms have been explored. The most popular approach uses the well-known Ordinary Least Squares method (OLS) which seeks to estimate unknown parameters in a linear regression model by minimizing the sum of squared residuals while assuming that the observations are subject to normally distributed random noise. Although this approach has been widely used on many applications, its assumption regarding the distribution of errors is unrealistic. An alternative model created to deal with non normal–random distributions of errors is the Total Least Square model (TLS). Although several approaches have been suggested for representing the errors in the design matrix, most of them fail in including errors with different stochastic characteristics truthfully. This paper presents an assessment of the quality of a Helmert transformation of 2D coordinates when different precisions in both the observation vector and the design matrix exist. Two algorithms were analyzed: the OLS, and the Improved Weighted Total Least Squares (IWTLS). The transformation parameters obtained through IWTLS and OLS show significant differences among them when the variances in the coordinate systems are different. When assessing the transformation of coordinates using control points, it becomes clear that both algorithms perform the same in terms of the quality of the transformation. Furthermore, it is shown that an analysis based on the residuals obtained in the least squares adjustment is not a reliable tool to assess the overall quality of the transformation.