Abstract Data fitting is an indispensable tool in modern metrology. However, the
most popular method, least squares, reaches its limit at the naonoscale. As the relative uncertainty in the dependent variable increases, the concept of an exactly known independent variable and an uncertain dependent variable, which is at the core of least squares, looses its validity. The increasing complexity of the measurement process may give rise to correlations. The use of reference samples leads to correlations among the dependent or independent variables, crosstalk between sensors may lead to correlation between independent and dependent variables. These problems can be treated with generalized least squares but only few implementations are available, none of which are suitable for general functions and general covariance matrices. A new algorithm - Optimal Estimation of Function Parameters by Iterated Linearization (OEFPIL) – has been recently suggested which can handle both a wide class of functions as well as general covariance matrices. In this work the OEFPIL algorithm is applied to the analysis of force distance curves in atomic force microscopy (AFM). Force distance curves are used to evaluate the mechanical properties of samples such as the Young’s modulus and adhesion. In this work we apply the new algorithm and compare the results to other methods. The uncertainties obtained by OEFPIL are in good agreement with uncertainties obtained by the Monte Carlo method.