Application of the method of ordinary least squares in the calibration of temperature, pressure and mass meters

Abstract

The goal of this paper is applied the Ordinary Least Squares (OLS) method as a strategy in order to reduce the uncertainty measurement associated to the relevant physical quantities. Methodology or method: This work was motivated due to the efforts made in the measurement sciences to investigate alternative methods that allow obtaining greater metrological reliability of the results, that is, reducing the uncertainty associated with the measurement. Thus, the applied methodology consisted of the development of a computational code using the MatLab tool, in which an algorithm was programmed that supports the application of the OLS method. Three of the most used physical magnitudes at an industrial level were evaluated: temperature, pressure and mass. These magnitudes were evaluated from the calibration of three measuring instruments: thermistor, manometer and digital scale. Results: The applied methodology allowed to: (i) obtain the matrix of the coefficients for polynomials of degrees 1, 2, 3 and 4 that adjust the experimental data; (ii) to draw the calibration curves for each of the obtained polynomials and (iii) to estimate the fit uncertainty (i.e.: the mean squared deviation) to specify the polynomial that best models the experimental data for a reliability level of 95.45 % (k = 2). The consolidated results confirmed a reduction in the uncertainty associated with the adjustment polynomial of 99.8% for the temperature measurement, 45.6% for the pressure measurement and 53.9% for the mass measurement. Conclusions: Finally, a discussion of the results is presented, confirming the effectiveness of the ordinary least squares method as a strategy to reduce the uncertainty associated with the calibration of measuring instruments.