Polynomial regression with reduced over-fitting—The PALS technique

Abstract

In order to reduce over-fitting, a new polynomial least-squares technique, named polynomial area least-squares (PALS), was developed. The technique is based on the minimization of the sum of the squared areas defined between the polynomial fitting function and the line segments connecting every two consecutive data points. A relatively simple system of linear equations was developed to compute the free parameters of the polynomial using the available training data. The ability of the PALS technique to generalize was evaluated through the generation of test data and statistical simulations. The PALS technique produces polynomials with reduced over-fitting, with little dependency on the order of the polynomial, especially when the polynomial order is high, so that, in some cases, increasing the order of the polynomial may even contribute to further over-fitting reduction. The major advantage of PALS is that it does not depend on any case-dependent parameter necessary for the data fitting procedure; the major limitation is that it can only be applied to polynomial regression with a single independent variable.