A new training set-based regularization for regression techniques

Abstract

The paper gives a new regularization criterion for the regression techniques where the overfitting problem may occur. The proposed criterion is not a penalization term calibrated from prior information but a penalization term calculated from the training set. It appears as an extension of the classic Tikhonov regularization constraint. It is shown that the statistical characterization of this penalization is possible. This characterization leads to an optimization criterion which does not depend on any hyperparameter. The method is applied to a parametric regression technique (polynomial regression) and to a nonparametric regression technique (kernel approximation). For the first technique, overfitting is avoided. For the second one, the method gives an estimation of the kernel spread close to the optimal value.