Automatic relevance determination for least squares support vector machine regression

Abstract

In the past automatic relevance determination (ARD) has been applied to multilayer perceptrons in order to select the relevant inputs of the regression problem at hand. This is obtained by inferring different regularization parameters for the input interconnection layer within the evidence framework. In this paper, the idea is extended to least squares support vector machines (LS-SVMs) for regression by weighting the inputs so as to obtain the most probable model for the given finite number of data points. On the first level of Bayesian inference, a probabilistic framework is related to the LS-SVM regression formulation. The hyper-parameters that control the trade-off between regularization and error minimization are inferred on the second level of inference. Model comparison is performed on the third bevel in order to select the parameters of the kernel function. ARD is performed by introducing a diagonal weighting matrix in the kernel function. These diagonal elements are obtained by evidence maximization on the third level of inference. Inputs with a low scaling in the radial basis function kernel are less relevant and can be removed.