Variational Inference based Automatic Relevance Determination Kernel for Embedded Feature Selection of Noisy Industrial Data

Abstract

In this paper, an embedded feature selection based on variational relevance vector machines is proposed to simultaneously perform feature selection and model construction. With the settings of specific hierarchical priors over the parameters of an automatic relevance determination kernel (ARDK) function, an approximate posterior distribution over these parameters is here derived and expressed as a multivariate Gaussian distribution, in which a first-order Taylor expansion-based Laplace approximation with respect to the parameters is introduced into the variational inference procedure. The posterior distributions, rather than generic pointwise estimates, over the rest of parameters of the model are also derived. The proposed method can simultaneously select relevant features and samples by adjusting the parameters of ARDK and the weighting vector, respectively. To verify the effectiveness of the proposed method, a synthetic dataset and a number of benchmark datasets, as well as a practical industrial dataset, are employed to solve the regression and classification problems. These experimental results indicate that the proposed method supports the mechanisms of feature selection and model construction while maintaining prediction performance, particularly in an industrial environment.