Simultaneous variable selection and component selection for regression density estimation with mixtures of heteroscedastic experts

Abstract

This paper is concerned with the problem of flexibly estimating the conditional density of a response variable given covariates. In our approach the density is modeled as a mixture of heteroscedastic normals with the means, variances and mixing probabilities all varying smoothly as functions of the covariates. We use the variational Bayes approach and propose a novel fast algorithm for simultaneous covariate selection, component selection and parameter estimation. Our method is able to deal with the local maxima problem inherent in mixture model fitting, and is applicable to high-dimensional settings where the number of covariates can be larger than the sample size. In the special case of the classical regression model, the proposed algorithm is similar to currently used greedy algorithms while having many attractive properties and working efficiently in high-dimensional problems. The methodology is demonstrated through simulated and real examples.