Kernelized Elastic Net Regularization based on Markov selective sampling

Abstract

This paper extends Kernelized Elastic Net Regularization (KENReg) algorithm from the assumption of independent and identically distributed (i.i.d.) samples to the case of non-i.i.d. samples. We first establish the generalization bounds of KENReg algorithm with uniformly ergodic Markov chain samples, then we prove that the KENReg algorithm with uniformly ergodic Markov chain samples is consistent and obtain the fast learning rate of KENReg algorithm with uniformly ergodic Markov chain samples. We also introduce the KENReg algorithm based on Markov selective sampling. Based on Gaussian kernels, the advantages of KENReg algorithm against the traditional one with i.i.d. samples are demonstrated on various real-world datasets. Compared to randomly independent sampling, experimental results show that the KENReg algorithm based on Markov selective sampling not only has much higher prediction accuracy in terms of mean square errors and generates simpler models in terms of the number of non-zero regression coefficients, but also has shorter total time of sampling and training. We compare the algorithm proposed in this paper with these known regularization algorithms, like kernelized Ridge regression and kernelized Least absolute shrinkage and selection operator (Lasso).