Dependent Hypothesis Space Research Articles

Group sparse additive machine with average top-k loss

Sparse additive models have shown competitive performance for high-dimensional variable selection and prediction due to their representation flexibility and interpretability. Despite their theoretical properties have been studied extensively, few works have addressed the robustness for the sparse additive models. In this paper, we employ the robust average top-k (ATk) loss as classification error measure and propose a new sparse algorithm, named ATkgroup sparse additive machine (ATk-GSAM). Besides the robust concern, the ATk-GSAM has well adaptivity by integrating the data dependent hypothesis space and group sparse regularizer together. Generalization error bound is established by the concentration estimate with empirical covering numbers. In particular, our error analysis shows that ATk-GSAM can achieve the learning rate O(n−1/2) under appropriate conditions. We further analyze the robustness of ATk-GSAM via a sample-weighted procedure interpretation, and the theoretical guarantees on grouped variable selection. Experimental evaluations on both simulated and benchmark datasets validate the effectiveness and robustness of the new algorithm.

Error analysis for l^{q}-coefficient regularized moving least-square regression

We consider the moving least-square (MLS) method by the coefficient-based regression framework with l^{q}-regularizer (1leq qleq2) and the sample dependent hypothesis spaces. The data dependent characteristic of the new algorithm provides flexibility and adaptivity for MLS. We carry out a rigorous error analysis by using the stepping stone technique in the error decomposition. The concentration technique with the l^{2}-empirical covering number is also employed in our study to improve the sample error. We derive the satisfactory learning rate that can be arbitrarily close to the best rate O(m^{-1}) under more natural and much simpler conditions.

Quantile Regression Based on Laplacian Manifold Regularizer with the Data Sparsity in <i>l</i>1 Spaces

In this paper, we consider the regularized learning schemes based on l1-regularizer and pinball loss in a data dependent hypothesis space. The target is the error analysis for the quantile regression learning. There is no regularized condition with the kernel function, excepting continuity and boundness. The graph-based semi-supervised algorithm leads to an extra error term called manifold error. Part of new error bounds and convergence rates are exactly derived with the techniques consisting of l1-empirical covering number and boundness decomposition.

Kernel-based sparse regression with the correntropy-induced loss

The correntropy-induced loss (C-loss) has been employed in learning algorithms to improve their robustness to non-Gaussian noise and outliers recently. Despite its success on robust learning, only little work has been done to study the generalization performance of regularized regression with the C-loss. To enrich this theme, this paper investigates a kernel-based regression algorithm with the C-loss and ℓ1-regularizer in data dependent hypothesis spaces. The asymptotic learning rate is established for the proposed algorithm in terms of novel error decomposition and capacity-based analysis technique. The sparsity characterization of the derived predictor is studied theoretically. Empirical evaluations demonstrate its advantages over the related approaches.

Asymptotic analysis of quantile regression learning based on coefficient dependent regularization

In this paper, we consider conditional quantile regression learning algorithms based on the pinball loss with data dependent hypothesis space and ℓ2-regularizer. Functions in this hypothesis space are linear combination of basis functions generated by a kernel function and sample data. The only conditions imposed on the kernel function are the continuity and boundedness which are pretty weak. Our main goal is to study the consistency of this regularized quantile regression learning. By concentration inequality with ℓ2-empirical covering numbers and operator decomposition techniques, satisfied error bounds and convergence rates are explicitly derived.

On empirical eigenfunction-based ranking with [formula omitted] norm regularization

The problem of ranking, in which the goal is to learn a real-valued ranking function that induces a ranking over an instance space, has recently gained increasing attention in machine learning. We study a learning algorithm for ranking generated by a regularized scheme with an ℓ1 regularizer. The algorithm is formulated in a data dependent hypothesis space. Such a space is spanned by empirical eigenfunctions which are constructed by a Mercer kernel and the learning data. We establish the computations of empirical eigenfunctions and the representer theorem for the algorithm. Particularly, we provide an analysis of the sparsity and convergence rates for the algorithm. The results show that our algorithm produces both satisfactory convergence rates and sparse representations under a mild condition, especially without assuming sparsity in terms of any basis.

Convergence Analysis of an Empirical Eigenfunction-Based Ranking Algorithm with Truncated Sparsity

We study an empirical eigenfunction-based algorithm for ranking with a data dependent hypothesis space. The space is spanned by certain empirical eigenfunctions which we select by using a truncated parameter. We establish the representer theorem and convergence analysis of the algorithm. In particular, we show that under a mild condition, the algorithm produces a satisfactory convergence rate as well as sparse representations with respect to the empirical eigenfunctions.

CONVERGENCE ANALYSIS OF COEFFICIENT-BASED REGULARIZATION UNDER MOMENT INCREMENTAL CONDITION

In this paper, we investigate coefficient-based regularized least squares regression problem in a data dependent hypothesis space. The learning algorithm is implemented with samples drawn by unbounded sampling processes and the error analysis is performed by a stepping-stone technique. A new error decomposition technique is proposed for the error analysis. The regularization parameters in our setting provide much more flexibility and adaptivity. Sharp learning rates are addressed by means of l2-empirical covering numbers under a moment hypothesis condition.

无界抽样情形下不定核的系数正则化回归

We investigate the coefficient-based regularized least squares regression with unbounded sampling in a data dependent hypothesis space. The learning scheme is essentially different from the standard one in a reproducing kernel Hilbert space: we do not need the kernel to be symmetric or positive semi-definite except for continuity and boundedness, the regularizer is the <italic>l</italic>²-norm of a function expansion involving samples and the unboudedness of the sampling output. This leads to additional difficulty in the error analysis. In this paper, the goal is to investigate some concentration estimates for the error based on <italic>l</italic>²-empirical covering numbers without the assumption of uniform boundedness for sampling. By introducing a suitable reproducing kernel Hilbert space and applying concentration techniques with <italic>l</italic>²-empirical covering numbers, we derive satisfactory learning rates in terms of regularity of the regression function and capacity of the hypothesis space.

Learning rates of multi-kernel regression by orthogonal greedy algorithm

We investigate the problem of regression from multiple reproducing kernel Hilbert spaces by means of orthogonal greedy algorithm. The greedy algorithm is appealing as it uses a small portion of candidate kernels to represent the approximation of regression function, and can greatly reduce the computational burden of traditional multi-kernel learning. Satisfied learning rates are obtained based on the Rademacher chaos complexity and data dependent hypothesis spaces.

Learning theory estimates for coefficient-based regularized regression

We consider a coefficient-based regularized regression in a data dependent hypothesis space. For a given set of samples, functions in this hypothesis space are defined to be linear combinations of basis functions generated by a kernel function and sample data. We do not need the kernel to be symmetric or positive semi-definite, which provides flexibility and adaptivity for the learning algorithm. Another advantage of this algorithm is that, it is computationally effective without any optimization processes. In this paper, we apply concentration techniques with ℓ2-empirical covering numbers to present an elaborate capacity dependent analysis for the algorithm, which yields shaper estimates in both confidence estimation and convergence rate. When the kernel is C∞, under a very mild regularity condition on the regression function, the rate can be arbitrarily close to m−1.

Learning rates for least square regressions with coefficient regularization

We analyze the learning rates for the least square regression with data dependent hypothesis spaces and coefficient regularization algorithms based on general kernels. Under a very mild regularity condition on the regression function, we obtain a bound for the approximation error by estimating the corresponding K-functional. Combining this estimate with the previous result of the sample error, we derive a dimensional free learning rate by the proper choice of the regularization parameter.

Learning Rates of Support Vector Machine Classifiers with Data Dependent Hypothesis Spaces

We study the error performances of -norm Support Vector Machine classifiers based on reproducing kernel Hilbert spaces. We focus on two category problem and choose the data-dependent polynomial kernels as the Mercer kernel to improve the approximation error. We also provide the standard estimation of the sample error, and derive the explicit learning rate.

An empirical feature-based learning algorithm producing sparse approximations

A learning algorithm for regression is studied. It is a modified kernel projection machine (Blanchard et al., 2004 [2]) in the form of a least square regularization scheme with ℓ1-regularizer in a data dependent hypothesis space based on empirical features (constructed by a reproducing kernel and the learning data). The algorithm has three advantages. First, it does not involve any optimization process. Second, it produces sparse representations with respect to empirical features under a mild condition, without assuming sparsity in terms of any basis or system. Third, the output function converges to the regression function in the reproducing kernel Hilbert space at a satisfactory rate. Our error analysis does not require any sparsity assumption about the underlying regression function.

Unified approach to coefficient-based regularized regression

In this paper, we consider the coefficient-based regularized least-squares regression problem with the l q -regularizer ( 1 ≤ q ≤ 2 ) and data dependent hypothesis spaces. Algorithms in data dependent hypothesis spaces perform well with the property of flexibility. We conduct a unified error analysis by a stepping stone technique. An empirical covering number technique is also employed in our study to improve sample error. Comparing with existing results, we make a few improvements: First, we obtain a significantly sharper learning rate that can be arbitrarily close to O ( m − 1 ) under reasonable conditions, which is regarded as the best learning rate in learning theory. Second, our results cover the case q = 1 , which is novel. Finally, our results hold under very general conditions.

Concentration estimates for learning with [formula omitted]-regularizer and data dependent hypothesis spaces

We consider the regression problem by learning with a regularization scheme in a data dependent hypothesis space and ℓ 1 -regularizer. The data dependence nature of the kernel-based hypothesis space provides flexibility for the learning algorithm. The regularization scheme is essentially different from the standard one in a reproducing kernel Hilbert space: the kernel is not necessarily symmetric or positive semi-definite and the regularizer is the ℓ 1 -norm of a function expansion involving samples. The differences lead to additional difficulty in the error analysis. In this paper we apply concentration techniques with ℓ 2 -empirical covering numbers to improve the learning rates for the algorithm. Sparsity of the algorithm is studied based on our error analysis. We also show that a function space involved in the error analysis induced by the ℓ 1 -regularizer and non-symmetric kernel has nice behaviors in terms of the ℓ 2 -empirical covering numbers of its unit ball.

Learning with sample dependent hypothesis spaces

Many learning algorithms use hypothesis spaces which are trained from samples, but little theoretical work has been devoted to the study of these algorithms. In this paper we show that mathematical analysis for these algorithms is essentially different from that for algorithms with hypothesis spaces independent of the sample or depending only on the sample size. The difficulty lies in the lack of a proper characterization of approximation error. To overcome this difficulty, we propose an idea of using a larger function class (not necessarily linear space) containing the union of all possible hypothesis spaces (varying with the sample) to measure the approximation ability of the algorithm. We show how this idea provides error analysis for two particular classes of learning algorithms in kernel methods: learning the kernel via regularization and coefficient based regularization. We demonstrate the power of this approach by its wide applicability.