Robust Regression Framework Research Articles

Unified distributed robust regression and variable selection framework for massive data

This paper proposes a unified distributed robust regression framework for distributed massive data, which can include many robust regressions in one setting. Specifically, we first transfer different types of robust regressions into an asymptotically equivalent least-squares problem. Then the resulting estimator can be calculated as a weighted average of robust local estimators, and the communication cost is reduced, since it involves only one round of communication. In addition, since the local data information is incorporated sufficiently, it is adaptive to the heterogeneity. The new estimator is proven to be equivalent with the corresponding global robust regression estimator. Furthermore, we conduct variable selection based on the unified robust regression framework and adaptive LASSO, and the path of solution can also be conveniently obtained by LARS algorithm. It is theoretically shown that the new variable selection method can select true relevant variables consistently by using a new distributed BIC-type tuning parameter selector. The simulation results confirm the effectiveness of the new methods and the correctness of the theoretical results.

Dual robust regression for pattern classification

Linear regression-based and extended methods have been widely used in pattern classification. These methods can be roughly divided into two categories: reconstruction error-based methods and discriminative methods. The reconstruction error-based methods search the target label by learning the representation coefficients to compute the minimal reconstruction error. The goal of the discriminative methods is to learn the projection matrix to predict the target label of the image. To combine the advantages of these two kinds of regression-based methods, this paper presents a dual robust regression framework (DualRR) for pattern classification. In the training stage, a double low-rank robust regression model (DLR) is proposed to learn the projection matrix. In DLR, low-rank robust regression motivates us to model the data as the sum of a low-rank clean data and sparse noise matrix. The low-rank is further used to constrain the projection matrix to enhance the discriminative performance. In the testing stage, the proposed framework employs a robust regression representation model to learn the optimal representation coefficients and obtain the reconstruction sample to approximate the test sample. We thus apply the reconstruction sample to search the classification label by using the projection matrix learned in the training stage. Extensive experiments are conducted on six public available databases, namely, LFW, FRGC, CUHK Sketch, PolyU Palm, NUST-RF and Caltech 101, demonstrating the merits of the proposed model over state-of-the-art regression-based classification methods.

Robust regression framework with asymmetrically analogous to correntropy-induced loss

This work proposes a robust loss function based on expectile penalty (named as rescaled expectile loss, RE-loss), which includes and generalizes the existing loss functions. Then some important properties of RE-loss are demonstrated such as asymmetry, nonconvexity, smoothness, boundedness and asymptotic approximation behaviors. From the viewpoints of correntropy, we analyze that the proposed RE-loss can be viewed as a correntropy-induced loss by a reproducing piecewise kernel. Furthermore, a sparse version of RE-loss (called SRE-loss function) is developed to improve sparsity by introducing a ϵ-insensitive zone. Following that, two robust regression frameworks are proposed with the proposed loss functions. However, the non-convexity of the proposed losses makes the problems difficult to optimize. We apply concave–convex procedure (CCCP) and dual theory to solve the problems effectively. The resulting algorithms converge linearly. To validate the proposed methods, we carry out numerical experiments in different scale datasets with different levels of noises and outliers, respectively. In three databases including artificial database, benchmark database and a practical application database, experimental results demonstrate that the proposed methods achieve better generalization than the traditional regression methods in most cases,especially when noise and outlier distribution are imbalance.

Modal Regression-Based Atomic Representation for Robust Face Recognition and Reconstruction.

Representation-based classification (RC) methods, such as sparse RC, have shown great potential in face recognition (FR) in recent years. Most previous RC methods are based on the conventional regression models, such as lasso regression, ridge regression, or group lasso regression. These regression models essentially impose a predefined assumption on the distribution of the noise variable in the query sample, such as the Gaussian or Laplacian distribution. However, the complicated noises in practice may violate the assumptions and impede the performance of these RC methods. In this paper, we propose a modal regression (MR)-based atomic representation and classification (MRARC) framework to alleviate such limitations. MR is a robust regression framework which aims to reveal the relationship between the input and response variables by regressing toward the conditional mode function. Atomic representation is a general atomic norm regularized linear representation framework which includes many popular representation methods, such as sparse representation, collaborative representation, and low-rank representation as special cases. Unlike previous RC methods, the MRARC framework does not require the noise variable to follow any specific predefined distributions. This gives rise to the capability of MRARC in handling various complex noises in reality. Using MRARC as a general platform, we also develop four novel RC methods for unimodal and multimodal FR, respectively. In addition, we devise a general optimization algorithm for the unified MRARC framework based on the alternating direction method of multipliers and half-quadratic theory. The experiments on real-world data validate the efficacy of MRARC for robust FR and reconstruction.

Robust multi-model adaptation regression with local feature space representation

Learning visual models of object categories generally requires a large number of training examples. We show in this paper that it is possible to uncover much discriminative information about a visual category from a small number of examples with just fewlabeled data. The key insight is that, rather than learning from scratch with the original feature representation, we learn an optimal target model with the local feature space representations by leveraging some prior models pre-trained on other related datasets, no matter how different they might be. This target model can be obtained by learning a robust regression scheme with a multi-source adaptation regularization term, which is formally formulated as a Robust Multi-model Adaptation Regression (RMAR) framework with local feature space representations of the target data. This framework is composed of three core components: (1) local consistency Laplacian regularization by constructing a self-adaptive affinity graph based on local feature space representation (LFSR), which aims to augment the discriminative space of the target domain; (2) robust regression learning scheme with feature selection based on l2,1-norm minimization by explicitly considering the regression noise/outlier as well as the regression residual; and (3) scatter constrained multi-model adaptation regularization for generalizing the established robust regression framework to exploit the prior models, under the assumption that multiple auxiliary discriminative models can help the semi-supervised learning from few labeled samples. Lastly, we further propose an effective extension to RMAR, i.e., ensemble LFSR-graph Laplacians regularization framework. The optimization algorithms for RMAR and its extension can be efficiently solved by the alternating iterative strategy, and the iteration convergence can be theoretically guaranteed. Experiments over several real datasets show the promising performance of our methods compared with several representative state-of-the-art works.

An outlier-robust kernel RLS algorithm for nonlinear system identification

The kernel recursive least squares (KRLS), a nonlinear counterpart of the famed RLS algorithm, performs linear regression in a high-dimensional feature space induced by a Mercer kernel. Despite the growing interest in the KRLS for nonlinear signal processing, the presence of outliers in the estimation data causes the resulting predictor’s performance to deteriorate considerably. Bearing this in mind, we introduce an approach to amalgamate the kernel-based learning framework that gives rise to the KRLS algorithm with the robust regression framework of M-estimators with the aim of building an outlier-robust variant for the KRLS. Initially, we develop the theoretical aspects of the proposed algorithm and then analyze its behavior in nonlinear system identification problems using synthetic and real-world datasets (including a large-scale one) contaminated with outliers. The obtained results indicate that the robust variant of the KRLS algorithm consistently outperforms the state-of-the-art in robust adaptive filtering algorithms, as the amount of outliers in the data increases.

A Robust Regression Framework with Laplace Kernel-Induced Loss.

This work proposes a robust regression framework with nonconvex loss function. Two regression formulations are presented based on the Laplace kernel-induced loss (LK-loss). Moreover, we illustrate that the LK-loss function is a nice approximation for the zero-norm. However, nonconvexity of the LK-loss makes it difficult to optimize. A continuous optimization method is developed to solve the proposed framework. The problems are formulated as DC (difference of convex functions) programming. The corresponding DC algorithms (DCAs) converge linearly. Furthermore, the proposed algorithms are applied directly to determine the hardness of licorice seeds using near-infrared spectral data with noisy input. Experiments in eight spectral regions show that the proposed methods improve generalization compared with the traditional support vector regressions (SVR), especially in high-frequency regions. Experiments on several benchmark data sets demonstrate that the proposed methods achieve better results than the traditional regression methods in most of data sets we have considered.

Robust Winsorized Regression Using Bootstrap Approach

The history of regression analysis when response and explanatory variables are subject to error dates back to Eisenhart (1939) and Wald (1940). Now the subject is discussed in the robust regression framework involving “leverage” points in addition to the traditional “outliers,” e.g., see Rousseeuw and Leroy (1987). In this article, we develop two variations of co-ordinatewise Winsorized-bootstrapped approach and study their properties using empirical methods and illustrative examples. The new methods are seen to provide significant improvement when the data are in a neighborhood of multivariate normal population without significant loss in the performance when the multivariate normal model holds.