Robust Regression Problem Research Articles

Least squares regression under weak moment conditions

In this paper we consider the robust regression problem when the output variable may be heavy-tailed. In such scenarios, the traditional least squares regression paradigm is usually thought to be not a good choice as it lacks robustness to outliers. By projecting the outputs onto an adaptive interval, we show the regularized least squares regression can still work well when the conditional distribution satisfies a weak moment condition. Fast convergence rates in various norm are derived by tuning the projection scale parameter and regularization parameter in according with the sample size and the moment condition.

Robust Personalized Federated Learning with Sparse Penalization

Federated learning (FL) is an emerging topic due to its advantage in collaborative learning with distributed data. Due to the heterogeneity in the local data-generating mechanism, it is important to consider personalization when developing federated learning methods. In this work, we propose a personalized federated learning (PFL) method to address the robust regression problem. Specifically, we aim to learn the regression weight by solving a Huber loss with the sparse fused penalty. Additionally, we designed our personalized federated learning for robust and sparse regression (PerFL-RSR) algorithm to solve the estimation problem in the federated system efficiently. Theoretically, we show that the proposed PerFL-RSR reaches a convergence rate of O ( 1 / T ) , and the proposed estimator is statistically consistent. Thorough experiments and real data analysis are conducted to corroborate the theoretical results of our proposed personalized federated learning method. Supplementary materials for this article are available online including a standardized description of the materials available for reproducing the work.

Generalization analysis of deep CNNs under maximum correntropy criterion

Convolutional neural networks (CNNs) have gained immense popularity in recent years, finding their utility in diverse fields such as image recognition, natural language processing, and bio-informatics. Despite the remarkable progress made in deep learning theory, most studies on CNNs, especially in regression tasks, tend to heavily rely on the least squares loss function. However, there are situations where such learning algorithms may not suffice, particularly in the presence of heavy-tailed noises or outliers. This predicament emphasizes the necessity of exploring alternative loss functions that can handle such scenarios more effectively, thereby unleashing the true potential of CNNs. In this paper, we investigate the generalization error of deep CNNs with the rectified linear unit (ReLU) activation function for robust regression problems within an information-theoretic learning framework. Our study demonstrates that when the regression function exhibits an additive ridge structure and the noise possesses a finite pth moment, the empirical risk minimization scheme, generated by the maximum correntropy criterion and deep CNNs, achieves fast convergence rates. Notably, these rates align with the mini-max optimal convergence rates attained by fully connected neural network model with the Huber loss function up to a logarithmic factor. Additionally, we further establish the convergence rates of deep CNNs under the maximum correntropy criterion when the regression function resides in a Sobolev space on the sphere.

Adaptive integral alternating minimization method for robust learning of nonlinear dynamical systems from highly corrupted data

This paper proposes an adaptive integral alternating minimization method (AIAMM) for learning nonlinear dynamical systems using highly corrupted measured data. This approach selects and identifies the system directly from noisy data using the integral model, encompassing unknown sparse coefficients, initial values, and outlier noisy data within the learning problem. It is defined as a sparse robust linear regression problem. An adaptive threshold parameter selection method is proposed to constrain model fitting errors and select appropriate threshold parameters for sparsity. The robustness and accuracy of the proposed AIAMM are demonstrated through several numerical experiments on typical nonlinear dynamical systems, including the van der Pol oscillator, Mathieu oscillator, Lorenz system, and 5D self-exciting homopolar disc dynamo. The proposed method is also compared to several advanced methods for sparse recovery, with the results indicating that the AIAMM demonstrates superior performance in processing highly corrupted data.

A truncated three-term conjugate gradient method with complexity guarantees with applications to nonconvex regression problem

In this paper, a truncated three-term conjugate gradient method is proposed for nonconvex unconstrained optimization. At each iteration, the search direction generated by this method is sufficiently descent independent of any line search. We investigate its global convergence with Wolfe line search under the standard assumptions. Moreover, we present its complexity analysis under the objective function with the Lipschitz continuously gradient and Armijo line search which implies that the proposed method is also globally convergent with Armijo line search. A remarkable point about its complexity is that the proposed method possesses the same complexity order as the classical gradient descent method without any restart strategy. Finally, the proposed method is applied to solve the nonconvex robust regression problems. Numerical results show that the proposed method is efficient.

Functional linear regression with Huber loss

In this paper, we consider the robust regression problem associated with Huber loss in the framework of functional linear model and reproducing kernel Hilbert spaces. We propose an Ivanov regularized empirical risk minimization estimation procedure to approximate the slope function of the linear model in the presence of outliers or heavy-tailed noises. By appropriately tuning the scale parameter of the Huber loss, we establish explicit rates of convergence for our estimates in terms of excess prediction risk under mild assumptions. Our study in the paper justifies the efficiency of Huber regression for functional data from a theoretical viewpoint.

Penalized weighted proportional hazards model for robust variable selection and outlier detection.

Identifying exceptional responders or nonresponders is an area of increased research interest in precision medicine as these patients may have different biological or molecular features and therefore may respond differently to therapies. Our motivation stems from a real example from a clinical trial where we are interested in characterizing exceptional prostate cancer responders. We investigate the outlier detection and robust regression problem in the sparse proportional hazards model for censored survival outcomes. The main idea is to model the irregularity of each observation by assigning an individual weight to the hazard function. By applying a LASSO-type penalty on both the model parameters and the log transformation of the weight vector, our proposed method is able to perform variable selection and outlier detection simultaneously. The optimization problem can be transformed to a typical penalized maximum partial likelihood problem and thus it is easy to implement. We further extend the proposed method to deal with the potential outlier masking problem caused by censored outcomes. The performance of the proposed estimator is demonstrated with extensive simulation studies and real data analyses in low-dimensional and high-dimensional settings.

A nonlinear conjugate gradient method with complexity guarantees and its application to nonconvex regression

Nonlinear conjugate gradients are among the most popular techniques for solving continuous optimization problems. Although these schemes have long been studied from a global convergence standpoint, their worst-case complexity properties have yet to be fully understood, especially in the nonconvex setting. In particular, it is unclear whether nonlinear conjugate gradient methods possess better guarantees than first-order methods such as gradient descent. Meanwhile, recent experiments have shown impressive performance of standard nonlinear conjugate gradient techniques on certain nonconvex problems, even when compared with methods endowed with the best known complexity guarantees. In this paper, we propose a nonlinear conjugate gradient scheme based on a simple line-search paradigm and a modified restart condition. These two ingredients allow for monitoring the properties of the search directions, which is instrumental in obtaining complexity guarantees. Our complexity results illustrate the possible discrepancy between nonlinear conjugate gradient methods and classical gradient descent. A numerical investigation on nonconvex robust regression problems as well as a standard benchmark illustrate that the restarting condition can track the behavior of a standard implementation. • Modified restart condition in nonlinear CG that leads to complexity guarantees. • Comparison with the complexity of gradient descent on nonconvex problems. • Numerical investigation of the new condition's practical impact.

Subgradient averaging for multi-agent optimisation with different constraint sets

We consider a multi-agent setting with agents exchanging information over a possibly time-varying network, aiming at minimising a separable objective function subject to constraints. To achieve this objective we propose a novel subgradient averaging algorithm that allows for non-differentiable objective functions and different constraint sets per agent. Allowing different constraints per agent simultaneously with a time-varying communication network constitutes a distinctive feature of our approach, extending existing results on distributed subgradient methods. To highlight the necessity of dealing with a different constraint set within a distributed optimisation context, we analyse a problem instance where an existing algorithm does not exhibit a convergent behaviour if adapted to account for different constraint sets. For our proposed iterative scheme we show asymptotic convergence of the iterates to a minimum of the underlying optimisation problem for step sizes of the form ηk+1, η>0. We also analyse this scheme under a step size choice of ηk+1, η>0, and establish a convergence rate of O(lnkk) in objective value. To demonstrate the efficacy of the proposed method, we investigate a robust regression problem and an ℓ2 regression problem with regularisation.

A Framework of Learning Through Empirical Gain Maximization.

We develop in this letter a framework of empirical gain maximization (EGM) to address the robust regression problem where heavy-tailed noise or outliers may be present in the response variable. The idea of EGM is to approximate the density function of the noise distribution instead of approximating the truth function directly as usual. Unlike the classical maximum likelihood estimation that encourages equal importance of all observations and could be problematic in the presence of abnormal observations, EGM schemes can be interpreted from a minimum distance estimation viewpoint and allow the ignorance of those observations. Furthermore, we show that several well-known robust nonconvex regression paradigms, such as Tukey regression and truncated least square regression, can be reformulated into this new framework. We then develop a learning theory for EGM by means of which a unified analysis can be conducted for these well-established but not fully understood regression approaches. This new framework leads to a novel interpretation of existing bounded nonconvex loss functions. Within this new framework, the two seemingly irrelevant terminologies, the well-known Tukey's biweight loss for robust regression and the triweight kernel for nonparametric smoothing, are closely related. More precisely, we show that Tukey's biweight loss can be derived from the triweight kernel. Other frequently employed bounded nonconvex loss functions in machine learning, such as the truncated square loss, the Geman-McClure loss, and the exponential squared loss, can also be reformulated from certain smoothing kernels in statistics. In addition, the new framework enables us to devise new bounded nonconvex loss functions for robust learning.

Graphical Convergence of Subgradients in Nonconvex Optimization and Learning

We investigate the stochastic optimization problem of minimizing population risk, where the loss defining the risk is assumed to be weakly convex. Compositions of Lipschitz convex functions with smooth maps are the primary examples of such losses. We analyze the estimation quality of such nonsmooth and nonconvex problems by their sample average approximations. Our main results establish dimension-dependent rates on subgradient estimation in full generality and dimension-independent rates when the loss is a generalized linear model. As an application of the developed techniques, we analyze the nonsmooth landscape of a robust nonlinear regression problem.

Robust Optimization for Models with Uncertain Second-Order Cone and Semidefinite Programming Constraints

In this paper, we consider uncertain second-order cone (SOC) and semidefinite programming (SDP) constraints with polyhedral uncertainty, which are in general computationally intractable. We propose to reformulate an uncertain SOC or SDP constraint as a set of adjustable robust linear optimization constraints with an ellipsoidal or semidefinite representable uncertainty set, respectively. The resulting adjustable problem can then (approximately) be solved by using adjustable robust linear optimization techniques. For example, we show that if linear decision rules are used, then the final robust counterpart consists of SOC or SDP constraints, respectively, which have the same computational complexity as the nominal version of the original constraints. We propose an efficient method to obtain good lower bounds. Moreover, we extend our approach to other classes of robust optimization problems, such as nonlinear problems that contain wait-and-see variables, linear problems that contain bilinear uncertainty, and general conic constraints. Numerically, we apply our approach to reformulate the problem on finding the minimum volume circumscribing ellipsoid of a polytope and solve the resulting reformulation with linear and quadratic decision rules as well as Fourier-Motzkin elimination. We demonstrate the effectiveness and efficiency of the proposed approach by comparing it with the state-of-the-art copositive approach. Moreover, we apply the proposed approach to a robust regression problem and a robust sensor network problem and use linear decision rules to solve the resulting adjustable robust linear optimization problems, which solve the problem to (near) optimality. Summary of Contribution: Computing robust solutions for nonlinear optimization problems with uncertain second-order cone and semidefinite programming constraints are of much interest in real-life applications, yet they are in general computationally intractable. This paper proposes a computationally tractable approximation for such problems. Extensive computational experiments on (i) computing the minimum volume circumscribing ellipsoid of a polytope, (ii) robust regressions, and (iii) robust sensor networks are conducted to demonstrate the effectiveness and efficiency of the proposed approach.

Recovery guarantees for polynomial coefficients from weakly dependent data with outliers

Learning non-linear systems from noisy, limited, and/or dependent data is an important task across various scientific fields including statistics, engineering, computer science, mathematics, and many more. In general, this learning task is ill-posed; however, additional information about the data’s structure or on the behavior of the unknown function can make the task well-posed. In this work, we study the problem of learning nonlinear functions from corrupted and weakly dependent data. The learning problem is recast as a sparse robust linear regression problem where we incorporate both the unknown coefficients and the corruptions in a basis pursuit framework. The main contribution of our paper is to provide a reconstruction guarantee for the associated ℓ1-optimization problem where the sampling matrix is formed from weakly dependent data. Specifically, we prove that the sampling matrix satisfies the null space property and the stable null space property, provided that the data is compact and satisfies a suitable concentration inequality. We show that our recovery results are applicable to various types of weakly dependent data such as exponentially strongly α-mixing data, geometrically C-mixing data, and uniformly ergodic Markov chain. Our theoretical results are verified via several numerical simulations.

Robust support vector machine with generalized quantile loss for classification and regression

A new robust loss function (called Lq-loss) is proposed based on the concept of quantile and correntropy, which can be seen as an improved version of quantile loss function. The proposed Lq-loss has some important properties such as asymmetry, non-convexity and boundedness, which has received a lot of attention recently. The Lq-loss includes and extends the traditional loss functions such as pinball loss, rescaled hinge loss, L1-norm loss and zero-norm loss. Additionally, we demonstrate that the Lq-loss is a kernel-induced loss by reproducing piecewise kernel function. Further, two robust SVM frameworks are presented to handle robust classification and regression problems by applying Lq-loss to support vector machine, respectively. Last but not least, we demonstrate that the proposed classification framework satisfies Bayes’ optimal decision rule. However, the non-convexity of the proposed Lq-loss makes it difficult to optimize. A non-convex optimization method, concave–convex procedure (CCCP) technique, is used to solve the proposed models, and the convergence of the algorithms is proved theoretically. For classification and regression tasks, experiments are carried out on three databases including UCI benchmark datasets, artificial datasets and a practical application dataset. Compared to some classical and advanced methods, numerical simulations under different noise setting and different evaluation criteria show that the proposed methods have good robustness to feature noise and outliers in both classification and regression applications.

A Robust Hybrid Filtering Method for Accurate Battery Remaining Useful Life Prediction

Accurate remaining useful life (RUL) prediction under the noisy environment is a big challenge for the health management of modern industrial systems since the extraction of the accurate data structure from heavily corrupted data is difficult. In recent years, the kernel adaptive filter (KAF) has been widely adopted to solve the robust regression problem due to its low-complexity and high-approximation capability and robustness while the applications in battery RUL prediction are still few and far between. Thus, this paper is concerned with long-term RUL prediction using the KAF method. At first, concretely speaking, a robust KAF algorithm is derived based on the double-Gaussian-mixture (DGM) cost function, which is used to learn the capacity degradation mechanism from contaminated capacity data and so as to build the long-term prediction model. Second, a robust unscented Kalman filter (UKF) algorithm employing the DGM-based cost function is developed, which is then combined with the KAF-based prediction model to realize a more accurate and reliable prediction. Under the hybrid prognostic framework, the proposed UKF algorithm is applied to filter the noisy observations. When the observation data are inaccessible, the predicted data from the off-line trained KAF-based prediction model are adopted as the approximated value of the real observations for the UKF algorithm to optimize the prediction results and to provide the uncertainty representation. The experimental results reveal that the proposed method has great robustness when the measurements contain noise and large outliers, which makes it possible to get satisfactory prediction performance without preprocessing the data manually.

Convergence of the Iterates in Mirror Descent Methods

We consider centralized and distributed mirror descent (MD) algorithms over a finite-dimensional Hilbert space, and prove that the problem variables converge to an optimizer of a possibly nonsmooth function when the step sizes are square summable but not summable. Prior literature has focused on the convergence of the function value to its optimum. However, applications from distributed optimization and learning in games require the convergence of the variables to an optimizer, which is generally not guaranteed without assuming strong convexity of the objective function. We provide numerical simulations comparing entropic MD and standard subgradient methods for the robust regression problem.

Computational and statistical analyses for robust non-convex sparse regularized regression problem

A robust and sparse estimation technique for linear regression problem is studied in this paper. Standard regression with Lasso, SCAD and MCP regularizations is not robust against outliers since it involves the least squares. To handle outliers, a two-stage procedure is proposed; at the first stage an initial estimator is calculated and then it is improved at the second stage by iteratively solving a sparse regression problem with reducing outlier effects. This procedure includes not only a random error but also a computational error. The convergence performance for the final estimator is investigated in both computational and statistical perspectives.

Robust feature matching via Gaussian field criterion for remote sensing image registration

Feature matching, which refers to establishing reliable feature correspondences between two images of the same scene, is a critical prerequisite in a wide range of remote sensing tasks including environment monitoring, multispectral image fusion, image mosaic, change detection, map updating. In this paper, we propose a method for robust feature matching and apply it to the problem of remote sensing image registration. We start by creating a set of putative feature matches which can contain a number of unknown false matches, and then focus on mismatch removal. This is formulated as a robust regression problem, and we customize a robust estimator, namely the Gaussian field criterion, to solve it. The robust criterion can handle both linear and nonlinear image transformations. In the linear case, we use a general homography to model the transformation, while in the nonlinear case, the non-rigid functions located in a reproducing kernel Hilbert space are considered, and a regularization term is added to the objective function to ensure its well-posedness. Moreover, we apply a sparse approximation to the non-rigid transformation and reduce the computational complexity from cubic to linear. Extensive experiments on various natural and remote sensing images show the effectiveness of our approach, which is able to yield superior results compared to other state-of-the-art methods.

Robustness analysis of a maximum correntropy framework for linear regression

In this paper we formulate a solution of the robust linear regression problem in a general framework of correntropy maximization. Our formulation yields a unified class of estimators which includes the Gaussian and Laplacian kernel-based correntropy estimators as special cases. An analysis of the robustness properties is then provided. The analysis includes a quantitative characterization of the informativity degree of the regression which is appropriate for studying the stability of the estimator. Using this tool, a sufficient condition is expressed under which the parametric estimation error is shown to be bounded. Explicit expression of the bound is given and discussion on its numerical computation is supplied. For illustration purpose, two special cases are numerically studied.

On a Class of Optimization-Based Robust Estimators

We consider in this paper the problem of estimating a parameter matrix from observations which are affected by two types of noise components: (i) a sparse noise sequence which, whenever nonzero can have arbitrarily large amplitude (ii) and a dense and bounded noise sequence of "moderate" amount. This is termed a robust regression problem. To tackle it, a quite general optimization-based framework is proposed and analyzed. When only the sparse noise is present, a sufficient bound is derived on the number of nonzero elements in the sparse noise sequence that can be accommodated by the estimator while still returning the true parameter matrix. While almost all the restricted isometry-based bounds from the literature are not verifiable, our bound can be easily computed through solving a convex optimization problem. Moreover, empirical evidence tends to suggest that it is generally tight. If in addition to the sparse noise sequence, the training data are affected by a bounded dense noise, we derive an upper bound on the estimation error.