Robust Loss Function Research Articles

Enhanced robustness in machine learning: Application of an adaptive robust loss function

In the field of machine learning, the selection of a loss function plays a pivotal role in determining the training dynamics and generalization capability of models. Traditional loss functions such as mean square error (MSE) and cross-entropy are often not equipped to handle noisy and anomalous data effectively, which can result in models that perform poorly in practical scenarios. To address these shortcomings, this study introduces a novel adaptive robust loss function, enhanced by a tunable parameter , which allows for flexibility in adjusting the robustness of the function according to the nature of the data being processed. Our research demonstrates that this new loss function significantly improves the performance of linear regression and multilayer perceptron models, particularly in environments laden with noisy and anomalous data. By adapting the parameter , the function can cater to varying levels of data irregularities, thus enhancing the models accuracy and reliability across diverse and complex data environments. This adaptive mechanism not only offers a substantial theoretical contribution to the understanding of robust loss functions but also provides a practical tool for machine learning practitioners to develop models that are resilient to data imperfections. The implications of this research are profound, suggesting a shift towards more adaptive and robust approaches in machine learning model development.

Global quantitative robustness of regression feed-forward neural networks

Neural networks are an indispensable model class for many complex learning tasks. Despite the popularity and importance of neural networks and many different established techniques from literature for stabilization and robustification of the training, the classical concepts from robust statistics have rarely been considered so far in the context of neural networks. Therefore, we adapt the notion of the regression breakdown point to regression neural networks and compute the breakdown point for different feed-forward network configurations and contamination settings. In an extensive simulation study, we compare the performance, measured by the out-of-sample loss, by a proxy of the breakdown rate and by the training steps, of non-robust and robust regression feed-forward neural networks in a plethora of different configurations. The results indeed motivate to use robust loss functions for neural network training.

A Noisy Sample Selection Framework Based on a Mixup Loss and Recalibration Strategy

Deep neural networks (DNNs) have achieved breakthrough progress in various fields, largely owing to the support of large-scale datasets with manually annotated labels. However, obtaining such datasets is costly and time-consuming, making high-quality annotation a challenging task. In this work, we propose an improved noisy sample selection method, termed “sample selection framework,” based on a mixup loss and recalibration strategy (SMR). This framework enhances the robustness and generalization abilities of models. First, we introduce a robust mixup loss function to pre-train two models with identical structures separately. This approach avoids additional hyperparameter adjustments and reduces the need for prior knowledge of noise types. Additionally, we use a Gaussian Mixture Model (GMM) to divide the entire training set into labeled and unlabeled subsets, followed by robust training using semi-supervised learning (SSL) techniques. Furthermore, we propose a recalibration strategy based on cross-entropy (CE) loss to prevent the models from converging to local optima during the SSL process, thus further improving performance. Ablation experiments on CIFAR-10 with 50% symmetric noise and 40% asymmetric noise demonstrate that the two modules introduced in this paper improve the accuracy of the baseline (i.e., DivideMix) by 1.5% and 0.5%, respectively. Moreover, the experimental results on multiple benchmark datasets demonstrate that our proposed method effectively mitigates the impact of noisy labels and significantly enhances the performance of DNNs on noisy datasets. For instance, on the WebVision dataset, our method improves the top-1 accuracy by 0.7% and 2.4% compared to the baseline method.

On Mean-Optimal Robust Linear Discriminant Analysis

Linear discriminant analysis (LDA) is widely used for dimensionality reduction under supervised learning settings. Traditional LDA objective aims to minimize the ratio of the squared Euclidean distances that may not perform optimally on noisy datasets. Multiple robust LDA objectives have been proposed to address this problem, but their implementations have two major limitations. One is that their mean calculations use the squared \(\ell_{2}\) -norm distance to center the data, which is not valid when the objective depends on other distance functions. The second problem is that there is no generalized optimization algorithm to solve different robust LDA objectives. In addition, most existing algorithms can only guarantee that the solution is locally optimal rather than globally optimal. In this article, we review multiple robust loss functions and propose a new and generalized robust objective for LDA. Besides, to remove the mean value within data better, our objective uses an optimal way to center the data through learning. As one important algorithmic contribution, we derive an efficient iterative algorithm to optimize the resulting non-smooth and non-convex objective function. We theoretically prove that our solution algorithm guarantees that both the objective and the solution sequences converge to globally optimal solutions at a sub-linear convergence rate. The results of comprehensive experimental evaluations demonstrate the effectiveness of our new method, achieving significant improvements compared to the other competing methods.

Robust adaptive learning framework for semi-supervised pattern classification

Hessian scatter regularized twin support vector machine (HSR-TSVM) employs hinge loss function and L2-distance metric, which makes it ineffective in dealing with outliers and noise data problems. Aiming to this problem, this paper a novel robust adaptive learning framework CL2,pHSR-TSVM is developed for semi-supervised classification tasks. In CL2,pHSR-TSVM, the generalized adaptive robust loss function Lδ(u) is first innovatively introduced to overcome the problem that hinge loss function is not sensitive to noise and outliers. Intuitively, Lδ(u) can improve the robustness of the model by selecting different robust loss functions for different learning tasks during the learning process via the adaptive parameter δ. Secondly, the robust distance metric capped L2,p-norm is introduced in CL2,pHSR-TSVM to reduce and eliminate the exaggerated influence of L2-distance metric on the learning process of outliers, especially when the outliers are far from the normal data distribution, by setting the appropriate parameters. Furthermore, to improve the computational efficiency of CL2,pHSR-TSVM, the fast CL2,pHSR-TSVM is presented for semi-supervised classification tasks. Finally, two effective algorithms are designed to solve our methods respectively, and the convergence and computational complexity are analyzed theoretically. Experimental results demonstrate the effectiveness and robustness of our methods.

Pinball-Huber boosted extreme learning machine regression: a multiobjective approach to accurate power load forecasting

Power load data frequently display outliers and an uneven distribution of noise. To tackle this issue, we present a forecasting model based on an improved extreme learning machine (ELM). Specifically, we introduce the novel Pinball-Huber robust loss function as the objective function in training. The loss function enhances the precision by assigning distinct penalties to errors based on their directions. We employ a genetic algorithm, combined with a swift nondominated sorting technique, for multiobjective optimization in the ELM-Pinball-Huber context. This method simultaneously reduces training errors while streamlining model structure. We practically apply the integrated model to forecast power load data in Taixing City, which is situated in the southern part of Jiangsu Province. The empirical findings confirm the method’s effectiveness.

On the convergence of gradient descent for robust functional linear regression

Functional data analysis offers a set of statistical methods concerned with extracting insights from intrinsically infinite-dimensional data and has attracted considerable amount of attentions in the past few decades. In this paper, we study robust functional linear regression model with a scalar response and a functional predictor in the framework of reproducing kernel Hilbert spaces. A gradient descent algorithm with early stopping is introduced to solve the corresponding empirical risk minimization problem associated with robust loss functions. By appropriately selecting the early stopping rule and the scaling parameter of the robust losses, the convergence of the proposed algorithm is established when the response variable is bounded or satisfies a moment condition. Explicit learning rates with respect to both estimation and prediction error are provided in terms of regularity of the regression function and eigenvalue decay rate of the integral operator induced by the reproducing kernel and covariance function.

Robust sparse concept factorization with graph regularization for subspace learning

Concept factorization (CF) is a powerful tool in subspace learning. Recently graph-based CF and local coordinate CF have been proposed to exploit the intrinsic geometrical structure of data, and have been shown quite successful in improving performance. However, these methods have limited robustness and might be sensitive to noises and disturbances in practical applications. In this paper, we propose a novel robust sparse CF framework (RSCF) for subspace learning. Specifically, we present a robust loss function to effectively eliminate the impact of the large outliers. The local coordinate constraint and the graph regularization term are incorporated into RSCF to simultaneously guarantee the sparsity of the coefficient matrix and maintain the local structure of the data. We prove that the local coordinate constraint implies the orthogonality of the coefficient matrix. By using the half-quadratic optimization technique, we transform the objective function of RSCF into a quadratic form and provide the iterative updating rules and the convergence analysis. Extensive experiments demonstrate the robustness and superiority of the proposed RSCF in comparison to state-of-the-art CF methods.

Mixture correntropy-based robust distance metric learning for classification

Metric learning is a branch of machine learning that aims to learn from the given training data a valid distance metric, with which the similarity between samples can be more effectively evaluated for classification. Metric learning has attracted significant attention, and a large number of models have been proposed in the past few years. However, the traditional methods adopt hinge loss which easily leads to noise sensitivity and instability. In this paper, to improve the robustness performance, we develop a mixture correntropy criterion where two Laplacian kernel functions are combined as the kernel function and induce a more general nonconvex robust loss function by the mixture correntropy. The properties related to the loss function are analysed and presented. The induced loss amalgamates the superiors of the state-of-the-art robust loss functions and is more effective. With this induced loss, we establish a robust metric learning model (called MCML) and design an effective iterative algorithm to optimize the nonconvex challenging problem. The computational complexity and convergence of algorithm are discussed in theory. Furthermore, a boosting version of MCML (BMCML) is derived, where the low-rank basis learning is jointly optimized with the metric to better uncover the data structure. Finally, extensive experiments are conducted on artificial datasets, UCI benchmark datasets and image datasets. The experimental results verify the robustness and effectiveness of the proposed methods.

Mitigating Label Noise through Data Ambiguation

Label noise poses an important challenge in machine learning, especially in deep learning, in which large models with high expressive power dominate the field. Models of that kind are prone to memorizing incorrect labels, thereby harming generalization performance. Many methods have been proposed to address this problem, including robust loss functions and more complex label correction approaches. Robust loss functions are appealing due to their simplicity, but typically lack flexibility, while label correction usually adds substantial complexity to the training setup. In this paper, we suggest to address the shortcomings of both methodologies by "ambiguating" the target information, adding additional, complementary candidate labels in case the learner is not sufficiently convinced of the observed training label. More precisely, we leverage the framework of so-called superset learning to construct set-valued targets based on a confidence threshold, which deliver imprecise yet more reliable beliefs about the ground-truth, effectively helping the learner to suppress the memorization effect. In an extensive empirical evaluation, our method demonstrates favorable learning behavior on synthetic and real-world noise, confirming the effectiveness in detecting and correcting erroneous training labels.

Robust Loss Functions for Training Decision Trees with Noisy Labels

We consider training decision trees using noisily labeled data, focusing on loss functions that can lead to robust learning algorithms. Our contributions are threefold. First, we offer novel theoretical insights on the robustness of many existing loss functions in the context of decision tree learning. We show that some of the losses belong to a class of what we call conservative losses, and the conservative losses lead to an early stopping behavior during training and noise-tolerant predictions during testing. Second, we introduce a framework for constructing robust loss functions, called distribution losses. These losses apply percentile-based penalties based on an assumed margin distribution, and they naturally allow adapting to different noise rates via a robustness parameter. In particular, we introduce a new loss called the negative exponential loss, which leads to an efficient greedy impurity-reduction learning algorithm. Lastly, our experiments on multiple datasets and noise settings validate our theoretical insight and the effectiveness of our adaptive negative exponential loss.

Toward Robust Self-Training Paradigm for Molecular Prediction Tasks.

Molecular prediction tasks normally demand a series of professional experiments to label the target molecule, which suffers from the limited labeled data problem. One of the semisupervised learning paradigms, known as self-training, utilizes both labeled and unlabeled data. Specifically, a teacher model is trained using labeled data and produces pseudo labels for unlabeled data. These labeled and pseudo-labeled data are then jointly used to train a student model. However, the pseudo labels generated from the teacher model are generally not sufficiently accurate. Thus, we propose a robust self-training strategy by exploring robust loss function to handle such noisy labels in two paradigms, that is, generic and adaptive. We have conducted experiments on three molecular biology prediction tasks with four backbone models to gradually evaluate the performance of the proposed robust self-training strategy. The results demonstrate that the proposed method enhances prediction performance across all tasks, notably within molecular regression tasks, where there has been an average enhancement of 41.5%. Furthermore, the visualization analysis confirms the superiority of our method. Our proposed robust self-training is a simple yet effective strategy that efficiently improves molecular biology prediction performance. It tackles the labeled data insufficient issue in molecular biology by taking advantage of both labeled and unlabeled data. Moreover, it can be easily embedded with any prediction task, which serves as a universal approach for the bioinformatics community.

Unified robust estimation

SummaryRobust estimation is primarily concerned with providing reliable parameter estimates in the presence of outliers. Numerous robust loss functions have been proposed in regression and classification, along with various computing algorithms. In modern penalised generalised linear models (GLMs), however, there is limited research on robust estimation that can provide weights to determine the outlier status of the observations. This article proposes a unified framework based on a large family of loss functions, a composite of concave and convex functions (CC‐family). Properties of the CC‐family are investigated, and CC‐estimation is innovatively conducted via the iteratively reweighted convex optimisation (IRCO), which is a generalisation of the iteratively reweighted least squares in robust linear regression. For robust GLM, the IRCO becomes the iteratively reweighted GLM. The unified framework contains penalised estimation and robust support vector machine (SVM) and is demonstrated with a variety of data applications.

Robust hybrid learning approach for adaptive neuro-fuzzy inference systems

The Adaptive Neuro-Fuzzy Inference System (ANFIS) is a regression model that uses fuzzy logic and neural networks, making it suitable for modeling the uncertainty of regression problems. However, the non-robust loss function in ANFIS's hybrid learning algorithm can make it susceptible to the direct effects of noise and outliers. This paper introduces a new procedure that uses robust loss functions to enhance the hybrid learning performance against noise and outliers. In addition, a new robust loss function is devised that can completely ignore outliers. Furthermore, a set of robust loss functions with mathematical relations are suggested. The proposed approach is evaluated on real-world problems, including weather forecasting and stock market prediction. Results suggested that the proposed model can reduce the Mean Square Error (MSE) in regression. Moreover, the new procedure enables utilizing different loss functions based on the application.

Robust time-of-arrival localization via ADMM

This article considers the problem of source localization (SL) using possibly unreliable time-of-arrival (TOA) based range measurements. Adopting the strategy of statistical robustification, we formulate TOA SL as minimization of a versatile loss that possesses resistance against the occurrence of outliers. We then present an alternating direction method of multipliers (ADMM) to tackle the nonconvex optimization problem in a computationally attractive iterative manner. Moreover, we prove that the solution obtained by the proposed ADMM will correspond to a Karush–Kuhn–Tucker point of the formulation when the algorithm converges, and discuss reasonable assumptions about the robust loss function under which the approach can be theoretically guaranteed to be convergent. Numerical investigations demonstrate the superiority of our method over many existing TOA SL schemes in terms of positioning accuracy and computational simplicity. In particular, the proposed ADMM achieves estimation results with mean square error performance closer to the Cramér–Rao lower bound than its competitors in our simulations of impulsive noise environments.

RoBoSS: A Robust, Bounded, Sparse, and Smooth Loss Function for Supervised Learning.

In the domain of machine learning, the significance of the loss function is paramount, especially in supervised learning tasks. It serves as a fundamental pillar that profoundly influences the behavior and efficacy of supervised learning algorithms. Traditional loss functions, though widely used, often struggle to handle outlier-prone and high-dimensional data, resulting in suboptimal outcomes and slow convergence during training. In this paper, we address the aforementioned constraints by proposing a novel robust, bounded, sparse, and smooth (RoBoSS) loss function for supervised learning. Further, we incorporate the RoBoSS loss within the framework of support vector machine (SVM) and introduce a new robust algorithm named LRoBoSS-SVM. For the theoretical analysis, the classification-calibrated property and generalization ability are also presented. These investigations are crucial for gaining deeper insights into the robustness of the RoBoSS loss function in classification problems and its potential to generalize well to unseen data. To validate the potency of the proposed LRoBoSS-SVM, we assess it on 88 benchmark datasets from KEEL and UCI repositories. Further, to rigorously evaluate its performance in challenging scenarios, we conducted an assessment using datasets intentionally infused with outliers and label noise. Additionally, to exemplify the effectiveness of LRoBoSS-SVM within the biomedical domain, we evaluated it on two medical datasets: the electroencephalogram (EEG) signal dataset and the breast cancer (BreaKHis) dataset. The numerical results substantiate the superiority of the proposed LRoBoSS-SVM model, both in terms of its remarkable generalization performance and its efficiency in training time. The code of the LRoBoSS-SVM is publicly accessible at https://github.com/mtanveer1/RoBoSS.

Iterative Unscented Kalman Filter With General Robust Loss Function for Power System Forecasting-Aided State Estimation

Iterative Unscented Kalman Filter With General Robust Loss Function for Power System Forecasting-Aided State Estimation

Study on Robust Loss Function for Artificial Neural Networks Models in Reliability Analysis

This paper presents a robust artificial neural network (ANN) loss function in solving samples with questionable data. The Latin hypercube sampling and the uniform experimental design are used as sample generation while ANN with various loss functions is the limit state function approaching method and Monte Carlo simulation is the failure probability computing technique. A marine lubricating oil cooler under a plus-minus triangular wave and internal pressure (1Mpa) is considered as representative of the dynamic implicit model. Three kinds of questionable data are considered: tiny vibrations(±δ1) caused by the surrounding environment, small deviations caused by equipment error(±δ2), and large deviations(±Δ) caused by accidental events. And two small probability events are considered: 1. Surrounding error + positive equipment error + a large shock (±Δ) in partial samples; 2. Surrounding error + negative equipment error + a large shock (±Δ) in partial samples. ANN with advanced loss function performs better than traditional loss function in dealing with samples including incorrect data.

Incomplete Gamma Kernels: Generalizing Locally Optimal Projection Operators.

We present incomplete gamma kernels, a generalization of Locally Optimal Projection (LOP) operators. In particular, we reveal the relation of the classical localized L1 estimator, used in the LOP operator for point cloud denoising, to the common Mean Shift framework via a novel kernel. Furthermore, we generalize this result to a whole family of kernels that are built upon the incomplete gamma function and each represents a localized Lp estimator. By deriving various properties of the kernel family concerning distributional, Mean Shift induced, and other aspects such as strict positive definiteness, we obtain a deeper understanding of the operator's projection behavior. From these theoretical insights, we illustrate several applications ranging from an improved Weighted LOP (WLOP) density weighting scheme and a more accurate Continuous LOP (CLOP) kernel approximation to the definition of a novel set of robust loss functions. These incomplete gamma losses include the Gaussian and LOP loss as special cases and can be applied to various tasks including normal filtering. Furthermore, we show that the novel kernels can be included as priors into neural networks. We demonstrate the effects of each application in a range of quantitative and qualitative experiments that highlight the benefits induced by our modifications.

Improve Noise Tolerance of Robust Loss via Noise-Awareness.

Robust loss minimization is an important strategy for handling robust learning issue on noisy labels. Current approaches for designing robust losses involve the introduction of noise-robust factors, i.e., hyperparameters, to control the trade-off between noise robustness and learnability. However, finding suitable hyperparameters for different datasets with noisy labels is a challenging and time-consuming task. Moreover, existing robust loss methods usually assume that all training samples share common hyperparameters, which are independent of instances. This limits the ability of these methods to distinguish the individual noise properties of different samples and overlooks the varying contributions of diverse training samples in helping models understand underlying patterns. To address above issues, we propose to assemble robust loss with instance-dependent hyperparameters to improve their noise tolerance with theoretical guarantee. To achieve setting such instance-dependent hyperparameters for robust loss, we propose a meta-learning method which is capable of adaptively learning a hyperparameter prediction function, called noise-aware-robust-loss-adjuster (NARL-Adjuster). Through mutual amelioration between hyperparameter prediction function and classifier parameters in our method, both of them can be simultaneously finely ameliorated and coordinated to attain solutions with good generalization capability. Four SOTA robust loss functions are attempted to be integrated with our algorithm, and comprehensive experiments substantiate the general availability and effectiveness of the proposed method in both its noise tolerance and performance. Meanwhile, the explicit parameterized structure makes the meta-learned prediction function ready to be transferrable and plug-and-play to unseen datasets with noisy labels. Specifically, we transfer our meta-learned NARL-Adjuster to unseen tasks, including several real noisy datasets, and achieve better performance compared with conventional hyperparameter tuning strategy, even with carefully tuned hyperparameters.