Robust Loss Function Research Articles

Structure-preserved dimension reduction using joint triplets sampling for multi-batch integration of single-cell transcriptomic data.

Dimension reduction (DR) plays an important role in single-cell RNA sequencing (scRNA-seq), such as data interpretation, visualization and other downstream analysis. A desired DR method should be applicable to various application scenarios, including identifying cell types, preserving the inherent structure of data and handling with batch effects. However, most of the existing DR methods fail to accommodate these requirements simultaneously, especially removing batch effects. In this paper, we develop a novel structure-preserved dimension reduction (SPDR) method using intra- and inter-batch triplets sampling. The constructed triplets jointly consider each anchor's mutual nearest neighbors from inter-batch, k-nearest neighbors from intra-batch and randomly selected cells from the whole data, which capture higher order structure information and meanwhile account for batch information of the data. Then we minimize a robust loss function for the chosen triplets to obtain a structure-preserved and batch-corrected low-dimensional representation. Comprehensive evaluations show that SPDR outperforms other competing DR methods, such as INSCT, IVIS, Trimap, Scanorama, scVI and UMAP, in removing batch effects, preserving biological variation, facilitating visualization and improving clustering accuracy. Besides, the two-dimensional (2D) embedding of SPDR presents a clear and authentic expression pattern, and can guide researchers to determine how many cell types should be identified. Furthermore, SPDR is robust to complex data characteristics (such as down-sampling, duplicates and outliers) and varying hyperparameter settings. We believe that SPDR will be a valuable tool for characterizing complex cellular heterogeneity.

Generalized Modified Blake–Zisserman Robust Sparse Adaptive Filters

In the past years, the generalized maximum correntropy criterion (GMCC) has been widely used in adaptive filters to provide robust behavior under non-Gaussian/impulsive noise environments. However, GMCC-based adaptive filters are affected by high steady-state misalignment. In order to enhance the robustness under non-Gaussian noise environments and reduce steady-state misalignment, a generalized modified Blake–Zisserman (GMBZ) robust loss function is introduced in this correspondence. Furthermore, a GMBZ adaptive filter (GMBZ-AF) has been developed that provides improved convergence performance over other existing algorithms. The proposed learning scheme has a computational complexity very similar to that of the GMCC-based adaptive filtering method. In order to further exploit the sparse nature of the system for identifying sparse systems and simultaneously provide robust convergence, two new robust sparse adaptive filters: 1) zero attracting GMBZ-AF (ZA-GMBZ-AF) and 2) reweighted ZA-GMBZ-AF (RZA-GMBZ-AF) have also been proposed. To further enhance the filter convergence performance, a new robust and sparsity-aware loss function called generalized modified dual Blake–Zisserman (GMDBZ) is also introduced in this correspondence and the corresponding GMDBZ adaptive filter (GMDBZ-AF) has been developed.

Zero-Shot Learning with Noisy Labels

Zero-shot learning (ZSL) is an attractive technique that can recognize novel object classes without any visual examples, but most existing methods assume that the class labels of the training instances from seen classes are accurate and reliable. However, in some real-world scenarios, the label quality may be compromised by various factors, and ZSL may suffer from performance degradation under label noise. In this paper, we propose an effective approach that utilizes a robust loss function to handle label noise in seen classes. Specifically, we construct a new denoising framework to reduce the impact of outliers and mislabeled samples. To mitigate overfitting to noisy labelled samples, a robust loss function called ramp-style loss is employed to filter out the negative influence of data with anomalous loss values, using only samples within a normal loss range for training, thereby enhancing the overall robust performance. Additionally, we introduce the concave-convex procedure (CCCP) method to address the non-convexity issue of ramp-style loss and devise an efficient update scheme based on ADMM. Extensive experiments conducted on several benchmark datasets (AWA2, SUN, CUB) demonstrate the superior performance of our framework over state-of-the-art ZSL methods in various noisy-label environments.

Robust loss function for class imbalanced semantic segmentation and image classification

Real world datasets are often unbalanced. Such an imbalance in data can limit the performance of modern deep learning based solutions by introducing a bias in the trained model. In particular, the trained model show weak performance on sub tasks or classes where the data availability is sparse. In this work, we address such a data imbalance problem and propose a novel modification to the existing cross-entropy loss function to mitigate the issue. Our proposed loss function can amplify the loss gradients generated during the back-propagation step. In particular, we penalize the predictions of the model which can result in higher loss values and gradients. We compare our proposed loss function with several recently proposed approaches and show superior performance. Our experiments show that proposed approach achieves state of the art performance on log-tailed image classification on CIFAR100/10-LT and Imagenet-LT datasets and on semantic segmentation task on Citiscapes dataset.

Predicting Returns, Volatilities and Correlations of Stock Indices Using Multivariate Conditional Autoregressive Range and Return Models

This paper extends the conditional autoregressive range (CARR) model to the multivariate CARR (MCARR) model and further to the two-stage MCARR-return model to model and forecast volatilities, correlations and returns of multiple financial assets. The first stage model fits the scaled realised Parkinson volatility measures using individual series and their pairwise sums of indices to the MCARR model to obtain the fitted volatilities. Then covariances are calculated to construct the fitted variance–covariance matrix of returns which are imputed into the stage-two return model to capture the heteroskedasticity of assets’ returns. We investigate different choices of mean functions to describe the volatility dynamics. Empirical applications are based on the Standard and Poor 500, Dow Jones Industrial Average and Dow Jones United States Financial Service Indices. Results show that the stage-one MCARR models using asymmetric mean functions give better in-sample model fits than those based on symmetric mean functions. They also provide better out-of-sample volatility forecasts than those using CARR models based on two robust loss functions. We also find that the stage-two return models with constant means and multivariate Student-t errors give better in-sample fits than the Baba–Engle–Kraft–Kroner generalised autoregressive conditional heteroskedasticity models. The estimates and forecasts of value-at-risk (VaR) and conditional VaR based on the best MCARR-return models for each asset are provided and tested using backtests to confirm the accuracy of the VaR forecasts.

Semi-supervised learning from coarse histopathology labels

ABSTRACT Ultrasound imaging is commonly used to guide sampling the prostate tissue in transrectal biopsies, followed by detection of cancer through histopathological analysis and coarse labelling of sampled tissue. Ideally, the procedure should be improved by developing machine learning solutions that can identify the presence of cancer in ultrasound images to guide the biopsy procedure. Training a fully supervised learning model using coarse histopathology labels suffers from weakly annotated data which introduce label noise for each image pixel. To address this challenge, we propose a semi-supervised framework for learning with noisy labels. We leverage a two-component mixture model to cluster the training data into clean and noisy label samples based on their loss values. Then, during the semi-supervised training phase, we utilise the well-known MixMatch algorithm which incorporates consistency regularisation, entropy minimisation, and the Mixup regularisation as well as the cross-entropy loss function for noisy and clean sets, respectively. We evaluate the proposed framework with prostate ultrasound data obtained from 71 subjects, while sampling 264 biopsy cores. We achieve balanced accuracy, sensitivity, and specificity of 78.6%, 80.0%, and 77.1%, respectively. In a detailed comparison study, we demonstrate that our proposed framework outperforms the fully supervised method with state-of-the-art robust loss functions.

The robust desparsified lasso and the focused information criterion for high-dimensional generalized linear models

The classical lasso estimation for sparse, high-dimensional regression models is typically biased and lacks the oracle properties. The desparsified versions of the lasso have been proposed in the literature that attempt to overcome these drawbacks. In this paper, we propose the outliers-robust version of the desparsified lasso for high dimensional generalized linear models. The robustness, consistency and high dimensional asymptotics are investigated rigorously in a general framework of M-estimation under potential model misspecification. The desparsification mechanism is subsequently utilized to construct the focused information criterion (FIC) thereby facilitating robust, focused model selection in high dimensions. The applications are demonstrated with the Poisson regression under robust quasilikelihood loss function. The empirical performance of the proposed methods is examined via simulations and a real data example.

Kernel methods with asymmetric and robust loss function

The least squares support vector machine (LSSVM) has achieved great success in various fields, but it still has certain limitations. Firstly, it treats all points equally and does not take into account the impact of different sample locations. Secondly, it is vulnerable to outliers and noise. Moreover, when class-imbalanced data are encountered, the decision boundary is biased toward the minority class. To address above problems, this paper introduces a bounded linear-exponential (BLINEX) loss function into LSSVM (LSKB), where the asymmetry and boundedness of the BLINEX loss function allow LSKB to assign distinct weights to each sample and be insensitive to noise and outliers. Further, this paper extends LSKB to a cost sensitive (CSLSKB) form to make it adapt to the class-imbalanced data. A fast optimization algorithm based on NESVM is developed to solve non-convex LSKB and CSLSKB (NEBLL). Numerous experiments demonstrate their effectiveness in dealing with class-balanced as well as class-imbalanced problems.

Asymmetric and robust loss function driven least squares support vector machine

Least squares support vector machine (LSSVM) considerably simplifies problem solving, however, there are restrictions. The first is that it treats samples on both sides of the proximal hyperplane equally and does not differentiate them based on locations; the second is that it is sensitive to noise and outliers. To address the aforementioned issues, this paper proposes an asymmetric robust least squares support vector machine termed QTLS by combining Quadratic Type Squared Error Loss Function (QTSELF) with LSSVM, in which QTSELF is introduced into machine learning for the first time. On the one hand, QTLS can impose various penalties on samples according to their locations and place greater emphasis on samples that are susceptible to misclassification. On the other hand, it improves the model robustness by imposing a tiny penalty on noise or outliers located far from the proximal hyperplane. Using Rademacher complexity theory, we investigate the generalization capacity of QTLS. The RMSProp technique is utilized to solve both linear and nonlinear QTLS. Extensive experiments indicate the effectiveness of QTLS in addressing binary classification problems.

Capped Linex Metric Twin Support Vector Machine for Robust Classification

In this paper, a novel robust loss function is designed, namely, capped linear loss function . Simultaneously, we give some ideal and important properties of , such as boundedness, nonconvexity and robustness. Furthermore, a new binary classification learning method is proposed via introducing , which is called the robust twin support vector machine (Linex-TSVM). Linex-TSVM can not only reduce the influence of outliers on Linex-SVM, but also improve the classification performance and robustness of Linex-SVM. Moreover, the effect of outliers on the model can be greatly reduced by introducing two regularization terms to realize the structural risk minimization principle. Finally, a simple and efficient iterative algorithm is designed to solve the non-convex optimization problem Linex-TSVM, and the time complexity of the algorithm is analyzed, which proves that the model satisfies the Bayes rule. Experimental results on multiple datasets demonstrate that the proposed Linex-TSVM can compete with the existing methods in terms of robustness and feasibility.

SRflow: Deep learning based super-resolution of 4D-flow MRI data.

Exploiting 4D-flow magnetic resonance imaging (MRI) data to quantify hemodynamics requires an adequate spatio-temporal vector field resolution at a low noise level. To address this challenge, we provide a learned solution to super-resolve in vivo 4D-flow MRI data at a post-processing level. We propose a deep convolutional neural network (CNN) that learns the inter-scale relationship of the velocity vector map and leverages an efficient residual learning scheme to make it computationally feasible. A novel, direction-sensitive, and robust loss function is crucial to learning vector-field data. We present a detailed comparative study between the proposed super-resolution and the conventional cubic B-spline based vector-field super-resolution. Our method improves the peak-velocity to noise ratio of the flow field by 10 and 30% for in vivo cardiovascular and cerebrovascular data, respectively, for 4 × super-resolution over the state-of-the-art cubic B-spline. Significantly, our method offers 10x faster inference over the cubic B-spline. The proposed approach for super-resolution of 4D-flow data would potentially improve the subsequent calculation of hemodynamic quantities.

Robust sparse portfolios for index tracking based on M-estimation

In order to save transaction costs, a sparse portfolio that consists of a small subset of the constituent stocks is commonly requested especially when the target index has a large number of assets. To the best of our knowledge, a large body of literature on sparse index tracking are mainly focus on the penalized least squares estimation under the no-short selling constraint. Accordingly, these procedures are highly sensitive to outliers and their efficiency may be significantly decreased when the noise distribution is unknown and possibly heavy-tailed. As promising alternatives, robust loss functions such as L 1 loss and Huber’s loss have been well characterized in penalized M-estimation techniques. This paper considers robust sparse portfolios to simultaneously perform stock selection and capital allocation for high-dimensional index tracking. Through the simulation, we show that newly proposed approach is resistant to heavy-tailed errors or outliers in the response. Finally, we illustrate the superior performance of the proposed method through tracking the Standard & Poor’s 500 Index.

A self-learning iterative weighted possibilistic fuzzy c-means clustering via adaptive fusion

Considering that weighted possibilistic fuzzy clustering does not obtain significant performance compared with possibilistic fuzzy clustering, so this paper proposes an enhanced self-adaptive weighted possibilistic fuzzy clustering algorithm. Firstly, the principle of maximum entropy is introduced to weighted possibilistic fuzzy clustering, and the weighted coefficients of fuzzy clustering and possibilistic clustering are subject to regularization entropy constraint and a novel self-learning iterative weighted possibilistic fuzzy clustering is obtained, and its convergence is strictly proved by Zangwill theorem and bordered Hessian matrix. Secondly, a series of clustering validity functions for the proposed algorithm are constructed to determine the optimal number of clusters in the data set. In the end, to enhance the anti-noise robustness of the proposed algorithm, a robust loss function is applied in the adaptive weighted possibilistic fuzzy clustering, and a robust algorithm is obtained for noisy data clustering. Experimental results show that the proposed algorithm outperforms existing possibilistic fuzzy clustering-related algorithms, and the validity functions for the proposed algorithm can accurately determine the optimal number of clusters in the data set, meanwhile, the corresponding robust algorithm effectively enhances the performance of the algorithm in the presence of noise.

Mind the Gap: Norm-Aware Adaptive Robust Loss for Multivariate Least-Squares Problems

Measurement outliers are unavoidable when solving real-world robot state estimation problems. A large family of robust loss functions (RLFs) exists to mitigate the effects of outliers, including newly developed adaptive methods that do not require parameter tuning. All of these methods assume that residuals follow a zero-mean Gaussian-like distribution. However, in multivariate problems the residual is often defined as a norm, and norms follow a Chi-like distribution with a non-zero mode value. This produces a “mode gap” that impacts the convergence rate and accuracy of existing RLFs. The proposed approach, “Adaptive MB,” accounts for this gap by first estimating the mode of the residuals using an adaptive Chi-like distribution. Applying an existing adaptive weighting scheme only to residuals greater than the mode leads to more robust performance and faster convergence times in two fundamental state estimation problems, point cloud alignment and pose averaging.

Improved generative adversarial network for retinal image super-resolution

Background and objectiveThe retina is the only organ in the body that can use visible light for non-invasive observation. By analyzing retinal images, we can achieve early screening, diagnosis and prevention of many ophthalmological and systemic diseases, helping patients avoid the risk of blindness. Due to the powerful feature extraction capabilities, many deep learning super-resolution reconstruction networks have been applied to retinal image analysis and achieved excellent results. MethodsGiven the lack of high-frequency information and poor visual perception in the current reconstruction results of super-resolution reconstruction networks under large-scale factors, we present an improved generative adversarial network (IGAN) algorithm for retinal image super-resolution reconstruction. Firstly, we construct a novel residual attention block, improving the reconstruction results lacking high-frequency information and texture details under large-scale factors. Secondly, we remove the Batch Normalization layer that affects the quality of image generation in the residual network. Finally, we use the more robust Charbonnier loss function instead of the mean square error loss function and the TV regular term to smooth the training results. ResultsExperimental results show that our proposed method significantly improves objective evaluation indicators such as peak signal-to-noise ratio and structural similarity. The obtained image has rich texture details and a better visual experience than the state-of-the-art image super-resolution methods. ConclusionOur proposed method can better learn the mapping relationship between low-resolution and high-resolution retinal images. This method can be effectively and stably applied to the analysis of retinal images, providing an effective basis for early clinical treatment.

VODRAC: Efficient and robust correspondence-based point cloud registration with extreme outlier ratios

Point set registration (PSR) from correspondences is a basic problem in the area of computer vision, robotics and remote sensing. Nevertheless, because of the limited accuracy of current correspondence building and feature matching technologies, PSR is frequently afflicted by the problem of outliers. In this work, we put forward VODRAC (VOting-based Double-point RAndom sampling with Compatibility weighting), a fast, highly robust and practically effective solution for the PSR problem as well as its real-world applications. To realize this, our first contribution is to integrate the scale-invariant constraint with a double-point random sampling framework to achieve the rapid seeking of inlier candidates and the rough rejection of outliers in the meantime. The second contribution is that we introduce the concept of weight matrix for PSR which is to use the robust loss function for weight computation between pairwise correspondences and also propose a time-efficient way to embed the construction of this matrix into the operation of the algorithm without requiring additional computational time. Moreover, we employ the correspondence voting technique to accelerate the consensus maximization (convergence) of the algorithm, as our third contribution. Through comprehensive experiments over multiple actual datasets, we demonstrate that VODRAC is fast, accurate, and highly robust against up to 99% of outliers (e.g. solving PSR within 2 s when there exist 99% outliers among 1000 correspondences), outperforming existing state-of-the-art (SOTA) robust algorithms.

A lightweight approach for network intrusion detection in industrial cyber-physical systems based on knowledge distillation and deep metric learning

With the rapid development of technology and science, machine learning approaches and deep learning methods have been widely applied in industrial Cyber-Physical Systems. However, there are still some challenging issues for anomaly detection to classify various attacks in industrial CPS to ensure the cyber security, especially when dealing with resource-constrained IoT devices. In this paper, we propose a Knowledge Distillation model based on Triplet Convolution Neural Network to improve the model performance and greatly enhance the speed of anomaly detection for industrial CPS as well as reduce the complexity of the model. Specifically, during the training process, we design a robust model loss function to improve the training stability of the model. A new neural network training method called K-fold cross training is also proposed to enhance the accuracy of anomaly detection. A lot of experimental results demonstrate that the performance metrics of KD-TCNN on the benchmark datasets NSL-KDD and CIC IDS2017 have significant advantages over traditional deep learning approaches and the recent state-of-the-art models. Furthermore, when compared to the original model, our model's computational cost and size are both reduced by roughly 86% with just 0.4% accuracy loss.

A robust real-time flood forecasting method based on error estimation for reservoirs

Abstract The observed discharge, an important input for flood forecasting systems, can significantly affect the accuracy of forecasting results. Since the reservoir inflow is not measured directly but calculated based on the observed reservoir stage and the reservoir outflow, it always contains gross errors causing some inflow to be outliers. In this study, a robust error estimation method for real-time flood forecasting has been developed for reservoirs’ data processing. The method differs from the conventional flood forecasting method, in that it represses the gross errors by a robust loss function based on the real error distribution of measurements. Furthermore, a fluctuation coefficient has been proposed to quantify the degree of fluctuation of a jagged reservoir inflow. The performance of the method is evaluated by both synthetic data and real cases. By using floods generated synthetically by an ideal model in which the true values and errors are known, the method is shown to be efficient as theoretically expected. The results show that the method is efficient and universally applicable in different cases. And the degree of forecast improvement is positively related to the fluctuation coefficient of the floods. The more severe the fluctuation of the flood hydrograph, the more robust the method.

Estimation Methods of the Multiple-Group One-Dimensional Factor Model: Implied Identification Constraints in the Violation of Measurement Invariance

Factor analysis is one of the most important statistical tools for analyzing multivariate data (i.e., items) in the social sciences. An essential case is the comparison of multiple groups on a one-dimensional factor variable that can be interpreted as a summary of the items. The assumption of measurement invariance is a frequently employed assumption that enables the comparison of the factor variable across groups. This article discusses different estimation methods of the multiple-group one-dimensional factor model under violations of measurement invariance (i.e., measurement noninvariance). In detail, joint estimation, linking methods, and regularized estimation approaches are treated. It is argued that linking approaches and regularization approaches can be equivalent to joint estimation approaches if appropriate (robust) loss functions are employed. Each of the estimation approaches defines identification constraints of parameters that quantify violations of measurement invariance. We argue in the discussion section that the fitted multiple-group one-dimensional factor analysis will likely be misspecified due to the violation of measurement invariance. Hence, because there is always indeterminacy in determining group comparisons of the factor variable under noninvariance, the preference of particular fitting strategies such as partial invariance over alternatives is unjustified. In contrast, researchers purposely define fitting functions that minimize the extent of model misspecification due to the choice of a particular (robust) loss function.

On Testing Equal Conditional Predictive Ability Under Measurement Error

Loss functions are widely used to compare several competing forecasts. However, forecast comparisons are often based on mismeasured proxy variables for the true target. We introduce the concept of exact robustness to measurement error for loss functions and fully characterize this class of loss functions as the Bregman class. Hence, only conditional mean forecasts can be evaluated exactly robustly. For such exactly robust loss functions, forecast loss differences are on average unaffected by the use of proxy variables and, thus, inference on conditional predictive ability can be carried out as usual. Moreover, we show that more precise proxies give predictive ability tests higher power in discriminating between competing forecasts. Simulations illustrate the different behavior of exactly robust and nonrobust loss functions. An empirical application to U.S. GDP growth rates demonstrates the nonrobustness of quantile forecasts. It also shows that it is easier to discriminate between mean forecasts issued at different horizons if a better proxy for GDP growth is used.