Heavy-tailed Noise Research Articles

Robust double machine learning model with application to omics data.

Recently, there has been a growing interest in combining causal inference with machine learning algorithms. Double machine learning model (DML), as an implementation of this combination, has received widespread attention for their expertise in estimating causal effects within high-dimensional complex data. However, the DML model is sensitive to the presence of outliers and heavy-tailed noise in the outcome variable. In this paper, we propose the robust double machine learning (RDML) model to achieve a robust estimation of causal effects when the distribution of the outcome is contaminated by outliers or exhibits symmetrically heavy-tailed characteristics. In the modelling of RDML model, we employed median machine learning algorithms to achieve robust predictions for the treatment and outcome variables. Subsequently, we established a median regression model for the prediction residuals. These two steps ensure robust causal effect estimation. Simulation study show that the RDML model is comparable to the existing DML model when the data follow normal distribution, while the RDML model has obvious superiority when the data follow mixed normal distribution and t-distribution, which is manifested by having a smaller RMSE. Meanwhile, we also apply the RDML model to the deoxyribonucleic acid methylation dataset from the Alzheimer's disease (AD) neuroimaging initiative database with the aim of investigating the impact of Cerebrospinal Fluid Amyloid 42 (CSF A 42) on AD severity. These findings illustrate that the RDML model is capable of robustly estimating causal effect, even when the outcome distribution is affected by outliers or displays symmetrically heavy-tailed properties.

Sequential Fusion for Multi-rate Multi-sensor Nonlinear Dynamic Systems with Heavy-Tailed Noise and Missing Measurements

This paper focuses on sequential fusion estimation for multi-rate multi-sensor nonlinear dynamic systems with heavy-tailed noise and missing measurements. On the basis of Bayesian inference, a sequential Student’s t-based unscented Kalman filter (SSTUKF), together with its square-root form (SR-SSTUKF), is proposed by using the unscented transform to calculate Student’s t weighted integrals. Considering the nonstationary measurement noise and/or accumulated computation error, adaptive factors are introduced by the t-test to suppress uncertainties. Additionally, the complexity computation and convergence analysis of the SR-SSTUKF are presented. The validity and robustness of the proposed sequential fusion method are illustrated by an example of agile target tracking. Simulation results indicate that the SR-SSTUKF with adaptive factors can further enhance accuracy and yield reliable estimations.

A nonlinear filter based on GSK for relative navigation using relative orbital elements

Due to the sensitivity of the relative orbital elements model to the measurement noise, the non-stationary heavy-tailed noise(NSHT) induced by the time-varying environment during the relative navigation usually leads to filter divergence. To address this problem, a new nonlinear filter based on Gaussian-Student's-Multivariate K(GSK) mixture distribution is proposed in this paper. A Dirichlet stochastic mixture vector fusing Gaussian, Student's t, and Multivariate K distributions is introduced, thus proposing a GSK mixture distribution modeling measurement likelihood; then the Kullback-Leibler Divergence (KLD) of the true posteriori probability density function(PDF) and the approximate posteriori PDF are minimized by a variational Bayesian(VB) technique to solve for the state and parameter approximate a posteriori estimations, and finally a new nonlinear filter based on the GSK mixture distribution is derived for angles-only relative navigation in time-varying environments. Simulation outcomes indicate that the filter can realize state estimation in non-stationary states effectively with 45.16% higher estimation accuracy than the existing advanced filters.

High-Probability Complexity Bounds for Non-smooth Stochastic Convex Optimization with Heavy-Tailed Noise

High-Probability Complexity Bounds for Non-smooth Stochastic Convex Optimization with Heavy-Tailed Noise

A spatial Durbin model for interval-valued data with t-distribution

Interval-valued data have received much attention due to their wide applications in finance, econometrics, meteorology and medicine. However, most regression models developed for interval-valued data assume that the observations are mutually independent, which can not be adapted to scenarios where individuals are spatially correlated. We propose a new linear model to account for areal-type spatial dependencies present in interval-valued data. Specifically, two spatial Durbin models are separately built for the centers and log-ranges of the interval-valued response. To make the new model more robust to heavy-tailed noise, we assume the error term follows a t-distribution. The parameters are obtained using an expectation-maximization (EM) algorithm. Numerical experiments are designed to examine the performance of the proposed method. We also use a weather dataset to demonstrate the usefulness of our model.

High Probability Bounds for Stochastic Subgradient Schemes with Heavy Tailed Noise]

High Probability Bounds for Stochastic Subgradient Schemes with Heavy Tailed Noise]

Robust Bilinear Form Identification: A Subgradient Method with Geometrically Decaying Stepsize in the Presence of Heavy-Tailed Noise

Robust Bilinear Form Identification: A Subgradient Method with Geometrically Decaying Stepsize in the Presence of Heavy-Tailed Noise

Robust Matrix Completion with Heavy-Tailed Noise

This article studies noisy low-rank matrix completion in the presence of heavy-tailed and possibly asymmetric noise, where we aim to estimate an underlying low-rank matrix given a set of highly incomplete noisy entries. Though the matrix completion problem has attracted much attention in the past decade, there is still lack of theoretical understanding when the observations are contaminated by heavy-tailed noises. Prior theory falls short of explaining the empirical results and is unable to capture the optimal dependence of the estimation error on the noise level. In this article, we adopt an adaptive Huber loss to accommodate heavy-tailed noise, which is robust against large and possibly asymmetric errors when the parameter in the Huber loss function is carefully designed to balance the Huberization biases and robustness to outliers. Then, we propose an efficient nonconvex algorithm via a balanced low-rank Burer-Monteiro matrix factorization and gradient descent with robust spectral initialization. We prove that under merely a bounded second-moment condition on the error distributions, rather than the sub-Gaussian assumption, the Euclidean errors of the iterates generated by the proposed algorithm decrease geometrically fast until achieving a minimax-optimal statistical estimation error, which has the same order as that in the sub-Gaussian case. The key technique behind this significant advancement is a powerful leave-one-out analysis framework. The theoretical results are corroborated by our numerical studies. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

A robust M-estimator for Gaussian ARMA time series based on the Whittle approximation

In the frequency domain, the Whittle approach using the classical periodogram is traditionally used to estimate the parameters of a stationary time series. The M-periodogram has recently become an alternative tool for analyzing the dependence of time series. It is particularly useful for dealing with outliers and heavy-tailed noise. This paper proposes an alternative to the standard Whittle approach, the M-Whittle estimator, built from the M-periodogram. We show that the proposed method is a consistent estimator of the true parameters of an ARMA process. A finite sample investigation is carried out to assess the performance of the estimator in the scenarios of contaminated and uncontaminated time series. As expected, for the uncontaminated data, the M-Whittle estimator performs similarly to the classical Whittle approach. However, the superiority of the first method is clear in terms of root mean squared error when the series has additive outliers. Two applications are considered to illustrate the methodologies in real data contexts. Regardless of whether or not the data are contaminated by additive outliers, the results presented here strongly motivate using the M-Whittle estimator in practical problems.

An improved multiple-outlier-robust GNSS/INS EKF filer based on multiple statistical similarity measure

Abstract High-precision and highly reliable location information is the key support for autonomous driving applications. Global Navigation satellite system (GNSS)/Inertial navigation system (INS) integrated positioning is a typical high-precision positioning technology. However, satellite signals are easily affected by various interferences such as signal blocking and multipath problems, leading to performance degradation in complex urban environments. The traditional extended Kalman filtering (EKF) cannot handle observation outlier, and the system’s positioning accuracy sharply declines when the observations are affected by non-Gaussian heavy-tailed noise. Therefore, this paper presents an improved multiple-outlier-robust extended Kalman filter (I-MOREKF) for GNSS/INS Loosely Coupled Integration. A multiple statistical similarity measure (MSSM) is built to evaluate the similarity between two random vectors from dimension to dimension. Then, the I-MOREKF is proposed by maximizing a cost function based on the MSSM, and considering the robust estimation with IGG III model. The proposed model effectively removes multiple types of errors in complex environments. A vehicular test was carried out to validate the feasibility and performance of the proposed model. Simulated random gross errors are introduced in GNSS measurements, and the measurement noises are corrupted by heavy-tail noise. The results show that simulated gross errors can be successfully detected with I-MOREKF model, and the 3D position Root Meas Square Error is 0.0148 m for I-MOREKF, which are improved by 97.2%,13.5% and 52.9%, as compared with EKF, IGG Ⅲ and MOREKF.

Stochastic wave equation with heavy-tailed noise: Uniqueness of solutions and past light-cone property

In this article, we study the stochastic wave equation in spatial dimensions d≤2 with multiplicative Lévy noise that can have infinite pth moments. Using the past light-cone property of the wave equation, we prove the existence and uniqueness of a solution, considering only the p-integrability of the Lévy measure ν for the region corresponding to the small jumps of the noise. For d=1, there are no restrictions on ν. For d=2, we assume that there exists a value p∈(0,2) for which ∫{|z|≤1}|z|pν(dz)<+∞.

VB-T PHD-SLAM: efficient SLAM under heavy-tailed noise

To address the challenge of simultaneous localization and mapping (SLAM) in the presence of heavy-tailed noise, this paper introduces a robust probability hypothesis density (PHD) SLAM algorithm. This algorithm models measurement noise using the Student's t-distribution, which better captures the heavy-tailed nature of the noise. Since the prior density is assumed to be Gaussian mixture form, the posterior density is no longer Gaussian mixture form after the likelihood update of the t-distribution. A variational Bayesian approach is employed to ensure computable multi-target densities during filtering, minimizing the Kullback-Leibler divergence to obtain an approximate solution for the new marginal likelihood function. Then a new closed-form recursion of PHD-SLAM is derived by using t-distribution. Simulation results and real-world validations demonstrate that the proposed algorithm outperforms PHD-SLAM 1.0 and PHD-SLAM 2.0 in terms of both localization and mapping accuracy while maintaining computational efficiency in SLAM scenarios affected by heavy-tailed noise.

Robust Kalman filters based on the sub-Gaussian [formula omitted]-stable distribution

Motivated by filtering tasks under a linear system with non-Gaussian heavy-tailed noise, various robust Kalman filters (RKFs) based on different heavy-tailed distributions have been proposed. Although the sub-Gaussian α-stable (SGαS) distribution captures heavy tails well and is applicable in various scenarios, its potential has not yet been explored for RKFs. The main hindrance is that there is no closed-form expression of its mixing density. This paper proposes a novel RKF framework, RKF-SGαS, where the process noise is assumed to be Gaussian and the heavy-tailed measurement noise is modelled by the SGαS distribution. The corresponding joint posterior distribution of the state vector and auxiliary random variables is approximated by the Variational Bayesian approach. Also, four different minimum mean square error (MMSE) estimators of the scale function are presented. The first two methods are based on the Importance Sampling (IS) and Gauss–Laguerre quadrature (GLQ), respectively. In contrast, the last two estimators combine a proposed Gamma series (GS) based method with the IS and GLQ estimators and hence are called GSIS and GSGL. Besides, the RKF-SGαS is compared with the state-of-the-art RKFs under three kinds of heavy-tailed measurement noises, and the simulation results demonstrate its estimation accuracy and efficiency.

Distributed Consensus Multi-Distribution Filter for Heavy-Tailed Noise

Distributed state estimation is one of the critical technologies in the field of target tracking, where the process noise and measurement noise may have a heavy-tailed distribution. Traditionally, heavy-tailed distributions like the student-t distribution are employed, but our observation reveals that Gaussian noise predominates in many instances, with occasional outliers. This sporadic reliance on heavy-tailed distributions can degrade performances or necessitate frequent parameter adjustments. To overcome this, we introduce a novel distributed consensus multi-distribution state estimation method that combines Gaussian and student-t filters. Our approach establishes a system model using both Gaussian and student-t distributions. We derive a multi-distribution filter for a single sensor, assigning probabilities to Gaussian and student-t noise models. Parallel estimation under both distributions, utilizing Gaussian and student-t filters, allows us to calculate the likelihood of each distribution. The fusion of these results yields a mixed-state estimation and corresponding error matrix. Recognizing the increasing degrees of freedom in the student-t distribution over time, we provide an effective approximation. An information consensus strategy for multi-distribution filters is introduced, achieving global estimation through consensus on fused local filter results via interaction with neighboring nodes. This methodology is extended to the distributed case, and the recursive process of the distributed multi-distribution consensus state estimation method is presented. Simulation results demonstrate that the estimation accuracy of the proposed algorithm improved by at least 20% compared to that of the traditional algorithm in scenarios involving both Gaussian and heavy-tailed distributions.

Robust convex biclustering with a tuning-free method

Biclustering is widely used in different kinds of fields including gene information analysis, text mining, and recommendation system by effectively discovering the local correlation between samples and features. However, many biclustering algorithms will collapse when facing heavy-tailed data. In this paper, we propose a robust version of convex biclustering algorithm with Huber loss. Yet, the newly introduced robustification parameter brings an extra burden to selecting the optimal parameters. Therefore, we propose a tuning-free method for automatically selecting the optimal robustification parameter with high efficiency. The simulation study demonstrates the more fabulous performance of our proposed method than traditional biclustering methods when encountering heavy-tailed noise. A real-life biomedical application is also presented. The R package RcvxBiclustr is available at https://github.com/YifanChen3/RcvxBiclustr.

Generative AI for Rapid Diffusion MRI with Improved Image Quality, Reliability and Generalizability

Abstract We use generative AI to enable rapid diffusion MRI (dMRI) with high fidelity, reproducibility, and generalizability across clinical and research settings. We employ a Swin UNEt Transformers (SWIN) model, trained on Human Connectome Project (HCP) data (n=1021) and conditioned on registered T1 scans, to perform generalized dMRI denoising. We also qualitatively demonstrate super-resolution with artificially downsampled HCP data. Remarkably, SWIN can be fine-tuned for an out-of-domain dataset with a single example scan, as we demonstrate on dMRI of children with neurodevelopmental disorders (n=40), adults with acute traumatic brain injury (n=40), and adolescents with intracerebral hemorrhage due to vascular malformations undergoing resection (n = 8), each cohort scanned on different scanner models with different imaging protocols at different sites. This robustness to scan acquisition parameters, patient populations, scanner types, and sites eliminates the advantages of self-supervised methods over our fully-supervised generative AI approach. We exceed current state-of-the-art denoising methods in accuracy and test-retest reliability of rapid diffusion tensor imaging (DTI) requiring only 90 seconds of scan time. SWIN denoising also achieves dramatic improvements over the state-of-the-art for test-retest reliability of intracellular volume fraction and free water fraction measurements and can remove heavy-tail noise, improving biophysical modeling fidelity. SWIN enables rapid diffusion MRI with unprecedented accuracy and reliability, especially at high diffusion-weighting for probing biological tissues at microscopic spatial scales. The code and model are publicly available at https://github.com/ucsfncl/dmri-swin.

Robust nonparametric regression based on deep ReLU neural networks

In this paper, we consider robust nonparametric regression using deep neural networks with ReLU activation function. While several existing theoretically justified methods are geared towards robustness against identical heavy-tailed noise distributions, the rise of adversarial attacks has emphasized the importance of safeguarding estimation procedures against systematic contamination. We approach this statistical issue by shifting our focus towards estimating conditional distributions. To address it robustly, we introduce a novel estimation procedure based on ℓ-estimation. Under a mild model assumption, we establish general non-asymptotic risk bounds for the resulting estimators, showcasing their robustness against contamination, outliers, and model misspecification. We then delve into the application of our approach using deep ReLU neural networks. When the model is well-specified and the regression function belongs to an α-Hölder class, employing ℓ-type estimation on suitable networks enables the resulting estimators to achieve the minimax optimal rate of convergence. Additionally, we demonstrate that deep ℓ-type estimators can circumvent the curse of dimensionality by assuming the regression function closely resembles the composition of several Hölder functions. To attain this, new deep fully-connected ReLU neural networks have been designed to approximate this composition class. This approximation result can be of independent interest.

Робастное обнаружение и оценивание широкополосных сигналов

An asymptotically robust invariant (ARI) algorithm for signal detection and time delay estimation is proposed. It is based on the q-point model of noise distributions. The algorithm is based on calculating the correlation statistics of the nonlinear transformation of the observed sample with the vector of reference signal samples. Time delay estimate is determined by special processing of the obtained statistics, taking into account the presence of mirror interference in the observed process. The algorithm is implemented in the frequency domain, which allows the use of the fast Fourier transform to reduce computational costs. The simulation results show that in the case of heavy-tailed noise distributions, ARI algorithm provides an energy gain compared to the classical correlation algorithm, and it has similar performance in the case of Gaussian noise.

Generalization analysis of deep CNNs under maximum correntropy criterion

Convolutional neural networks (CNNs) have gained immense popularity in recent years, finding their utility in diverse fields such as image recognition, natural language processing, and bio-informatics. Despite the remarkable progress made in deep learning theory, most studies on CNNs, especially in regression tasks, tend to heavily rely on the least squares loss function. However, there are situations where such learning algorithms may not suffice, particularly in the presence of heavy-tailed noises or outliers. This predicament emphasizes the necessity of exploring alternative loss functions that can handle such scenarios more effectively, thereby unleashing the true potential of CNNs. In this paper, we investigate the generalization error of deep CNNs with the rectified linear unit (ReLU) activation function for robust regression problems within an information-theoretic learning framework. Our study demonstrates that when the regression function exhibits an additive ridge structure and the noise possesses a finite pth moment, the empirical risk minimization scheme, generated by the maximum correntropy criterion and deep CNNs, achieves fast convergence rates. Notably, these rates align with the mini-max optimal convergence rates attained by fully connected neural network model with the Huber loss function up to a logarithmic factor. Additionally, we further establish the convergence rates of deep CNNs under the maximum correntropy criterion when the regression function resides in a Sobolev space on the sphere.

Renewable Quantile Regression with Heterogeneous Streaming Datasets

The renewable statistical inference has received much attention since the advent of streaming data collection techniques. However, most existing online updating methods are developed based on a homogeneity assumption and gradients; all data batches are required to be either independent and identically distributed or share the same regression parameters, and objective functions must be smooth concerning parameters. To our best knowledge, the only existing approach that allows some regression parameters to be different for different data batches, was proposed by Luo and Song who required the homogeneous structure to be known, which is difficult to guarantee in actual application. In this article, we develop an online renewable quantile regression method that relies only on the current data and summary statistics of historical data, for both homogeneous and heterogeneous streaming data. The proposed methods are computationally efficient, can automatically detect the unknown potential homogeneous structure, and are robust to heavy-tailed noise and data with outliers. Asymptotic properties show that the proposed renewable estimators can achieve the same statistical efficiency as the oracle estimators based on individual-level data. A numerical simulation and a real data analysis illustrate that the proposed methods perform well. Supplementary materials for this article are available online.