Skewed Noise Research Articles

Advancing robust regression: Addressing asymmetric noise with the BLINEX loss function

In real-world applications, regression performance is significantly impeded by complex noise. The choice of loss function is pivotal in building robust regression models, which broadly fall into two categories: bounded symmetric loss functions and asymmetric loss functions. The former struggle with asymmetric noise, while the latter are formulated as segmented functions involving complicates optimization. To handle various noise within a concise framework, we propose a robust regression model based on the asymmetric bounded linear-exponential (BLINEX) loss function, called BXSVR. Firstly, the asymmetry of the BLINEX loss function enables BXSVR to apply different penalties to positively and negatively skewed noise with equal regression error magnitudes to balance the effect of asymmetric noise. Secondly, the boundedness of the BLINEX loss function allows BXSVR to mitigate the detrimental effects of large noise. Thirdly, the smoothness and differentiability of the BLINEX loss function contributes to the optimization of BXSVR. Due to the above properties, BSXVR can adaptively and effectively manage diverse noise scenarios. We use the Nesterov accelerated gradient (NAG) algorithm to optimize BSXVR and analyze the complexity of our algorithm. Extensive experiments on synthetic datasets, real-world datasets, and an application dataset confirm that BXSVR is more competitive than other benchmark methods in terms of various measures.

Adaptive locally weighted support vector algorithm with asymmetrically parametric insensitive/margin model

In many fields such as finance and electrical load, locally weighted support vector regression (LWSVR) method can achieve better prediction results by constructing different sub-models for each prediction target. However, it selected a fixed and single nearest neighbor parameter of kglobal for each query, and adopted the traditional k-nearest neighbor (KNN) algorithm, which is suboptimal. The weights in LWSVR assigned to the neighboring sample points also affected the accuracy of the model predictions. In this paper, we propose an adaptive locally weighted support vector regression with asymmetrically parametric insensitive/margin model, called Asy-par-v-ALWSVR. Since the selection of similar sample for query points is significant. We propose a novel locally q-neighbour selection algorithm that adaptively selects the most suitable number of neighbours for each query point based on Mahalanobis distance, rather than using a fixed kglobal. This significantly improves prediction accuracy. The weights of neighbours for each query point in Asy-par-v-ALWSVR is based on the distance within the kernel-induced feature space. This effectively discerns the importance of samples during the training process. In particular, Asy-par-v-ALWSVR can handle skewed noise with heteroscedastic structure in regression problems. Finally, we evaluate the proposed model on a synthetic sinc dataset and 14 real datasets. The results demonstrate that Asy-par-v-ALWSVR outperforms six state-of-the-art SVR-based models.

Skew Filtering for Online State Estimation and Control

Process control can become challenging when the measurements are affected by irregular noise. Classical approaches utilize Gaussian methods to alleviate the sensory noise. However, many industries involve skewed noise in their processes. While the closed skew-normal (CSN) distribution generalizes a Gaussian distribution with additional parameters, its dimension increases during recursive estimation, making it impractical. Even though there are some techniques for the solution, they are typically too complicated or inaccurate for higher-dimensional problems. This study proposes a novel online optimization scheme to reduce the dimensionality of a CSN distribution while considering the properties of the complete empirical distribution. Since the objective function used during the optimization step considers the geometry of the metric space, the proposed scheme achieves higher accuracy without sacrificing computational efficiency. The proposed filter is applied to two pilot-scale experiments. The results indicate it is beneficial for recursive state estimation in the presence of skewed noise.

Robust partially linear trend filtering for regression estimation and structure discovery

Partially linear additive models (PLAMs) have attracted much attention in the statistical machine learning community due to their interpretability and flexibility in data-driven prediction and inference. Since the performance of PLAMs is closely related to the structure information of linear and nonlinear components, several approaches have been proposed for regression estimation and data-driven structure discovery. However, the existing automatic discovery strategy is limited to the mean regression framework and is usually sensitive to non-Gaussian noises, e.g. skewed noise and heavy-tailed noise. To further improve the robustness of PLAMs, this paper proposes a Robust Partially Linear Trend Filtering (RPLTF) for regression estimation and structure discovery by integrating the mode-induced error metric and the trend filtering-based nonlinear approximation into regularized PLAMs. Besides the computing algorithm of RPLTF, we establish its upper bound on generalization error in theory. Empirical examples are provided to validate the effectiveness of the proposed method.

Robust partially linear models for automatic structure discovery

Partially linear models (PLMs), rooted in the combination of linear and nonlinear approximation, are recognized to be capable of modeling complex data. Indeed, the performance of PLMs depends heavily on the choice of model structure, such as which covariates have linear or nonlinear effects on the response. Nevertheless, most existing PLMs are limited to the mean regression, resulting in sensitivity to non-Gaussian noises, such as skewed noise and heavy-tailed noise. In order to mitigate the influence of noise in structure discovery, this paper proposes a Robust Linear And Nonlinear Discovery algorithm (RLAND) by integrating the modal regression and PLMs. Statistical analysis on generalization bound and structure discovery consistency are established to characterize its learning theory foundations. Computation analysis illustrates that the RLAND can be efficiently realized by half quadratic optimization and the quadratic programming. Empirical evaluations on simulation and real-world data validate the competitive performance of the proposed method.

Sparse Modal Additive Model.

Sparse additive models have been successfully applied to high-dimensional data analysis due to the flexibility and interpretability of their representation. However, the existing methods are often formulated using the least-squares loss with learning the conditional mean, which is sensitive to data with the non-Gaussian noises, e.g., skewed noise, heavy-tailed noise, and outliers. To tackle this problem, we propose a new robust regression method, called as sparse modal additive model (SpMAM), by integrating the modal regression metric, the data-dependent hypothesis space, and the weighted lq,1 -norm regularizer (q ≥ 1) into the additive models. Specifically, the modal regression metric assures the model robustness to complex noises via learning the conditional mode, the data-dependent hypothesis space offers the model adaptivity via sample-based presentation, and the lq,1 -norm regularizer addresses the algorithmic interpretability via sparse variable selection. In theory, the proposed SpMAM enjoys statistical guarantees on asymptotic consistency for regression estimation and variable selection simultaneously. Experimental results on both synthetic and real-world benchmark data sets validate the effectiveness and robustness of the proposed model.

Dissociating error-based and reinforcement-based loss functions during sensorimotor learning.

It has been proposed that the sensorimotor system uses a loss (cost) function to evaluate potential movements in the presence of random noise. Here we test this idea in the context of both error-based and reinforcement-based learning. In a reaching task, we laterally shifted a cursor relative to true hand position using a skewed probability distribution. This skewed probability distribution had its mean and mode separated, allowing us to dissociate the optimal predictions of an error-based loss function (corresponding to the mean of the lateral shifts) and a reinforcement-based loss function (corresponding to the mode). We then examined how the sensorimotor system uses error feedback and reinforcement feedback, in isolation and combination, when deciding where to aim the hand during a reach. We found that participants compensated differently to the same skewed lateral shift distribution depending on the form of feedback they received. When provided with error feedback, participants compensated based on the mean of the skewed noise. When provided with reinforcement feedback, participants compensated based on the mode. Participants receiving both error and reinforcement feedback continued to compensate based on the mean while repeatedly missing the target, despite receiving auditory, visual and monetary reinforcement feedback that rewarded hitting the target. Our work shows that reinforcement-based and error-based learning are separable and can occur independently. Further, when error and reinforcement feedback are in conflict, the sensorimotor system heavily weights error feedback over reinforcement feedback.

Skewed noise

We study the attitude of decision makers to skewed noise. For a binary lottery that yields the better outcome with probability p, we identify noise around p with a compound lottery that induces a distribution over the exact value of the probability and has an average value p. We propose and characterize a new notion of skewed distributions, and use a recursive non-expected utility to provide conditions under which rejection of symmetric noise implies rejection of negatively skewed noise, yet does not preclude acceptance of some positively skewed noise, in agreement with recent experimental evidence. In the context of decision making under uncertainty, our model permits the co-existence of aversion to symmetric ambiguity (as in Ellsberg's paradox) and ambiguity seeking for low likelihood “good” events.

Expectile Matrix Factorization for Skewed Data Analysis

Matrix factorization is a popular approach to solving matrix estimation problems based on partial observations. Existing matrix factorization is based on least squares and aims to yield a low-rank matrix to interpret the conditional sample means given the observations. However, in many real applications with skewed and extreme data, least squares cannot explain their central tendency or tail distributions, yielding undesired estimates. In this paper, we propose expectile matrix factorization by introducing asymmetric least squares, a key concept in expectile regression analysis, into the matrix factorization framework. We propose an efficient algorithm to solve the new problem based on alternating minimization and quadratic programming. We prove that our algorithm converges to a global optimum and exactly recovers the true underlying low-rank matrices when noise is zero. For synthetic data with skewed noise and a real-world dataset containing web service response times, the proposed scheme achieves lower recovery errors than the existing matrix factorization method based on least squares in a wide range of settings.

Skewed Noise

Skewed Noise

Asymmetric [formula omitted]-tube support vector regression

Finding a tube of small width that covers a certain percentage of the training data samples is a robust way to estimate a location: the values of the data samples falling outside the tube have no direct influence on the estimate. The well-known ν-tube Support Vector Regression (ν-SVR) is an effective method for implementing this idea in the context of covariates. However, the ν-SVR considers only one possible location of this tube: it imposes that the amount of data samples above and below the tube are equal. The method is generalized such that those outliers can be divided asymmetrically over both regions. This extension gives an effective way to deal with skewed noise in regression problems. Numerical experiments illustrate the computational efficacy of this extension to the ν-SVR.

Skewed Noise

Skewed Noise

Signal detection in nearly Gaussian skewed noise

The problem of detecting a constant signal in nearly Gaussian skewed noise is studied in this paper. Sensitivity of both the linear detector and the sign detector are examined. It is shown that the performance of the linear detector deteriorates when the noise distribution becomes skewed. On the other hand, the sign detector appears to be less sensitive to the noise skewness. Furthermore, a modification of the linear detector is proposed. This modified scheme, when the noise skewness measure is small, will behave almost like the linear detector in Gaussian noise.