Corrigendum: An Unbiased Risk Estimator for Partial Label Learning with Augmented Classes

Multi-label learning (MLL) usually requires assigning multiple relevant labels to each instance. While a fully supervised MLL dataset needs a large amount of labeling effort, using complementary labels can help alleviate this burden. However, current approaches to learning from complementary labels are mainly designed for multi-class learning and assume that each instance has a single relevant label. This means that these approaches cannot be easily applied to MLL when only complementary labels are provided, where the number of relevant labels is unknown and can vary across instances. In this paper, we first propose the unbiased risk estimator for the multi-labeled complementary label learning (MLCLL) problem. We also provide an estimation error bound to ensure the convergence of the empirical risk estimator. In some cases, the unbiased estimator may give unbounded gradients for certain loss functions and result in overfitting. To mitigate this problem, we improve the risk estimator by minimizing a proper loss function, which has been shown to improve gradient updates. Our experimental results demonstrate the effectiveness of the proposed approach on various datasets.

We address the problem of suppressing background noise from noisy speech within a risk estimation framework, where the clean signal is estimated from the noisy observations by minimizing an unbiased estimate of a chosen risk function. For Gaussian noise, such a risk estimate was derived by Stein, which eventually went on to be called Stein's unbiased risk estimate (SURE). Stein's formalism is restricted to Gaussian noise and exclusive risk estimators have been developed for each noise type. On the other hand, we consider linear denoising functions and derive an unbiased risk estimate without making any assumption about the noise distribution. The proposed unbiased estimate depends only on the second-order statistics of noise and makes the proposed framework applicable to many practical denoising problems where the noise distribution is not known a priori, but one has access only to the samples of noise. We demonstrate the usefulness of the proposed methodology for speech enhancement using subband shrinkage, where the shrinkage parameters are obtained by minimizing the newly developed risk estimator. The proposed methodology is also applicable to nonstationary noise conditions. We show that the proposed denoising algorithm outperforms the state-of-the art algorithms in terms of standard speech-quality evaluation metrics.

Partial Label Learning (PLL) is a typical weakly supervised learning task, which assumes each training instance is annotated with a set of candidate labels containing the ground-truth label. Recent PLL methods adopt identification-based disambiguation to alleviate the influence of false positive labels and achieve promising performance. However, they require all classes in the test set to have appeared in the training set, ignoring the fact that new classes will keep emerging in real applications. To address this issue, in this article, we focus on the problem of Partial Label Learning with Augmented Class (PLLAC), where one or more augmented classes are not visible in the training stage but appear in the inference stage. Specifically, we propose an unbiased risk estimator with theoretical guarantees for PLLAC, which estimates the distribution of augmented classes by differentiating the distribution of known classes from unlabeled data and can be equipped with arbitrary PLL loss functions. Besides, we provide a theoretical analysis of the estimation error bound of the estimator, which guarantees the convergence of the empirical risk minimizer to the true risk minimizer as the number of training data tends to infinity. Furthermore, we add a risk-penalty regularization term in the optimization objective to alleviate the influence of the over-fitting issue caused by negative empirical risk. Extensive experiments on benchmark, UCI, and real-world datasets demonstrate the effectiveness of the proposed approach.

Let π1, …, πp be p (p ≥2) independent populations with πi being binomial bin(l, θ) with an unknown parameter θii = 1, …, p. Suppose independent random samples of sizes n1, …, np are drawn from the populations π1, …, πp, respectively, and let $${erline X}_i=X_i/n_i$$, where $$X_{i}={Sigma^{ni}_{j=1}}X_{ij},i=1,$$ …, P. We call the population associated with the largest of θi ’s the best population. Suppose a population is selected using the Gupta’s (Gupta, S. S. (1965). On some multiple decision(selection and ranking) rules. Technometrics 7, 225–245) subset selection procedure. In this paper, we consider simultaneous estimation of the parameters of the selected populations. It is shown that neither the unbiased estimator nor the riskunbiased estimator (corresponding to the normalized squared error loss function) exists based on a single-stage sample. When additional observations are available from the selected populations, we derive an unbiased and risk-unbiased estimators for the selected subset and also prove that the natural estimators are positively biased. Finally, the bias and the risk of the natural, unbiased and risk-unbiased estimators are computed using Monte-Carlo simulation method.

Suppose \(\overline{X}=(\overline{X}_1,...\overline{X}_p), (p\geq 2)\); where \(\overline{X}_i\)represents the mean of a random sample of size ni drawn from binomial \(bin(1,\theta_i)\) population. Assume the parameters \(\theta_1,...,\theta_p\) are unknown and the populations \(bin(1,\theta_1),...,bin(1,\theta_p)\) are independent. A subset of random size is selected using Gupta's (Gupta, S. S. (1965). On some multiple decision(selection and ranking) rules. Technometrics 7,225-245) subset selection procedure. In this paper, we estimate of the average worth of the parameters for the selected subset under squared error loss and normalized squared error loss functions. First, we show that neither the unbiased estimator nor the risk- unbiased estimator of the average worth (corresponding to the normalized squared error loss function) exist based on a single-stage sample. Second, when additional observations are available from the selected populations, we derive an unbiased and risk-unbiased estimators of the average worth and also prove that the natural estimator of the average worth is positively biased. Finally, the bias and risk of the natural, unbiased and risk-unbiased estimators are computed and compared using Monti Carlo simulation method.

The goal of speech enhancement algorithms is to provide an estimate of clean speech starting from noisy observations. The often-employed cost function is the mean square error (MSE). However, the MSE can never be computed in practice. Therefore, it becomes necessary to find practical alternatives to the MSE. In image denoising problems, the cost function (also referred to as risk) is often replaced by an unbiased estimator. Motivated by this approach, we reformulate the problem of speech enhancement from the perspective of risk minimization. Some recent contributions in risk estimation have employed Stein's unbiased risk estimator (SURE) together with a parametric denoising function, which is a linear expansion of threshold/bases (LET). We show that the first-order case of SURE-LET results in a Wiener-filter type solution if the denoising function is made frequency-dependent. We also provide enhancement results obtained with both techniques and characterize the improvement by means of local as well as global SNR calculations.

The task of partial label (PL) learning is to learn a multi-class classifier from training examples each associated with a set of candidate labels, among which only one corresponds to the ground-truth label. It is well known that for inducing multi-class predictive model, the most straightforward solution is binary decomposition which works by either one-vs-rest or one-vs-one strategy. Nonetheless, the ground-truth label for each PL training example is concealed in its candidate label set and thus not accessible to the learning algorithm, binary decomposition cannot be directly applied under partial label learning scenario. In this paper, a novel approach is proposed to solving partial label learning problem by adapting the popular one-vs-one decomposition strategy. Specifically, one binary classifier is derived for each pair of class labels, where PL training examples with distinct relevancy to the label pair are used to generate the corresponding binary training set. After that, one binary classifier is further derived for each class label by stacking over predictions of existing binary classifiers to improve generalization. Experimental studies on both artificial and real-world PL data sets clearly validate the effectiveness of the proposed binary decomposition approach w.r.t state-of-the-art partial label learning techniques.

Partial label (PL) learning tackles the problem where each training instance is associated with a set of candidate labels, among which only one is the true label. In this paper, we propose a simple but effective batch-based partial label learning algorithm named PL-BLC, which tackles the partial label learning problem with batch-wise label correction (BLC). PL-BLC dynamically corrects the label confidence matrix of each training batch based on the current prediction network, and adopts a MixUp data augmentation scheme to enhance the underlying true labels against the redundant noisy labels. In addition, it introduces a teacher model through a consistency cost to ensure the stability of the batch-based prediction network update. Extensive experiments are conducted on synthesized and real-world partial label learning datasets, while the proposed approach demonstrates the state-of-the-art performance for partial label learning.

We address the problem of image denoising for an additive white noise model without placing any restrictions on the statistical distribution of noise. We assume knowledge of only the first- and second-order noise statistics. In the recent mean-square error (MSE) minimization approaches for image denoising, one considers a particular noise distribution and derives an expression for the unbiased risk estimate of the MSE. For additive white Gaussian noise, an unbiased estimate of the MSE is Stein's unbiased risk estimate (SURE), which relies on Stein's lemma. We derive an unbiased risk estimate without using Stein's lemma or its counterparts for additive white noise model irrespective of the noise distribution. We refer to the MSE estimate as the generic risk estimate (GenRE). We demonstrate the effectiveness of GenRE using shrinkage in the undecimated Haar wavelet transform domain as the denoising function. The estimated peak-signal-to-noise-ratio (PSNR) using GenRE is typically within 1% of the PSNR obtained when optimizing with the oracle MSE. The performance of the proposed method is on par with SURE for Gaussian noise distribution, and better than SURE-based methods for other noise distributions such as uniform and Laplacian distribution in terms of both PSNR and structural similarity (SSIM).

Guided image filtering has recently emerged as an effective technique for noise reduction and edge-preserving smoothing operation due to its appealing properties. It outperforms conventional bilateral filtering in a variety of applications in terms of both quality and computational cost. However, the combination of smoothing parameters (e and Ω s ) that delivers optimal results has not yet been reported, and the contrast of the filtered output is considerably reduced. In this paper, we present a restricted version of guided filtering that has better contrast-preserving characteristics, and use Stein's unbiased risk estimate (SURE) with an exhaustive search or a linear expansion of threshold (LET) to optimally tune the two above parameters. With SURE, the mean squared error (MSE) can be unbiasedly estimated without the requirement of the noise-free image. Experiments verified the accuracy of the SURE derivation and its effectiveness with respect to providing a better trade off between two interrelated objectives - noise reduction and edge-preservation.

Stein’s formula states that a random variable of the form z⊤f(z)−divf(z) is mean-zero for all functions f with integrable gradient. Here, divf is the divergence of the function f and z is a standard normal vector. This paper aims to propose a second-order Stein formula to characterize the variance of such random variables for all functions f(z) with square integrable gradient, and to demonstrate the usefulness of this second-order Stein formula in various applications. In the Gaussian sequence model, a remarkable consequence of Stein’s formula is Stein’s Unbiased Risk Estimate (SURE), an unbiased estimate of the mean squared risk for almost any given estimator μˆ of the unknown mean vector. A first application of the second-order Stein formula is an Unbiased Risk Estimate for SURE itself (SURE for SURE): an unbiased estimate providing information about the squared distance between SURE and the squared estimation error of μˆ. SURE for SURE has a simple form as a function of the data and is applicable to all μˆ with square integrable gradient, for example, the Lasso and the Elastic Net. In addition to SURE for SURE, the following statistical applications are developed: (1) upper bounds on the risk of SURE when the estimation target is the mean squared error; (2) confidence regions based on SURE and using the second-order Stein formula; (3) oracle inequalities satisfied by SURE-tuned estimates under a mild Lipschtiz assumption; (4) an upper bound on the variance of the size of the model selected by the Lasso, and more generally an upper bound on the variance of the empirical degrees-of-freedom of convex penalized estimators; (5) explicit expressions of SURE for SURE for the Lasso and the Elastic Net; (6) in the linear model, a general semiparametric scheme to de-bias a differentiable initial estimator for the statistical inference of a low-dimensional projection of the unknown regression coefficient vector, with a characterization of the variance after debiasing; and (7) an accuracy analysis of a Gaussian Monte Carlo scheme to approximate the divergence of functions f:Rn→Rn.

We study the effective degrees of freedom of the lasso in the framework of Stein’s unbiased risk estimation (SURE). We show that the number of nonzero coefficients is an unbiased estimate for the degrees of freedom of the lasso—a conclusion that requires no special assumption on the predictors. In addition, the unbiased estimator is shown to be asymptotically consistent. With these results on hand, various model selection criteria—Cp, AIC and BIC—are available, which, along with the LARS algorithm, provide a principled and efficient approach to obtaining the optimal lasso fit with the computational effort of a single ordinary least-squares fit.

For Gaussian regression, we develop and analyse methods for combining estimators from various models. For squared-error loss, an unbiased estimator of the risk of a mixture of general estimators is developed. Special attention is given to the case that the components are least-squares projections into arbitrary linear subspaces. We relate the unbiased risk estimate for the mixture estimator to estimates of the risks achieved by the components. This results in accurate bounds on the risk and its unbiased estimate - without advance knowledge of which model is best, the resulting performance is comparable to what is achieved by the best of the individual models

The effect of multiplicative noise on a signal when compared with that of additive noise is very large. In this paper, we address the problem of suppressing multiplicative noise in one-dimensional signals. To deal with signals that are corrupted with multiplicative noise, we propose a denoising algorithm based on minimization of an unbiased estimator (MURE) of meansquare error (MSE). We derive an expression for an unbiased estimate of the MSE. The proposed denoising is carried out in wavelet domain (soft thresholding) by considering time-domain MURE. The parameters of thresholding function are obtained by minimizing the unbiased estimator MURE. We show that the parameters for optimal MURE are very close to the optimal parameters considering the oracle MSE. Experiments show that the SNR improvement for the proposed denoising algorithm is competitive with a state-of-the-art method.

For a seemingly unrelated regression system with the assumption of normality, a necessary and sufficient condition for the existence of the Uniformly Minimum Risk Unbiased (UMRU) estimator of regression coefficients under strictly convex loss is obtained; it is proved that any unbiased estimator can not improve the least squares estimator; it is also shown that no UMRU estimator exists under missing observations.