Bayesian Minimum Mean Square Error Research Articles

Robust importance sampling for error estimation in the context of optimal Bayesian transfer learning

SummaryClassification has been a major task for building intelligent systems because it enables decision-making under uncertainty. Classifier design aims at building models from training data for representing feature-label distributions—either explicitly or implicitly. In many scientific or clinical settings, training data are typically limited, which impedes the design and evaluation of accurate classifiers. Atlhough transfer learning can improve the learning in target domains by incorporating data from relevant source domains, it has received little attention for performance assessment, notably in error estimation. Here, we investigate knowledge transferability in the context of classification error estimation within a Bayesian paradigm. We introduce a class of Bayesian minimum mean-square error estimators for optimal Bayesian transfer learning, which enables rigorous evaluation of classification error under uncertainty in small-sample settings. Using Monte Carlo importance sampling, we illustrate the outstanding performance of the proposed estimator for a broad family of classifiers that span diverse learning capabilities.

Robust Importance Sampling for Error Estimation in the Context of Optimal Bayesian Transfer Learning

Classification has been a major task for building intelligent systems as it enables decision-making under uncertainty. Classifier design aims at building models from training data for representing feature-label distributions--either explicitly or implicitly. In many scientific or clinical settings, training data are typically limited, which makes designing accurate classifiers and evaluating their classification error extremely challenging. While transfer learning (TL) can alleviate this issue by incorporating data from relevant source domains to improve learning in a different target domain, it has received little attention for performance assessment, notably in error estimation. In this paper, we fill this gap by investigating knowledge transferability in the context of classification error estimation within a Bayesian paradigm. We introduce a novel class of Bayesian minimum mean-square error (MMSE) estimators for optimal Bayesian transfer learning (OBTL), which enables rigorous evaluation of classification error under uncertainty in a small-sample setting. Using Monte Carlo importance sampling, we employ the proposed estimator to evaluate the classification accuracy of a broad family of classifiers that span diverse learning capabilities. Experimental results based on both synthetic data as well as real-world RNA sequencing (RNA-seq) data show that our proposed OBTL error estimation scheme clearly outperforms standard error estimators, especially in a small-sample setting, by tapping into the data from other relevant domains.

Accurate Tensor Completion via Adaptive Low-Rank Representation.

Low-rank representation-based approaches that assume low-rank tensors and exploit their low-rank structure with appropriate prior models have underpinned much of the recent progress in tensor completion. However, real tensor data only approximately comply with the low-rank requirement in most cases, viz., the tensor consists of low-rank (e.g., principle part) as well as non-low-rank (e.g., details) structures, which limit the completion accuracy of these approaches. To address this problem, we propose an adaptive low-rank representation model for tensor completion that represents low-rank and non-low-rank structures of a latent tensor separately in a Bayesian framework. Specifically, we reformulate the CANDECOMP/PARAFAC (CP) tensor rank and develop a sparsity-induced prior for the low-rank structure that can be used to determine tensor rank automatically. Then, the non-low-rank structure is modeled using a mixture of Gaussians prior that is shown to be sufficiently flexible and powerful to inform the completion process for a variety of real tensor data. With these two priors, we develop a Bayesian minimum mean-squared error estimate framework for inference. The developed framework can capture the important distinctions between low-rank and non-low-rank structures, thereby enabling more accurate model, and ultimately, completion. For various applications, compared with the state-of-the-art methods, the proposed model yields more accurate completion results.

Bayesian MMSE Estimation of a Gaussian Source in the Presence of Bursty Impulsive Noise

In this letter, we derive the minimum mean square error (MMSE) optimal Bayesian estimation (OBE) for a Gaussian source, in the presence of bursty impulsive noise, as essentially encountered within power substations. Clearly, it is observed that the presence of bursty impulsive noise makes the input–output characteristics of MMSE OBE non-linear. To handle the non-linearity, we propose a novel MMSE estimator, based on the detection of the unobservable states of the noise process, using the maximum a posteriori (MAP) detector. Resultantly, the proposed MAP-based MMSE estimator is shown to achieve the lower bound derived for the proposed scenario and outperform the various MMSE estimators that neglect the noise memory.

Moments and root-mean-square error of the Bayesian MMSE estimator of classification error in the Gaussian model

The most important aspect of any classifier is its error rate, because this quantifies its predictive capacity. Thus, the accuracy of error estimation is critical. Error estimation is problematic in small-sample classifier design because the error must be estimated using the same data from which the classifier has been designed. Use of prior knowledge, in the form of a prior distribution on an uncertainty class of feature-label distributions to which the true, but unknown, feature-distribution belongs, can facilitate accurate error estimation (in the mean-square sense) in circumstances where accurate completely model-free error estimation is impossible. This paper provides analytic asymptotically exact finite-sample approximations for various performance metrics of the resulting Bayesian Minimum Mean-Square-Error (MMSE) error estimator in the case of linear discriminant analysis (LDA) in the multivariate Gaussian model. These performance metrics include the first, second, and cross moments of the Bayesian MMSE error estimator with the true error of LDA, and therefore, the root-mean-square (RMS) error of the estimator. We lay down the theoretical groundwork for Kolmogorov double-asymptotics in a Bayesian setting, which enables us to derive asymptotic expressions of the desired performance metrics. From these we produce analytic finite-sample approximations and demonstrate their accuracy via numerical examples. Various examples illustrate the behavior of these approximations and their use in determining the necessary sample size to achieve a desired RMS. The Supplementary Material contains derivations for some equations and added figures.

On MMSE Estimation: A Linear Model Under Gaussian Mixture Statistics

In a Bayesian linear model, suppose observation y = Hx + n stems from independent inputs x and n which are Gaussian mixture (GM) distributed. With known matrix H, the minimum mean square error (MMSE) estimator for x , has analytical form. However, its performance measure, the MMSE itself, has no such closed form. Because existing Bayesian MMSE bounds prove to have limited practical value under these settings, we instead seek analytical bounds for the MMSE, both upper and lower. This paper provides such bounds, and relates them to the signal-to-noise-ratio (SNR).

Bayesian Minimum Mean-Square Error Estimation for Classification Error—Part II: Linear Classification of Gaussian Models

In this paper, Part II of a two-part study, we derive a closed-form analytic representation of the Bayesian minimum mean-square error (MMSE) error estimator for linear classification assuming Gaussian models. This is presented in a general framework permitting a structure on the covariance matrices and a very flexible class of prior parameter distributions with four free parameters. Closed-form solutions are provided for known, scaled identity, and arbitrary covariance matrices. We examine performance in small sample settings via simulations on both synthetic and real genomic data, and demonstrate the robustness of these error estimators to false Gaussian modeling assumptions by applying them to Johnson distributions.

Bayesian Minimum Mean-Square Error Estimation for Classification Error—Part I: Definition and the Bayesian MMSE Error Estimator for Discrete Classification

With the advent of high-throughput genomic and proteomic technologies, in conjunction with the difficulty in obtaining even moderately sized samples, small-sample classifier design has become a major issue in the biological and medical communities. With small samples, training-data error estimation becomes mandatory. Yet none of the popular error estimation techniques have been rigorously designed based on statistical inference and optimization. In this investigation, we place classifier error estimation into the framework of optimal mean-square error (MSE) signal estimation in the presence of uncertainty, which results in a Bayesian approach to error estimation based on a parameterized family of feature-label distributions with the prior distribution of the parameters governing the choice of feature-label distribution. These Bayesian error estimators are optimal when averaged over a given family of distributions, unbiased when averaged over a given family and all samples, and analytically address a trade-off between robustness (modeling assumptions) and accuracy (minimum mean-square error). In this paper, Part I of a two-part study, we define the minimum mean-square error (MMSE) error estimator, discuss its basic properties, provide closed-form analytic estimator representation for discrete classifiers with both non-informative and informative prior distributions, and examine the performance and robustness of the MMSE error estimator via simulations. In Part II of this paper, in this same issue of IEEE Transactions on Signal Processing, we address all of these issues, in particular, closed-form representation for linear classification in the Gaussian model with known and unknown covariance matrices. For both the discrete and Gaussian cases, the MMSE error estimator has especially good performance for distributions having moderate true errors.

A Lower Bound on the Bayesian MSE Based on the Optimal Bias Function

A lower bound on the minimum mean-squared error (MSE) in a Bayesian estimation problem is proposed in this paper. This bound utilizes a well-known connection to the deterministic estimation setting. Using the prior distribution, the bias function which minimizes the Cramer-Rao bound can be determined, resulting in a lower bound on the Bayesian MSE. The bound is developed for the general case of a vector parameter with an arbitrary probability distribution, and is shown to be asymptotically tight in both the high and low signal-to-noise ratio (SNR) regimes. A numerical study demonstrates several cases in which the proposed technique is both simpler to compute and tighter than alternative methods.

Distributed Linear Parameter Estimation over Wireless Sensor Networks

In this paper, we consider distributed linear least squares (LLS) and Bayesian minimum mean square error (BMMSE) parameter estimation over sensor networks. In particular, we propose distributed iterative algorithms that asymptotically converge to the centralized solutions. These algorithms are first studied for the case of unclustered (flat architecture) sensor networks; in this venue, we provide necessary and sufficient conditions for the distributed algorithm to converge. Subsequently, we extend our analysis to clustered sensor networks with pulsed inter-cluster updates. In this latter scenario, inter-cluster communications occur every ß time steps (with ß > 1) and the corresponding updates are held until the next update instant. Depending on sensor locations and the employed network topology construction algorithm, it may be the case that inter-cluster communications require higher transmitter power support than intracluster communications. For energy-constrained sensor networks, it will therefore be beneficial-from a power efficiency (or alternately, network lifetime) point of view-to limit the extent of inter-cluster communication, without significantly deteriorating the convergence properties of the distributed estimation algorithm. We anticipate that a pulsed inter-cluster update scheme will also be useful for applications such as ground or airborne sensor networks, where low probability of detection and interception is essential. Our analysis provides sufficient conditions under which such distributed estimation algorithms, operating on a pulsed inter-cluster update scheme, converge. Simulation results illustrating the dependence of the convergence rate of the algorithm on the hold interval ß conclude the paper.

A Bayesian MMSE Turbo Receiver for Coded MIMO Systems

In conventional interference cancellation (IC)-based turbo receivers, extrinsic information (EXT) from the soft-input-soft-output (SISO) decoders is used to compute the statistical mean (SM) of the interfering signals. In this paper, we present a class of Bayesian minimum mean-square-error (MMSE) turbo receivers for coded multiple-input-multiple-output (MIMO) systems. Instead of using the EXT to estimate the prior SM, we use it in the posterior Bayesian MMSE estimation of the interfering signals. The estimation accuracy is, therefore, improved, leading to better bit-error-rate (BER) performance from our proposed Bayesian receivers. For the cases that we have studied, the proposed receiver with two iterations has a lower BER than the conventional one with five iterations, thus significantly reducing the receiver processing delay and the implementation complexity. Its superior performance is also demonstrated by the higher detector output mutual information and fewer iterations to achieve convergence, which is obtained from the extrinsic information transfer (EXIT) chart analysis.

Wavelet-based image denoising with the normal inverse Gaussian prior and linear MMSE estimator

A new spatially adaptive wavelet-based method is introduced for reducing noise in images corrupted by additive white Gaussian noise. It is shown that a symmetric normal inverse Gaussian distribution is highly suitable for modelling the wavelet coefficients. In order to estimate the parameters of the distribution, a maximum- likelihood-based technique is proposed, wherein the Gauss-Hermite quadrature approximation is exploited to perform the maximisation in a computationally efficient way. A Bayesian minimum mean-squared error (MMSE) estimator is developed utilising the proposed distribution. The variances corresponding to the noise- free coefficients are obtained from the Bayesian estimates using a local neighbourhood. A modified linear MMSE estimator that incorporates both intra-scale and inter-scale dependencies is proposed. The performance of the proposed method is studied using typical noise-free images corrupted with simulated noise and compared with that of the other state-of-the-art methods. It is shown that the proposed method gives higher values of the peak signal-to-noise ratio compared with most of the other denoising techniques and provides images of good visual quality. Also, the performance of the proposed method is quite close to that of the state-of-the-art Gaussian scale mixture (GSM) method, but with much less complexity.

A Bayesian Robust Adaptive Beamforming Based on Phase-Invariant MMSE Criterion

The Bayesian beamformer under minimum mean square error (MMSE) criterion has been reported to perform well against uncertainty in source direction of arrival (DOA), but its performance depends on the location of phase reference. This letter proposes a phase-invariant MMSE criterion under which the Bayesian beamformer is independent of the location of phase reference. An iterative algorithm is developed for designing such a new Bayesian beamformer. Numerical examples demonstrate that the new Bayesian beamformer can provide better performance than the existing Bayesian MMSE beamformer for either uniform or non-uniform linear array.

Optimal estimation of acoustic intensity from pressure in the presence of mean flow

Bayesian minimum mean-squared error (MMSE) estimators are constructed for obtaining pressure, velocity, and intensity in the near- or far-field of an extended acoustic source from near-field pressure measurements alone, in the presence of noise and mean flow. In the absence of mean flow, there is a unique relation between the local pressure gradient and the local velocity, but in the presence of mean flow this uniqueness is destroyed [D. Munro and K. Ingard, ‘‘On acoustic intensity measurements in the presence of mean flow,’’ J. Acoust. Soc. Am. 65, 1402–1406 (1979)]. This nonuniqueness makes an exact reconstruction of the velocity and intensity fields impossible, even in the absence of noise; however, accurate approximate reconstruction may be accomplished with a sufficient array of receivers. The MMSE estimation framework provides a prediction of the estimation accuracy which may be used to assess the sufficiency of a given pressure sensor array. Numerical examples of array performance assessment are presented for intensity estimation in the near field of an extended source in mean flows of Mach 0 to 0.8.