Statistical Estimation Theory Research Articles

Generalization error guaranteed auto-encoder-based nonlinear model reduction for operator learning

Many physical processes in science and engineering are naturally represented by operators between infinite-dimensional function spaces. The problem of operator learning, in this context, seeks to extract these physical processes from empirical data, which is challenging due to the infinite or high dimensionality of data. An integral component in addressing this challenge is model reduction, which reduces both the data dimensionality and problem size. In this paper, we utilize low-dimensional nonlinear structures in model reduction by investigating Auto-Encoder-based Neural Network (AENet). AENet first learns the latent variables of the input data and then learns the transformation from these latent variables to corresponding output data. Our numerical experiments validate the ability of AENet to accurately learn the solution operator of nonlinear partial differential equations. Furthermore, we establish a mathematical and statistical estimation theory that analyzes the generalization error of AENet. Our theoretical framework shows that the sample complexity of training AENet is intricately tied to the intrinsic dimension of the modeled process, while also demonstrating the robustness of AENet to noise.

Quantum Onsager relations

Abstract Using quantum information geometry, I derive quantum generalizations of the Onsager rate equations, which model the dynamics of an open system near a steady state. The generalized equations hold for a flexible definition of the forces as well as a large class of statistical divergence measures and quantum-Fisher-information metrics beyond the conventional definition of entropy production. I also derive quantum Onsager-Casimir relations for the transport tensors by proposing a general concept of time reversal and detailed balance for open quantum systems. The results establish a remarkable connection between statistical mechanics and parameter estimation theory.

Statistical estimation theory detection limits for label-free imaging.

The emergence of label-free microscopy techniques has significantly improved our ability to precisely characterize biochemical targets, enabling non-invasive visualization of cellular organelles and tissue organization. However, understanding each label-free method with respect to the specific benefits, drawbacks, and varied sensitivities under measurement conditions across different types of specimens remains a challenge. We link all of these disparate label-free optical interactions together and compare the detection sensitivity within the framework of statistical estimation theory. To achieve this goal, we introduce a comprehensive unified framework for evaluating the bounds for signal detection with label-free microscopy methods, including second-harmonic generation, third-harmonic generation, coherent anti-Stokes Raman scattering, coherent Stokes Raman scattering, stimulated Raman loss, stimulated Raman gain, stimulated emission, impulsive stimulated Raman scattering, transient absorption, and photothermal effect. A general model for signal generation induced by optical scattering is developed. Based on this model, the information obtained is quantitatively analyzed using Fisher information, and the fundamental constraints on estimation precision are evaluated through the Cramér-Rao lower bound, offering guidance for optimal experimental design and interpretation. We provide valuable insights for researchers seeking to leverage label-free techniques for non-invasive imaging applications for biomedical research and clinical practice.

Photon Phase Delay Sensing with Sub-Attosecond Uncertainty.

The application of statistical estimation theory to Hong-Ou-Mandel interferometry led to enticing results in terms of the detection limit for photon reciprocal delay and polarisation measurement. In the following paper, a fully fibre-coupled setup operating in the telecom wavelength region proves to achieve, for the first time, in common-path Hong-Ou-Mandel-based interferometry, a detection limit for photon phase delay at the zeptosecond scale. The experimental results are then framed in a theoretical model by calculating the Cramer-Rao bound (CRB) and, after comparison with the obtained experimental results, it is shown that our setup attains the optimal measurement, nearly saturating CRB.

Stochastic Arbitrage Opportunities: Set Estimation and Statistical Testing

We provide a formal statistical theory of consistent estimation of the set of all arbitrage portfolios that meet the description of being a stochastic arbitrage opportunity. Two empirical likelihood ratio tests are developed: one for the null that a given arbitrage portfolio is qualified, and another for the alternative that the portfolio is not qualified. Apart from considering generalized concepts and hypotheses based on multiple host portfolios, the statistical assumption framework is also more general than in earlier studies that focused on special cases with a single benchmark portfolio. Various extensions and generalizations of the theory are discussed. A Monte Carlo simulation study shows promising statistical size and power properties for testing the null, for representative data dimensions. The results of an empirical application illustrate the importance of selecting a proper blocking structure and moment estimation method.

On the estimation of the core for TU games

The core of a transferable utility (TU) game, if it is not empty, is prescribed by the set of all stable allocations. The exact determination of the core reaches exponential time complexity. Therefore, its exact computation is often avoided as the number of players increases. In this work, we propose an estimator for the core of a TU game based on the statistical theory of set estimation. Concretely, we provide a core reconstruction that is obtained in polynomial time for general dimension. Additionally, convergence rates for the estimation error are derived. Finally, a consistent core-center estimator is established as a geometrical application of this methodology.

Condition numbers of the generalized ridge regression and its statistical estimation

<abstract><p>In this paper, we considered the condition number theory of a new generalized ridge regression model. The explicit expressions of different types of condition numbers were derived to measure the ill-conditionness of the generalized ridge regression problem with respect to different circumstances. To overcome the computational difficulty of computing the exact value of the condition number, we employed the statistical condition estimation theory to design efficient condition number estimators, and the numerical examples were also given to illustrate its efficiency.</p></abstract>

Optimal covariance cleaning for heavy-tailed distributions: Insights from information theory.

In optimal covariance cleaning theory, minimizing the Frobenius norm between the true population covariance matrix and a rotational invariant estimator is a key step. This estimator can be obtained asymptotically for large covariance matrices, without knowledge of the true covariance matrix. In this study, we demonstrate that this minimization problem is equivalent to minimizing the loss of information between the true population covariance and the rotational invariant estimator for normal multivariate variables. However, for Student's t distributions, the minimal Frobenius norm does not necessarily minimize the information loss in finite-sized matrices. Nevertheless, such deviations vanish in the asymptotic regime of large matrices, which might extend the applicability of random matrix theory results to Student's t distributions. These distributions are characterized by heavy tails and are frequently encountered in real-world applications such as finance, turbulence, or nuclear physics. Therefore, our work establishes a connection between statistical random matrix theory and estimation theory in physics, which is predominantly based on information theory.

Quantum information criteria for model selection in quantum state estimation

Quantum state estimation (or state tomography) is an indispensable task in quantum information processing. Because full state tomography that determines all elements of the density matrix is computationally demanding, one usually takes the strategy of assuming a certain model of quantum states and identifying the model parameters. However, it is difficult to make a valid assumption given little prior knowledge on a quantum state of interest, and thus we need a reasonable model selection method for quantum state estimation. Actually, in the classical statistical estimation theory, several types of information criteria have been established and widely used in practice for appropriately choosing a classical statistical model. In this study, we propose quantum information criteria for evaluating the quality of the estimated quantum state in terms of the quantum relative entropy, which is a natural quantum analogue of the classical information criterion defined in terms of Kullback–Leibler divergence. In particular, we derive two quantum information criteria depending on the type of an estimator for the quantum relative entropy; one uses the log-likelihood and the other uses the classical shadow. The general role of information criteria is to predict the performance of an estimated model for unseen data, although it is a function of only sampled data; this generalization capability of the proposed quantum information criteria is evaluated in numerical simulations.

A simple analytical expression of quantum Fisher and Skew information and their dynamics under decoherence channels

In statistical estimation theory, it has been shown previously that the Wigner–Yanase skew information is bounded by the quantum Fisher information associated with the phase parameter. Besides, the quantum Cramér–Rao inequality is expressed in terms of skew information. Since these two fundamental quantities are based on the concept of quantum uncertainty, we derive here their analytical formulas for arbitrary two qubit X-states using the same analytical procedures. A comparison of these two informational quantifiers for two quasi-Werner states composed of two bipartite superposed coherent states is examined. Moreover, we investigated the decoherence effects on such quantities generated by the phase damping, depolarization and amplitude damping channels. We showed that decoherence strongly influences the quantum criteria during the evolution and these quantities exhibit similar dynamic behaviors. This current work is characterized by the fact that these two concepts play the same role and capture similar properties in quantum estimation protocols.

Optimal Measurement of Telecom Wavelength Single Photon Polarisation via Hong-Ou-Mandel Interferometry

The use of statistical estimation theory to boost the metrological performance of the measurement apparatus is becoming increasingly popular in a wide range of applications. Recently, such an approach has been adopted in Hong Ou Mandel interferometry, setting a new record in time delay and polarization measurement. Here, we extend these pioneering experiments in the telecom range to unlock the full potential of the information-based approach combined with a versatile spectral range, aiming for its adoption in fiber-coupled devices of up to hundreds of kilometers long as bobines or optical networks. Our measurement saturates the Cramér-Rao bound and in a long lasting experiment returns an Allan deviation of the polarization angle of 0.002 degs in 1 h of integration time.

Rapid nanometer-precision autocorrelator.

The precise measurement of a target depth has applications in biophysics and nanophysics, and non-linear optical methods are sensitive to intensity changes on very small length scales. By exploiting the high sensitivity of an autocorrelator's dependency on path length, we propose a technique that achieves ≈30 nm depth precision for each pixel in 30 seconds. Our method images up-converted pulses from a non-linear crystal using a sCMOS (scientific Complementary Metal-Oxide-Semiconductor) camera and converts the intensity recorded by each pixel to a delay. By utilising statistical estimation theory and using the data from a set of 32×32 pixels, the standard error (SE) of the detected delay falls below 1 nm after 30 seconds of measurement. Numerical simulations show that this result is extremely close to what can be achieved with a shot-noise-limited source and is consistent with the precision that can be achieved with a sCMOS camera.

Track Segment Association With Dual Contrast Neural Network

Track interruption will cause serious interference to target tracking, situation awareness, and information fusion. And it is often triggered by a variety of reasons, such as target maneuver, the blind spot of Doppler radar, and excessive long sampling interval, among other causes. All the existing algorithms are based on the hypothetical target motion models and adopt the theory of statistical estimation to measure the similarity between predicted tracks to associate, which have some disadvantages such as unreasonable hypotheses, inapplicable models, and uncertain thresholds. In order to address these problems, in this article, we propose a track segment association dual contrast neural network (TSADCNN) to complete track segment association tasks. First, based on deep metric learning, we propose a temporal and spatial information extraction module to extract track information, especially symmetric correlation information and raise dimension. Second, by constructing an appropriate optimization function, TSADCNN makes track segments belonging to the same target closer, track segments belonging to different targets farther in high-dimensional space. Finally, TSADCNN will pick the nearest neighbor vectors in high-dimensional space as associated tracks. Extensive experiments on simulation data are conducted to evaluate the effectiveness of TSADCNN, and the results demonstrate that this method has good adaptability and anti-noise ability. Furthermore, the contrast experiment shows that it can outperform other TSA methods in both quality and efficiency. Compared with the best performance, the average true association rate of TSADCNN can be increased by 1.3% in long interrupt intervals.

Noisy quantum amplitude estimation without noise estimation

Many quantum algorithms contain an important subroutine---the quantum amplitude estimation. As the name implies, this is essentially the parameter estimation problem and thus can be handled via the established statistical estimation theory. However, this problem has an intrinsic difficulty that the system, i.e., the real quantum computing device, inevitably introduces unknown noise; the probability distribution model then has to incorporate many nuisance noise parameters, resulting that the construction of an optimal estimator becomes inefficient and difficult. For this problem we apply the theory of nuisance parameters (more specifically, the parameter orthogonalization method) to precisely compute the maximum likelihood estimator for only the target amplitude parameter by removing the other nuisance noise parameters. That is, we can estimate the amplitude parameter without estimating the noise parameters. We validate the parameter orthogonalization method in a numerical simulation and study the performance of the estimator in the experiment using a real superconducting quantum device.

Limit theorems for Toeplitz-type quadratic functionals of stationary processes and applications

This is a survey of recent results on central and non-central limit theorems for quadratic functionals of stationary processes. The underlying processes are Gaussian, linear or Lévy-driven linear processes with memory, and are defined either in discrete or continuous time. We focus on limit theorems for Toeplitz and tapered Toeplitz type quadratic functionals of stationary processes with applications in parametric and nonparametric statistical estimation theory. We discuss questions concerning Toeplitz matrices and operators, Fejér-type singular integrals, and Lévy-Itô-type and Stratonovich-type multiple stochastic integrals. These are the main tools for obtaining limit theorems.

Assessing the Global and Local Uncertainty of Scientific Evidence in the Presence of Model Misspecification

Scientists need to compare the support for models based on observed phenomena. The main goal of the evidential paradigm is to quantify the strength of evidence in the data for a reference model relative to an alternative model. This is done via an evidence function, such as ΔSIC, an estimator of the sample size scaled difference of divergences between the generating mechanism and the competing models. To use evidence, either for decision making or as a guide to the accumulation of knowledge, an understanding of the uncertainty in the evidence is needed. This uncertainty is well characterized by the standard statistical theory of estimation. Unfortunately, the standard theory breaks down if the models are misspecified, as is commonly the case in scientific studies. We develop non-parametric bootstrap methodologies for estimating the sampling distribution of the evidence estimator under model misspecification. This sampling distribution allows us to determine how secure we are in our evidential statement. We characterize this uncertainty in the strength of evidence with two different types of confidence intervals, which we term “global” and “local.” We discuss how evidence uncertainty can be used to improve scientific inference and illustrate this with a reanalysis of the model identification problem in a prominent landscape ecology study using structural equations.

RTop-k: A Statistical Estimation Approach to Distributed SGD

The large communication cost for exchanging gradients between different nodes significantly limits the scalability of distributed training for large-scale learning models. Motivated by this observation, there has been significant recent interest in techniques that reduce the communication cost of distributed Stochastic Gradient Descent (SGD), with gradient sparsification techniques such as top-k and random-k shown to be particularly effective. The same observation has also motivated a separate line of work in distributed statistical estimation theory focusing on the impact of communication constraints on the estimation efficiency of different statistical models. The primary goal of this paper is to connect these two research lines and demonstrate how statistical estimation models and their analysis can lead to new insights in the design of communication-efficient training techniques. We propose a simple statistical estimation model for the stochastic gradients which captures the sparsity and skewness of their distribution. The statistically optimal communication scheme arising from the analysis of this model leads to a new sparsification technique for SGD, which concatenates random-k and top-k, considered separately in the prior literature. We show through extensive experiments on both image and language domains with CIFAR-10, ImageNet, and Penn Treebank datasets that the concatenated application of these two sparsification methods consistently and significantly outperforms either method applied alone.

Data-Driven and Knowledge-Based Algorithms for Gene Network Reconstruction on High-Dimensional Data.

Previous efforts in gene network reconstruction have mainly focused on data-driven modeling, with little attention paid to knowledge-based approaches. Leveraging prior knowledge, however, is a promising paradigm that has been gaining momentum in network reconstruction and computational biology research communities. This paper proposes two new algorithms for reconstructing a gene network from expression profiles with and without prior knowledge in small sample and high-dimensional settings. First, using tools from the statistical estimation theory, particularly the empirical Bayesian approach, the current research estimates a covariance matrix via the shrinkage method. Second, estimated covariance matrix is employed in the penalized normal likelihood method to select the Gaussian graphical model. This formulation allows the application of prior knowledge in the covariance estimation, as well as in the Gaussian graphical model selection. Experimental results on simulated and real datasets show that, compared to state-of-the-art methods, the proposed algorithms achieve better results in terms of both PR and ROC curves. Finally, the present work applies its method on the RNA-seq data of human gastric atrophy patients, which was obtained from the EMBL-EBI database. The source codes and relevant data can be downloaded from: https://github.com/AbbaszadehO/DKGN.

Set-Convergence and Its Application: A Tutorial

Optimization problems, generalized equations, and a multitude of other variational problems invariably lead to the analysis of sets and set-valued mappings as well as their approximations. We review the central concept of set-convergence and explain its role in defining a notion of proximity between sets, especially for epigraphs of functions and graphs of set-valued mappings. The development leads to an approximation theory for optimization problems and generalized equations with profound consequences for the construction of algorithms. We also introduce the role of set-convergence in variational geometry and subdifferentiability with applications to optimality conditions. Examples illustrate the importance of set-convergence in stability analysis, error analysis, construction of algorithms, statistical estimation, and probability theory.

Atom column detection from simultaneously acquired ABF and ADF STEM images

In electron microscopy, the maximum a posteriori (MAP) probability rule has been introduced as a tool to determine the most probable atomic structure from high-resolution annular dark-field (ADF) scanning transmission electron microscopy (STEM) images exhibiting low contrast-to-noise ratio (CNR). Besides ADF imaging, STEM can also be applied in the annular bright-field (ABF) regime. The ABF STEM mode allows to directly visualize light-element atomic columns in the presence of heavy columns. Typically, light-element nanomaterials are sensitive to the electron beam, limiting the incoming electron dose in order to avoid beam damage and leading to images exhibiting low CNR. Therefore, it is of interest to apply the MAP probability rule not only to ADF STEM images, but to ABF STEM images as well. In this work, the methodology of the MAP rule, which combines statistical parameter estimation theory and model-order selection, is extended to be applied to simultaneously acquired ABF and ADF STEM images. For this, an extension of the commonly used parametric models in STEM is proposed. Hereby, the effect of specimen tilt has been taken into account, since small tilts from the crystal zone axis affect, especially, ABF STEM intensities. Using simulations as well as experimental data, it is shown that the proposed methodology can be successfully used to detect light elements in the presence of heavy elements.