Fisher Information Matrix Research Articles

Bayesian Frequency Estimation Under Local Differential Privacy With an Adaptive Randomized Response Mechanism

Frequency estimation plays a critical role in many applications involving personal and private categorical data. Such data are often collected sequentially over time, making it valuable to estimate their distribution online while preserving privacy. We propose AdOBEst-LDP, a new algorithm for adaptive, online Bayesian estimation of categorical distributions under local differential privacy (LDP). The key idea behind AdOBEst-LDP is to enhance the utility of future privatized categorical data by leveraging inference from previously collected privatized data. To achieve this, AdOBEst-LDP uses a new adaptive LDP mechanism to collect privatized data. This LDP mechanism constrains its output to a subset of categories that ‘predicts’ the next user's data. By adapting the subset selection process to the past privatized data via Bayesian estimation, the algorithm improves the utility of future privatized data. To quantify utility, we explore various well-known information metrics, including (but not limited to) the Fisher information matrix, total variation distance, and information entropy. For Bayesian estimation, we utilize posterior sampling through stochastic gradient Langevin dynamics, a computationally efficient approximate Markov chain Monte Carlo (MCMC) method. We provide a theoretical analysis showing that (i) the posterior distribution of the category probabilities targeted with Bayesian estimation converges to the true probabilities even for approximate posterior sampling, and (ii) AdOBEst-LDP eventually selects the optimal subset for its LDP mechanism with high probability if posterior sampling is performed exactly. We also present numerical results to validate the estimation accuracy of AdOBEst-LDP. Our comparisons show its superior performance against non-adaptive and semi-adaptive competitors across different privacy levels and distributional parameters.

An Analysis of Positioning Accuracy Based on Fisher Information Matrix for Distance-Based Positioning

An Analysis of Positioning Accuracy Based on Fisher Information Matrix for Distance-Based Positioning

Estimating the Lifetime Parameters of the Odd-Generalized-Exponential–Inverse-Weibull Distribution Using Progressive First-Failure Censoring: A Methodology with an Application

This paper investigates statistical methods for estimating unknown lifetime parameters using a progressive first-failure censoring dataset. The failure mode’s lifetime distribution is modeled by the odd-generalized-exponential–inverse-Weibull distribution. Maximum-likelihood estimators for the model parameters, including the survival, hazard, and inverse hazard rate functions, are obtained, though they lack closed-form expressions. The Newton–Raphson method is used to compute these estimations. Confidence intervals for the parameters are approximated via the normal distribution of the maximum-likelihood estimation. The Fisher information matrix is derived using the missing information principle, and the delta method is applied to approximate the confidence intervals for the survival, hazard rate, and inverse hazard rate functions. Bayes estimators were calculated with the squared error, linear exponential, and general entropy loss functions, utilizing independent gamma distributions for informative priors. Markov-chain Monte Carlo sampling provides the highest-posterior-density credible intervals and Bayesian point estimates for the parameters and reliability characteristics. This study evaluates these methods through Monte Carlo simulations, comparing Bayes and maximum-likelihood estimates based on mean squared errors for point estimates, average interval widths, and coverage probabilities for interval estimators. A real dataset is also analyzed to illustrate the proposed methods.

Optimal Constant-Stress Accelerated Life Tests for the Marshall-Olkin Extended Lindley Distribution under Complete Sampling

This study introduces an extended Lindley distribution utilizing a new family of Marshall-Olkin distributions, providing a more flexible framework for modeling lifetime data. The distribution is examined within the context of complete sampling for k-level constant stress accelerated life testing. A reliability analysis of the proposed model and a discussion on parameter estimation using maximum likelihood estimation are presented. Asymptotic confidence intervals for the parameters are derived through the Fisher information matrix. Bayesian estimation procedures and the Markov Chain Monte Carlo (MCMC) approach are also explored. Real data is analyzed to assess the model’s fit, followed by simulation studies to illustrate its performance. The paper concludes with a summary of findings and implications for future research.

A Varying Precision Beta Prime Autoregressive Moving Average Model With Application to Water Flow Data

ABSTRACTWe introduce a dynamic model tailored for positively valued time series. It accommodates both autoregressive and moving average dynamics and allows for explanatory variables. The underlying assumption is that each random variable follows, conditional on the set of previous information, the beta prime distribution. A novel feature of the proposed model is that both the conditional mean and conditional precision evolve over time. The model thus comprises two dynamic submodels, one for each parameter. The proposed model for the conditional precision parameter is parsimonious, incorporating first‐order time dependence. Changes over time in the shape of the density are determined by the time evolution of two parameters, and not just of the conditional mean. We present simple closed‐form expressions for the model's conditional log‐likelihood function, score vector, and Fisher's information matrix. Monte Carlo simulation results are presented. Finally, we use the proposed approach to model and forecast two seasonal water flow time series. Specifically, we model the inflow and outflow rates of the reservoirs of two hydroelectric power plants. Overall, the forecasts obtained using the proposed model are more accurate than those yielded by alternative models.

Gill and Massar type bound for estimation of SU(2) channel

Abstract In the estimation for a parametric family of quantum state on a Hilbert space H, the Gill and Massar bound is known as a lower bound of weighted traces of covariances of unbiased estimators. The Gill and Massar bound is derived by considering the convexity of the set of classical Fisher information matrices, and the bound is locally achievable by using randomized strategies when H=C2. In this paper, we show that estimation for a parametric SU(2) unitary channel model has a similar convex structure as qubit state model, and a Gill and Massar type lower bound of weighted traces of covariances of unbiased estimators can be derived for any weight matrix. We show that the Gill and Massar type lower bound is achievable by using randomized strategies when certain conditions are satisfied. To derive a convex strucutre of the set of classical Fisher information matrices, we introduce a Fisher information matrix J(U) for a SU(2) unitary channel model, and we show a upper bound of inverse J(U) weighted trace of classical Fisher information matrix. The optimal randomized strategy we construct in this paper does not require ancilla systems in many cases.

Order-restricted statistical inference and optimal censoring scheme for Gompertz distribution based on joint type-II progressive censored data

This paper considers the order-restricted statistical inference for two populations based on the joint type-II progressive censoring scheme. The lifetime distributions of the two populations are supposed to follow the Gompertz distribution with the same shape parameter but different scale parameters. The maximum likelihood estimates of the unknown parameters are derived by employing the Newton-Raphson algorithm and the expectation-maximization algorithm, respectively. The Fisher information matrix is then employed to construct asymptotic confidence intervals. For Bayes estimation, we assume an ordered Beta-Gamma prior for the scale parameters and a Gamma prior for the common shape parameter. Bayes estimations and the highest posterior density credible intervals for unknown parameters under two different loss functions are obtained with the importance sampling technique. To evaluate the performance of order-restricted inference, extensive Monte Carlo simulations are performed and two air-conditioning systems datasets are used to illustrate the proposed inference methods. In addition, the results are compared with the case when there is no order restriction on the parameters. Finally, the optimal censoring scheme is obtained by four optimality criteria.

Random Natural Gradient

Hybrid quantum-classical algorithms appear to be the most promising approach for near-term quantum applications. An important bottleneck is the classical optimization loop, where the multiple local minima and the emergence of barren plateaux make these approaches less appealing. To improve the optimization the Quantum Natural Gradient (QNG) method \cite{stokes2020quantum} was introduced – a method that uses information about the local geometry of the quantum state-space. While the QNG-based optimization is promising, in each step it requires more quantum resources, since to compute the QNG one requires O(m2) quantum state preparations, where m is the number of parameters in the parameterized circuit. In this work we propose two methods that reduce the resources/state preparations required for QNG, while keeping the advantages and performance of the QNG-based optimization. Specifically, we first introduce the Random Natural Gradient (RNG) that uses random measurements and the classical Fisher information matrix (as opposed to the quantum Fisher information used in QNG). The essential quantum resources reduce to linear O(m) and thus offer a quadratic "speed-up", while in our numerical simulations it matches QNG in terms of accuracy. We give some theoretical arguments for RNG and then benchmark the method with the QNG on both classical and quantum problems. Secondly, inspired by stochastic-coordinate methods, we propose a novel approximation to the QNG which we call Stochastic-Coordinate Quantum Natural Gradient that optimizes only a small (randomly sampled) fraction of the total parameters at each iteration. This method also performs equally well in our benchmarks, while it uses fewer resources than the QNG.

Prospect of Detecting Magnetic Fields from Strong-magnetized Binary Neutron Stars

Binary neutron star mergers are unique sources of gravitational waves in multi-messenger astronomy. The inspiral phase of binary neutron stars can emit gravitational waves as chirp signals. The present waveform models of gravitational waves only considered the gravitational interaction. In this paper, we derive the waveform of the gravitational wave signal taking into account the presence of magnetic fields. We found that the electromagnetic interaction and radiation can introduce different frequency-dependent power laws for both the amplitude and frequency of the gravitational wave. We show from the results of the Fisher information matrix that the third-generation observation may detect magnetic dipole moments if the magnetic field is ∼1017 G.

Detecting the tidal heating with the generic extreme mass-ratio inspirals

The horizon of a classical black hole (BH), functioning as a one-way membrane, plays a vital role in the dynamic evolution of binary BHs, capable of absorbing fluxes entirely.Tidal heating, stemming from this phenomenon, exerts a notable influence on the production of gravitational waves (GWs). If at least one member of a binary is an exotic compact object (ECO) instead of a BH, the absorption of fluxes is expected to be incomplete and the tidal heating would be different. Thus, tidal heating can be utilized for model-independent investigations into the nature of compact object. In this paper, assuming that the extreme mass-ratio inspiral (EMRI) contains a stellar-mass compact object orbiting around a massive ECO with a reflective surface, we compute the GWs from the generic EMRI orbits. Using the accurate and analytic flux formulas in the black hole spacetime, we adapted these formulas in the vicinity of the ECO surface by incorporating a reflectivity parameter.Under the adiabatic approximation, we can evolve the orbital parameters and compute the EMRI waveforms. The effect of tidal heating for the spinning and non-spinning objects can be used to constrain the reflectivity of the surface at the level of 𝒪(10-6) by computing the mismatch and fisher information matrix.

Natural Gradient Variational Bayes Without Fisher Matrix Analytic Calculation and Its Inversion

This article introduces a method for efficiently approximating the inverse of the Fisher information matrix, a crucial step in achieving effective variational Bayes inference. A notable aspect of our approach is the avoidance of analytically computing the Fisher information matrix and its explicit inversion. Instead, we introduce an iterative procedure for generating a sequence of matrices that converge to the inverse of Fisher information. The natural gradient variational Bayes algorithm without analytic expression of the Fisher matrix and its inversion is provably convergent and achieves a convergence rate of order O ( log s / s ) , with s the number of iterations. We also obtain a central limit theorem for the iterates. Implementation of our method does not require storage of large matrices, and achieves a linear complexity in the number of variational parameters. Our algorithm exhibits versatility, making it applicable across a diverse array of variational Bayes domains, including Gaussian approximation and normalizing flow Variational Bayes. We offer a range of numerical examples to demonstrate the efficiency and reliability of the proposed variational Bayes method. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Natural gradient hybrid variational inference with application to deep mixed models

Stochastic models with global parameters and latent variables are common, and for which variational inference (VI) is popular. However, existing methods are often either slow or inaccurate in high dimensions. We suggest a fast and accurate VI method for this case that employs a well-defined natural gradient variational optimization that targets the joint posterior of the global parameters and latent variables. It is a hybrid method, where at each step the global parameters are updated using the natural gradient and the latent variables are generated from their conditional posterior. A fast to compute expression for the Tikhonov damped Fisher information matrix is used, along with the re-parameterization trick, to provide a stable natural gradient. We apply the approach to deep mixed models, which are an emerging class of Bayesian neural networks with random output layer coefficients to allow for heterogeneity. A range of simulations show that using the natural gradient is substantially more efficient than using the ordinary gradient, and that the approach is faster and more accurate than two cutting-edge natural gradient VI methods. In a financial application we show that accounting for industry level heterogeneity using the deep mixed model improves the accuracy of asset pricing models. MATLAB code to implement the method and replicate the results can be found at https://github.com/WeibenZhang07/NG-HVI

Inference for depending competing risks from Marshall–Olikin bivariate Kies distribution under generalized progressive hybrid censoring

This paper explores inferences for a competing risk model with dependent causes of failure. When the lifetimes of competing risks are modelled by a Marshall–Olikin bivariate Kies distribution, classical and Bayesian estimations are studied under generalized progressive hybrid censoring. The existence and uniqueness results for maximum likelihood estimators of unknown parameters are established, whereas approximate confidence intervals are constructed using the observed Fisher information matrix. In addition, Bayes estimates are explored based on a flexible Gamma-Dirichlet prior information. Furthermore, when there is a priori order information on competing risk parameters being available, traditional classical likelihood and Bayesian estimates are also established under restricted parameter case. The behavior of the proposed estimators is evaluated through extensive simulation studies, and a real data study is presented for illustrative purposes.

Comparison of estimation limits for quantum two-parameter estimation

Measurement estimation bounds for local quantum multiparameter estimation, which provide lower bounds on possible measurement uncertainties, have so far been formulated in two ways: by extending the classical Cramér-Rao bound (e.g., the quantum Cramér-Rao bound and the Nagaoka Cramér-Rao bound) and by incorporating the parameter estimation framework with the uncertainty principle, as in the Lu-Wang uncertainty relation. In this work, we present a general framework that allows a direct comparison between these different types of estimation limits. Specifically, we compare the attainability of the Nagaoka Cramér-Rao bound and the Lu-Wang uncertainty relation, using analytical and numerical techniques. We show that these two limits can provide different information about the physically attainable precision. We present an example where both limits provide the same attainable precision and an example where the Lu-Wang uncertainty relation is not attainable even for pure states. We further demonstrate that the unattainability in the latter case arises because the figure of merit underpinning the Lu-Wang uncertainty relation (the difference between the quantum and classical Fisher information matrices) does not necessarily agree with the conventionally used figure of merit (mean-squared error). The results offer insights into the general attainability and applicability of the Lu-Wang uncertainty relation. Furthermore, our proposed framework for comparing bounds of different types may prove useful in other settings. Published by the American Physical Society 2024

Learning under singularity: an information criterion improving WBIC and sBIC

AbstractWe introduce a novel information criterion, termed learning under singularity (LS), designed to enhance the functionality of the widely applicable Bayes information criterion (WBIC) and the singular Bayesian information criterion (sBIC). LS is effective without regularity constraints and demonstrates stability. Watanabe defined a statistical model or a learning machine as regular if the mapping from a parameter to a probability distribution is one-to-one and its Fisher information matrix is positive definite. In contrast, models not meeting these conditions are termed singular. Over the past decade, several information criteria for singular cases have been proposed, including WBIC and sBIC. WBIC is applicable in non-regular scenarios but faces challenges with large sample sizes and redundant estimation of known learning coefficients. Conversely, sBIC is limited in its broader application due to its dependence on maximum likelihood estimates. LS addresses these limitations by enhancing the utility of both WBIC and sBIC. It incorporates the empirical loss from the widely applicable information criterion (WAIC) to represent the goodness of fit to the statistical model, along with a penalty term similar to that of sBIC. This approach offers a flexible and robust method for model selection, free from regularity constraints.

Constraining the EdGB theory with extreme mass-ratio inspirals

The Einstein-dilaton-Gauss–Bonnet (EdGB) theory is a modified theory of gravity which include a scalar field to couple with the higher order curvature terms. It has already been constrained with various observations include the gravitational wave (GW) with LIGO, Virgo and KAGRA (LVK) Collaboration. In this work, we study the capability for space-borne GW detectors to constrain the EdGB theory using the signal of Extreme Mass-Ratio Inspiral (EMRIs). We use the “numerical kludge (NK)” method to construct the waveform of EMRI in the EdGB theory, focusing on the case when the central black hole is spinless. We then study how a future space-borne gravitational wave detector, TianQin, for example, can place constraints on the EdGB theory through the detection of EMRIs. With the analysis using mismatch and Fisher Information Matrix (FIM), we find that the EdGB parameter α\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sqrt{\\alpha }$$\\end{document} is expected to be constrained to the level of ∼O(0.1)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim \\mathcal {O}(0.1)$$\\end{document} km.

Statistical inference for marshall-olkin bivariate Kumaraswamy distribution under adaptive progressive hybrid censored dependent competing risks data

Abstract The statistical inference under competing risks model is of great significance in reliability analysis and it is more practical to assume that they have dependent competing causes of failure in actual situations. In this article, we make inference for unknown parameters of a Marshall-Olkin bivariate Kumaraswamy distribution under adaptive progressive hybrid censoring mechanism. The maximum likelihood estimations of the unknown parameters are derived, and the Fisher information matrix is then employed to construct asymptotic confidence intervals. Bayes estimates are evaluated against squared error and linex loss functions assuming ordered Gamma-Dirichlet and Gamma-Dirichlet prior distributions for order restriction and without order restriction cases respectively. The Metropolis-Hasting and Lindley techniques are applied to acquire the estimates of all unknown parameters. A thorough simulation analysis is demonstrated to assess the performance of the supplied approaches across various sample sizes. The usefulness of the techniques is illustrated using real engineering data to prove their versatility in practical applications.

Analysis and comparison of two-level KFAC methods for training deep neural networks

As a second-order method, the Natural Gradient Descent (NGD) has the ability to accelerate training of neural networks. However, due to the prohibitive computational and memory costs of computing and inverting the Fisher Information Matrix (FIM), efficient approximations are necessary to make NGD scalable to Deep Neural Networks (DNNs). Many such approximations have been attempted. The most sophisticated of these is KFAC, which approximates the FIM as a block-diagonal matrix, where each block corresponds to a layer of the neural network. By doing so, KFAC ignores the interactions between different layers. In this work, we investigate the interest of restoring some low-frequency interactions between the layers by means of two-level methods. Inspired from domain decomposition, several two-level corrections to KFAC using different coarse spaces are proposed and assessed. The obtained results show that incorporating the layer interactions in this fashion does not really improve the performance of KFAC. This suggests that it is safe to discard the off-diagonal blocks of the FIM, since the block-diagonal approach is sufficiently robust, accurate and economical in computation time.

Calculating the power of a planned individual participant data meta-analysis to examine prognostic factor effects for a binary outcome.

Collecting data for an individual participant data meta-analysis (IPDMA) project can be time consuming and resource intensive and could still have insufficient power to answer the question of interest. Therefore, researchers should consider the power of their planned IPDMA before collecting IPD. Here we propose a method to estimate the power of a planned IPDMA project aiming to synthesise multiple cohort studies to investigate the (unadjusted or adjusted) effects of potential prognostic factors for a binary outcome. We consider both binary and continuous factors and provide a three-step approach to estimating the power in advance of collecting IPD, under an assumption of the true prognostic effect of each factor of interest. The first step uses routinely available (published) aggregate data for each study to approximate Fisher's information matrix and thereby estimate the anticipated variance of the unadjusted prognostic factor effect in each study. These variances are then used in step 2 to estimate the anticipated variance of the summary prognostic effect from the IPDMA. Finally, step 3 uses this variance to estimate the corresponding IPDMA power, based on a two-sided Wald test and the assumed true effect. Extensions are provided to adjust the power calculation for the presence of additional covariates correlated with the prognostic factor of interest (by using a variance inflation factor) and to allow for between-study heterogeneity in prognostic effects. An example is provided for illustration, and Stata code is supplied to enable researchers to implement the method.

Observability-Aware Differential Dynamic Programming with Impulsive Maneuvers

In autonomous space systems, the reliability of navigation systems is essential. Observability in autonomous orbit determination techniques depends on the spacecraft’s orbital motion, making the design of autonomous navigation systems and orbital maneuvers a coupled process. This study develops a stable and efficient algorithm based on differential dynamic programming to design maneuver sequences that improve navigation performance. Our approach incorporates the Fisher information matrix into a cost function to quantify state observability and facilitates its convergence using a semi-analytic gradient and Hessian derived under impulsive maneuvers. Two numerical examples show the validity and effectiveness of our algorithm. The results indicate the stability and efficiency in determining maneuver sequences and the improvement of state estimation accuracy along an optimized trajectory. It is also applicable to other observability-aware optimal control problems because the algorithm is independent of specific systems.