Fisher Information Matrix Research Articles

CL-BPUWM: continuous learning with Bayesian parameter updating and weight memory

Catastrophic forgetting in neural networks is a common problem, in which neural networks lose information from previous tasks after training on new tasks. Although adopting a regularization method that preferentially retains the parameters important to the previous task to avoid catastrophic forgetting has a positive effect; existing regularization methods cause the gradient to be near zero because the loss is at the local minimum. To solve this problem, we propose a new continuous learning method with Bayesian parameter updating and weight memory (CL-BPUWM). First, a parameter updating method based on the Bayes criterion is proposed to allow the neural network to gradually obtain new knowledge. The diagonal of the Fisher information matrix is then introduced to significantly minimize computation and increase parameter updating efficiency. Second, we suggest calculating the importance weight by observing how changes in each network parameter affect the model prediction output. In the process of model parameter updating, the Fisher information matrix and the sensitivity of the network are used as the quadratic penalty terms of the loss function. Finally, we apply dropout regularization to reduce model overfitting during training and to improve model generalizability. CL-BPUWM performs very well in continuous learning for classification tasks on CIFAR-100 dataset, CIFAR-10 dataset, and MNIST dataset. On CIFAR-100 dataset, it is 0.8%, 1.03% and 0.75% higher than the best performing regularization method (EWC) in three task partitions. On CIFAR-10 dataset, it is 2.25% higher than the regularization method (EWC) and 0.7% higher than the scaled method (GR). It is 0.66% higher than the regularization method (EWC) on the MNIST dataset. When the CL-BPUWM method was combined with the brain-inspired replay model under the CIFAR-100 and CIFAR-10 datasets, the classification accuracy was 2.35% and 5.38% higher than that of the baseline method, BI-R + SI.

Variational quantum algorithm for experimental photonic multiparameter estimation

Variational quantum metrology represents a powerful tool to optimize estimation strategies, resulting particularly beneficial for multiparameter estimation problems that often suffer from limitations due to the curse of dimensionality and computational complexity. To overcome these challenges, we develop a variational approach able to efficiently optimize a quantum multiphase sensor. Leveraging the reconfigurability of an integrated photonic device, we implement a hybrid quantum-classical feedback loop able to enhance the estimation performances. The quantum circuit evaluations are used to compute the system partial derivatives by applying the parameter-shift rule, and thus reconstruct experimentally the Fisher information matrix. This in turn is adopted as the cost function of a classical learning algorithm run to optimize the measurement settings. Our experimental results showcase significant improvements in estimation accuracy and noise robustness, highlighting the potential of variational techniques for practical applications in quantum sensing and more generally in quantum information processing using photonic circuits.

RIS-assisted near-field localization using practical phase shift model

Our research focuses on examining the problem of localizing user equipment (UE) in the uplink scenario using reconfigurable intelligent surfaces (RIS) based lens. We carry out a thorough analysis of the Fisher information matrix (FIM) and assess the influence of various RIS-based lens configurations using an actual RIS phase-dependent amplitude variations model. Furthermore, to reduce the complexity of the maximum likelihood (ML) estimator, a simple localization algorithm-based angular expansion is presented. Simulation results show superior localization performance when prior location information is available for directional and positional channel configurations. The position error bound (PEB) and the root mean square error (RMSE) are studied to evaluate the localization accuracy of the user utilizing the realistic RIS phase-dependent amplitude model in the near-field region. Furthermore, the achievable data rate is obtained in the same region using the realistic RIS phase-dependent amplitude model. It is noticed that adopting the actual RIS phase-dependent amplitude model under the near-field channel increases the localization error and degrades the data rate performance for amplitude value less than one so, the unity assumption of the RIS phase shift model used widely in the literature is inaccurate.

Continual online learning-based optimal tracking control of nonlinear strict-feedback systems: application to unmanned aerial vehicles

A novel optimal trajectory tracking scheme is introduced for nonlinear continuous-time systems in strict feedback form with uncertain dynamics by using neural networks (NNs). The method employs an actor-critic-based NN backstepping technique for minimizing a discounted value function along with an identifier to approximate unknown system dynamics that are expressed in augmented form. Novel online weight update laws for the actor and critic NNs are derived by using both the NN identifier and Hamilton-Jacobi-Bellman residual error. A new continual lifelong learning technique utilizing the Fisher Information Matrix via Hamilton-Jacobi-Bellman residual error is introduced to obtain the significance of weights in an online mode to overcome the issue of catastrophic forgetting for NNs, and closed-loop stability is analyzed and demonstrated. The effectiveness of the proposed method is shown in simulation by contrasting the proposed with a recent method from the literature on an underactuated unmanned aerial vehicle, covering both its translational and attitude dynamics.

Dynamic Weighted Cumulative Residual Entropy Estimators for Laplace Distribution: Bayesian Approach

Objectives: To develop Bayesian estimators of dynamic weighted cumulative residual entropy (DWCRE) for Laplace distribution and to investigate posterior risks using various priors and loss functions. Methods: Weighted entropy measure of information is provided by a probabilistic experiment whose basic events are described by their objective probabilities and some qualitative (objective or subjective) weights. In this paper, we have used priors (Jeffrey’s, Hartigan, Uniform and Gumble Type II) and several loss functions. Findings: Bayesian estimators and associated posterior risks for Laplace distribution have been derived for different priors and loss functions. Monte Carlo Simulation study and graphical analyses have also been presented along with the conclusion. Through the comprehensive simulation study in the paper, it has been observed that Hartigan prior is better than other priors in terms of the posterior risk whereas Uniform prior has always higher posterior risk. Novelty: The introduction of new Bayesian estimators and their posterior risks for dynamic weighted cumulative residual entropy (DWCRE) of Laplace distribution. Keywords: Bayesian estimators, Laplace distribution, Fisher information matrix, Loss functions, Priors

Quantum Fisher information matrix of quantum metrology in a Heisenberg XXZ model

Quantum Fisher information matrix of quantum metrology in a Heisenberg XXZ model

Deep learning forecasts of cosmic acceleration parameters from DECi-hertz Interferometer Gravitational-wave Observatory

Context. Validating the accelerating expansion of the universe is an important aspect in improving our understanding of the evolution of the universe. By constraining the cosmic acceleration parameter XH, we can discriminate between the cosmological constant plus cold dark matter (ΛCDM) model and the Lemaître–Tolman–Bondi (LTB) model. Aims. In this paper, we explore the possibility of constraining the cosmic acceleration parameter with the inspiral gravitational waveform of neutron star binaries (NSBs) in the frequency range of 0.1 Hz–10 Hz, which can be detected by the second-generation space-based gravitational wave detector DECIGO. Methods. We used a convolutional neural network (CNN) and a long short-term memory (LSTM) network combined with a gated recurrent unit (GRU), along with a Fisher information matrix to derive constraints on the cosmic acceleration parameter, XH. Results. We assumed that our networks estimate the cosmic acceleration parameter without biases (the expected value of the estimation is equal to the true value). Under this assumption, based on the simulated gravitational wave data with a time duration of one month, we conclude that CNN can limit the relative error to 15.71%, while LSTM network combined with GRU can limit the relative error to 14.14%. Additionally, using a Fisher information matrix for gravitational wave data with a five-year observation can limit the relative error to 32.94%. Conclusions. Under the assumption of an unbiased estimation, the neural networks can offer a high-precision estimation of the cosmic acceleration parameter at different redshifts. Therefore, DECIGO is expected to provide direct measurements of the acceleration of the universe by observing the chirp signals of coalescing binary neutron stars.

The Best Fit Bayesian Hierarchical Generalized Linear Model Selection Using Information Complexity Criteria in the MCMC Approach

Both frequentist and Bayesian statistics schools have improved statistical tools and model choices for the collected data or measurements. Model selection approaches have advanced due to the difficulty of comparing complicated hierarchical models in which linear predictors vary by grouping variables, and the number of model parameters is not distinct. Many regression model selection criteria are considered, including the maximum likelihood (ML) point estimation of the parameter and the logarithm of the likelihood of the dataset. This paper demonstrates the information complexity (ICOMP), Bayesian deviance information, or the widely applicable information criterion (WAIC) of the BRMS to hierarchical linear models fitted with repeated measures with a simulation and two real data examples. The Fisher information matrix for the Bayesian hierarchical model considering fixed and random parameters under maximizing a posterior estimation is derived. Using Gibbs sampling and Hybrid Hamiltonian Monte Carlo approaches, six different models were fitted for three distinct application datasets. The best-fitted candidate models were identified under each application dataset with the two MCMC approaches. In this case, the Bayesian hierarchical (mixed effect) linear model with random intercepts and random slopes estimated using the Hamiltonian Monte Carlo method best fits the two application datasets. Information complexity (ICOMP) is a better indicator of the best-fitted models than DIC and WAIC. In addition, the information complexity criterion showed that hierarchical models with gradient-based Hamiltonian Monte Carlo estimation are the best fit and have supper convergence relative to the gradient-free Gibbs sampling methods.

Data Augmentation for Regression Machine Learning Problems in High Dimensions

Machine learning approaches are currently used to understand or model complex physical systems. In general, a substantial number of samples must be collected to create a model with reliable results. However, collecting numerous data is often relatively time-consuming or expensive. Moreover, the problems of industrial interest tend to be more and more complex, and depend on a high number of parameters. High-dimensional problems intrinsically involve the need for large amounts of data through the curse of dimensionality. That is why new approaches based on smart sampling techniques have been investigated to minimize the number of samples to be given to train the model, such as active learning methods. Here, we propose a technique based on a combination of the Fisher information matrix and sparse proper generalized decomposition that enables the definition of a new active learning informativeness criterion in high dimensions. We provide examples proving the performances of this technique on a theoretical 5D polynomial function and on an industrial crash simulation application. The results prove that the proposed strategy outperforms the usual ones.

Joint energy beamforming and bandwidth allocation for cooperative localization in passive sensor networks

Passive sensor networks (PSNs) harvest continuous radio-frequency (RF) energy to work. However, the location information is difficult to maintain with unstable power supply for each passive sensor node. Cooperative localization is a promising solution utilizes the cooperation between the nodes with unknown positions to maintain the location information. And the localization performance relies on the attained energy from the energy access point (E-AP) and also the allocated bandwidth for each node. In this paper, we analyze the impacts of energy beamforming and bandwidth allocation on the cooperative localization performance of PSNs. We formulate the Fisher information matrix (FIM) and deduce the squared position error bound (SPEB) for the proposed network. Then, we propose an alternative optimization method for jointly energy beamforming and bandwidth allocation. Firstly, we employ the first-order Taylor expansions based on the trust region method to allocate the bandwidth of each node. Secondly, the Lagrangian multiplier based energy beamforming strategy is proposed. Then, the bandwidth allocation and energy beamforming calculations are executed alternatively and iteratively until the global optimal solution is achieved. The simulation results indicate that our proposed two-step method can achieve high precision positioning in different scenarios, and the SPEBs of proposed method reach 10−3 m2.

Observed Fisher information for radar clutter modeled with sub-Gaussian α-stable distribution

It is well known that the clutter in such applications as synthetic aperture radar (SAR) follows heavy-tailed distributions. In such applications, knowledge of the exact statistical properties and/or errors in estimation of the clutter parameters plays an important role in successful implementation of the related applications like target tracking using SAR. In this study, the observed Fisher information (OFI) matrix will be derived for the received clutter assuming that the statistical behavior of clutter follows a sub-Gaussian α-stable (SGS) distribution along with a density function for an SGS distribution based on a newly proposed expansion. The new expansion is characterized with high speed computation in one hand, and avoids issues related to numerical computation of SGS density function, on the other. The minimum standard error of the clutter parameters, then, will be derived as the OFI matrix is inversely related to the Cramer-Rao lower bound (CRLB) of the clutter's estimated parameters. In order to compute the density function and OFI of the SGS distribution, the performance of the proposed methodology has been tested through numerical simulation where the implemented computer code of the proposed numerical method has been demonstrated to be fast and accurate. The R code is available at https://github.com/mahditeimouri/Sub-Gaussian/blob/main/OFI-sub-Gaussian.r.

Log‐density gradient covariance and automatic metric tensors for Riemann manifold Monte Carlo methods

AbstractA metric tensor for Riemann manifold Monte Carlo particularly suited for nonlinear Bayesian hierarchical models is proposed. The metric tensor is built from symmetric positive semidefinite log‐density gradient covariance (LGC) matrices, which are also proposed and further explored here. The LGCs generalize the Fisher information matrix by measuring the joint information content and dependence structure of both a random variable and the parameters of said variable. Consequently, positive definite Fisher/LGC‐based metric tensors may be constructed not only from the observation likelihoods as is current practice, but also from arbitrarily complicated nonlinear prior/latent variable structures, provided the LGC may be derived for each conditional distribution used to construct said structures. The proposed methodology is highly automatic and allows for exploitation of any sparsity associated with the model in question. When implemented in conjunction with a Riemann manifold variant of the recently proposed numerical generalized randomized Hamiltonian Monte Carlo processes, the proposed methodology is highly competitive, in particular for the more challenging target distributions associated with Bayesian hierarchical models.

Deep learning for parameter estimation of supermassive binary black holes with simulated LISA data

Previous studies have demonstrated that deep learning is an effective approach for processing gravitational wave (GW) data obtained from ground-based detectors, as it can enhance the efficiency of data processing and offer great potential for real-time parameter estimation. In this paper, we explore three different deep learning architectures (MCD, TFP, and VI) for inferring the chirp mass (Mc) and the luminosity distance (DL) of supermassive binary black holes (SMBBHs) using simulated data from the Laser Interferometer Space Antenna (LISA). We train the neural networks with the simulated data and evaluate their performance on predicting the parameters (Mc,DL). The results show that more than 97.5% of the true values fall within the 3σ confidence interval of the predicted values with the optimal network. To verify the accuracy of the network on parameter estimation, we also calculate the estimation error of the parameters using the Fisher Information Matrix (FIM) with the same simulated data. By comparing the root-mean-square error (RMSE) between deep learning and FIM, we find that the two methods are comparable, which implies that deep learning can be reliable for GW parameter estimation.

Closed-Loop Estimation Method of Neurostimulation Strength-Duration Curve Using Fisher Information Optimization.

Strength-duration (SD) curve, rheobase and chronaxie parameters provide insights about neural activation dynamics and interdependence between pulse amplitude and duration, for diagnostics and therapeutic applications. The existing SD curve estimation methods are based on open-loop uniform and/or random pulse durations, which are chosen without feedback from neuronal data. To develop a method for closed-loop estimation of the SD curve, where the pulse durations are adjusted iteratively using the neuronal data. In the proposed method, after the selection of each pulse duration through Fisher information matrix (FIM) optimization, the corresponding motor threshold (MT) is computed, the SD curve estimation is updated, and the process continues until satisfaction of a stopping rule based on the successive convergence of the SD curve parameters. The results are compared with various iterative uniform and random sampling techniques, where the SD curve estimation is updated after each sample. 250 simulation cases were run. The FIM method satisfied the stopping rule in 225 (90%) runs, and estimated the rheobase (chronaxie in parenthesis) with an average absolute relative error (ARE) of 1.57% (2.15%), with an average of 85 samples. At the FIM termination sample, methods with two and all random pulse durations, and uniform methods with descending, ascending and random orders led to 5.69% (20.09%), 2.22% (3.93%), 7.34% (40.90%), 3.10% (4.44%), and 2.05% (3.45%) AREs. In all 250 runs, the FIM method has chosen the minimum and maximum pulse durations as the optimal pulse durations for the SD curve estimation. As proposed by the FIM method, the SD curve is identifiable by fitting to the data of the minimum and maximum pulse durations. However, the range of pulse duration should cover the vertical and horizontal parts of the SD curve. Also, iterative random or uniform samples from only the vertical or horizontal areas of the curve might not result in satisfactory estimation. This paper provides insights about pulse durations selection for closed-loop and open-loop SD curve estimation.

Оптимальное планирование эксперимента на основе винеровской деградационной модели с ковариатами с использованием условной информационной матрицы Фишера

The Wiener degradation models with covariates are widely used for reliability analysis by degradation index paths. In this paper, we suggest an algorithm for constructing the optimal design based on the Wiener degradation model, which involves determining optimal stress levels, the number of objects under each stress level and the time moments for measuring degradation index, taking into account the limitation on the duration of the experiment and the critical value of the degradation index. The proposed algorithm is based on optimizing the sum of functionals from the Fisher information matrix (FIM) and the conditional Fisher information matrix (CFIM) under given restrictions on the stress levels, the duration of the experiment and the minimal interval between the time points of measuring the degradation index. As an example, the application of the developed algorithm is considered for the LED reliability analysis.

Sparsified federated learning with differential privacy for intrusion detection in VANETs based on Fisher Information Matrix.

With the continuous development of vehicular ad hoc networks (VANET) security, using federated learning (FL) to deploy intrusion detection models in VANET has attracted considerable attention. Compared to conventional centralized learning, FL retains local training private data, thus protecting privacy. However, sensitive information about the training data can still be inferred from the shared model parameters in FL. Differential privacy (DP) is sophisticated technique to mitigate such attacks. A key challenge of implementing DP in FL is that non-selectively adding DP noise can adversely affect model accuracy, while having many perturbed parameters also increases privacy budget consumption and communication costs for detection models. To address this challenge, we propose FFIDS, a FL algorithm integrating model parameter pruning with differential privacy. It employs a parameter pruning technique based on the Fisher Information Matrix to reduce the privacy budget consumption per iteration while ensuring no accuracy loss. Specifically, FFIDS evaluates parameter importance and prunes unimportant parameters to generate compact sub-models, while recording the positions of parameters in each sub-model. This not only reduces model size to lower communication costs, but also maintains accuracy stability. DP noise is then added to the sub-models. By not perturbing unimportant parameters, more budget can be reserved to retain important parameters for more iterations. Finally, the server can promptly recover the sub-models using the parameter position information and complete aggregation. Extensive experiments on two public datasets and two F2MD simulation datasets have validated the utility and superior performance of the FFIDS algorithm.

Reliability analysis for two populations Nadarajah-Haghighi distribution under Joint progressive type-II censoring

<abstract><p>In order to evaluate the competitive advantages and dependability of two products in a competitive environment, comparative lifespan testing becomes essential. We examine the inference problems that occur when two product lines follow the Nadarajah-Haghighighi distribution in the setting of joint type-II censoring. In the present study, we derived the maximum likelihood estimates for the Nadarajah-Haghighi population parameters. Additionally, a Fisher information matrix was constructed based on these maximum likelihood estimations. Furthermore, Bayesian estimators and their corresponding posterior risks were calculated, considering both gamma and non-informative priors under symmetric and asymmetric loss functions. To assess the performance of the overall parameter estimators, we conducted a Monte Carlo simulation using numerical methods. Lastly, a real data analysis was carried out to validate the accuracy of the models and methods discussed.</p></abstract>

Inference and other aspects for $ q- $Weibull distribution via generalized order statistics with applications to medical datasets

<abstract><p>This work utilizes generalized order statistics (GOSs) to study the $ q $-Weibull distribution from several statistical perspectives. First, we explain how to obtain the maximum likelihood estimates (MLEs) and utilize Bayesian techniques to estimate the parameters of the model. The Fisher information matrix (FIM) required for asymptotic confidence intervals (CIs) is generated by obtaining explicit expressions. A Monte Carlo simulation study is conducted to compare the performances of these estimates based on type Ⅱ censored samples. Two well-established measures of information are presented, namely extropy and weighted extropy. In this context, the order statistics (OSs) and sequential OSs (SOSs) for these two measures are studied based on this distribution. A bivariate $ q $-Weibull distribution based on the Farlie-Gumbel-Morgenstern (FGM) family and its relevant concomitants are studied. Finally, two concrete instances of medical real data are ultimately provided.</p></abstract>

Joint Optimization Method of Trajectory of UAV Swarm and MIMO Radar Radiation Power based on Posterior Fisher Information Matrix

Joint Optimization Method of Trajectory of UAV Swarm and MIMO Radar Radiation Power based on Posterior Fisher Information Matrix

An Inflated Modelling of Zero Truncated Poisson Ailamujia Distribution and Its Application to Child Mortality and Genetics Count Data

In this paper, an attempt has been made to develop a model as Inflated form of zero truncated Poisson Ailamujia distribution. The proposed model involved two parameters, estimated by the maximum likelihood estimation method and derived as an expression of the Fisher Information Matrix. To evaluate the adequacy and applicability of the proposed model, it is applied to real data sets on child mortality and genetic count data.