Fisher Information Metric Research Articles

On statistics which are almost sufficient from the viewpoint of the Fisher metrics

Given a statistical model, a statistic on the model is sufficient if the Fisher metric of the induced model coincides with the original Fisher metric, according to the definition by Ay-Jost-Lê-Schwachhöfer. We introduce and study its quantitative version: for 0<δ≤1, we call a statistic δ-almost sufficient if δ2g(v,v)≤g′(v,v) for every tangent vector v of the parameter space, where g and g′ are the Fisher metric of the original and the induced model, respectively. By the monotonicity theorem due to Amari-Nagaoka and Ay-Jost-Lê-Schwachhöfer, the Fisher metric g′ of the induced model for such a statistic is bi-Lipschitz equivalent to the original one g, which means that the information loss of the statistic is uniformly bounded. We characterize such statistics in terms of the conditional probability or by the existence of a certain decomposition of the density function in a way similar to the characterizations of sufficient statistics due to Ay-Jost-Lê-Schwachhöfer and Fisher-Neyman.

A Comprehensive Method for Example-Based Color Transfer with Holistic-Local Balancing and Unit-Wise Riemannian Information Gradient Acceleration.

Color transfer, an essential technique in image editing, has recently received significant attention. However, achieving a balance between holistic color style transfer and local detail refinement remains a challenging task. This paper proposes an innovative color transfer method, named BHL, which stands for Balanced consideration of both Holistic transformation and Local refinement. The BHL method employs a statistical framework to address the challenge of achieving a balance between holistic color transfer and the preservation of fine details during the color transfer process. Holistic color transformation is achieved using optimal transport theory within the generalized Gaussian modeling framework. The local refinement module adjusts color and texture details on a per-pixel basis using a Gaussian Mixture Model (GMM). To address the high computational complexity inherent in complex statistical modeling, a parameter estimation method called the unit-wise Riemannian information gradient (uRIG) method is introduced. The uRIG method significantly reduces the computational burden through the second-order acceleration effect of the Fisher information metric. Comprehensive experiments demonstrate that the BHL method outperforms state-of-the-art techniques in both visual quality and objective evaluation criteria, even under stringent time constraints. Remarkably, the BHL method processes high-resolution images in an average of 4.874 s, achieving the fastest processing time compared to the baselines. The BHL method represents a significant advancement in the field of color transfer, offering a balanced approach that combines holistic transformation and local refinement while maintaining efficiency and high visual quality.

Beyond Linear Response: Equivalence between Thermodynamic Geometry and Optimal Transport.

A fundamental result of thermodynamic geometry is that the optimal, minimal-work protocol that drives a nonequilibrium system between two thermodynamic states in the slow-driving limit is given by a geodesic of the friction tensor, a Riemannian metric defined on control space. For overdamped dynamics in arbitrary dimensions, we demonstrate that thermodynamic geometry is equivalent to L^{2} optimal transport geometry defined on the space of equilibrium distributions corresponding to the control parameters. We show that obtaining optimal protocols past the slow-driving or linear response regime is computationally tractable as the sum of a friction tensor geodesic and a counterdiabatic term related to the Fisher information metric. These geodesic-counterdiabatic optimal protocols are exact for parametric harmonic potentials, reproduce the surprising nonmonotonic behavior recently discovered in linearly biased double well optimal protocols, and explain the ubiquitous discontinuous jumps observed at the beginning and end times.

Understanding Higher-Order Interactions in Information Space.

Methods used in topological data analysis naturally capture higher-order interactions in point cloud data embedded in a metric space. This methodology was recently extended to data living in an information space, by which we mean a space measured with an information theoretical distance. One such setting is a finite collection of discrete probability distributions embedded in the probability simplex measured with the relative entropy (Kullback-Leibler divergence). More generally, one can work with a Bregman divergence parameterized by a different notion of entropy. While theoretical algorithms exist for this setup, there is a paucity of implementations for exploring and comparing geometric-topological properties of various information spaces. The interest of this work is therefore twofold. First, we propose the first robust algorithms and software for geometric and topological data analysis in information space. Perhaps surprisingly, despite working with Bregman divergences, our design reuses robust libraries for the Euclidean case. Second, using the new software, we take the first steps towards understanding the geometric-topological structure of these spaces. In particular, we compare them with the more familiar spaces equipped with the Euclidean and Fisher metrics.

Dually flat structure of binary choice models

In this study, we consider parametric binary choice models from the perspective of information geometry. The set of models is a dually flat manifold with dual connections, naturally derived from the Fisher information metric. Under the dual connections, the canonical divergence and the Kullback–Leibler divergence of the binary choice model coincide if and only if the model is a logit model.

Natural gradient ascent in evolutionary games

We consider evolutionary games with a continuous trait space where the replicator dynamics are restricted to the manifold of multivariate Gaussian distributions. We demonstrate that replicator dynamics are gradient flows with respect to the Fisher information metric. The potential function for these gradient flows is closely related to the mean fitness. Our findings extend previous results on natural gradient ascent in evolutionary games with a finite strategy set. Throughout the paper we pursue an information-geometric point of view on evolutionary games. This sheds a new light on the replicator dynamics as a learning process, realizing the compromise between maximization of the mean fitness and preservation of the diversity.

Geometric relations in Classical and Quantum Information Theory using the Lambert-Tsallis Wq(x) function

This work presents relations using the Lambert-Tsallis function Wq(x) that provide geometric characteristics in classical and quantum information theory. Firstly, the Wq(x) function is used to estimate parameters in discrete and continuous distributions knowing the a priori probability value. A lower bound for the probability density derived from the branch point of Wq(x). Secondly, we will represent the direct relationship between the Fisher distance in the HF2 hyperbolic model of normal distributions as a function of the distance associated with the Kulback-Leibler divergence in the argument of the Lambert-Tsallis function. Subsequently, using the disentropy based on Rényi, a functional that uses Wq(x) as its kernel, can be used as a measure of purity in the qubit state space (Bloch sphere), as well as for calculating quantum disentanglement of pure states of two qubits. Finally, representations for quantum fidelity and quantum affinity are defined as a function of Wq(x) once the Fisher and Wigner-Yanase quantum information metrics were known, connecting the Lambert-Tsallis function to the quantum speed limit (QSL) theory.

Information theoretic waveform design with applications to adaptive‐on‐transmit radar

AbstractThe marginal Fisher information (MFI) metric is used to design waveforms for the sake of informationally optimal adaptive‐on‐transmit radar operation. A framework for MFI waveform design is developed and the Polyphase‐Coded FM (PCFM) waveform model is utilised to produce a constant‐modulus, spectrally contained signal amenable to transmission with high‐power amplifiers. The efficacy of the MFI waveform design and minimum mean square error (MMSE) estimation is experimentally demonstrated and extended into an adaptive and dynamic sensing paradigm. The radar transmit waveform is optimised to maximise the Fisher information with respect to the range profile. Upon observing new information from radar echoes, the iterative MMSE (iMMSE) estimator then minimises the estimation error variance according to prior observations. Sequential information maximisation (via waveform design) and error minimisation (via iMMSE) tends towards the Cramér–Rao lower bound (CRLB) with additional measurements improving radar resolution and accuracy. These concepts maximise the information extracted by a radar operating in a congested spectrum where the available bandwidth is limited.

Pullback Bundles and the Geometry of Learning.

Explainable Artificial Intelligence (XAI) and acceptable artificial intelligence are active topics of research in machine learning. For critical applications, being able to prove or at least to ensure with a high probability the correctness of algorithms is of utmost importance. In practice, however, few theoretical tools are known that can be used for this purpose. Using the Fisher Information Metric (FIM) on the output space yields interesting indicators in both the input and parameter spaces, but the underlying geometry is not yet fully understood. In this work, an approach based on the pullback bundle, a well-known trick for describing bundle morphisms, is introduced and applied to the encoder-decoder block. With constant rank hypothesis on the derivative of the network with respect to its inputs, a description of its behavior is obtained. Further generalization is gained through the introduction of the pullback generalized bundle that takes into account the sensitivity with respect to weights.

Differential geometry for the optimal design of the contingent valuation method

AbstractThe contingent valuation method (CVM) is a widely used experimental method to measure the monetary value of goods. However, CVM estimates are sensitive to experiment design. In this study, we formulated the optimal design problem as a minimization problem of the Fisher information metric of a gradient vector field generated by using the statistical model of the CVM. Furthermore, a necessary and sufficient condition of the optimal design was proven.

Generalized speed limits for classical stochastic systems and their applications to relaxation, annealing, and pumping processes

We extend the speed limit of a distance between two states evolving by different generators for quantum systems [K. Suzuki and K. Takahashi, Phys. Rev. Res. 2, 032016(R) (2020)] to the classical stochastic processes described by the master equation. We demonstrate that the trace distance between arbitrary evolving states is bounded from above by using a geometrical metric. The geometrical bound reduces to the Fisher information metric for the distance between the time-evolved state and the initial state. We compare the bound in relaxation and annealing processes with a different type of bound known for nonequilibrium thermodynamical systems. For dynamical processes such as annealing and pumping processes, the distance between the time-evolved state and the instantaneous stationary state becomes a proper choice and the bound is represented by the Fisher information metric of the stationary state. The metric is related to the counterdiabatic term defined from the time dependence of the stationary state.

Fisher-like Metrics Associated with ϕ-Deformed (Naudts) Entropies

The paper defines and studies new semi-Riemannian generalized Fisher metrics and Fisher-like metrics, associated with entropies and divergences. Examples of seven such families are provided, based on exponential PDFs. The particular case when the basic entropy is a ϕ-deformed one, in the sense of Naudts, is investigated in detail, with emphasis on the variation of the emergent scalar curvatures. Moreover, the paper highlights the impact on these geometries determined by the addition of some group logarithms.

Riemannian geometry of optimal driving and thermodynamic length and its application to chemical reaction networks

It is known that the trajectory of an endoreversibly driven system with minimal dissipation is a geodesic on the equilibrium state space. Thereby, the state space is equipped with the Riemannian metric given by the Hessian of the free energy function, known as Fisher information metric. However, the derivations given until now require both the system and the driving reservoir to be in local equilibrium. In the present article, we rederive the framework for chemical reaction networks (CRN) and thereby enhance its scope of applicability to the nonequilibrium situation. Moreover, because our results are derived without restrictive assumptions, we are able to discuss phenomena that could not be seen previously. We introduce a suitable weighted Fisher information metric on the space of chemical concentrations and show that it characterizes the dissipation caused by diffusive driving, with arbitrary diffusion rate constants. This allows us to consider driving far from equilibrium. As the main result, we show that the isometric embedding of a steady-state manifold into the concentration space yields a lower bound for the dissipation when the system is driven along the manifold. We give an analytic expression for this bound and for the corresponding geodesic, and thereby are able to dissect the contributions from the driving kinetics and from thermodynamics. Finally, we discuss in detail the application to quasithermostatic steady states.

The information geometry of two-field functional integrals

Two-field functional integrals (2FFI) are an important class of solution methods for generating functions of dissipative processes, including discrete-state stochastic processes, dissipative dynamical systems, and decohering quantum densities. The stationary trajectories of these integrals describe a conserved current by Liouville’s theorem, despite the absence of a conserved kinematic phase space current in the underlying stochastic process. We develop the information geometry of generating functions for discrete-state classical stochastic processes in the Doi-Peliti 2FFI form, and exhibit two quantities conserved along stationary trajectories. One is a Wigner function, familiar as a semiclassical density from quantum-mechanical time-dependent density-matrix methods. The second is an overlap function, between directions of variation in an underlying distribution and those in the directions of relative large-deviation probability that can be used to interrogate the distribution, and expressed as an inner product of vector fields in the Fisher information metric. To give an interpretation to the time invertibility implied by current conservation, we use generating functions to represent importance sampling protocols, and show that the conserved Fisher information is the differential of a sample volume under deformations of the nominal distribution and the likelihood ratio. We derive a pair of dual affine connections particular to Doi-Peliti theory for the way they separate the roles of the nominal distribution and likelihood ratio, distinguishing them from the standard dually-flat connection of Nagaoka and Amari defined on the importance distribution, and show that dual flatness in the affine coordinates of the coherent-state basis captures the special role played by coherent states in Doi-Peliti theory.

Weighted Relative Group Entropies and Associated Fisher Metrics.

A large family of new -weighted group entropy functionals is defined and associated Fisher-like metrics are considered. All these notions are well-suited semi-Riemannian tools for the geometrization of entropy-related statistical models, where they may act as sensitive controlling invariants. The main result of the paper establishes a link between such a metric and a canonical one. A sufficient condition is found, in order that the two metrics be conformal (or homothetic). In particular, we recover a recent result, established for and for non-weighted relative group entropies. Our conformality condition is “universal”, in the sense that it does not depend on the group exponential.

Probabilistic Autoencoder Using Fisher Information.

Neural networks play a growing role in many scientific disciplines, including physics. Variational autoencoders (VAEs) are neural networks that are able to represent the essential information of a high dimensional data set in a low dimensional latent space, which have a probabilistic interpretation. In particular, the so-called encoder network, the first part of the VAE, which maps its input onto a position in latent space, additionally provides uncertainty information in terms of variance around this position. In this work, an extension to the autoencoder architecture is introduced, the FisherNet. In this architecture, the latent space uncertainty is not generated using an additional information channel in the encoder but derived from the decoder by means of the Fisher information metric. This architecture has advantages from a theoretical point of view as it provides a direct uncertainty quantification derived from the model and also accounts for uncertainty cross-correlations. We can show experimentally that the FisherNet produces more accurate data reconstructions than a comparable VAE and its learning performance also apparently scales better with the number of latent space dimensions.

Space from entanglement: An information-geometric perspective

In this paper, we study the construction of classical geometry from the quantum entanglement structure by using information geometry. In the information geometry of classical spacetime, the Fisher information metric is related to a blurred metric of a classical physical space. We first show that a local information metric can be obtained from the entanglement contour in a local subregion. This local information metric measures the fine structure of entanglement spectra inside the subregion, which suggests a quantum origin of the information-geometric blurred space. We study both the continuous and the classical limits of the quantum-originated blurred space by using the techniques from the statistical sampling algorithms, the sampling theory of spacetime and the projective limit. A scheme for going from a blurred space with quantum features to a classical geometry is also explored.

Bridging the information and dynamics attributes of neural activities

The brain works as a dynamic system to process information. Various challenges remain in understanding the connection between information and dynamics attributes in the brain. The present research pursues exploring how the characteristics of neural information functions are linked to neural dynamics. We attempt to bridge dynamics (e.g., Kolmogorov-Sinai entropy) and information (e.g., mutual information and Fisher information) metrics on the stimulus-triggered stochastic dynamics in neural populations. On the one hand, our unified analysis identifies various essential features of the information-processing-related neural dynamics. We discover spatiotemporal differences in the dynamic randomness and chaotic degrees of neural dynamics during neural information processing. On the other hand, our framework reveals the fundamental role of neural dynamics in shaping neural information processing. The neural dynamics creates an oppositely directed variation of encoding and decoding properties under specific conditions, and it determines the neural representation of stimulus distribution. Overall, our findings demonstrate a potential direction to explain the emergence of neural information processing from neural dynamics and help understand the intrinsic connections between the informational and the physical brain.

On the Entanglement Entropy in Gaussian cMERA

AbstractThe continuous Multi Scale Entanglement Renormalization Anstaz (cMERA) consists of a variational method which carries out a real space renormalization scheme on the wavefunctionals of quantum field theories. In this work we calculate the entanglement entropy of the half space for a free scalar theory through a Gaussian cMERA circuit. We obtain the correct entropy written in terms of the optimized cMERA variational parameter, the local density of disentanglers. Accordingly, using the entanglement entropy production per unit scale, we study local areas in the bulk of the tensor network in terms of the differential entanglement generated along the cMERA flow. This result spurs us to establish an explicit relation between the cMERA variational parameter and the radial component of a dual AdS geometry through the Ryu‐Takayanagi formula. Finally, we argue that the entanglement entropy for the half space can be written as an integral along the renormalization scale whose measure is given by the Fisher information metric of the cMERA circuit. Consequently, a straightforward relation between AdS geometry and the Fisher information metric is also established.

Pathological Spectra of the Fisher Information Metric and Its Variants in Deep Neural Networks.

The Fisher information matrix (FIM) plays an essential role in statistics and machine learning as a Riemannian metric tensor or a component of the Hessian matrix of loss functions. Focusing on the FIM and its variants in deep neural networks (DNNs), we reveal their characteristic scale dependence on the network width, depth, and sample size when the network has random weights and is sufficiently wide. This study covers two widely used FIMs for regression with linear output and for classification with softmax output. Both FIMs asymptotically show pathological eigenvalue spectra in the sense that a small number of eigenvalues become large outliers depending on the width or sample size, while the others are much smaller. It implies that the local shape of the parameter space or loss landscape is very sharp in a few specific directions while almost flat in the other directions. In particular, the softmax output disperses the outliers and makes a tail of the eigenvalue density spread from the bulk. We also show that pathological spectra appear in other variants of FIMs: one is the neural tangent kernel; another is a metric for the input signal and feature space that arises from feedforward signal propagation. Thus, we provide a unified perspective on the FIM and its variants that will lead to more quantitative understanding of learning in large-scale DNNs.