Generalized Information Measures Research Articles

Characterization of Two Parametric New Generalized Information Measure and Its Corresponding Code-word Length

Characterization of Two Parametric New Generalized Information Measure and Its Corresponding Code-word Length

On some dynamic generalized information measures for bivariate lifetimes

ABSTRACTSeveral generalizations to the concept of Kullback-Leibler divergence measure and Kerridge inaccuracy measure are available in the literature. In a recent paper Kundu (Metrika, 78:415–435, 2015) considered a generalized K-L divergence measure of order (α, β). Nath (Metrika, 13:123–135, 1968) has also proposed generalized inaccuracy measure of order α. Here we address the question of extending these measures to higher dimensions with reference to residual lifetimes. In the present work, the generalized divergence and inaccuracy measures are extended for conditional lifetimes of two components having possibly different ages. Several properties, including monotonicity, and bounds of these measures are obtained for conditional random variables. Moreover, we study the effect of (increasing) monotone transformation on these generalized measures.

Some Inequalities in Information Theory Using Tsallis Entropy

We presenta relation betweenTsallis’s entropy and generalizedKerridge inaccuracywhich is called generalizedShannon inequalityand is well-known generalization ininformation theoryand then give its application in coding theory. The objective of the paper is to establish a result on noiseless coding theorem for the proposed mean code length in terms of generalized information measure of orderξ.

On Dynamic Generalized Measures of Inaccuracy

Generalized information measures play an important role in the measurement of uncertainty of certain random variables, where the standard practice of applying ordinary uncertainty measures fails to fit. Based on the Renyi entropy and its divergence, we propose a generalized measure of inaccuracy of order α(≠1)>0 between two residual and past lifetime distributions of a system. We study some important properties and characterizations of these measures. A numerical example is given to illustrate the usefulness of the proposed measure.

The Generalization of Maximum Entropy Principle for Generalized Information Measures

The Generalization of Maximum Entropy Principle for Generalized Information Measures

Groups, information theory, and Einstein's likelihood principle.

We propose a unifying picture where the notion of generalized entropy is related to information theory by means of a group-theoretical approach. The group structure comes from the requirement that an entropy be well defined with respect to the composition of independent systems, in the context of a recently proposed generalization of the Shannon-Khinchin axioms. We associate to each member of a large class of entropies a generalized information measure, satisfying the additivity property on a set of independent systems as a consequence of the underlying group law. At the same time, we also show that Einstein's likelihood function naturally emerges as a byproduct of our informational interpretation of (generally nonadditive) entropies. These results confirm the adequacy of composable entropies both in physical and social science contexts.

An Overview of the Pathway Idea and Its Applications in Statistical and Physical Sciences

Pathway idea is a switching mechanism by which one can go from one functional form to another, and to yet another. It is shown that through a parameter α, called the pathway parameter, one can connect generalized type-1 beta family of densities, generalized type-2 beta family of densities, and generalized gamma family of densities, in the scalar as well as the matrix cases, also in the real and complex domains. It is shown that when the model is applied to physical situations then the current hot topics of Tsallis statistics and superstatistics in statistical mechanics become special cases of the pathway model, and the model is capable of capturing many stable situations as well as the unstable or chaotic neighborhoods of the stable situations and transitional stages. The pathway model is shown to be connected to generalized information measures or entropies, power law, likelihood ratio criterion or λ - criterion in multivariate statistical analysis, generalized Dirichlet densities, fractional calculus, Mittag-Leffler stochastic process, Krätzel integral in applied analysis, and many other topics in different disciplines. The pathway model enables one to extend the current results on quadratic and bilinear forms, when the samples come from Gaussian populations, to wider classes of populations.

Combining Expected Utility and Weighted Gini-Simpson Index into a Non-Expected Utility Device

We present and discuss a conceptual decision-making procedure supported by a mathematical device combining expected utility and a generalized information measure: the weighted Gini-Simpson index, linked to the scientific fields of information theory and ecological diversity analysis. After a synthetic review of the theoretical background relative to those themes, such a device—an EU-WGS framework denoting a real function defined with positive utility values and domain in the simplex of probabilities—is analytically studied, identifying its range with focus on the maximum point, using a Lagrange multiplier method associated with algorithms, exemplified numerically. Yet, this EU-WGS device is showed to be a proper analog of an expected utility and weighted entropy (EU-WE) framework recently published, both being cases of mathematical tools that can be referred to as non-expected utility methods using decision weights, framed within the field of decision theory linked to information theory. This kind of decision modeling procedure can also be interpreted to be anchored in Kurt Lewin utility’s concept and may be used to generate scenarios of optimal compositional mixtures applied to generic lotteries associated with prospect theory, financial risk assessment, security quantification and natural resources management. The epistemological method followed in the reasoned choice procedure that is presented in this paper is neither normative nor descriptive in an empirical sense, but instead it is heuristic and hermeneutical in its conception.

On bounds of a generalized ‘useful’ mean codeword length

In the present paper a new generalized ‘useful’ mean codeword length is defined and characterized a new generalized information measure by obtaining bounds ofa new generalized ‘useful’ mean codeword length in terms of a new generalized information measure using Lagrange’s Multiplier method. The Shannon’s Noiseless coding theorem is verified by considering Huffman and Shannon -Fano coding schemes on taking empirical data.. Mathematics Subject Classification: 94A15, 94A17

On Power-Law Kernels, Corresponding Reproducing Kernel Hilbert Space and Applications

The role of kernels is central to machine learning. Motivated by the importance of power-law distributions in statistical modeling, in this paper, we propose the notion of power-law kernels to investigate power-laws in learning problem. We propose two power-law kernels by generalizing Gaussian and Laplacian kernels. This generalization is based on distributions, arising out of maximization of a generalized information measure known as nonextensive entropy that is very well studied in statistical mechanics. We prove that the proposed kernels are positive definite, and provide some insights regarding the corresponding Reproducing Kernel Hilbert Space (RKHS). We also study practical significance of both kernels in classification and regression, and present some simulation results.

Unifying complexity and information.

Complex systems, arising in many contexts in the computer, life, social, and physical sciences, have not shared a generally-accepted complexity measure playing a fundamental role as the Shannon entropy H in statistical mechanics. Superficially-conflicting criteria of complexity measurement, i.e. complexity-randomness (C-R) relations, have given rise to a special measure intrinsically adaptable to more than one criterion. However, deep causes of the conflict and the adaptability are not much clear. Here I trace the root of each representative or adaptable measure to its particular universal data-generating or -regenerating model (UDGM or UDRM). A representative measure for deterministic dynamical systems is found as a counterpart of the H for random process, clearly redefining the boundary of different criteria. And a specific UDRM achieving the intrinsic adaptability enables a general information measure that ultimately solves all major disputes. This work encourages a single framework coving deterministic systems, statistical mechanics and real-world living organisms.

Gaze and informativeness during category learning: Evidence for an inverse relation

In what follows, we explore the general relationship between eye gaze during a category learning task and the information conveyed by each member of the learned category. To understand the nature of this relationship empirically, we used eye tracking during a novel object classification paradigm. Results suggest that the average fixation time per object during learning is inversely proportional to the amount of information that object conveys about its category. This inverse relationship may seem counterintuitive; however, objects that have a high-information value are inherently more representative of their category. Therefore, their generality captures the essence of the category structure relative to less representative objects. As such, it takes relatively less time to process these objects than their less informative companions. We use a general information measure referred to as representational information theory (Vigo, 2011a, 2013a) to articulate and interpret the results from our experiment and compare its predictions to those of three models of prototypicality.

Data-Processing Inequalities Based on a Certain Structured Class of Information Measures With Application to Estimation Theory

We study data-processing inequalities that are derived from a certain class of generalized information measures, where a series of convex functions and multiplicative likelihood ratios is nested alternately. While these information measures can be viewed as a special case of the most general Zakai-Ziv generalized information measure, this special nested structure calls for attention and motivates our study. Specifically, a certain choice of the convex functions leads to an information measure that extends the notion of the Bhattacharyya distance (or the Chernoff divergence): While the ordinary Bhattacharyya distance is based on the (weighted) geometric mean of two replicas of the channel's conditional distribution, the more general information measure allows an arbitrary number of such replicas. We apply the data-processing inequality induced by this information measure to a detailed study of lower bounds of parameter estimation under additive white Gaussian noise (AWGN) and show that in certain cases, tighter bounds can be obtained by using more than two replicas. While the resulting lower bound may not compete favorably with the best bounds available for the ordinary AWGN channel, the advantage of the new lower bound, relative to the other bounds, becomes significant in the presence of channel uncertainty, like unknown fading. This different behavior in the presence of channel uncertainty is explained by the convexity property of the information measure.

Data Processing Theorems and the Second Law of Thermodynamics

We draw relationships between the generalized data processing theorems of Zakai and Ziv (1973 and 1975) and the dynamical version of the second law of thermodynamics, a.k.a. the Boltzmann H-Theorem, which asserts that the Shannon entropy, $H(X_t)$, pertaining to a finite--state Markov process $\{X_t\}$, is monotonically non-decreasing as a function of time $t$, provided that the steady-state distribution of this process is uniform across the state space (which is the case when the process designates an isolated system). It turns out that both the generalized data processing theorems and the Boltzmann H-Theorem can be viewed as special cases of a more general principle concerning the monotonicity (in time) of a certain generalized information measure applied to a Markov process. This gives rise to a new look at the generalized data processing theorem, which suggests to exploit certain degrees of freedom that may lead to better bounds, for a given choice of the convex function that defines the generalized mutual information.

New Information Measures for the Generalized Normal Distribution

We introduce a three-parameter generalized normal distribution, which belongs to the Kotz type distribution family, to study the generalized entropy type measures of information. For this generalized normal, the Kullback-Leibler information is evaluated, which extends the well known result for the normal distribution, and plays an important role for the introduced generalized information measure. These generalized entropy type measures of information are also evaluated and presented.

Logarithmic Sobolev Inequalities for Information Measures

For <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ges</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</i> , the new Vajda-type information measure <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">J</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(X</i> ) is a quantity generalizing Fisher's information (FI), to which it is reduced for <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">=</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</i> . In this paper, a corresponding generalized entropy power <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(X</i> ) is introduced, and the inequality <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(X</i> ) <b xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">J</b> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> ) ges <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> is proved, which is reduced to the well-known inequality of Stam for <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">=</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</i> . The cases of equality are also determined. Furthermore, the Blachman-Stam inequality for the FI of convolutions is generalized for the Vajda information <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">J</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(X</i> ) and both families of results in the context of measure of information are discussed. That is, logarithmic Sobolev inequalities (LSIs) are written in terms of new more general entropy-type information measure, and therefore, new information inequalities are arisen. This generalization for special cases yields to the well known information measures and relative bounds.

Axiomatic Characterizations of Information Measures

Axiomatic characterizations of Shannon entropy, Kullback I-divergence, and some generalized information measures are surveyed. Three directions are treated: (A) Characterization of functions of probability distributions suitable as information measures. (B) Characterization of set functions on the subsets of {1; : : : ;N} representable by joint entropies of components of an N-dimensional random vector. (C) Axiomatic characterization of MaxEnt and related inference rules. The paper concludes with a brief discussion of the relevance of the axiomatic approach for information theory.

Some Dynamic Generalized Information Measures In The Context Of Weighted Models

In this paper, we study some dynamic generalized information measures between a true distribution and an observed (weighted) distribution, useful in life length studies. Further, some bounds and inequalities related to these measures are also studied.

Generalized phase retrieval algorithm based on information measures

An iterative phase retrieval algorithm based on the maximum entropy method (MEM) is presented. Introducing a new generalized information measure, we derive a novel class of algorithms which includes the conventionally used error reduction algorithm and a MEM-type iterative algorithm which is presented for the first time. These different phase retrieval methods are unified on the basis of the framework of information measures used in information theory.

A non-rigid registration approach for quantifying myocardial contraction in tagged MRI using generalized information measures

We address the problem of quantitatively assessing myocardial function from tagged MRI sequences. We develop a two-step method comprising (i) a motion estimation step using a novel variational non-rigid registration technique based on generalized information measures, and (ii) a measurement step, yielding local and segmental deformation parameters over the whole myocardium. Experiments on healthy and pathological data demonstrate that this method delivers, within a reasonable computation time and in a fully unsupervised way, reliable measurements for normal subjects and quantitative pathology-specific information. Beyond cardiac MRI, this work redefines the foundations of variational non-rigid registration for information-theoretic similarity criteria with potential interest in multimodal medical imaging.