Information Theory Concepts Research Articles

Fundamental properties of relative entropy and Lin divergence for Choquet integral

Entropy is the most important concept used in information theory and measuring uncertainty. In Choquet calculus, Sugeno (2013) [10] and Torra and Narukawa (2016) [2] studied Choquet integral and derivative with respect to monotone measures on the real line. Then as a very challenging problem, the definition of entropy and relative entropy on monotone measures for infinite sets based on Choquet integral was proposed by Torra (2017) [1] and Agahi (2019) [12]. These results show that based on the submodularity condition on monotone measures, entropy and relative entropy for Choquet integral are non-negative.In this paper, we first introduce the concept of Lin divergence (Lin, 1991, [8]), including Choquet integral and derivative with respect to monotone measures. Then some fundamental properties of this concept in information theory are given. In special case, we show that we can omit the submodularity condition in previous results on entropy and relative entropy for Choquet integral.

Positive maps and trace polynomials from the symmetric group

With techniques borrowed from quantum information theory, we develop a method to systematically obtain operator inequalities and identities in several matrix variables. These take the form of trace polynomials: polynomial-like expressions that involve matrix monomials Xα1,…,Xαr and their traces tr(Xα1,…,Xαr). Our method rests on translating the action of the symmetric group on tensor product spaces into that of matrix multiplication. As a result, we extend the polarized Cayley–Hamilton identity to an operator inequality on the positive cone, characterize the set of multilinear equivariant positive maps in terms of Werner state witnesses, and construct permutation polynomials and tensor polynomial identities on tensor product spaces. We give connections to concepts in quantum information theory and invariant theory.

A Scale-Invariant Generalization of the Rényi Entropy, Associated Divergences and Their Optimizations Under Tsallis’ Nonextensive Framework

Entropy and relative or cross entropy measures are two very fundamental concepts in information theory and are also widely used for statistical inference across disciplines. The related optimization problems, in particular the maximization of the entropy and the minimization of the cross entropy or relative entropy (divergence), are essential for general logical inference in our physical world. In this paper, we discuss a two parameter generalization of the popular R\'{e}nyi entropy and associated optimization problems. We derive the desired entropic characteristics of the new generalized entropy measure including its positivity, expandability, extensivity and generalized (sub-)additivity. More importantly, when considered over the class of sub-probabilities, our new family turns out to be scale-invariant. We also propose the corresponding cross entropy and relative entropy measures and discuss their geometric properties including generalized Pythagorean results over $\beta$-convex sets. The maximization of the new entropy and the minimization of the corresponding cross or relative entropy measures are carried out explicitly under the non-extensive (`third-choice') constraints given by the Tsallis' normalized $q$ expectations which also correspond to the $\beta$-linear family of probability distributions. Important properties of the associated forward and reverse projection rules are discussed along with their existence and uniqueness. In this context, we have come up with, for the first time, a class of entropy measures -- a subfamily of our two-parameter generalization -- that leads to the classical (extensive) exponential family of MaxEnt distributions under the non-extensive constraints. Other members of the new entropy family, however, lead to the power-law type generalized $q$-exponential MaxEnt distributions which is in conformity with Tsallis' nonextensive theory.

Triad model: simulation - functional tensometry - information database in the assessment of the reliability of technological machines

The concept of a multidisciplinary approach to assessing the resource of individual components of the machine by combining database information based on simulation techniques and functional tensomethration is proposed. Simulations determine the reperative points of the tendometric sensors. The creation of a diagnostic model using basic concepts of information theory has allowed the development of a synergistic model for the recognition of the area of displacement of areas of uncertainty, which will ensure the identification of the defect (risk-denial). The formation of an electronic database of parametric data on the nature of loads as a diagnostic indicator of the change in the accuracy of pairing in machine systems is justified. Experimental studies were conducted on the model of the quick-capler. Hierarchical structuring of the machine to the level of mating parts with digital control of the criticality of the magnitude of external and internal loads ensures reliability control throughout the entire service life of the machine. When disposing of machines, this data allows you to obtain information about the residual resources of the elements for their reuse or the feasibility of restoration. This, in turn, will ensure the environmental friendliness and economy of the process.

Role of accuracy and quantity of field tests in engineering-geotechnical researches for construction

The aim of this work is to summarize previously conducted studies on the optimization of the unequal geotechnical testing program and on the selection of the desired calculation indicator based on the results of such tests. The approximate, but quick and cheap tests (“express methods”) are recommended to be performed on a large scale and considered as a means of assessing the geotechnical structure of the site as a whole. It is proposed to carry out expensive “accurate” tests in a reduced volume and to use them as a means of correcting approximate tests. In the article, these issues are considered by the example of determining the bearing capacity of piles according to the data of static sounding (cone penetration testing – CPT), dynamic and static tests of full-scale piles. We propose the mathematical model for evaluating the informative content of the test complex, based on the concepts of information theory. The site is mentally divided into several sections, each of which is characterized by one of the possible values of the ultimate resistance of piles of a certain length. All variants of “placement in the plan” of possible values of pile resistances (“site images”) are considered. Initially, when nothing is known about the true value of the pile resistances in each section, all possible values of the pile resistances are assumed to be equally probable, i.e. the uncertainty of the situation is maximum. In the theory of information, such uncertainty is quantified by the value called entropy. When any test is performed at the site, the uncertainty decreases, and the more accurate the test the more significant is the decrease. The difference in entropy before and after the test represents the amount of information (in bits) that these tests carry. The calculations using this model showed that the information content of a large number of approximate tests can (due to heterogeneity of the soil) exceed the information content of small exact tests. Only one approximate test method can lead to the systematic error (overestimation or underestimation of the average value of the desired indicator). It is necessary to carry out control “exact” tests and approximate tests to eliminate such a danger. A technique is proposed for adjusting approximate estimates based on data from “accurate” tests, which ensures optimal “safety margins” in decisions being made.

ЗНАК – ТЕКСТ – ІНФОРМАЦІЯ (ВІДМІННОСТІ, СПОРІДНЕНІСТЬ, ВЗАЄМООБУМОВЛЕНІСТЬ)

Розглядаються знак - текст - інформація як явище однієї природи у їх взаємозв’язку і та взаємообумовленості. Інформація має знакову природу. Процеси, пов’язані с передачею, переробкою і зберіганням інформації протікають з використанням знаків та знакових систем, тобто є семіотичними. Семіотика постає як основа теорії інформації.

Selection of seismic intensity measures for prescribed limit states using alternative nonlinear dynamic analysis methods

AbstractIn performance‐based earthquake engineering, the suitability of an intensity measure (IM) is expressed through its efficiency and sufficiency. An efficient IM leads to a small record‐to‐record variability in the estimation of demand given seismic intensity. A sufficient IM is one that renders the estimation of demand for all intensity levels independent of all other ground motion parameters. Given that establishing sufficiency is not a trivial task, the relative sufficiency measure (RSM) has been proposed previously based on information theory concepts. RSM can be employed for quantifying the relative sufficiency of an IM with respect to another IM by the amount of extra information that it relays on average about the ground motion for the estimation of a demand parameter of interest. RSM has been so far conditioned on a linear logarithmic regression probability model, better known as the Cloud Analysis (CA), which relies on unscaled ground‐motion records. This work lays out the methodology for estimating both the efficiency and RSM in terms of the damage measure directly (instead of demand parameter) and by employing alternative nonlinear dynamic analysis procedures (NDAPs), such as, a modified version of CA that considers the collapse‐cases explicitly and the incremental dynamic analysis. It is demonstrated that the RSM can be sensitive to the NDAP employed, while it does not demonstrate significant sensitivity to the limit state of interest. An alternative measure of efficiency (known in literature as proficiency), directly measured as the dispersion in the fragility curve, shows more sensitivity to the limit state.

The Role of Corporate Diversification in Resource Allocation Adjustments of Multi-Business Firms

Resource allocation is fundamental to both strategic management scholarship and business practice. While a major research stream in strategy has generated a well-developed understanding of the link between resource allocation and firm performance, we are yet struggling to understand why certain firms allocate their resources more dynamically – i.e., they adjust their allocations very strongly over time – while other firms keep their allocations quite persistent. Drawing on the concept of private information, we develop and test theory in order to address this puzzle in the literature. Results from our study indicate that those firms that are less diversified act more dynamically in terms of adjusting their resource allocation. In addition, our findings suggest that several critical factors related to the main stages of the process of utilizing private information moderate the diversification level–resource adjustment relationship, allowing highly diversified firms to reduce their agility disadvantages.

Handling uncertainty in optimal design of reservoir water quality monitoring systems

In the present paper, a scenario-based many-objective optimization model is developed for the spatio-temporal optimal design of reservoir water quality monitoring systems considering uncertainties. The proposed methodology is based on the concept of nonlinear interval number programming and information theory, while handling uncertainties of temperature, reservoir inflow, and inflow constituent concentration. A reference-point-based non-dominated sorting genetic algorithm (NSGA-III) is used to deal with the many-objective optimization problem. The proposed model is developed for the Karkheh reservoir system in Iran as a real-world problem. The results show excellent performance of the optimized water quality sampling locations instead of all potential ones in providing adequate information about the reservoir water quality status. The presented uncertainty-based model leads to a 55.73% reduction in the radius of the uncertain interval caused by different scenarios. Handling uncertainties in a spatio-temporal many-objective optimization problem is the main contribution of this study, yielding a reliable and robust design of a reservoir monitoring system that is less sensitive to various scenarios.

Convergence Behavior of DNNs with Mutual-Information-Based Regularization.

Information theory concepts are leveraged with the goal of better understanding and improving Deep Neural Networks (DNNs). The information plane of neural networks describes the behavior during training of the mutual information at various depths between input/output and hidden-layer variables. Previous analysis revealed that most of the training epochs are spent on compressing the input, in some networks where finiteness of the mutual information can be established. However, the estimation of mutual information is nontrivial for high-dimensional continuous random variables. Therefore, the computation of the mutual information for DNNs and its visualization on the information plane mostly focused on low-complexity fully connected networks. In fact, even the existence of the compression phase in complex DNNs has been questioned and viewed as an open problem. In this paper, we present the convergence of mutual information on the information plane for a high-dimensional VGG-16 Convolutional Neural Network (CNN) by resorting to Mutual Information Neural Estimation (MINE), thus confirming and extending the results obtained with low-dimensional fully connected networks. Furthermore, we demonstrate the benefits of regularizing a network, especially for a large number of training epochs, by adopting mutual information estimates as additional terms in the loss function characteristic of the network. Experimental results show that the regularization stabilizes the test accuracy and significantly reduces its variance.

Dynamics of virtual water networks: Role of national socio-economic indicators across the world

Intensified water usage due to rapid industrialization is often dictated by economic policies based on monetary growth rather than sustainable use of environmental resources. In addition, interdependence within economic sectors further interweaves water usage through product transactions, which further makes it difficult to quantify the dynamics of hydro-economic systems at regional, national and global scale. In this study, we investigated the dynamics of domestic virtual water networks (VWN) of 189 countries based on concept of information theory by quantifying network metrics that describes VWN flow capacity, robustness, efficiency and flexibility. These networks represent virtual water interconnected through economic sectors within a specified country built based on environmentally extended multi region input output (EE-MRIO) approach. We further estimated trends associated with network metrics, as well as coupling intensity between metrics with respect to socio-economic indicators, such as, population, Gross Domestic Product (GDP) and Gross National Income (GNI). It was observed that capacity and flexibility of VWNs are strongly and positively correlated indicating that a high capacity VWN can be more flexible. Our results also indicate that, in general a higher percentage of developing countries (i.e. both least developing and developing nations) have exhibited increasing trends in capacity, robustness, efficiency and flexibility of VWN compared to developed nations. It was revealed that the dynamics of VWNs are positively coupled with socio-economic growth for few countries, which indicates the sustainable behavior of VWN with socio-economic growth. Our results argue that the information theory-based metrics by embedding water footprints can holistically capture sustainability aspect of the VWN dynamics.

Update of Infrared Atmospheric Sounding Interferometer (IASI) channel selection with correlated observation errors for numerical weather prediction (NWP)

Abstract. The Infrared Atmospheric Sounding Interferometer (IASI) is an essential instrument for numerical weather prediction (NWP). It measures radiances at the top of the atmosphere using 8461 channels. The huge amount of observations provided by IASI has led the community to develop techniques to reduce observations while conserving as much information as possible. Thus, a selection of the 300 most informative channels was made for NWP based on the concept of information theory. One of the main limitations of this method was to neglect the covariances between the observation errors of the different channels. However, many centres have shown a significant benefit for weather forecasting to use them. Currently, the observation-error covariances are only estimated on the current IASI channel selection, but no studies to make a new selection of IASI channels taking into account the observation-error covariances have yet been carried out. The objective of this paper was therefore to perform a new selection of IASI channels by taking into account the observation-error covariances. The results show that with an equivalent number of channels, accounting for the observation-error covariances, a new selection of IASI channels can reduce the analysis error on average in temperature by 3 %, humidity by 1.8 % and ozone by 0.9 % compared to the current selection. Finally, we go one step further by proposing a robust new selection of 400 IASI channels to further reduce the analysis error for NWP.

Influence measurement in a complex dynamical model: an information theoretic approach

We address the link between the controllability or observability of a stochastic complex system and concepts of information theory. We show that the most influential degrees of freedom can be detected without acting on the system, by measuring the time-delayed multi-information. Numerical and analytical results support this claim, which is developed in the case of a simple stochastic model on a graph, the so-called voter model. The importance of the noise when controlling the system is demonstrated, leading to the concept of control length. The link with classical control theory is given, as well as the interpretation of controllability in terms of the capacity of a communication canal.

Towards a Unified Theory of Learning and Information.

In this paper, we introduce the notion of “learning capacity” for algorithms that learn from data, which is analogous to the Shannon channel capacity for communication systems. We show how “learning capacity” bridges the gap between statistical learning theory and information theory, and we will use it to derive generalization bounds for finite hypothesis spaces, differential privacy, and countable domains, among others. Moreover, we prove that under the Axiom of Choice, the existence of an empirical risk minimization (ERM) rule that has a vanishing learning capacity is equivalent to the assertion that the hypothesis space has a finite Vapnik–Chervonenkis (VC) dimension, thus establishing an equivalence relation between two of the most fundamental concepts in statistical learning theory and information theory. In addition, we show how the learning capacity of an algorithm provides important qualitative results, such as on the relation between generalization and algorithmic stability, information leakage, and data processing. Finally, we conclude by listing some open problems and suggesting future directions of research.

Fluctuation–response inequality out of equilibrium

We present an approach to response around arbitrary out-of-equilibrium states in the form of a fluctuation-response inequality (FRI). We study the response of an observable to a perturbation of the underlying stochastic dynamics. We find that the magnitude of the response is bounded from above by the fluctuations of the observable in the unperturbed system and the Kullback-Leibler divergence between the probability densities describing the perturbed and the unperturbed system. This establishes a connection between linear response and concepts of information theory. We show that in many physical situations, the relative entropy may be expressed in terms of physical observables. As a direct consequence of this FRI, we show that for steady-state particle transport, the differential mobility is bounded by the diffusivity. For a "virtual" perturbation proportional to the local mean velocity, we recover the thermodynamic uncertainty relation (TUR) for steady-state transport processes. Finally, we use the FRI to derive a generalization of the uncertainty relation to arbitrary dynamics, which involves higher-order cumulants of the observable. We provide an explicit example, in which the TUR is violated but its generalization is satisfied with equality.

Probabilistic Ensemble of Deep Information Networks.

We describe a classifier made of an ensemble of decision trees, designed using information theory concepts. In contrast to algorithms C4.5 or ID3, the tree is built from the leaves instead of the root. Each tree is made of nodes trained independently of the others, to minimize a local cost function (information bottleneck). The trained tree outputs the estimated probabilities of the classes given the input datum, and the outputs of many trees are combined to decide the class. We show that the system is able to provide results comparable to those of the tree classifier in terms of accuracy, while it shows many advantages in terms of modularity, reduced complexity, and memory requirements.

Location Verification for Emerging Wireless Vehicular Networks

The work reported here utilizes the best aspects of information theory and deep-learning concepts so as to provide, for the first time, a solution for a real-world location verification system (LVS) in the context of vehicular networks. it is well established that global positioning system coordinates supplied by vehicles will be a vital component of such emerging networks. This supplied location information, if erroneous and not verified, can seriously degrade the overall system performance and lead to significant safety issues. A number of location verification protocols and systems have been developed to address this important problem but all have operational constraints and performance limitations due to their requirement for ideal static channel conditions and assumed threat models. In this article, we remove such limitations by designing a neural-network-based LVS (NN-LVS) that can accommodate a priori unknown channel conditions and unknown threat models. Under most channel conditions, the NN-LVS shows a performance improvement of 50%, or more, relative to other LVSs. We also derive a new information-theoretic bound on the total error for an LVS and show how this new bound allows for a useful tradeoff in learning-time versus verification-performance for the NN-LVS. We demonstrate an improved performance for the NN-LVS within the context of vehicular networks using time of arrival measurements of the vehicles’ transmitted signals measured at multiple verifying base stations. The work reported here, we believe, paves the way to the actual deployment in real-world conditions of LVSs for emerging vehicular networks.

Towards a Fisher-Information Description of Complexity in de Sitter Universe

Recent developments on holography and quantum information physics suggest that quantum information theory has come to play a fundamental role in understanding quantum gravity. Cosmology, on the other hand, plays a significant role in testing quantum gravity effects. How to apply this idea to a realistic universe is still unknown. Here, we show that some concepts in quantum information theory have cosmological descriptions. Particularly, we show that the complexity of a tensor network can be regarded as a Fisher information measure (FIM) of a dS universe, followed by several observations: (i) the holographic entanglement entropy has a tensor-network description and admits a information-theoretical interpretation, (ii) on-shell action of dS spacetime has a same description of FIM, (iii) complexity/action(CA) duality holds for dS spacetime. Our result is also valid for f ( R ) gravity, whose FIM exhibits the same features of a recent proposed L n norm complexity.

11th Asia-Europe Workshop on Concepts in Information Theory

11th Asia-Europe Workshop on Concepts in Information Theory

On the Solvability of the Mind\u2013Body Problem

The mind–body problem is analyzed in a physicalist perspective. By combining the concepts of emergence and algorithmic information theory in a thought experiment, employing a basic nonlinear process, it is shown that epistemologically emergent properties may develop in a physical system. Turning to the significantly more complex neural network of the brain it is subsequently argued that consciousness is epistemologically emergent. Thus reductionist understanding of consciousness appears not possible; the mind–body problem does not have a reductionist solution. The ontologically emergent character of consciousness is then identified from a combinatorial analysis relating to universal limits set by quantum mechanics, implying that consciousness is fundamentally irreducible to low-level phenomena.