Jensen-Shannon Divergences Research Articles

Fisher and Bayes-Fisher information measures for finite mixture distributions

. In this work, we consider Fisher information and Bayes-Fisher information measures for the mixing parameter vector of a finite mixture density function and develop some associated results for this model. We provide several interesting connections between these measures and some known informational measures, such as chi-square divergence, Shannon entropy, Kullback-Leibler, Jeffreys, and Jensen-Shannon divergences. Finally, to demonstrate the usefulness of the Bayes-Fisher information measure, we apply it to a real example in image processing and present some numerical results. Our findings in this regard show that this information measure is an effective criteria for quantifying the similarity between two images.

Neuro-Robotic Synergy: Crafting the Secure Future of Industries in the Post Pandemic Era

In recent years, ICSs have become increasingly commonplace in virtually every industry. The abbreviation “ICSs” refers to industrial control systems. These are specially designed computers used for monitoring, managing, and controlling procedures and tasks across a wide range of industries and vital infrastructure sectors. Production, power, disinfection of water, transport, and other sectors all greatly benefit from ICS use. The authors of this paper aim to detect ICS cyber hazards in industry. This article is the result of the writers’ extensive research on ICS programs and the impact of cyberattacks on them as well. The study narrowed its attention to just three ICS applications because there are simply too many to count: power plants, water reservoirs, and gas pipelines. The present paper focuses on the development and evaluation of neural networks for use in cyberattacks. An early form of neural network, the residual system, came first in the field. When a breach is detected in the ICS, the neural network sorts it into one of several categories. The produced datasets must not compromise users’ privacy or cause harm to the relevant industry if they fall into the wrong hands. An encoding device, decoder, pseudo-encoder, and critical model neural networks work together to generate random data. Finally, a set of trials is conducted in which a residual neural network is utilized to classify cyberattacks based on both the created and original datasets. Results from a series of studies indicate that using the created dataset is an effective technique to train high-quality neural networks for use in cybersecurity on a large amount of data without sacrificing the accuracy of the models. The Kullback-Leibler and Jensen-Shannon divergences also serve as the theoretical foundation and technique, respectively. In particular, the paper recommends operational and maintenance cybersecurity standards for ICS. This entails such things as secure password practices, patch management, and anti-malware defense. Physical safeguards for ICS is another topic that is covered.

Dataset of Program Source Codes Solving Unique Programming Exercises Generated by Digital Teaching Assistant

This paper presents a dataset containing automatically collected source codes solving unique programming exercises of different types. The programming exercises were automatically generated by the Digital Teaching Assistant (DTA) system that automates a massive Python programming course at MIREA—Russian Technological University (RTU MIREA). Source codes of the small programs grouped by the type of the solved task can be used for benchmarking source code classification and clustering algorithms. Moreover, the data can be used for training intelligent program synthesizers or benchmarking mutation testing frameworks, and more applications are yet to be discovered. We describe the architecture of the DTA system, aiming to provide detailed insight regarding how and why the dataset was collected. In addition, we describe the algorithms responsible for source code analysis in the DTA system. These algorithms use vector representations of programs based on Markov chains, compute pairwise Jensen–Shannon divergences of programs, and apply hierarchical clustering algorithms in order to automatically discover high-level concepts used by students while solving unique tasks. The proposed approach can be incorporated into massive programming courses when there is a need to identify approaches implemented by students.

Parameter estimation with the current generation of phenomenological waveform models applied to the black hole mergers of GWTC-1

ABSTRACT We consider the 10 confidently detected gravitational-wave signals in the GWTC-1 catalog, which are consistent with mergers of binary black hole systems, and perform a thorough parameter estimation re-analysis. This is made possible by using computationally efficient waveform models of the current (fourth) generation of the IMRPhenom family of phenomenological waveform models, which consists of the IMRPhenomX frequency-domain models and the IMRPhenomT time-domain models.The analysis is performed with both precessing and non-precessing waveform models with and without subdominant spherical harmonic modes. Results for all events are validated with convergence tests, discussing in particular the events GW170729 and GW151226. For the latter and the other two lowest-mass events, we also compare results between two independent sampling codes, bilbyand lalinference. We find overall consistent results with the original GWTC-1 results, with all Jensen–Shannon divergences between the previous results using IMRPhenomPv2 and our default IMRPhenomXPHM posteriors below 0.045 bits. However, we also discuss cases where subdominant harmonics and/or precession influence the posteriors.

Jensen-Shannon Divergence of Two Eddy Current Distributions Induced by Circular and Fractal Koch Excitation Coils

Eddy current distribution is important to the performance of planar eddy current probes. In this paper, the Jensen-Shannon divergences of tangential intersection angle spectrum and radial direction energy spectrum were proposed to evaluate the difference between eddy current distributions generated by circular and fractal Koch excitation coils. By the simulation for the circular and Koch shape excitation coils, it works out that the difference of the eddy current distributions between the two kinds of coils becomes larger and larger with an increase in the values of the two Jensen-Shannon divergences. At the same time, the correlation between the change of Jensen-Shannon divergence and the detectability of the short crack in the special direction was discussed through simulation and experiment results. It is found that, relative to the crack in 0° direction, the detectability of the Koch and circular differential pickup probes to the crack in 90° direction has a correlation with the Jensen-Shannon divergence of tangential intersection angle spectrum. The width of each signal generated by the two probes has a correlation with the Jensen-Shannon divergence of radial direction energy spectrum.

On a Variational Definition for the Jensen-Shannon Symmetrization of Distances Based on the Information Radius.

We generalize the Jensen-Shannon divergence and the Jensen-Shannon diversity index by considering a variational definition with respect to a generic mean, thereby extending the notion of Sibson’s information radius. The variational definition applies to any arbitrary distance and yields a new way to define a Jensen-Shannon symmetrization of distances. When the variational optimization is further constrained to belong to prescribed families of probability measures, we get relative Jensen-Shannon divergences and their equivalent Jensen-Shannon symmetrizations of distances that generalize the concept of information projections. Finally, we touch upon applications of these variational Jensen-Shannon divergences and diversity indices to clustering and quantization tasks of probability measures, including statistical mixtures.

Discrete Versions of Jensen-Fisher, Fisher and Bayes-Fisher Information Measures of Finite Mixture Distributions.

In this work, we first consider the discrete version of Fisher information measure and then propose Jensen–Fisher information, to develop some associated results. Next, we consider Fisher information and Bayes–Fisher information measures for mixing parameter vector of a finite mixture probability mass function and establish some results. We provide some connections between these measures with some known informational measures such as chi-square divergence, Shannon entropy, Kullback–Leibler, Jeffreys and Jensen–Shannon divergences.

Generalized word shift graphs: a method for visualizing and explaining pairwise comparisons between texts

A common task in computational text analyses is to quantify how two corpora differ according to a measurement like word frequency, sentiment, or information content. However, collapsing the texts’ rich stories into a single number is often conceptually perilous, and it is difficult to confidently interpret interesting or unexpected textual patterns without looming concerns about data artifacts or measurement validity. To better capture fine-grained differences between texts, we introduce generalized word shift graphs, visualizations which yield a meaningful and interpretable summary of how individual words contribute to the variation between two texts for any measure that can be formulated as a weighted average. We show that this framework naturally encompasses many of the most commonly used approaches for comparing texts, including relative frequencies, dictionary scores, and entropy-based measures like the Kullback–Leibler and Jensen–Shannon divergences. Through a diverse set of case studies ranging from presidential speeches to tweets posted in urban green spaces, we demonstrate how generalized word shift graphs can be flexibly applied across domains for diagnostic investigation, hypothesis generation, and substantive interpretation. By providing a detailed lens into textual shifts between corpora, generalized word shift graphs help computational social scientists, digital humanists, and other text analysis practitioners fashion more robust scientific narratives.

Metrics induced by Jensen-Shannon and related divergences on positive definite matrices

We study metric properties of symmetric divergences on Hermitian positive definite matrices. In particular, we prove that the square root of these divergences is a distance metric. As a corollary we obtain a proof of the metric property for Quantum Jensen-Shannon-(Tsallis) divergences (parameterized by α∈[0,2]). When specialized to α=1, we obtain as a corollary a proof of the metric property of the Quantum Jensen-Shannon divergence that was conjectured by Lamberti et al. (2008) [13], and recently also proved by Virosztek (2019) [28]. A more intricate argument also establishes metric properties of Jensen-Rényi divergences (for α∈(0,1)); this argument develops a technique that may be of independent interest.

Quantification of Network Structural Dissimilarities Based on Graph Embedding

Identifying and quantifying structural dissimilarities between complex networks is a fundamental and challenging problem in network science. Previous network comparison methods are based on the structural features, such as the length of shortest path, degree and graphlet, which may only contain part of the topological information. Therefore, we propose an efficient network comparison method based on network embedding, i.e., \textit{DeepWalk}, which considers the global structural information. In detail, we calculate the distance between nodes through the vector extracted by \textit{DeepWalk} and quantify the network dissimilarity by spectral entropy based Jensen-Shannon divergences of the distribution of the node distances. Experiments on both synthetic and empirical data show that our method outperforms the baseline methods and can distinguish networks perfectly by only using the global embedding based distance distribution. In addition, we show that our method can capture network properties, e.g., average shortest path length and link density. Moreover, the experiments of modularity further implies the functionality of our method.

Fuzzy Integral-Based CNN Classifier Fusion for 3D Skeleton Action Recognition

Action recognition based on skeleton key joints has gained popularity due to its cost effectiveness and low complexity. Existing Convolutional Neural Network (CNN) based models mostly fail to capture various aspects of the skeleton sequence. To this end, four feature representations, which capture complementary characteristics of the sequence of key joints, are extracted with novel contribution of features estimated from angular information, and kinematics of the human actions. Single channel grayscale images are used to encode these features for classification using four CNNs, with the complementary nature verified through Kullback-Leibler (KL) and Jensen-Shannon (JS) divergences. As opposed to straightforward classifier combination generally used in existing literature, fuzzy fusion through the Choquet integral leverages the degree of uncertainty of decision scores obtained from four CNNs. Experimental results support the efficacy of fuzzy combination of CNNs to adaptively generate final decision score based upon confidence of each information source. Impressive results on the challenging UTD-MHAD, HDM05, G3D, and NTU RGB+D 60 and 120 datasets demonstrate the effectiveness of the proposed method. The source code for our method is available at https://github.com/theavicaster/fuzzy-integral-cnn-fusion-3d-har

On a Generalization of the Jensen-Shannon Divergence and the Jensen-Shannon Centroid.

The Jensen–Shannon divergence is a renown bounded symmetrization of the Kullback–Leibler divergence which does not require probability densities to have matching supports. In this paper, we introduce a vector-skew generalization of the scalar -Jensen–Bregman divergences and derive thereof the vector-skew -Jensen–Shannon divergences. We prove that the vector-skew -Jensen–Shannon divergences are f-divergences and study the properties of these novel divergences. Finally, we report an iterative algorithm to numerically compute the Jensen–Shannon-type centroids for a set of probability densities belonging to a mixture family: This includes the case of the Jensen–Shannon centroid of a set of categorical distributions or normalized histograms.

On the Jensen-Shannon Symmetrization of Distances Relying on Abstract Means.

The Jensen–Shannon divergence is a renowned bounded symmetrization of the unbounded Kullback–Leibler divergence which measures the total Kullback–Leibler divergence to the average mixture distribution. However, the Jensen–Shannon divergence between Gaussian distributions is not available in closed form. To bypass this problem, we present a generalization of the Jensen–Shannon (JS) divergence using abstract means which yields closed-form expressions when the mean is chosen according to the parametric family of distributions. More generally, we define the JS-symmetrizations of any distance using parameter mixtures derived from abstract means. In particular, we first show that the geometric mean is well-suited for exponential families, and report two closed-form formula for (i) the geometric Jensen–Shannon divergence between probability densities of the same exponential family; and (ii) the geometric JS-symmetrization of the reverse Kullback–Leibler divergence between probability densities of the same exponential family. As a second illustrating example, we show that the harmonic mean is well-suited for the scale Cauchy distributions, and report a closed-form formula for the harmonic Jensen–Shannon divergence between scale Cauchy distributions. Applications to clustering with respect to these novel Jensen–Shannon divergences are touched upon.

Spectral Entropies as Information-Theoretic Tools for Complex Network Comparison

Any physical system can be viewed from the perspective that information is implicitly represented in its state. However, the quantification of this information when it comes to complex networks has remained largely elusive. In this work, we use techniques inspired by quantum statistical mechanics to define an entropy measure for complex networks and to develop a set of information-theoretic tools, based on network spectral properties, such as Rényi q entropy, generalized Kullback-Leibler and Jensen-Shannon divergences, the latter allowing us to define a natural distance measure between complex networks. First, we show that by minimizing the Kullback-Leibler divergence between an observed network and a parametric network model, inference of model parameter(s) by means of maximum-likelihood estimation can be achieved and model selection can be performed with appropriate information criteria. Second, we show that the information-theoretic metric quantifies the distance between pairs of networks and we can use it, for instance, to cluster the layers of a multilayer system. By applying this framework to networks corresponding to sites of the human microbiome, we perform hierarchical cluster analysis and recover with high accuracy existing community-based associations. Our results imply that spectral-based statistical inference in complex networks results in demonstrably superior performance as well as a conceptual backbone, filling a gap towards a network information theory.Received 25 August 2016DOI:https://doi.org/10.1103/PhysRevX.6.041062Published by the American Physical Society under the terms of the Creative Commons Attribution 3.0 License. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasNetwork structureQuantum computationPhysical SystemsMultilayer & multiplex networksReal world networksWatts-Strogatz networkTechniquesInformation theoryCondensed Matter, Materials & Applied Physics

Learning With Jensen-Tsallis Kernels.

Jensen-type [Jensen-Shannon (JS) and Jensen-Tsallis] kernels were first proposed by Martins et al. (2009). These kernels are based on JS divergences that originated in the information theory. In this paper, we extend the Jensen-type kernels on probability measures to define positive-definite kernels on Euclidean space. We show that the special cases of these kernels include dot-product kernels. Since Jensen-type divergences are multidistribution divergences, we propose their multipoint variants, and study spectral clustering and kernel methods based on these. We also provide experimental studies on benchmark image database and gene expression database that show the benefits of the proposed kernels compared with the existing kernels. The experiments on clustering also demonstrate the use of constructing multipoint similarities.

Multi-scale similarities in stochastic neighbour embedding: Reducing dimensionality while preserving both local and global structure

Stochastic neighbour embedding (SNE) and its variants are methods of nonlinear dimensionality reduction that involve soft Gaussian neighbourhoods to measure similarities for all pairs of data. In order to build a suitable embedding, these methods try to reproduce in a low-dimensional space the neighbourhoods that are observed in the high-dimensional data space. Previous works have investigated the immunity of such similarities to norm concentration, as well as enhanced cost functions, like sums of Jensen–Shannon divergences. This paper proposes an additional refinement, namely multi-scale similarities, which are averages of soft Gaussian neighbourhoods with exponentially growing bandwidths. Such multi-scale similarities can replace the regular, single-scale neighbourhoods in SNE-like methods. Their objective is then to maximise the embedding quality on all scales, with the best preservation of both local and global neighbourhoods, and also to exempt the user from having to fix a scale arbitrarily. Experiments with several data sets show that the proposed multi-scale approach captures better the structure of data and improves significantly the quality of dimensionality reduction.

Relativistic global and local divergences in hydrogenic systems: A study in position and momentum spaces

Relativistic effects in one-particle densities of hydrogenic systems are quantified by means of global and local density functionals: the Jensen-Shannon and the Jensen-Fisher divergences, respectively. The Schr\"odinger and Dirac radial densities are compared, providing complementary results in position and momentum spaces. While the electron cloud gets compressed towards the origin in the Dirac case, the momentum density spreads out over its domain, and the raising of minima in position space does not occur in the momentum space. Regarding the dependence on the nuclear charge and the state quantum numbers for all divergences here considered, as well as their mutual interconnection, accurate powerlike laws $y\ensuremath{\approx}C{x}^{a}$ are found systematically. The parameters ${C,a}$ defining the respective dependences are extremely sensitive to the closeness of the system to the ground and/or the circular state. Particularly interesting are the analyses of (i) the plane subtended by the Jensen-Shannon and Jensen-Fisher divergences, in a given space (position or momentum), and (ii) either of the above two divergences in the position-momentum plane. These kinds of results show the complementary role of global and local divergences and that of both conjugate spaces.

An information theoretic approach to hierarchical clustering combination

In Hierarchical clustering, a set of patterns are partitioned into a sequence of groups represented as a dendrogram. The dendrogram is a tree representation where each node is associated with merging of two (or more) partitions and hence each partition is nested into the next partition. Hierarchical representation has properties that are useful for visualization and interpretation of clustering results. On one hand, different hierarchical clustering algorithms usually produce different dendrograms. On the other hand, clustering combination methods have received considerable interest in recent years and they yield superior results for clustering problems. This paper proposes a framework for combining various hierarchical clustering results which preserves the structural contents of input hierarchies. In this method, first a description matrix is created for each hierarchy, and then the description matrices of the input hierarchies are aggregated to form a consensus matrix from which the final hierarchy is derived. In this framework, we use two new measures for aggregating the description matrices, namely Rényi and Jensen–Shannon Divergences. The experimental and comparative analysis of our proposed framework shows the effectiveness of these two aggregators in hierarchical clustering combination.

Seven Means, Generalized Triangular Discrimination, and Generating Divergence Measures

Jensen-Shannon, J-divergence and Arithmetic-Geometric mean divergences are three classical divergence measures known in the information theory and statistics literature. These three divergence measures bear interesting inequality among the three non-logarithmic measures known as triangular discrimination, Hellingar’s divergence and symmetric chi-square divergence. However, in 2003, Eve studied seven means from a geometrical point of view, which are Harmonic, Geometric, Arithmetic, Heronian, Contra-harmonic, Root-mean square and Centroidal. In this paper, we have obtained new inequalities among non-negative differences arising from these seven means. Correlations with generalized triangular discrimination and some new generating measures with their exponential representations are also presented.

Discriminating Different Classes of Biological Networks by Analyzing the Graphs Spectra Distribution

The brain's structural and functional systems, protein-protein interaction, and gene networks are examples of biological systems that share some features of complex networks, such as highly connected nodes, modularity, and small-world topology. Recent studies indicate that some pathologies present topological network alterations relative to norms seen in the general population. Therefore, methods to discriminate the processes that generate the different classes of networks (e.g., normal and disease) might be crucial for the diagnosis, prognosis, and treatment of the disease. It is known that several topological properties of a network (graph) can be described by the distribution of the spectrum of its adjacency matrix. Moreover, large networks generated by the same random process have the same spectrum distribution, allowing us to use it as a “fingerprint”. Based on this relationship, we introduce and propose the entropy of a graph spectrum to measure the “uncertainty” of a random graph and the Kullback-Leibler and Jensen-Shannon divergences between graph spectra to compare networks. We also introduce general methods for model selection and network model parameter estimation, as well as a statistical procedure to test the nullity of divergence between two classes of complex networks. Finally, we demonstrate the usefulness of the proposed methods by applying them to (1) protein-protein interaction networks of different species and (2) on networks derived from children diagnosed with Attention Deficit Hyperactivity Disorder (ADHD) and typically developing children. We conclude that scale-free networks best describe all the protein-protein interactions. Also, we show that our proposed measures succeeded in the identification of topological changes in the network while other commonly used measures (number of edges, clustering coefficient, average path length) failed.