Kullback-Leibler Research Articles

Generative Adversarial Networks with Radiomics Supervision for Lung Lesion Generation.

Data-driven methods for lesion generation are quickly emerging due to the need for realistic imaging targets for image quality assessment and virtual clinical trials. We proposed a generative adversarial network (GAN) architecture for conditional generation of lung lesions based on user-specified classes of lesion size and solidity. The network consists of two discriminators, one for volumetric lesion data, and one for radiomics features derived from the lesion volume. A Wasserstein loss with gradient penalty was adopted for each discriminator. Training data were drawn from contoured and annotated lesions from a public lung CT database. Four quantitative evaluation methods were devised to assess the network performance: 1) overfitting (similarity between generated and real lesions), 2) diversity (similarity among generated lesions), 3) conditional consistency (capability of generating lesions according to user-specified classes), and 4) similarity in distributions of various lesion properties between the generated and real lesions. Ablation studies were also performed to investigate the importance of individual network component. The proposed network was found to generate lesions that resemble real lesions by visual inspection. Solid lesions are distinct from non-solid ones, and lesion sizes largely correspond to their specified classes. With a classifier trained on real lesions, the classification accuracies of generated and real lesions in both solid and non-solid classes are similar. Radiomics features of generated and real lesions were found to have similar distributions, indicated by the relatively low Kullback-Leibler (KL) divergence values. Furthermore, the correlations between pairwise radiomics features in generated lesions were comparable to those of real lesions. The proposed network presents a promising approach for generating realistic lesions with clinically relevant features crucial for the comprehensive assessment of medical imaging systems.

State-of-health estimation for lithium-ion batteries based on Kullback–Leibler divergence and a retentive network

Accurate state-of-health (SOH) estimation is crucial for the lithium-ion battery industry, as it underpins the safety, durability, and reliability of lithium-ion batteries. Currently, most researchers use various methods of health indicator (HI) extraction for the SOH estimation of batteries. However, these methods may require certain expertise and prior knowledge to achieve accurate modeling, being affected by measurement noise and other factors. To solve the abovementioned problems, three Kullback–Leibler (KL) divergence features based on partial voltage sequences are proposed as new HIs that are independent of prior knowledge and strongly correlated with SOH. Moreover, a modified retentive network is proposed to enhance SOH estimation accuracy and better utilize HIs than traditional deep learning methods, which have high training costs and insufficient accuracy. To ensure consistent extraction of KL divergence features across various experimental conditions and time intervals, a B-spline algorithm is utilized for interpolation. The effectiveness of the proposed method is validated through analysis of Pearson correlation coefficients and experiments conducted in four dimensions. Additionally, the potential of using the proposed method to compress data on the cloud-side is explored.

Optimal innovation-based attacks against remote state estimation with side information and historical data

This work investigates the attack strategy design problem of the false data injection attack against the cyber-physical system. Distinct from the relevant results which utilise the current intercepted data or additionally consider side information, a more universal attack model is proposed which combines historical data and side information with the current intercepted data to synergistically deteriorate the system estimation performance. In order to quantify the impact resulting from the proposed attack strategy, the optimisation objective is characterised by deriving the error covariance matrix under the attacks, which becomes more intricate since the proposed attack model introduces more decision variables and coupled terms. Take the stealthiness which is characterised by Kullback-Leibler divergence as the constraints, the problem investigated in this work is equivalently transformed into the constrained multi-variable non-convex optimisation problems, which are not able to be solved directly by the methods in the relevant results. By utilising the Lagrange multiplier method, the structural characteristic of the optimal mean and the optimal covariance which only related to the Lagrange multiplier are derived, such that the optimal distribution of the modified innovation is able to be obtained by a simple search procedure. Following that, the design of the optimal attack strategy is completed by using semi-definite programming to derive the optimal attack matrices. Finally, the simulation examples are given such that the validity of the results is verified.

Adversarial Variational Autoencoders to Extend and Improve Generative Model

Generative artificial intelligence (GenAI) has been advancing with many notable achievements like ChatGPT and Bard. The deep generative model (DGM) is a branch of GenAI, which is preeminent in generating raster data such as image and sound due to the strong role of deep neural networks (DNNs) in inference and recognition. The built-in inference mechanism of DNN, which simulates and aims at synaptic plasticity of the human neuron network, fosters the generation ability of DGM, which produces surprising results with the support of statistical flexibility. Two popular approaches in DGM are the variational autoencoder (VAE) and generative adversarial network (GAN). Both VAE and GAN have their own strong points although they share and imply the underlying theory of statistics as well as significant complex via hidden layers of DNN when DNN becomes effective encoding/decoding functions without concrete specifications. This research unifies VAE and GAN into a consistent and consolidated model called the adversarial variational autoencoder (AVA) in which the VAE and GAN complement each other; for instance, the VAE is a good data generator by encoding data via the excellent ideology of Kullback–Leibler divergence and the GAN is a significantly important method to assess the reliability of data as to whether it is real or fake. In other words, the AVA aims to improve the accuracy of generative models; besides, the AVA extends the function of simple generative models. In methodology, this research focuses on the combination of applied mathematical concepts and skillful techniques of computer programming in order to implement and solve complicated problems as simply as possible.

Open-Set Recognition Model for SAR Target Based on Capsule Network with the KLD

Synthetic aperture radar (SAR) automatic target recognition (ATR) technology has seen significant advancements. Despite these advancements, the majority of research still operates under the closed-set assumption, wherein all test samples belong to classes seen during the training phase. In real-world applications, however, it is common to encounter targets not previously seen during training, posing a significant challenge to the existing methods. Ideally, an ATR system should not only accurately identify known target classes but also effectively reject those belonging to unknown classes, giving rise to the concept of open set recognition (OSR). To address this challenge, we propose a novel approach that leverages the unique capabilities of the Capsule Network and the Kullback-Leibler divergence (KLD) to distinguish unknown classes. This method begins by deeply mining the features of SAR targets using the Capsule Network and enhancing the separability between different features through a specially designed loss function. Subsequently, the KLD of features between a testing sample and the center of each known class is calculated. If the testing sample exhibits a significantly larger KLD compared to all known classes, it is classified as an unknown target. The experimental results of the SAR-ACD dataset demonstrate that our method can maintain a correct identification rate of over 95% for known classes while effectively recognizing unknown classes. Compared to existing techniques, our method exhibits significant improvements.

Computing marginal and conditional divergences between decomposable models with applications in quantum computing and earth observation

AbstractThe ability to compute the exact divergence between two high-dimensional distributions is useful in many applications, but doing so naively is intractable. Computing the $$\alpha \beta $$ α β -divergence—a family of divergences that includes the Kullback–Leibler divergence and Hellinger distance—between the joint distribution of two decomposable models, i.e., chordal Markov networks, can be done in time exponential in the treewidth of these models. Extending this result, we propose an approach to compute the exact $$\alpha \beta $$ α β -divergence between any marginal or conditional distribution of two decomposable models. In order to do so tractably, we provide a decomposition over the marginal and conditional distributions of decomposable models. We then show how our method can be used to analyze distributional changes by first applying it to the benchmark image dataset QMNIST and a dataset containing observations from various areas at the Roosevelt Nation Forest and their cover type. Finally, based on our framework, we propose a novel way to quantify the error in contemporary superconducting quantum computers.

SGFNet: A semantic graph-based multimodal network for financial invoice information extraction

To meet the demand for a large amount of invoice entry work in the financial industry and improve the low accuracy of traditional manual entry, we construct SGFNet, a financial invoice information extraction network that integrates semantic graph associations and multimodal modeling. First, we construct a graph of strong and weak semantic associations between data within each modality based on the correlation of text content. Subsequently, we model the multimodal data in a unified structure, extract the text modal information of invoices along with corresponding image and layout modal information, and guide the fusion and embedding of multimodal data through semantic associations in the graph to produce a richer feature representation. Furthermore, semantically linked multimodal information is fed into an aggregated multimodal self-attention mechanism to establish effective connection between modalities. Finally, with the combination of supervised contrastive learning and smoothed Kullback–Leibler divergence in terms of loss functions, the accuracy degradation problem incurred by sample imbalance and convergence instability is reduced. In our experiments, we achieved F1 scores of 93.71% for the English financial invoice dataset and 96.27% for the Chinese dataset, indicating that the proposed method successfully extracts feature information from different data modalities, thereby achieving satisfactory information extraction results.

Extending global fits of 4D Composite Higgs Models with partially composite leptons

We perform the first convergent Bayesian global fits of 4D Composite Higgs Models with partially-composite third generation quarks and leptons based on the minimal SO(5) → SO(4) symmetry breaking pattern. We consider two models with the τ lepton and its associated neutrino in different representations of SO(5). Fitting each model with a wide array of experimental constraints allows us to analyse the Bayesian evidence and currently-observed fine-tuning of each model by calculating the Kullback-Leibler divergence between their respective priors and posteriors. Notably both models are found to be capable of satisfying all constraints simultaneously at the 3σ level at scales of < 5 TeV. From a Bayesian viewpoint of naturalness the model with leptons in the 14 and 10 representations is preferred over those in the 5 representation due to its lower fine-tuning. Finally, we consider the experimental signatures for the preferred parameters in these models, including lepton partner decay signatures and gluon-fusion produced Higgs signal strengths, and discuss their potential phenomenology at future high-luminosity LHC runs.

Variational Bayesian Approximation (VBA): Implementation and Comparison of Different Optimization Algorithms.

In any Bayesian computations, the first step is to derive the joint distribution of all the unknown variables given the observed data. Then, we have to do the computations. There are four general methods for performing computations: Joint MAP optimization; Posterior expectation computations that require integration methods; Sampling-based methods, such as MCMC, slice sampling, nested sampling, etc., for generating samples and numerically computing expectations; and finally, Variational Bayesian Approximation (VBA). In this last method, which is the focus of this paper, the objective is to search for an approximation for the joint posterior with a simpler one that allows for analytical computations. The main tool in VBA is to use the Kullback-Leibler Divergence (KLD) as a criterion to obtain that approximation. Even if, theoretically, this can be conducted formally, for practical reasons, we consider the case where the joint distribution is in the exponential family, and so is its approximation. In this case, the KLD becomes a function of the usual parameters or the natural parameters of the exponential family, where the problem becomes parametric optimization. Thus, we compare four optimization algorithms: general alternate functional optimization; parametric gradient-based with the normal and natural parameters; and the natural gradient algorithm. We then study their relative performances on three examples to demonstrate the implementation of each algorithm and their efficiency performance.

Computationally feasible bounds for the free energy of nonequilibrium steady states, applied to simple models of heat conduction

In this paper, we study computationally feasible bounds for relative free energies between two many-particle systems. Specifically, we consider systems out of equilibrium that admit a nonequilibrium steady state that is reached asymptotically in the long-time limit. The bounds that we suggest are based on the well-known Bogoliubov inequality and variants of Gibbs' and Donsker–Varadhan variational principles. As a general paradigm, we consider systems of oscillators coupled to heat baths at different temperatures. For such systems, we define the free energy of the system relative to any given reference system in terms of the Kullback–Leibler divergence between steady states. By employing a two-sided Bogoliubov inequality and a mean-variance approximation of the free energy (or cumulant generating function), we can efficiently estimate the free energy cost needed in passing from the reference system to the system out of equilibrium (characterised by a temperature gradient). A specific test case to validate our bounds are harmonic oscillator chains with ends that are coupled to Langevin thermostats at different temperatures; such a system is simple enough to allow for analytic calculations and general enough to be used as a prototype to estimate, e.g. heat fluxes or interface effects in a larger class of nonequilibrium particle systems.

Investigation on the propagation of uncertainties of a timber floor under human excitation

Due to the characteristics of high stiffness-weight ratio, timber floors are prone to annoying vibrations under human excitation. Given the natural origin of timber, its mechanical properties exhibit significant variability. The randomness inherent in human excitation cannot be overlooked during structural dynamic analysis. Consequently, the adoption of a stochastic approach is imperative for attaining reliable serviceability evaluation results. However, current research on human-induced vibrations in the timber floor, accounting for this randomness, remains inadequate. In this paper, an experimental investigation is conducted on the dynamic properties and human-induced responses of a timber floor composed of glued laminated timber and oriented strand board. A finite element model is developed and subsequently validated for accuracy in terms of modal properties and dynamic responses. The probability density evolution method is introduced for stochastic analysis, which demonstrates that both the natural frequency and dynamic responses of the floor exhibit considerable variability when uncertainty factors are considered. The Kullback–Leibler divergence indices are used to assess the impact of each uncertain variable quantitatively. The results indicate that the longitudinal elastic modulus has the greatest influence on the natural frequency, while the first dynamic load factor, αz1, exerts the most significant impact on dynamic responses. Notably, both material mechanical properties and load model parameters contribute to the uncertainty of dynamic responses, with the influence of the load model parameters being significantly greater than that of material mechanical properties.

Computer-Aided Diagnosis of Diabetic Retinopathy Lesions Based on Knowledge Distillation in Fundus Images

At present, the early diagnosis of diabetic retinopathy (DR), a possible complication of diabetes due to elevated glucose concentrations in the blood, is usually performed by specialists using a manual inspection of high-resolution fundus images based on lesion screening, leading to problems such as high work-intensity and accessibility only in specialized health centers. To support the diagnosis of DR, we propose a deep learning-based (DL) DR lesion classification method through a knowledge distillation (KD) strategy. First, we use the pre-trained DL architecture, Inception-v3, as a teacher model to distill the dataset. Then, a student model, also using the Inception-v3 model, is trained on the distilled dataset to match the performance of the teacher model. In addition, a new combination of Kullback–Leibler (KL) divergence and categorical cross-entropy (CCE) loss is used to measure the difference between the teacher and student models. This combined metric encourages the student model to mimic the predictions of the teacher model. Finally, the trained student model is evaluated on a validation dataset to assess its performance and compare it with both the teacher model and another competitive DL model. Experiments are conducted on the two datasets, corresponding to an imbalanced and a balanced dataset. Two baseline models (Inception-v3 and YOLOv8) are evaluated for reference, obtaining a maximum training accuracy of 66.75% and 90.90%, respectively, and a maximum validation accuracy of 35.94% and 81.52%, both for the imbalanced dataset. On the other hand, the proposed DR classification model achieves an average training accuracy of 99.01% and an average validation accuracy of 97.30%, overcoming the baseline models and other state-of-the-art works. Experimental results show that the proposed model achieves competitive results in DR lesion detection and classification tasks, assisting in the early diagnosis of diabetic retinopathy.

Efficient First-Order Algorithms for Large-Scale, Non-Smooth Maximum Entropy Models with Application to Wildfire Science.

Maximum entropy (MaxEnt) models are a class of statistical models that use the maximum entropy principle to estimate probability distributions from data. Due to the size of modern data sets, MaxEnt models need efficient optimization algorithms to scale well for big data applications. State-of-the-art algorithms for MaxEnt models, however, were not originally designed to handle big data sets; these algorithms either rely on technical devices that may yield unreliable numerical results, scale poorly, or require smoothness assumptions that many practical MaxEnt models lack. In this paper, we present novel optimization algorithms that overcome the shortcomings of state-of-the-art algorithms for training large-scale, non-smooth MaxEnt models. Our proposed first-order algorithms leverage the Kullback-Leibler divergence to train large-scale and non-smooth MaxEnt models efficiently. For MaxEnt models with discrete probability distribution of n elements built from samples, each containing m features, the stepsize parameter estimation and iterations in our algorithms scale on the order of O(mn) operations and can be trivially parallelized. Moreover, the strong ℓ1 convexity of the Kullback-Leibler divergence allows for larger stepsize parameters, thereby speeding up the convergence rate of our algorithms. To illustrate the efficiency of our novel algorithms, we consider the problem of estimating probabilities of fire occurrences as a function of ecological features in the Western US MTBS-Interagency wildfire data set. Our numerical results show that our algorithms outperform the state of the art by one order of magnitude and yield results that agree with physical models of wildfire occurrence and previous statistical analyses of wildfire drivers.

A Measure of Departure from Marginal Homogeneity using Continuation Odds for Square Contingency Tables with Ordered Categories

For the analysis of square contingency tables with ordinal classifications, the marginal homogeneity (MH) model is the one of important models. Some measures for analyzing the degree of departure from the MH model have been proposed. This study proposes a new measure using the continuation odds. Continuation odds may be considered as discrete time hazard. The proposed measure is expressed in the form of Cressie-Read’s power-divergence (including the Kullback-Leibler divergence) or Patil and Taillie’s diversity index (including Shannon entropy). This study derives a plug-in estimator of the proposed measure and an approximate confidence interval for the proposed measure. Through numerical examples, we evaluate the performances of them. Additionally. the usefulness of the proposed measure is demonstrated by applying it to real data that the row and column variables are the discrete survival time variables.

New Improvements of the Jensen–Mercer Inequality for Strongly Convex Functions with Applications

In this paper, we use the generalized version of convex functions, known as strongly convex functions, to derive improvements to the Jensen–Mercer inequality. We achieve these improvements through the newly discovered characterizations of strongly convex functions, along with some previously known results about strongly convex functions. We are also focused on important applications of the derived results in information theory, deducing estimates for χ-divergence, Kullback–Leibler divergence, Hellinger distance, Bhattacharya distance, Jeffreys distance, and Jensen–Shannon divergence. Additionally, we prove some applications to Mercer-type power means at the end.

Taming numerical imprecision by adapting the KL divergence to negative probabilities

The Kullback–Leibler (KL) divergence is frequently used in data science. For discrete distributions on large state spaces, approximations of probability vectors may result in a few small negative entries, rendering the KL divergence undefined. We address this problem by introducing a parameterized family of substitute divergence measures, the shifted KL (sKL) divergence measures. Our approach is generic and does not increase the computational overhead. We show that the sKL divergence shares important theoretical properties with the KL divergence and discuss how its shift parameters should be chosen. If Gaussian noise is added to a probability vector, we prove that the average sKL divergence converges to the KL divergence for small enough noise. We also show that our method solves the problem of negative entries in an application from computational oncology, the optimization of Mutual Hazard Networks for cancer progression using tensor-train approximations.

An efficient method of predicting S-wave velocity using sparse Gaussian process regression for a tight sandstone reservoir

The shear wave (S-wave) velocity plays a crucial role in interpreting the lithology in seismic data, identifying fluids and predicting reservoirs. However, S-wave velocity is often unavailable due to the high cost of measurement and technical constraints. Conventional methods exhibit limitations that potentially impact the accuracy or efficiency on predicting S-wave velocity. Moreover, these methods always ignore the uncertainty quantification associated with the predicted results. This paper proposes a sparse Gaussian process regression (SGPR) method to predict the S-wave velocity in tight sandstone reservoirs. SGPR is a highly efficient regression technique that is based on the Gaussian process regression (GPR) method. In the SGPR method, inducing inputs are introduced to approximate the kernel matrix to decrease the computational complexity. A sparse set of inducing inputs and kernel hyperparameters are optimized through minimizing the Kullback-Leibler (KL) divergence between the exact posterior distribution and the approximate one. In this study, we select several types of logging data, which include porosity, water saturation, shale content, lithology and P-wave velocity, as the inputs for the SGPR method to predict S-wave velocity. To validate its effectiveness, we use the SGPR method to predict S-wave velocity in tight sandstone and compare the results with those from the GPR method, the bidirectional long short-term memory (BiLSTM) method and the Xu-White model. Additionally, we conduct cross-validation to demonstrate the robustness of the SGPR method. Our findings indicate that the SGPR method presents better performance and significant advantages about the accuracy and efficiency. Moreover, the SGPR method offers uncertainty quantification for the predicted S-wave velocity.

Improved Non-Negative Matrix Factorization-Based Noise Reduction of Leakage Acoustic Signals.

The detection of gas leaks using acoustic signals is often compromised by environmental noise, which significantly impacts the accuracy of subsequent leak identification. Current noise reduction algorithms based on non-negative matrix factorization (NMF) typically utilize the Euclidean distance as their objective function, which can exacerbate noise anomalies. Moreover, these algorithms predominantly rely on simple techniques like Wiener filtering to estimate the amplitude spectrum of pure signals. This approach, however, falls short in accurately estimating the amplitude spectrum of non-stationary signals. Consequently, this paper proposes an improved non-negative matrix factorization (INMF) noise reduction algorithm that enhances the traditional NMF by refining both the objective function and the amplitude spectrum estimation process for reconstructed signals. The improved algorithm replaces the conventional Euclidean distance with the Kullback-Leibler (KL) divergence and incorporates noise and sparse constraint terms into the objective function to mitigate the adverse effects of signal amplification. Unlike traditional methods such as Wiener filtering, the proposed algorithm employs an adaptive Minimum Mean-Square Error-Log Spectral Amplitude (MMSE-LSA) method to estimate the amplitude spectrum of non-stationary signals adaptively across varying signal-to-noise ratios. Comparative experiments demonstrate that the INMF algorithm significantly outperforms existing methods in denoising leakage acoustic signals.

A Novel Job Similarity Index for Career Transition in the Mining Industry

In this study, with the primary goal of capturing ongoing digital transformation and automation impacts on the mining industry and its workforce, we conduct several interviews with mining industry experts in the USA and analyze our survey reports qualitatively and quantitatively through exploratory analysis. After the interpretation of the insights of industry experts, we proceed to generate a personalized and customized data analysis through a novel metric based on skills, knowledge, competencies, and occupational requirements, which quantifies the job similarities for occupations in the mining industry based on the publicly available database of the United States Department of Labor. We utilize text analytics to tokenize and classify the interviews to capture a better understanding of major response categories. The temporal analysis shows that the critical competency needs in the data science and autonomy category increases from 28% in current demands to 43%. In defining our metric, we also calculate Kullback–Leibler (KL) divergence for each job profile that enables determining whether and to what extent that job is transitionary in our test set based on the mean, standard deviation, and kurtosis of each job of interest. Our analysis reveals that the in-group job transitions are significantly easier than the between-group transitions, proving our initial assumptions and common sense. The generated heat maps provide the opportunity to present the gap between the current job and desired job profiles that provide feasible career change options, among others, offering individualized career paths for job seekers and promoting potential job transitions. Through the collection of industry-specific individual employee data, the AI system is envisaged to continue to learn as end users engage with the system, thus creating a central data hub specifically for the future workforce in the mining industry. Although the study has limitations on generalizability for qualitative assessments, it presents itself as a valuable application of how qualitative and quantitative approaches could be of value for future worker training in the mining sector.

Exact Expressions for Kullback-Leibler Divergence for Multivariate and Matrix-Variate Distributions.

The Kullback-Leibler divergence is a measure of the divergence between two probability distributions, often used in statistics and information theory. However, exact expressions for it are not known for multivariate or matrix-variate distributions apart from a few cases. In this paper, exact expressions for the Kullback-Leibler divergence are derived for over twenty multivariate and matrix-variate distributions. The expressions involve various special functions.