Kullback-Leibler Research Articles

Just-in-time framework for robust soft sensing based on robust variational autoencoder

Modeling with high-dimensional data subject to abnormal observations have always been a practical interest. In this paper, under the just-in-time learning (JITL) framework, a robust soft sensor modeling approach is developed based on robust Variational Autoencoder (VAE). Unlike the vanilla VAE that extracts features from the given dataset under the Gaussian prior assumption, robust VAE employs Student’s t-distribution as prior distribution to handle abnormal data. Under assumption of the Student’s t-prior, the proposed robust VAE model is capable of describing collected data contaminated with outliers. Once the robust VAE model is trained, each robust feature variable in the latent space can be determined. Subsequently, similarity measure is calculated using robust Kullback-Leibler divergence between two Student’s t-distributions, that is, the distribution of a new data sample and that of each historical data sample. After completing similarity measurement for a query sample, the weights for input-output historical data can be determined. Based on these weighted historical data samples, a robust probabilistic principal component regression (PPCR) is utilized to perform local modeling for prediction. Numerical simulations, including the Tennessee Eastman and Penicillin fermentation benchmark processes, are utilized to validate the proposed JITL-based robust soft sensor modeling method.

Visualizing departures from marginal homogeneity for square contingency tables with ordered categories

Square contingency tables are a special case commonly used in various fields to analyze categorical data. Although several analysis methods have been developed to examine marginal homogeneity (MH) in these tables, existing measures are single-summary ones. To date, a visualization approach has yet to be proposed to intuitively depict the results of MH analysis. Current measures used to assess the degree of departure from MH are based on entropy such as the Kullback-Leibler divergence and do not satisfy distance postulates. Hence, the current measures are not conducive to visualization. Herein we present a measure utilizing the Matusita distance and introduce a visualization technique that employs sub-measures of categorical data. Through multiple examples, we demonstrate the meaningfulness of our analytical framework approach and validate its usefulness to provide insightful interpretations.

Some Improvements on Good Lattice Point Sets

Good lattice point (GLP) sets are a type of number-theoretic method widely utilized across various fields. Their space-filling property can be further improved, especially with large numbers of runs and factors. In this paper, Kullback-Leibler (KL) divergence is used to measure GLP sets. The generalized good lattice point (GGLP) sets obtained from linear-level permutations of GLP sets have demonstrated that the permutation does not reduce the criterion maximin distance. This paper confirms that linear-level permutation may lead to greater mixture discrepancy. Nevertheless, GGLP sets can still enhance the space-filling property of GLP sets under various criteria. For small-sized cases, the KL divergence from the uniform distribution of GGLP sets is lower than that of the initial GLP sets, and there is nearly no difference for large-sized points, indicating the similarity of their distributions. This paper incorporates a threshold-accepting algorithm in the construction of GGLP sets and adopts Frobenius distance as the space-filling criterion for large-sized cases. The initial GLP sets have been included in many monographs and are widely utilized. The corresponding GGLP sets are partially included in this paper and will be further calculated and posted online in the future. The performance of GGLP sets is evaluated in two applications: computer experiments and representative points, compared to the initial GLP sets. It shows that GGLP sets perform better in many cases.

Enhanced predictive PDF control of stochastic distribution systems with neural network compensation and its application

Compared to the conventional control algorithms that use mean and variance as indicators, predictive probability density function (PDF) control can effectively handle the output PDF control problem of non-Gaussian stochastic distribution systems. However, the existing predictive PDF control method does not consider the construction error between output PDF and weight, thus the control performance is still unsatisfactory. Therefore, this paper proposes a new Enhanced Predictive PDF control method (En-PDF) to improve the output PDF control performance of stochastic distribution systems. The proposed method mainly consists of two parts: the predictive PDF control part and the neural network compensation control part aiming to reduce the bias of output PDF. First, the Radical Basis Functions (RBFs) are used to approximate the PDF of the stochastic systems output, and then a prediction model representing the relationship between input and weight is established using the subspace identification algorithm to design the predictive PDF control for the stochastic systems. Next, the Kullback-Leibler (KL) divergence is used to measure the similarity between the output PDF and the set PDF, combined with the weight error and compensation to design a new performance index. Based on this, the parameters of the neural network are adjusted using the gradient descent algorithm to obtain the optimal compensation, and the stability and tracking performance of the proposed algorithm are analyzed using inductive reasoning method. Finally, the predictive PDF control input with the compensation work together on the controlled plant to achieve high-performance control of the output PDF of non-Gaussian stochastic distribution systems. Both simulation experiments and physical control experiments validate the effectiveness and superiority of the proposed method.

Evaluating model fit for type II censored data: a Bayesian non-parametric approach based on the Kullback-Leibler divergence estimation

Model checking evaluates the appropriateness of a statistical model based on the observed data, and it is essential to make valid statistical analyses. In this paper, a new procedure for model checking type II censored data is proposed. This procedure combines the Kullback-Leibler divergence, the Dirichlet process, and the relative belief ratio. The method is implemented via a computational algorithm and is explained through several examples.

Asymptotic Properties of a Statistical Estimator of the Jeffreys Divergence: The Case of Discrete Distributions

We investigate the asymptotic properties of the plug-in estimator for the Jeffreys divergence, the symmetric variant of the Kullback–Leibler (KL) divergence. This study focuses specifically on the divergence between discrete distributions. Traditionally, estimators rely on two independent samples corresponding to two distinct conditions. However, we propose a one-sample estimator where the condition results from a random event. We establish the estimator’s asymptotic unbiasedness (law of large numbers) and asymptotic normality (central limit theorem). Although the results are expected, the proofs require additional technical work due to the randomness of the conditions.

Discounted fully probabilistic design of decision rules

Axiomatic fully probabilistic design (FPD) of optimal decision rules strictly extends the decision making (DM) theory represented by Markov decision processes (MDP). This means that any MDP task can be approximated by an explicitly found FPD task whereas many FPD tasks have no MDP equivalent. MDP and FPD model the closed loop — the coupling of an agent and its environment — via a joint probability density (pd) relating the involved random variables, referred to as behaviour. Unlike MDP, FPD quantifies agent's aims and constraints by an ideal pd. The ideal pd is high on the desired behaviours, small on undesired behaviours and zero on forbidden ones. FPD selects the optimal decision rules as the minimiser of Kullback-Leibler's divergence of the closed-loop-modelling pd to its ideal twin. The proximity measure choice follows from the FPD axiomatics.MDP minimises the expected total loss, which is usually the sum of discounted partial losses. The discounting reflects the decreasing importance of future losses. It also diminishes the influence of errors caused by:▪ the imperfection of the employed environment model;▪ roughly-expressed aims;▪ the approximate learning and decision-rules design.The established FPD cannot currently account for these important features. The paper elaborates the missing discounted version of FPD. This non-trivial filling of the gap in FPD also employs an extension of dynamic programming, which is of an independent interest.

Feature Vector Effectiveness Evaluation for Pattern Selection in Computational Lithography

Pattern selection is crucial for optimizing the calibration process of optical proximity correction (OPC) models in computational lithography. However, it remains a challenge to achieve a balance between representative coverage and computational efficiency. This work presents a comprehensive evaluation of the feature vectors’ (FVs’) effectiveness in pattern selection for OPC model calibration, leveraging key performance indicators (KPIs) based on Kullback–Leibler divergence and distance ranking. Through the construction of autoencoder-based FVs and fast Fourier transform (FFT)-based FVs, we compare their efficacy in capturing critical pattern features. Validation experimental results indicate that autoencoder-based FVs, particularly augmented with the lithography domain knowledge, outperform FFT-based counterparts in identifying anomalies and enhancing lithography model performance. These results also underscore the importance of adaptive pattern representation methods in calibrating the OPC model with evolving complexities.

Remote Sensing Image Segmentation Based on Probabilistic Fuzzy Local Information Clustering

Abstract. In this paper, a remote sensing image segmentation based on probabilistic fuzzy local information clustering algorithm is proposed. First, assuming that the spectral measure within the same ground object follows the same probability distribution. The dissimilarity between pixel and object is modeled by the negative logarithm of the Gaussian probability density function. It can improve the noise sensitive problem caused by the Euclidean distance which can only describe the data with isotropic distribution. Then, in order to consider the effect of local spatial constraint, on the one hand, the probability measure is used to modify the local fuzzy factor to establish the dissimilarity measure with spatial constraints. On the other hand, the hidden Markov random field is used to model the prior probability model of pixel membership. Next, the entropy regularization term of the objective function is built by combining the Kullback-Leibler(KL) maximum entropy model to further improve the robustness and noise resistance. The qualitative and quantitative analysis of simulated image and different types of real remote sensing images show that the proposed algorithm can effectively overcome the above problems and further improve the accuracy of image segmentation to over 95%.

Local inconsistency detection using the Kullback–Leibler divergence measure

BackgroundThe standard approach to local inconsistency assessment typically relies on testing the conflict between the direct and indirect evidence in selected treatment comparisons. However, statistical tests for inconsistency have low power and are subject to misinterpreting a p-value above the significance threshold as evidence of consistency.MethodsWe propose a simple framework to interpret local inconsistency based on the average Kullback–Leibler divergence (KLD) from approximating the direct with the corresponding indirect estimate and vice versa. Our framework uses directly the mean and standard error (or posterior mean and standard deviation) of the direct and indirect estimates obtained from a local inconsistency method to calculate the average KLD measure for selected comparisons. The average KLD values are compared with a semi-objective threshold to judge the inconsistency as acceptably low or material. We exemplify our novel interpretation approach using three networks with multiple treatments and multi-arm studies.ResultsAlmost all selected comparisons in the networks were not associated with statistically significant inconsistency at a significance level of 5%. The proposed interpretation framework indicated 14%, 66%, and 75% of the selected comparisons with an acceptably low inconsistency in the corresponding networks. Overall, information loss was more notable when approximating the posterior density of the indirect estimates with that of the direct estimates, attributed to indirect estimates being more imprecise.ConclusionsUsing the concept of information loss between two distributions alongside a semi-objectively defined threshold helped distinguish target comparisons with acceptably low inconsistency from those with material inconsistency when statistical tests for inconsistency were inconclusive.

Simple variational inference based on minimizing Kullback–Leibler divergence

Simple variational inference based on minimizing Kullback–Leibler divergence

Disrupted gray matter connectome in vestibular migraine: a combined machine learning and individual-level morphological brain network analysis

BackgroundAlthough gray matter (GM) volume alterations have been extensively documented in previous voxel-based morphometry studies on vestibular migraine (VM), little is known about the impact of this disease on the topological organization of GM morphological networks. This study investigated the altered network patterns of the GM connectome in patients with VM.MethodsIn this study, 55 patients with VM and 57 healthy controls (HCs) underwent structural T1-weighted MRI. GM morphological networks were constructed by estimating interregional similarity in the distributions of regional GM volume based on the Kullback–Leibler divergence measure. Graph-theoretical metrics and interregional morphological connectivity were computed and compared between the two groups. Partial correlation analyses were performed between significant GM connectome features and clinical parameters. Logistic regression (LR), support vector machine (SVM), and random forest (RF) classifiers were used to examine the performance of significant GM connectome features in distinguishing patients with VM from HCs.ResultsCompared with HCs, patients with VM exhibited increased clustering coefficient and local efficiency, as well as reduced nodal degree and nodal efficiency in the left superior temporal gyrus (STG). Furthermore, we identified one connected component with decreased morphological connectivity strength, and the involved regions were mainly located in the STG, temporal pole, prefrontal cortex, supplementary motor area, cingulum, fusiform gyrus, and cerebellum. In the VM group, several connections in the identified connected component were correlated with clinical measures (i.e., symptoms and emotional scales); however, these correlations did not survive multiple comparison corrections. A combination of significant graph- and connectivity-based features allowed single-subject classification of VM versus HC with significant accuracy of 77.68%, 77.68%, and 72.32% for the LR, SVM, and RF models, respectively.ConclusionPatients with VM had aberrant GM connectomes in terms of topological properties and network connections, reflecting potential dizziness, pain, and emotional dysfunctions. The identified features could serve as individualized neuroimaging markers of VM.

Sexual somatic dimorphism in connection with ecological factors

Introduction. Analysis of correlations of sexual somatic dimorphism (SD) with the degree of urbanization (quantity of population) and geographic latitude of the residence place is carried out using the material of monitoring of Russian children and adolescents. Material and methods. The study embraces the wide specter of ethno-territorial samples of children and adolescents from Russia and neighboring countries (literary data), examined through the historic interval 1930-2010 years. To estimate the direction and degree of associations of anthropometric traits (height, weight, chest girth) and ecological factors the classic correlation analysis for pairwise combinations of variables was implemented for age groups of 9-year-old children and 13-year-old adolescents. The quantitative estimation of SD was carried out using Kullback divergence, the analogue of Makhalanobis distance. Results. Significant correlations of SD of height with both ecological factors for 13-year-old adolescents and SD of weight for 9-year-old children with the quantity of population were fixed on base of the whole data massive. The more homogeneous data base (only Slavonic groups examined in 1960s-1970s) shows significant correlations of latitude with SD of height for 13-year-old adolescents only in combination with the quantity of population of the residence place up to 500 thousands of people; and latitude with SD of weight for 9-year-old children only for million-plus cities. Significant correlations of quantity of population were fixed for SD of height of 13-year-old adolescents in more southern ranges of latitude (37-50degrees), for SD of weight for 9-year-old children in more northern ranges of latitude (50 degrees and more). Conclusion. The study shows complex superposition of ecological factors, which influence somatic variability of children through growth process. High degree of urbanization conceal anthropoecological correlations SD-latitude for ecosensitive age 13 years, but reveal them for “neutral” age, 9 years, with the opposite vector. The most southern range of latitudes (about 40 degrees) reveal positive associations of SD with the degree of urbanization for 13-year-olds, which are not fixed in more northern latitudes. More frequent anthropoecological correlations of SD of height, as genetic marker, of 13-year-olds, and SD of weight as the marker of nutrition status and life style of 9-year-olds reflect the physiological essence of age periods - increase of intersex differences for adolescents and decrease of intersex differences for second childhood children

Comparison of traditional and deep learning optimisation for the design of acoustic metamaterials

This paper presents a comparative analysis of traditional and Deep learning (DL) optimisation approaches for AMM design. A case study comparing a generalised pattern search (GPS) optimisation to a previously trained inverse design deep neural network (IDDNN) of an acoustic meta-absorber using various metrics is considered. For benchmarking, a baseline design is realised with arbitrarily chosen geometric parameters in a CAD software. This design is 3D printed and experimentally tested. The peak absorption and corresponding resonant frequency obtained from experimental absorption, along with other geometric parameters, are used as inputs to IDDNN and GPS. Results show that IDDNN provides superior inference speed, requiring 7.24e-3 [s] to generate predictions on a single input instance, compared to GPS which took about 30.5464 [s]. While GPS exhibits slightly closer alignment to experimental curve based on Kullback-Leibler divergence (2.79e-3) compared to IDDNN (3.31e-3), IDDNN still achieves a slightly higher accuracy of 92%, compared to 90% for GPS. It is emphasised that trade-offs may be necessary in practice to determine the most suitable design approach for a given application. Traditional optimisation offers ease of design flexibility, allowing for on-the-fly model improvement and modification. In contrast, DL may require re-training, leading to delays in process workflow

Separation of music signal spectrograms using copula models

The focus of this paper is on utilizing a copula-based technique for separating music signals. In this method, copulas are utilized to model the dependency structure of the source components. The objective is to minimize the Kullback-Leibler divergence between the copula density of the estimated source components and the copula density of the source components, which is assumed to be unknown. We give an application for the separation of stereo music recording using different copula models. We show through this application that by modeling the dependency between music signals, we can gain insights into the underlying structure of music signals and also we can ameliorate the separation performance.

Environmental management and restoration under unified risk and uncertainty using robustified dynamic Orlicz risk

Environmental management and restoration should be designed such that the risk and uncertainty owing to nonlinear stochastic systems can be successfully addressed. We apply the robustified dynamic Orlicz risk to the modeling and analysis of environmental management and restoration to consider both the risk and uncertainty within a unified theory. We focus on the control of a jump-driven hybrid stochastic system that represents macrophyte dynamics. The dynamic programming equation based on the Orlicz risk is first obtained heuristically, from which the associated Hamilton–Jacobi–Bellman (HJB) equation is derived. In the proposed Orlicz risk, the risk aversion of the decision-maker is represented by a power coefficient that resembles a certainty equivalence, whereas the uncertainty aversion is represented by the Kullback–Leibler divergence, in which the risk and uncertainty are handled consistently and separately. The HJB equation includes a new state-dependent discount factor that arises from the uncertainty aversion, which leads to a unique, nonlinear, and nonlocal term. The link between the proposed and classical stochastic control problems is discussed with a focus on control-dependent discount rates. We propose a finite difference method for computing the HJB equation. Finally, the proposed model is applied to an optimal harvesting problem for macrophytes in a brackish lake that contains both growing and drifting populations.

Dynamic robustness evaluation for automated model selection in operation

Context:The increasing use of artificial neural network (ANN) classifiers in systems, especially safety-critical systems (SCSs), requires ensuring their robustness against out-of-distribution (OOD) shifts in operation, which are changes in the underlying data distribution from the data training the classifier. However, measuring the robustness of classifiers in operation with only unlabeled data is challenging. Additionally, machine learning engineers may need to compare different models or versions of the same model and switch to an optimal version based on their robustness. Objective:This paper explores the problem of dynamic robustness evaluation for automated model selection. We aim to find efficient and effective metrics for evaluating and comparing the robustness of multiple ANN classifiers using unlabeled operational data. Methods:To quantitatively measure the differences between the model outputs and assess robustness under OOD shifts using unlabeled data, we choose distance-based metrics. An empirical comparison of five such metrics, suitable for higher-dimensional data like images, is performed. The selected metrics include Wasserstein distance (WD), maximum mean discrepancy (MMD), Hellinger distance (HL), Kolmogorov–Smirnov statistic (KS), and Kullback–Leibler divergence (KL), known for their efficacy in quantifying distribution differences. We evaluate these metrics on 20 state-of-the-art models (ten CIFAR10-based models, five CIFAR100-based models, and five ImageNet-based models) from a widely used robustness benchmark (RobustBench) using data perturbed with various types and magnitudes of corruptions to mimic real-world OOD shifts. Results:Our findings reveal that the WD metric outperforms others when ranking multiple ANN models for CIFAR10- and CIFAR100-based models, while the KS metric demonstrates superior performance for ImageNet-based models. MMD can be used as a reliable second option for both datasets. Conclusion:This study highlights the effectiveness of distance-based metrics in ranking models’ robustness for automated model selection. It also emphasizes the significance of advancing research in dynamic robustness evaluation.

Extended matrix modeling of integrated energy systems considering network dynamic characteristics and source–load uncertainty

A refined and efficient modeling approach is indispensable for enhancing the timeliness of scheduling strategies for the integrated energy system (IES). Non-flowing variables such as pressure and temperature introduced by energy devices, however, worsen the complexity while improving the completeness for models. Nonlinear and uncertain variables associated with network dynamic characteristics and source–load uncertainty in the systems seriously postpone the modeling processes. An extended matrix modeling approach is proposed in this paper to tackle such issues. New nodes, branches, definition rules and topological matrices are first presented to create an extended matrix model that is compatible with both flowing and non-flowing variables. To boost the modeling efficiency, adaptive piecewise linearization (APL) with extended matrix form and modified Taylor expansion are applied to approximate various nonlinear characteristics. The convex approximation based on Kullback–Leibler (KL) divergence and the strong duality theory characterized by Wasserstein distance are adopted to deal with uncertain variables, which adapt to combine with the above matrix model. A linearized matrix modeling framework applicable to highly automated by computers is thus established. The distributionally robust optimization (DRO) scheduling strategy is verified in a standard test system with the proposed modeling. The effect of dynamic characteristics and uncertain factors on the strategy is discussed. The results show that the extended matrix model simulates the response for real physical systems more effectively, which exhibits the better performances in modeling, objectives and scheduling strategies.

A Nonlinear Filter based on GSK for Relative Navigation Using Relative Orbital Elements

Due to the sensitivity of the relative orbital elements model to the measurement noise, the non-stationary heavy-tailed noise(NSHT) induced by the time-varying environment during the relative navigation usually leads to filter divergence. To address this problem, a new nonlinear filter based on Gaussian-Student's-Multivariate K(GSK) mixture distribution is proposed in this paper. A Dirichlet stochastic mixture vector fusing Gaussian, Student's t, and Multivariate K distributions is introduced, thus proposing a GSK mixture distribution modeling measurement likelihood; then the Kullback-Leibler Divergence (KLD) of the true posteriori probability density function(PDF) and the approximate posteriori PDF are minimized by a variational Bayesian(VB) technique to solve for the state and parameter approximate a posteriori estimations, and finally a new nonlinear filter based on the GSK mixture distribution is derived for angles-only relative navigation in time-varying environments. Simulation outcomes indicate that the filter can realize state estimation in non-stationary states effectively with 45.16% higher estimation accuracy than the existing advanced filters.

A novel self-training framework for semi-supervised soft sensor modeling based on indeterminate variational autoencoder

In modern industrial processes, the high acquisition cost of labeled data can lead to a large number of unlabeled samples, which greatly impacts the accuracy of traditional soft sensor models. To this end, this paper proposes a novel semi-supervised soft sensor framework that can fully utilize the unlabeled data to expand the original labeled data, and ultimately improve the prediction accuracy. Specifically, an indeterminate variational autoencoder (IVAE) is first proposed to obtain the pseudo-labels of unlabeled data as well as their uncertainty. On this basis, the IVAE-based self-training (ST-IVAE) framework is further naturally proposed to expand the original small labeled dataset through continuous circulation. Among them, a variance-based oversampling (VOS) strategy is introduced to better utilize the pseudo-label uncertainty. By determining similar sample sets through the comparison of Kullback-Leibler (KL) divergence obtained by the proposed IVAE model, each sample can be independently modeled for prediction. The effectiveness of the proposed semi-supervised framework is verified on two real industrial processes and the realization results illustrate that the ST-IVAE framework can still predict well even in the presence of missing input data compared to state-of-the-art methodologies in addressing semi-supervised soft sensing challenges.