Entropy Of Probability Distribution Research Articles

Effect of lake waves on manoeuvring performance of a model ship

ABSTRACT In this study, we investigate how natural environmental conditions affect a training vessel using a ‘mathematical model of wind and wave’ specific to Lake Silm near Kamionka, Poland. Wind and wave measurements on the lake were conducted. We present the vessel’s trajectory under varying conditions: navigating in restricted waters simulating port conditions with speed readings, and during propeller operation. The control system of the nonlinear autonomous object employed two controllers: a linear matrix inequality controller and an overriding trajectory controller. We assess the impact of nonlinearities in the measured waves. It is shown that the maximum entropy probability distribution conforms more closely to the empirical frequency histogram than the Gaussian distribution. The wave height probability distributions diverge from Rayleigh distributions, indicating slight nonlinear effects. These results enhance the mathematical model of Silm Lake’s wave dynamics.

Negation of a probability distribution: An information theoretic analysis

Yager (2014) proposed a transformation for negating the happening of an uncertain event which opposes the occurrence of any uncertain event by redistributing its probability equally among the other outcomes. Yager’s negation applied repeatedly on any probability distribution converges to the uniform distribution (all events having identical probability of occurrence). In most cases, uniform distribution is the maximum entropy probability distribution (MEPD) as it has maximum uncertainty inherent in it. However, if some information is available regarding the occurrence of events associated with a random experiment, then MEPD may or may not be a uniform distribution. In this work, we have first explored some new properties of Yager’s negation and then investigated Yager’s negation when we have some additional information about the probability distribution other than the natural constraints. The MEPD in the presence of this additional information is determined using the negation of probabilities. It is shown that the existence of MEPD (and as a result the maximum entropy) largely depends on the parameters of the additional constraints. Some numerical examples have been considered for comparing the MEPD with and without additional constraints.

Time-dependent uncertainty of critical care transitions in very old patients - lessons for time-limited trials.

Prognostication for patients with critical conditions remains challenging, especially for very old individuals. Time-limited trials (TLT) are used to decrease prognostic uncertainty in the individual patient by monitoring the response to treatment over a pre-determined period of time. However, there are substantial difficulties with determining the length of that period. This study presents a probabilistic method to estimate a suitable duration of a TLT based on temporal profiles of uncertainty about critical care and outcome. The study included very old patients (age ≥ 80 years, n = 1209) from the VIP2 study cohort who were admitted to the ICU for between 2 and 14 days, with respiratory or circulatory support from day 1 and with either no limitations of life-sustaining treatment or a decision to withdraw that treatment, as well as with complete data. Multi-state modelling of critical care trajectories to obtain time-dependent probabilities for transitions between distinct levels of organ support and to outcome states. The extent of uncertainty is quantified by Shannon's entropy of probability distributions at discrete points in time. We detected periods of enhanced prognostic uncertainty of up to 7 days after admission. The duration of these periods depends on patient characteristics at baseline (frailty, severity of critical illness) and the extent of organ support. Time-dependent patterns of uncertainty concerning the response to critical care can inform decisions about the duration of TLTs which may last up to a week in very old patients.

Predicting Abnormal Laboratory Blood Test Results in the Intensive Care Unit Using Novel Features Based on Information Theory and Historical Conditional Probability: Observational Study.

BackgroundRedundancy in laboratory blood tests is common in intensive care units (ICUs), affecting patients’ health and increasing health care expenses. Medical communities have made recommendations to order laboratory tests more judiciously. Wise selection can rely on modern data-driven approaches that have been shown to help identify low-yield laboratory blood tests in ICUs. However, although conditional entropy and conditional probability distribution have shown the potential to measure the uncertainty of yielding an abnormal test, no previous studies have adapted these techniques to include them in machine learning models for predicting abnormal laboratory test results.ObjectiveThis study aimed to address the limitations of previous reports by adapting conditional entropy and conditional probability to extract features for predicting abnormal laboratory blood test results.MethodsWe used an ICU data set collected across Alberta, Canada, which included 55,689 ICU admissions from 48,672 patients. We investigated the features of conditional entropy and conditional probability by comparing the performances of 2 machine learning approaches for predicting normal and abnormal results for 18 blood laboratory tests. Approach 1 used patients’ vitals, age, sex, and admission diagnosis as features. Approach 2 used the same features plus the new conditional entropy–based and conditional probability–based features. Both approaches used 4 different machine learning models (fuzzy model, logistic regression, random forest, and gradient boosting trees) and 10 metrics (sensitivity, specificity, accuracy, precision, negative predictive value [NPV], F1 score, area under the curve [AUC], precision-recall AUC, mean G, and index balanced accuracy) to assess the performance of the approaches.ResultsApproach 1 achieved an average AUC of 0.86 for all 18 laboratory tests across the 4 models (sensitivity 78%, specificity 84%, precision 82%, NPV 75%, F1 score 79%, and mean G 81%), whereas approach 2 achieved an average AUC of 0.89 (sensitivity 84%, specificity 84%, precision 83%, NPV 81%, F1 score 83%, and mean G 84%). We found that the inclusion of the new features resulted in significant differences for most of the metrics in favor of approach 2. Sensitivity significantly improved for 8 and 15 laboratory tests across the different classifiers (minimum P<.001 and maximum P=.04). Mean G and index balanced accuracy, which are balanced performance metrics, also improved significantly across the classifiers for 6 to 10 and 6 to 11 laboratory tests. The most relevant feature was the pretest probability feature, which is the probability that a test result was normal when a certain number of consecutive prior tests was already normal.ConclusionsThe findings suggest that conditional entropy–based features and pretest probability improve the capacity to discriminate between normal and abnormal laboratory test results. Detecting the next laboratory test result is an intermediate step toward developing guidelines for reducing overtesting in the ICU.

Entropic Out-of-Distribution Detection: Seamless Detection of Unknown Examples

In this article, we argue that the unsatisfactory out-of-distribution (OOD) detection performance of neural networks is mainly due to the SoftMax loss anisotropy and propensity to produce low entropy probability distributions in disagreement with the principle of maximum entropy. On the one hand, current OOD detection approaches usually do not directly fix the SoftMax loss drawbacks, but rather build techniques to circumvent it. Unfortunately, those methods usually produce undesired side effects (e.g., classification accuracy drop, additional hyperparameters, slower inferences, and collecting extra data). On the other hand, we propose replacing SoftMax loss with a novel loss function that does not suffer from the mentioned weaknesses. The proposed IsoMax loss is isotropic (exclusively distance-based) and provides high entropy posterior probability distributions. Replacing the SoftMax loss by IsoMax loss requires no model or training changes. Additionally, the models trained with IsoMax loss produce as fast and energy-efficient inferences as those trained using SoftMax loss. Moreover, no classification accuracy drop is observed. The proposed method does not rely on outlier/background data, hyperparameter tuning, temperature calibration, feature extraction, metric learning, adversarial training, ensemble procedures, or generative models. Our experiments showed that IsoMax loss works as a seamless SoftMax loss drop-in replacement that significantly improves neural networks' OOD detection performance. Hence, it may be used as a baseline OOD detection approach to be combined with current or future OOD detection techniques to achieve even higher results.

Probabilistic analysis of shear strength of intact rock in triaxial compression: a case study of Jinping II project

Triaxial compression testing can yield realistic simulations of in-situ rock properties in underground excavation. This paper presents laboratory triaxial compression tests for marble from Jinping II hydropower station project, southwestern China. The test results were used to determine shear strength parameters (i.e., angles of internal friction and cohesive strengths) of the marble. Prominent uncertainties were found among the shear strength parameters. Modelling and quantifying these uncertainties in rock properties are initial, critical steps in probabilistic analysis and design of rock engineering. However, disagreement exists among researchers about which probability distribution should be used to characterize the rock angle of internal friction and the cohesive strength. This paper proposes an alternative universal form of probability distributions for these shear strength parameters, i.e., maximum entropy probability density functions. The entropy distributions are obtained from maximum entropy method, a combination of maximum entropy principle and a second order Akaike information criterion. The maximum entropy principle is used to determine a probability density function that maximizes Shannon’s entropy subjected to constraints in terms of sample integral moments. The second order Akaike information criterion can provide strong consistent and unbiased estimate for Kullback-Leibler entropy, and thus the most unbiased and objective probability density distribution with optimum order can be determined from an available sample. Kolmogorov-Smirnov test is used to establish the maximum entropy probability distributions, which are compared with conventional normal and lognormal probability distributions to illustrate the advantages.

Information Theoretic Study of the Ground-Level Ozone and Its Precursors Over Kolkata, India, During the Summer Monsoon

The present paper reports a study on the dependence of tropospheric ozone on some of its precursors, namely NOx, PM10, and SO2 during summer monsoon over a highly polluted Kolkata region megacity of India. The intrinsic multicollinearity has been explored through Bartlett’s sphericity test. Subsequently, principal component analysis has been implemented to identify the principal components with maximum factor loadings, and accordingly, the most influential ozone precursors have been extracted. Finally, after fitting some continuous and discrete distributions to the extracted precursors, the intrinsic uncertainty has been investigated by maximizing Shannon entropy. The normal distribution has been identified to generate the maximum entropy probability distribution for surface ozone and its precursors. Future directions have been stated at the end of the study.

Determining representative sample size for validation of continuous, large continental remote sensing data

The validation of global remote sensing data comprises multiple methods including comparison to field measurements, cross-comparisons and verification of physical consistency. Physical consistency and cross-comparisons are typically assessed for all pixels of the entire product extent, which requires intensive computing. This paper proposes a statistically representative sampling approach to reduce time and efforts associated with validations of remote sensing data having big data volume. A progressive sampling approach, as typically applied in machine learning to train algorithms, combined with two performance measures, was applied to estimate the required sample size. The confidence interval (CI) and maximum entropy probability distribution were used as indicators to represent accuracy. The approach was tested on 8 continental remote sensing-based data products over the Middle East and Africa. Without the consideration of climate classes, a sample size of 10,000–100,000, dependent on the product, met the nominally set CI and entropy indicators. This corresponds to <0.01 % of the total image for the high-resolution images. All continuous datasets showed the same trend of CI and entropy with increasing sample size. The actual evapotranspiration and interception (ETIa) product was further analysed based on climate classes, which increased the sample size required to meet performance requirements, but was still determined to be significantly less than the entire dataset size. The proposed approach can significantly reduce the processing time while still providing a statistically valid representation of a large remote sensing dataset. This can be useful as more high-resolution remote sensing data becomes available.

Estimation on Reliability Models of Bearing Failure Data

The failure data of bearing products is random and discrete and shows evident uncertainty. Is it accurate and reliable to use Weibull distribution to represent the failure model of product? The Weibull distribution, log-normal distribution, and an improved maximum entropy probability distribution were compared and analyzed to find an optimum and precise reliability analysis model. By utilizing computer simulation technology and k-s hypothesis testing, the feasibility of three models was verified, and the reliability of different models obtained via practical bearing failure data was compared and analyzed. The research indicates that the reliability model of two-parameter Weibull distribution does not apply to all situations, and sometimes, two-parameter log-normal distribution model is more precise and feasible; compared to three-parameter log-normal distribution model, the three-parameter Weibull distribution manifests better accuracy but still does not apply to all cases, while the novel proposed model of improved maximum entropy probability distribution fits not only all kinds of known distributions but also poor information issues with unknown probability distribution, prior information, or trends, so it is an ideal reliability analysis model with least error at present.

Analysis of Finite Buffer Queue: Maximum Entropy Probability Distribution With Shifted Fractional Geometric and Arithmetic Means

A theoretical method based on maximum Shannon entropy framework (MSEF) in the presence of the geometric/or shifted fractional geometric mean of the queue size is applied to study the finite buffer system. Analytical expression of the loss probability for large buffer size is found to depict power law behavior. The maximum entropy framework is extended to incorporate additional shifted fractional arithmetic mean constraint to yield the expression of loss probability which is similar to the one derived by Kim and Shroff who have employed an entirely different approach based on maximum variance asymptotic (MVA) approximation. An important finding is the relationship between fractional order and the Hurst parameter. The advantage of MSEF is that it has enabled one to derive the analytical closed form generalized expression of the probability distribution of queue size in finite buffer system. Further, MSEF with shifted fractional geometric mean constraint gives the same distribution of queue size as the one obtained by maximizing Tsallis entropy subject to the fractional arithmetic mean of queue size.

Modeling and Estimation of a Bivariate Pareto Distribution using the Principle of Maximum Entropy

In this paper we modeled a bivariate Pareto I distribution using the method of principle of maximum entropy probability distribution. Properties of the model are discussed. Further the estimation of the parameters involved in the model is done in two stages using two different methods namely, principle of maximum entropy estimation (POME) and maximum likelihood estimation. From the simulation study conducted to compare the performance of the estimates obtained by the above two methods, we conclude that POME method is performing better than MLE and the two methods are comparable. DOI: http://dx.doi.org/10.4038/sljastats.v15i3.7795

Cation Disorder Regulation by Microstate Configurational Entropy in Photovoltaic Absorber Materials Cu2ZnSn(S,Se)4

Cation disorder plays a major role in the performance deterioration of Cu2ZnSn(S,Se)4 (CZTSSe) photovoltaic (PV) absorbers. Unfortunately the quantitative impact of cation disorder in this material is not well understood. Here, we show that changes in microstate configurational entropy (SMCE), predicted by a combination of statistical mechanics, density functional theory, and phonon calculations, can quantitatively describe cation disorder and the associated properties in CZTSSe. For example, the predicted critical temperature of the second-order phase transition and the thermal expansion anomaly of CZTS based on SMCE are in good agreement with experiments. We further reveal that the more separated is the same kind of cations (Cu, Zn, or Sn), the lower the energy of the corresponding microstate will be, implying the maximum entropy probability distribution of cations at high temperatures. It is suggested that the introduction of SMCE can serve as a framework to quantitatively understand and tailor cation disorder and associated properties for synthesis of the high-quality CZTS-based solar cells.

Low energy structures of lithium-ion battery materials Li(MnxNixCo1−2x)O2 revealed by first-principles calculations

A long-standing issue regarding the low energy structures for the partially disordered cathode materials Li(MnxNixCo1−2x)O2 has been probed by first-principles calculations. It is found that the transitional metals Mn, Ni, and Co in Li(MnxNixCo1−2x)O2 follow the maximum entropy probability distribution (MEPD), instead of the random distribution, according to the distributions of the minimal partial radial distribution functions and the correlation functions. Here, the MEPD is proposed to understand the low energy structures of the partially disordered lithium-ion battery materials.

Empirical Protein Partition Functions

In the present paper, we outline how to construct the partition function for a protein using empirical heat capacity data. The procedure is based on the calculation of a set of energy moments from the temperature dependence of the heat capacity. Given a set of energy moments, one can then use the maximum-entropy method to calculate an approximate energy distribution for the protein; the more energy moments one has, the better the approximation. The energy distribution can then be used to calculate the probability that the molecule is in a given energy level, which, using standard statistical mechanics, gives the degeneracy of the particular energy level. The degeneracy as a function of energy is the central ingredient in the construction of the partition function. Given the partition function, one can calculate all of the thermodynamic functions of the protein (free energy, energy, entropy, heat capacity, and energy probability distribution) as a function of temperature. The three-dimensional plot of the probability that the protein has a given energy at a given temperature tells one graphically (without imposing the assumption) whether or not it is a good approximation to divide the terms in the partition function into two or more groups, reflecting, for example, the presence of distinct native and denatured species.

Maximising entropy on the nonparametric predictive inference model for multinomial data

The combination of mathematical models and uncertainty measures can be applied in the area of data mining for diverse objectives with as final aim to support decision making. The maximum entropy function is an excellent measure of uncertainty when the information is represented by a mathematical model based on imprecise probabilities. In this paper, we present algorithms to obtain the maximum entropy value when the information available is represented by a new model based on imprecise probabilities: the nonparametric predictive inference model for multinomial data (NPI-M), which represents a type of entropy-linear program. To reduce the complexity of the model, we prove that the NPI-M lower and upper probabilities for any general event can be expressed as a combination of the lower and upper probabilities for the singleton events, and that this model can not be associated with a closed polyhedral set of probabilities. An algorithm to obtain the maximum entropy probability distribution on the set associated with NPI-M is presented. We also consider a model which uses the closed and convex set of probability distributions generated by the NPI-M singleton probabilities, a closed polyhedral set. We call this model A-NPI-M. A-NPI-M can be seen as an approximation of NPI-M, this approximation being simpler to use because it is not necessary to consider the set of constraints associated with the exact model.

Comment on “Critique of multinomial coefficient method for evaluating Tsallis and Rényi entropies” by A.S. Parvan

Parvan [A.S. Parvan, Critique of multinomial coefficients method for evaluating Tsallis and Rényi entropies, Physica A 389 (2010) 5645] has recently presented some calculations in order to demonstrate the incorrectness of the results obtained from the generalized multinomial coefficients (GMC) presented in Oikonomou (2007) [1]. According to Parvan, the aforementioned approach of studying maximum entropy probability distributions is erroneous. In this comment I demonstrate that Parvan’s arguments do not hold true and that the obtained results from GMC do not present either mathematical or physical discrepancies.

Phase Transitions in the Distribution of Bipartite Entanglement of a Random Pure State

Using a Coulomb gas method, we compute analytically the probability distribution of the Renyi entropies (a standard measure of entanglement) for a random pure state of a large bipartite quantum system. We show that, for any order q>1 of the Renyi entropy, there are two critical values at which the entropy's probability distribution changes shape. These critical points correspond to two different transitions in the corresponding charge density of the Coulomb gas: the disappearance of an integrable singularity at the origin and the detachment of a single-charge drop from the continuum sea of all the other charges. These transitions, respectively, control the left and right tails of the entropy's probability distribution, as verified also by Monte Carlo numerical simulations of the Coulomb gas equilibrium dynamics.

A boundary corrected expansion of the moments of nearest neighbor distributions

In this article, the moments of nearest neighbor distance distributions are examined. While the asymptotic form of such moments is well-known, the boundary effect has this far resisted a rigorous analysis. Our goal is to develop a new technique that allows a closed-form high order expansion, where the boundaries are taken into account up to the first order. The resulting theoretical predictions are tested via simulations and found to be much more accurate than the first order approximation obtained by neglecting the boundaries. While our results are of theoretical interest, they definitely also have important applications in statistics and physics. As a concrete example, we mention estimating Rényi entropies of probability distributions. Moreover, the algebraic technique developed may turn out to be useful in other, related problems including estimation of the Shannon differential entropy.© 2010 Wiley Periodicals, Inc. Random Struct. Alg., 2010

Transformations preserving the Hausdorff-Besicovitch dimension

Abstract Continuous transformations preserving the Hausdorff-Besicovitch dimension (“DP-transformations”) of every subset of R 1 resp. [0, 1] are studied. A class of distribution functions of random variables with independent s-adic digits is analyzed. Necessary and sufficient conditions for dimension preservation under functions which are distribution functions of random variables with independent s-adic digits are found. In particular, it is proven that any strictly increasing absolutely continuous distribution function from the above class is a DP-function. Relations between the entropy of probability distributions, their Hausdorff-Besicovitch dimension and their DP-properties are discussed. Examples are given of singular distribution functions preserving the fractal dimension and of strictly increasing absolutely continuous functions which do not belong to the DP-class.

On some entropy functionals derived from Rényi information divergence

We consider the maximum entropy problems associated with Rényi Q-entropy, subject to two kinds of constraints on expected values. The constraints considered are a constraint on the standard expectation, and a constraint on the generalized expectation as encountered in nonextensive statistics. The optimum maximum entropy probability distributions, which can exhibit a power-law behaviour, are derived and characterized. The Rényi entropy of the optimum distributions can be viewed as a function of the constraint. This defines two families of entropy functionals in the space of possible expected values. General properties of these functionals, including nonnegativity, minimum, convexity, are documented. Their relationships as well as numerical aspects are also discussed. Finally, we work out some specific cases for the reference measure Q ( x ) and recover in a limit case some well-known entropies.