Kullback-Leibler Number Research Articles

The Kullback–Leibler Number and Thermodynamics

In this article, we establish a connection between the Kullback–Leibler number and thermodynamics in using the canonical ensemble themodynamic probability in the most probable distribution. As a result, we obtain many expressions, each connecting the Kullback–Leibler number to one, two or many of the following thermodynamic quantities: the canonical ensemble partition function, the entropy of the canonical ensemble in the most probable distribution, the canonical ensemble average of energy, the Helmholtz free energy, the standard deviation in the energy of the canonical ensemble, the heat capacity at constant volume, the absolute temperature T and the temperature parameter β.

Measures of dependence for the multivariate t distribution with applications to the stock market

Two measures of dependence for multivariate t and Cauchy random variables are developed based on Kullback–Leibler number. The mutual information number T(X) is obtained in a closed expression form, as well as its asymptotic distribution. A dependence coefficient ρ1, is defined (based on the Kullback–Leibler number) with the properties of ρ1=0 indicating independence and ρ1=1indicating degeneracy. Two real life examples from the stock market are used to analyze the level of dependence and correlation among stocks.

An asymptotic test of independence for multivariate t and cauchy random variables with applications

An asymptotic test for independence for multivariate t and Cauchy random variables is developed based on Kullback-Leibler number. The mutual information number T( X) is obtained for the multivariate t and Cauchy distribution in a closed expression form as well as the moment generating function and asymptotic distribution for its sample estimator T ̂ (X) . This test is applied to the multiple regression problem of stocks returns.

A test for population collinearity a Kullback-Leibler information approach

We study collinearity as a population characteristic. The intention of this article is to measure degree of population collinearity and statistically determine whether collinearity exists in the original population of the predictors. We use concepts of cybernetics and information theory to define and measure the degree of collinearity by means of the Kullback-Leibler number. This Kullback-Leibler Information approach proposes a test for detecting population collinearity and analyzing its sample distribution. We provide 5% and 1% cutoff-point tables for the test H o) “no population collinearity” and also provide formulas for the cumulative distribution function of test statistic for collinearity.

Sample mean based index policies by O(log n) regret for the multi-armed bandit problem

We consider a non-Bayesian infinite horizon version of the multi-armed bandit problem with the objective of designing simple policies whose regret increases slowly with time. In their seminal work on this problem, Lai and Robbins had obtained a O(log n) lower bound on the regret with a constant that depends on the Kullback–Leibler number. They also constructed policies for some specific families of probability distributions (including exponential families) that achieved the lower bound. In this paper we construct index policies that depend on the rewards from each arm only through their sample mean. These policies are computationally much simpler and are also applicable much more generally. They achieve a O(log n) regret with a constant that is also based on the Kullback–Leibler number. This constant turns out to be optimal for one-parameter exponential families; however, in general it is derived from the optimal one via a ‘contraction' principle. Our results rely entirely on a few key lemmas from the theory of large deviations.

Sample mean based index policies by O(log n) regret for the multi-armed bandit problem

We consider a non-Bayesian infinite horizon version of the multi-armed bandit problem with the objective of designing simple policies whose regret increases slowly with time. In their seminal work on this problem, Lai and Robbins had obtained a O(log n) lower bound on the regret with a constant that depends on the Kullback–Leibler number. They also constructed policies for some specific families of probability distributions (including exponential families) that achieved the lower bound. In this paper we construct index policies that depend on the rewards from each arm only through their sample mean. These policies are computationally much simpler and are also applicable much more generally. They achieve a O(log n) regret with a constant that is also based on the Kullback–Leibler number. This constant turns out to be optimal for one-parameter exponential families; however, in general it is derived from the optimal one via a ‘contraction' principle. Our results rely entirely on a few key lemmas from the theory of large deviations.

Limit theorems for the logarithm of sample spacings

Many statistical problems can be reformulated in terms of tests of uniformity. Some strong laws of large numbers and a central limit theorem for the logarithm of transformed spacings are obtained. These theorems provide a characterization of the uniform distribution. A general information-type inequality is deduced which gives a quantitative measurement (using the Kullback-Leibler number) of the discrepancy between an arbitrary distribution and the uniform distribution.

A Probabilistic Distance Measure for Hidden Markov Models

We propose a probabilistic distance measure for measuring the dissimilarity between pairs of hidden Markov models with arbitrary observation densities. The measure is based on the Kullback-Leibler number and is consistent with the reestimation technique for hidden Markov models. Numerical examples that demonstrate the utility of the proposed distance measure are given for hidden Markov models with discrete densities. We also discuss the effects of various parameter deviations in the Markov models on the resulting distance, and study the relationships among parameter estimates (obtained from reestimation), initial guesses of parameter values, and observation duration through the use of the measure.