Kullback-Leibler Information Measure Research Articles

European regional science association

The specification of cross-sectional models is usually solved following a traditional procedure, highly supported by practitioners. In the first step, a simple model is proposed that will be subsequently improved with different elements if the evidence so advises. This procedure expedites the econometric solution and fits well into the Lagrange Multiplier approach, which contributes to explain its current popularity. However, there are other methods that could also be used, and some of them are considered in this paper. Specifically, we turn our attention to the Vuong test, developed in the context of the Kullback-Leibler information measure. This test represents an intermediate solution between the complexity inherent in the Wald test and the simplicity of the Lagrange Multiplier principle.

Iterative and recursive estimators for hidden Markov errors-in-variables models

In this paper we propose maximum-likelihood (ML) estimation of errors in variables models with finite-state Markovian disturbances. Such models have applications in econometrics, speech processing, communication systems, and neurobiological signal processing. We derive the maximum likelihood (ML) model estimates using the expectation maximization (EM) algorithm. Then two recursive or "on-line" estimation schemes are derived for estimating such models. The first on-line algorithm is based on the EM algorithm and uses stochastic approximations to maximize the Kullback-Leibler (KL) information measure. The second on-line algorithm we propose is a gradient-based scheme and uses stochastic approximations to maximize the log likelihood.

On-line estimation of dynamic shock-error models based on the Kullback Leibler information measure

Develops two sequential or "on-line" estimation schemes in the time domain for dynamic shock-error models which are special cases of errors-in-variables models. The author's approach utilizes a state-space representation of the model, Kalman filtering techniques, and on-line algorithms. The first on-line algorithm is based on the expectation-maximization algorithm and uses a recursive Gauss-Newton scheme to maximize the Kullback Leibler information measure. The second on-line algorithm the author proposes is a gradient-based scheme and uses stochastic approximations to maximize the log likelihood. In comparison to the off-line maximum likelihood estimation scheme used in Ghosh (1989), the author's on-line algorithms have significantly reduced computational costs and negligible memory requirements. Simulations illustrate the satisfactory performance of the algorithms in estimating errors-in-variables systems with parameters that vary slowly with time or undergo infrequent jump changes.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Kullback-Leibler information measure for studying convergence rates of densities and distributions

The Kullback-Leibler (KL) information measure l(f/sub 1/:f/sub 2/) is proposed as an index for studying rates of convergence of densities and distribution functions. To this end, upper bounds in terms of l(f/sub 1/:f/sub 2/) for several distance functions for densities and for distribution functions are obtained. Many illustrations of the use of this technique are given. It is shown, for example, that the sequence of KL information measures converges to zero more slowly for a normalized sequence of gamma random variables converging to its limiting normal distribution than for a normalized sequence of largest order statistics from an exponential distribution converging to its limiting extreme value distribution. Furthermore, a sequence of KL information measures for log-normal random variables approaching normality converges more slowly to zero than for a sequence of normalized gamma random variables.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Relative sufficiency

By the classical definition of a sufficient statistic, either a statistic is or is not sufficient. A measure called relative sufficiency, by which the degree of sufficiency can be defined for the problem of random parameter estimation is presented. It is shown that relative sufficiency is equivalent to the Kullback-Leibler information measure. Its calculation and interpretation are illustrated for the estimation of a scalar random variable. >

On-line estimation of hidden Markov model parameters based on the Kullback-Leibler information measure

Sequential or online hidden Markov model (HMM) signal processing schemes are derived, and their performance is illustrated by simulation. The online algorithms are sequential expectation maximization (EM) schemes and are derived by using stochastic approximations to maximize the Kullback-Leibler information measure. The schemes can be implemented either as filters or fixed-lag or sawtooth-lag smoothers. They yield estimates of the HMM parameters including transition probabilities, Markov state levels, and noise variance. In contrast to the offline EM algorithm (Baum-Welch scheme), which uses the fixed-interval forward-backward scheme, the online schemes have significantly reduced memory requirements and improved convergence, and they can estimate HMM parameters that vary slowly with time or undergo infrequent jump changes. Similar techniques are used to derive online schemes for extracting finite-state Markov chains imbedded in a mixture of white Gaussian noise (WGN) and deterministic signals of known functional form with unknown parameters. >

Sequential algorithms for parameter estimation based on the Kullback-Leibler information measure

Methods of stochastic approximation are used to convert iterative algorithms for maximizing the Kullback-Leibler information measure into sequential algorithms. Special attention is given to the case of incomplete data, and several algorithms are presented to deal with situations of this kind. The application of these algorithms to the identification of finite impulse response systems is considered. >

Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses

In this paper, we propose a classical approach to model selection. Using the Kullback-Leibler Information measure, we propose simple and directional likelihood-ratio tests for discriminating and choosing between two competing models whether the models are nonnested, overlapping or nested and whether both, one, or neither is misspecified. As a prerequisite, we fully characterize the asymptotic distribution of the likelihood ratio statistic under the most general conditions.

Minimaxity and nonminimaxity of a preliminary test estimator for the multivariate normal mean

The problem is to estimate the mean of ap-dimensional normal distribution in the situation where there is vague information that the mean vector might be equal to zero vector. Minimax property of the preliminary test estimator obtained by the use of AIC (Akaike's Information Criterion) procedure is discussed under a loss function which is based on Kullback-Leibler information measure and evaluates both an error of model selection and that of estimation. Whenp is even, the minimaxity is shown to hold for small values ofp but not for large values.

On the information criterion for selecting regressors

It has been asserted in the past that any Bayesian treatment of the model selection problem in regression using some form of continuous loss structure would always lead to using the largest possible model (Leamer 1979; Chow 1981). We show in this paper that, provided the distinction between the choice of a model and the estimation of its parameters is maintained, the Kullback-Leibler information measure can be used in a Bayesian context to derive a criterion which may lead to parsimony of parameters in regression analysis.

A derivation of the information criteria for selecting autoregressive models

The Akaike information criterion, AIC, for autoregressive model selection is derived by adopting −2Ttimes the expected predictive density of a future observation of an independent process as a loss function, whereTis the length of the observed time series. The conditions under which AIC provides an asymptotically unbiased estimator of the corresponding risk function are derived. When the unbiasedness property fails, the use of AIC is justified heuristically. However, a method for estimating the risk function, which is applicable for all fitted orders, is given. A derivation of the generalized information criterion, AICα, is also given; the loss function used being obtained by a modification of the Kullback-Leibler information measure. Results paralleling those for AIC are also obtained for the AICαcriterion.

A derivation of the information criteria for selecting autoregressive models

The Akaike information criterion, AIC, for autoregressive model selection is derived by adopting −2T times the expected predictive density of a future observation of an independent process as a loss function, where T is the length of the observed time series. The conditions under which AIC provides an asymptotically unbiased estimator of the corresponding risk function are derived. When the unbiasedness property fails, the use of AIC is justified heuristically. However, a method for estimating the risk function, which is applicable for all fitted orders, is given. A derivation of the generalized information criterion, AICα, is also given; the loss function used being obtained by a modification of the Kullback-Leibler information measure. Results paralleling those for AIC are also obtained for the AICα criterion.

Minimaxity of a preliminary test estimator for the mean of normal distribution

The problem is to estimate the mean of the normal distribution under the situation where there is vague information that the mean might be equal to zero. A minimax property of the preliminary test estimator obtained by the use of AIC (Akaike information Criterion) procedure is proved under a loss function based on the Kullback-Leibler information measure.

Minimum Cross Entropy and Informational Approaches for Spectrum Estimation

Abstract In this paper, we present the methods to estimate spectra based upon the principle of minimum cross entropy. First, we review briefly the principle of minimum cross entropy and introduce the spectral estimation method by Shore and Johnson. Next we present an extension of their work in case of partial correlation data obtained uncertainly. Furthermore, by the principle of minimum cross entropy, we show a direct extension of Burg's continuous spectral estimation method which has been developed based on the principle of maximum entropy. The extension work is achieved by applying the spectral expression of cross entropy (Kullback Leibler's information measure of discrimination) between two zero mean Gaussian stationary time series, where a prior spectrum is assumed as the one generated by autoregressive (AR) model. Two numerical examples are included for verifying our proposed spectral estimation methods based on the principle of minimum cross entropy.

Admissibility of some preliminary test estimators for the mean of normal distribution

Let a random variableX follow ap-variate normal distributionNp(θ, Ip) with an unknownp×1 vector θ andp×p identity matrixIp. The admissibility of a preliminary test estimator using AIC (Akaike's Information Criterion) procedure will be shown ifp=1 and its inadmissibility will be shown ifp≧3 under the loss function based on Kullback-Leibler information measure. Furthermore the two sample case is also considered.

An Extension of Kullback-Leibler Information Measure to Linear Structural Relations

This paper attemps to clarify and simplify the theory of identification of linear structural relations (the identification of linear functional relations between random variables all of which are subject to observation error). It is based on the ideas of Kullback-Leibler information measure as used recently by Akaike (1973,1974,1976).The paper begins with a review of the ideas and formulae of Kullback-Leibler information measure as they apply to the least-square problem. The linear structural relation problem is then introduced and the Gaussian case discussed by maximum likelihood. It is shown how, from the maximum-likelihood theory, a measure of deviation arises which appears to be natural extension of the Kullback-Leibler information measure of this problem.Finally the formulae are worked out for the systems-analysis case of identification of linear systems with simultaneous input- and output noise.