Kullback-Leibler Information Number Research Articles

Asymptotic optimality of the square-root transformation on the gamma distribution using the Kullback–Leibler information number criterion

The Kullback–Leibler information number criterion is a useful criterion for determining an optimal power transformation to approximate the target distribution. When the normal distribution with a fixed variance is set as the target distribution, this criterion may be used to show the asymptotic optimality of the square-root transformation. In particular, when the Box–Cox transformation corresponding to the square-root transformation is applied, the Kullback–Leibler information number of the transformed gamma distribution with the scale parameter equal to the fixed variance converges to zero asymptotically as the shape parameter approaches infinity. That provides a stronger result on the mode of convergence of the variance-stabilizing power transformation on the gamma distribution. Moreover, an analogous result may be obtained by setting the Laplace distribution with a fixed scale parameter as the target distribution. This result utilizes a novel application of the asymptotic expansion for the normalized upper incomplete gamma function at the transition point.

General Gaussian estimation

This article proposes a general Gaussian estimation approach for situations where the implementation of maximum likelihood estimation is difficult. The primary task is to construct a Gaussian estimation function by putting exact expressions of the mean vector and the variance–covariance matrix of the response vector into the log-likelihood function of the multivariate normal distribution. A Gaussian estimator is derived by maximizing the Gaussian estimation function. This construction can induce an optimality condition that the true parameter vector is the unique maximizer of the expected value of the Gaussian estimation function. The optimality condition is equally important to that given by the Kullback–Leibler information number in the maximum likelihood approach. It is a major condition in the derivation of nice theoretical properties such as consistency and asymptotic normality. The general Gaussian estimation approach can significantly reduce the computational burden when the log-likelihood function of a statistical model contains intractable high-dimensional integrals. By applying it to the Poisson-lognormal model, a special case of generalized linear mixed effect models, a closed-form (i.e., without intractable integrals) estimation approach for fixed effect parameters and variance components is derived. The simulation study shows that the resulting estimator is precise, reliable, and computationally efficient.

Distributed Quickest Detection in Sensor Networks via Two-Layer Large Deviation Analysis

We propose a distributed Bayesian quickest detection algorithm for sensor networks, based on a random gossip inter-sensor communication structure. Without a control or fusion center, each sensor executes its local change detection procedure in a parallel and distributed fashion, interacting with its neighboring sensors via random inter-sensor communications to propagate information. By modeling the information propagation dynamics in the network as a Markov process, a two-layer large deviation analysis is presented to analyze the performance of the proposed algorithm. The first-layer analysis shows that the relation between the probability of false alarm and the conditional averaged detection delay satisfies the large deviation principle, where the distributed Kullback–Leibler information number is established as a crucial factor. The second-layer analysis studies the probability that not all observations are available at one sensor. It shows that this probability decays exponentially fast to zero as the averaged rounds of communication increases. The large deviation upper and lower bounds for the converge rate are then derived. Finally, we show that the performance of the distributed algorithm converges exponentially fast to that of the centralized optimal one.

Assessing compatibility of two laboratories: formulations as a statistical hypothesis testing problem

A decision problem frequently occurring in metrology is that of compatibility of data obtained by two (or several) different laboratories, methods or instruments. One laboratory can be a prestigious national metrology institute whose data are considered to be a gold standard or a certified reference material interval. When each laboratory presents its results in the form of a coverage interval for the measurand, several statistical approaches to this conformity assessment problem are reviewed including the classical ‘equality of means’ hypotheses tests. A new compatibility hypothesis is formulated in terms of consistency of laboratory results and compliance with a maximum permitted uncertainty. The power functions of these tests are compared numerically. The Kullback–Leibler information number is suggested as a directional (asymmetric) interchangeability index.

Channel-Aware Decentralized Detection via Level-Triggered Sampling

We consider decentralized detection through distributed sensors that perform level-triggered sampling and communicate with a fusion center (FC) via noisy channels. Each sensor computes its local log-likelihood ratio (LLR), samples it using the level-triggered sampling mechanism, and at each sampling instant transmits a single bit to the FC. Upon receiving a bit from a sensor, the FC updates the global LLR and performs a sequential probability ratio test (SPRT) step. We derive the fusion rules under various types of channels. We further provide an asymptotic analysis on the average decision delay for the proposed channel-aware scheme, and show that the asymptotic decision delay is characterized by a Kullback-Leibler information number. The delay analysis facilitates the choice of the appropriate signaling schemes under different channel types for sending the 1-bit information from the sensors to the FC.

Relationship of target recognition performance and radar waveform parameters

Target recognition performance can be affected by radar waveform parameters. In this paper, we established rigorous relationship between target recognition efficiency and the parameters of a repeatedly transmitted waveform. It is based on Kullback-Leibler Information Number of single observation (KLINs), which measures the dissimilarity between targets depicted by a range-velocity double spread density function in frequency domain. We considered two signal models which are different in the coherence of the observations. The method we proposed takes advantage of the methodology of sequential hypothesis test, and then the recognition performance in terms of correct classification rate is expressed by Receiver Operating Characteristic (ROC). Simulation results about the parameters of LFM signal show the validity of the method.

Normalization of the origin-shifted exponential distribution for control chart construction

This study demonstrates that a location parameter of an exponential distribution significantly influences normalization of the exponential. The Kullback–Leibler information number is shown to be an appropriate index for measuring data normality using a location parameter. Control charts based on probability limits and transformation are compared for known and estimated location parameters. The probabilities of type II error (β-risks) and average run length (ARL) without a location parameter indicate an ability to detect an out-of-control signal of an individual chart using a power transformation similar to using probability limits. The β-risks and ARL of control charts with an estimated location parameter deviate significantly from their theoretical values when a small sample size of n≤50 is used. Therefore, without taking into account of the existence of a location parameter, the control charts result in inaccurate detection of an out-of-control signal regardless of whether a power or natural logarithmic transformation is used. The effects of a location parameter should be eliminated before transformation. Two examples are presented to illustrate these findings.

Large deviations for M-estimators

We study the large deviation principle for M-estimators (and maximum likelihood estimators in particular). We obtain the rate function of the large deviation principle for M-estimators. For exponential families, this rate function agrees with the Kullback–Leibler information number. However, for location or scale families this rate function is smaller than the Kullback–Leibler information number. We apply our results to obtain confidence regions of minimum size whose coverage probability converges to one exponentially. In the case of full exponential families, the constructed confidence regions agree with the ones obtained by inverting the likelihood ratio test with a simple null hypothesis.

Multivariate exponential families and the Taguchi loss function

Taguchi uses a loss function concept to quantify quality as loss due to functional variation. When there is one variable under study a quadratic loss function is used. When the system under consideration is defined by p-variables which may or may not be correlated a more general loss function should be defined. We use information theory to define a multivariate loss function based on a divergence measure between two probability distributions that belong to the multivariate exponential family. The two distributions are the ideal or factory distribution that our data should follow and the other is the real distribution of the multivariate data. This divergence measure uses the concept of the Kullback—Leibler information number.

An efficient algorithm to find the MLE of prior probabilities of a mixture in pattern recognition

A number of techniques have been proposed to determine the parameters which define the unknown components of a mixture in pattern recognition. The most common method is the maximum likelihood estimation (MLE). A direct ML approach requires solution to maximize the likelihood function of the unknown prior probabilities of classes in a mixture. This is a complicated multiparameter optimization problem. The direct approach tends to be computationally complex and time consuming. In this study, we use the concave property of the Kullback-Leibler information number to derive a simple and accurate algorithm which can find the MLE of the prior probability of each class. The results of a Monte Carlo simulation study with normal and exponential distributions are presented to demonstrate the favorable prior estimation for the algorithm.

Maximum smoothed likelihood density estimation

We interpret the kernel estimator for the density as the solution to a maximum smoothed likelihood problem, and show under suitable conditions that it converges to the true densityin the sense that the Kullback-Leibler information number of the estimator relative to the true density converges to 0 in probability. We also consider the convergence in probability of the naiveestimator for the entropy number of the density. The conditions under which all this happens say essentially that the true density as well as the smoothed true density have finite entropy. The significance of the maximum (smoothed) likelihood set-up is that the relevant random variables are nonnegative, and all we have to do is show that their expected values converge to 0.

Boundary Crossing Problems for Sample Means

Motivated by several classical sequential decision problems, we study herein the following type of boundary crossing problems for certain nonlinear functions of sample means. Let $X_1, X_2,\ldots$ be i.i.d. random vectors whose common density belongs to the $k$-dimensional exponential family $h_\theta(x) = \exp\{\theta'x - \psi(\theta)\}$ with respect to some nondegenerate measure $\nu$. Let $\bar{X}_n = (X_1 + \cdots + X_n)/n, \hat\theta_n = (\nabla\psi)^{-1}(\bar{X}_n)$, and let $I(\theta, \lambda) = E_\theta\log\{h_\theta(X_1)/h_\lambda(X_1)\}$ ( = Kullback-Leibler information number). Consider stopping times of the form $T_c(\lambda) = \inf\{n: I(\hat\theta_n, \lambda) \geq n^{-1}g(cn)\}, c > 0$, where $g$ is a positive function such that $g(t) \sim \alpha \log t^{-1}$ as $t \rightarrow 0$. We obtain asymptotic approximations to the moments $E_\theta T^r_c(\lambda)$ as $c \rightarrow 0$ that are uniform in $\theta$ and $\lambda$ with $|\lambda - \theta|^2/c \rightarrow \infty$. We also study the probability that $\bar{X}_{Tc(\lambda)}$ lies in certain cones with vertex $\nabla\psi (\lambda)$. In particular, in the one-dimensional case with $\lambda > \theta$, we consider boundary crossing probabilities of the form $P_\theta\{\hat\theta_n \geq \lambda \text{and} I(\hat\theta_n, \lambda) \geq n^{-1} g(cn) \text{for some} n\}$. Asymptotic approximations (as $c \rightarrow 0$) to these boundary crossing probabilities are obtained that are uniform in $\theta$ and $\lambda$ with $|\lambda - \theta|^2/c \rightarrow \infty$.

On asymptotic optimality of bayes empirical bayes estimators

In an empirical Bayes decision problem, a prior distribution ≱ is placed on a one-dimensfonal family G of priors Gw, weΩ, to produce a Bayes empirical Bayes estimator, The asymptotic optimaiity of the Bayes estimator is established when the support of ≱ is Ω and the marginal distributions Hw have monotone likelihood ratio and continuous Kullback-Leibler information number.

On-line surveillance of a grinding process via a Kullback-Leibler information number

The dynamic signals emitted in grinding processes, such as grinding chatter, grinding forces and grinding sounds, can be successfully used for on-line surveillance of a grinding operation. A new criterion called the Kullback-Leibler information number based on time-series analysis and information theory, is suggested to monitor the performance of a wheel and grinder under different working conditions and to prognosticate the remaining wheel life. A computer algorithm for supervision of the grinding cycles is also designed and testified.

An algorithm for maximizing expected log investment return

Let the random (stock market) vector X \geq 0 be drawn according to a known distribution function F(x), x \in R^{m} . A log-optimal portfolio b^{\ast} is any portfolio b achieving maximal expected \log return W^{\ast}=\sup_{b} E \ln b^{t}X , where the supremum is over the simplex b \geq 0, \sum_{i=1}^{m} b_{i} = 1 . An algorithm is presented for finding b^{\ast} . The algorithm consists of replacing the portfolio b by the expected portfolio b^{'}, b_{i}^{'} = E(b_{i}X_{i}/b^{t}X) , corresponding to the expected proportion of holdings in each stock after one market period. The improvement in W(b) after each iteration is lower-bounded by the Kullback-Leibler information number D(b^{'}\|b) between the current and updated portfolios. Thus the algorithm monotonically improves the return W . An upper bound on W^{\ast} is given in terms of the current portfolio and the gradient, and the convergence of the algorithm is established.

Some strong ε-equivalence of random variables

Strongness and related error evaluations are investigated on type (B)d, type (A) and type (I) e-equivalence of random variables, which are based on Kolmogorov-Smirnov distance, a difference of random variables and Kullback-Leibler information number, respectively. As an application the Prohorov-LeCam type binomial-Poisson approximation problem is discussed and is given the best possible constant for the problem. Similar discussions are made on the negative binomial-Poisson approximation.