Entropy Constraint Research Articles

Robust coding for a class of sources: Applications in control and reliable communication over limited capacity channels

This paper is concerned with the control of a class of dynamical systems over finite capacity communication channels. Necessary conditions for reliable data reconstruction and stability of a class of dynamical systems are derived. The methodology is information theoretic. It introduces the notion of entropy for a class of sources, which is defined as a maximization of the Shannon entropy over a class of sources. It also introduces the Shannon information transmission theorem for a class of sources, which states that channel capacity should be greater or equal to the mini-max rate distortion (maximization is over the class of sources) for reliable communication. When the class of sources is described by a relative entropy constraint between a class of source densities, and a given nominal source density, the explicit solution to the maximum entropy, is given, and its connection to Rényi entropy is illustrated. Furthermore, this solution is applied to a class of controlled dynamical systems to address necessary conditions for reliable data reconstruction and stability of such systems.

Lagrangian Vector Quantization With Combined Entropy and Codebook Size Constraints

In this paper, the Lagrangian formulation of variable-rate vector quantization is extended to quantization with simultaneous constraints on entropy and codebook size, including variable- and fixed-rate quantization as special cases. The formulation leads to a Lloyd quantizer design algorithm and generalizations of Gersho's approximations characterizing optimal performance for asymptotically large rate. A variation of Gersho's approach is shown to yield rigorous results partially characterizing the asymptotically optimal performance.

Robust Affine Invariant Feature Extraction for Image Matching

A new approach is presented to extract more robust affine invariant features for image matching. The novelty of our approach is a hierarchical filtering strategy for affine invariant feature detection, which is based on information entropy and spatial dispersion quality constraints. The concept of spatial dispersion quality is introduced to quantify the spatial distribution of features. Moreover, an integrated algorithm combined by the filtering strategy, maximally stable extremal region (MSER) and scale invariant feature transform, is introduced for affine invariant feature extraction. Since Mikolajczyk et al. identified that MSER is the best detector in many cases, we design an experiment to compare our approach (ED-MSER) with the standard MSER. By using two stereo pairs and an image sequence with different types of imagery, the experiment indicates that ED-MSER can always get much higher repeatability and matching score compared to the standard MSER and other algorithms, thus benefiting the subsequent image matching and many other applications.

Multiple Descriptions With Symbol Based Turbo Codes Over Noisy Channels With Packet Loss: Design and Performance Analysis

Channel optimized multiple description (MD) vector quantization with symbol-based turbo codes are proposed, where the decoder exploits both nonuniformity of descriptions and their dependencies. The joint source channel coding strategy is devised by setting an entropy constraint for quantizer design, which together with the MD index assignment (IA), control the level of redundancy at the MD source coder output. This is in turn exploited by the source and channel decoders for robust transmission in presence of noise and packet loss. At the source coder, the IA is designed for an M-description vector quantizer using an efficient simulated annealing algorithm. Various numerical results and comparisons are provided to investigate the effect of different system and design parameters on the performance. The results specifically indicate improved performance in comparison to the prior art, and also in multiple input multiple output communications.

Formulating viscous hydrodynamics for large velocity gradients

Viscous corrections to relativistic hydrodynamics, which are usually formulated for small velocity gradients, have recently been extended from Navier-Stokes formulations to a class of treatments based on Israel-Stewart equations. Israel-Stewart treatments, which treat the spatial components of the stress-energy tensor ${\ensuremath{\tau}}_{\mathit{ij}}$ as dynamical objects, introduce new parameters, such as the relaxation times describing nonequilibrium behavior of the elements ${\ensuremath{\tau}}_{\mathit{ij}}$. By considering linear response theory and entropy constraints, we show how the additional parameters are related to fluctuations of ${\ensuremath{\tau}}_{\mathit{ij}}$. Furthermore, the Israel-Stewart parameters are analyzed for their ability to provide stable and physical solutions for sound waves. Finally, it is shown how these parameters, which are naturally described by correlation functions in real time, might be constrained by lattice calculations, which are based on path-integral formulations in imaginary time.

Finite Horizon Robust State Estimation for Uncertain Finite-Alphabet Hidden Markov Models with Conditional Relative Entropy Constraints

We consider a robust state estimation problem for time-varying uncertain discrete-time, homogeneous, first-order, finite-state finite-alphabet hidden Markov models (HMMs). A class of time-varying uncertain HMMs is considered in which the uncertainty is sequentially described by a conditional relative entropy constraint on perturbed conditional probability measures given a realized observation sequence. For this class of uncertain HMMs, the robust state estimation problem is formulated as a constrained optimization problem. Using a Lagrange multiplier technique and a variational formula for conditional relative entropy, the above problem is converted into an unconstrained optimization problem and a problem related to partial information risk-sensitive filtering. A measure transformation technique and an information state method are employed to solve this equivalent problem related to risk-sensitive filtering. A characterization of the solution to the robust state estimation problem is also presented.

Uniform and Non-Uniform Optimum Scalar Quantizers Performances: A Comparative Study

The aim of this research is to investigate source coding, the representation of information source output by finite R bits/symbol. The performance of optimum quantisers subject to an entropy constraint has been studied. The definitive work in this area is best summarised by Shannon' s source coding theorem, that is, a source with entropy H can be encoded with arbitrarily small error probability at any rate R (bits/source output) as long as R>H. Conversely, If R<H the error probability will be driven away from zero, independent of the complexity of the encoder and the decoder employed. In this context, the main objective of engineers is however to design the optimum code. Unfortunately, the rate­distortion theorem does not provide the recipe for such a design. T he t heorem d oes, h owever, p rovide t he t heoretical l imit s o t hat w e know how close we are to the optimum. The full understanding of the theorem also helps in setting the direction to achieve such an optimum. In this research, we have investigated the performances of two practical scalar quantisers, i.e., a Lloyd­Max quantiser and the uniformly defined one and also a well­known entropy coding scheme, i.e., Huffman coding against their theoretically attainable optimum performance due to Shannon' s limit R. I t ha s be en s hown that our uniformly defined quantiser could demonstrate superior performance. The performance improvements, in fact, are more noticeable at higher bit rates.

Variable Dimension Trellis-Coded Quantization of Sinusoidal Parameters

In this letter, we propose joint quantization of the parameters of a set of sinusoids based on the theory of trellis-coded quantization. A particular advantage of this approach is that it allows for joint quantization of a variable number of sinusoids, which is particularly relevant in variable rate parametric audio coding. Under high-resolution assumptions and based on a perceptually relevant distortion measure, we derive analytical expressions for the optimal design subject to an entropy constraint. Numerical experiments show a significant performance gain compared to optimal spherical quantization at the cost of a slight increase in computational complexity.

Preserving monotonicity in anisotropic diffusion

We show that standard algorithms for anisotropic diffusion based on centered differencing (including the recent symmetric algorithm) do not preserve monotonicity. In the context of anisotropic thermal conduction, this can lead to the violation of the entropy constraints of the second law of thermodynamics, causing heat to flow from regions of lower temperature to higher temperature. In regions of large temperature variations, this can cause the temperature to become negative. Test cases to illustrate this for centered asymmetric and symmetric differencing are presented. Algorithms based on slope limiters, analogous to those used in second order schemes for hyperbolic equations, are proposed to fix these problems. While centered algorithms may be good for many cases, the main advantage of limited methods is that they are guaranteed to avoid negative temperature (which can cause numerical instabilities) in the presence of large temperature gradients. In particular, limited methods will be useful to simulate hot, dilute astrophysical plasmas where conduction is anisotropic and the temperature gradients are enormous, e.g., collisionless shocks and disk-corona interface.

Improving a radiative plus collisional energy loss model for application to RHIC and LHC

With the QGP opacity computed perturbatively and with the global entropy constraints imposed by the observed dNch/dy ≈ 1000, radiative energy loss alone cannot account for the observed suppression of single non-photonic electrons. Collisional energy loss is comparable in magnitude to radiative loss for both light and heavy jets. Two aspects that significantly affect the collisional energy loss are examined: the role of fluctuations and the effect of introducing a running QCD coupling as opposed to the fixed αs = 0.3 used previously.

Relative entropy and spectral constraints: some invariance properties of the ARMA class

Abstract. We determine the form of spectral densities of multidimensional scalar processes which minimize a relative entropy under a finite number of general moment‐type constraints. The obtained theoretical results are applied to spectral densities of weakly stationary processes under covariances, inverse covariances and cepstral or impulse response constraints. Invariance properties of the class of autoregressive moving‐average (ARMA) processes are shown to hold under the relative entropy minimization principle for many choices of entropy.

Informationally optimal correlation

This papers studies an optimization problem under entropy constraints arising from repeated games with signals. We provide general properties of solutions and a full characterization of optimal solutions for 2 × 2 sets of actions. As an application we compute the minmax values of some repeated games with signals.

Stochastic Uncertain Systems Subject to Relative Entropy Constraints: Induced Norms and Monotonicity Properties of Minimax Games

Entropy and relative entropy are fundamental concepts on which information theory is founded on, and in general, telecommunication systems design. On the other hand, dissipation inequalities, minimax strategies, and induced norms are the basic concepts on which robustness of uncertain control and estimation of systems are founded on. In this paper, the precise relation between these notions is investigated. In particular, it will be shown that the higher the dissipation the higher the entropy of the system, which has implications in computing the induced norm associated with robustness. These connections are obtained by considering stochastic optimal uncertain control systems, in which uncertainty is described by a relative entropy constraint between the nominal and uncertain measures, while the pay-off is a linear functional of the uncertain measure. This is a minimax game, in which the controller measure seeks to minimize the pay-off, while the disturbance measure aims at maximizing the pay-off. Salient properties of the minimax solution are derived, including a characterization of the optimal sensitivity reduction, computation of the induced norm, monotonicity properties of minimax solution, and relations between dissipation and relative entropy of the system. The theory is developed in an abstract setting and then applied to nonlinear partially observable continuous-time uncertain controlled systems, in which the nominal and uncertain systems are described by conditional distributions. In addition, existence of the optimal control policy among the class of policies known as wide-sense control laws is shown, and an explicit formulae for the worst case conditional measure is derived. The results are applied to linear-quadratic-Gaussian problems

High-Resolution Spherical Quantization of Sinusoidal Parameters

Sinusoidal coding is an often employed technique in low bit-rate audio coding. Therefore, methods for efficient quantization of sinusoidal parameters are of great importance. In this paper, we use high-resolution assumptions to derive analytical expressions for the optimal entropy-constrained unrestricted spherical quantizers for the amplitude, phase, and frequency parameters of the sinusoidal model. This is done both for the case of a single sinusoid, and for the more practically relevant case of multiple sinusoids distributed across multiple segments. To account for psychoacoustical effects of the auditory system, a perceptual distortion measure is used. The optimal quantizers minimize a high-resolution approximation of the expected perceptual distortion, while the corresponding quantization indices satisfy an entropy constraint. The quantizers turn out to be flexible and of low complexity, in the sense that they can be determined easily for varying bit rate requirements, without any sort of retraining or iterative procedures. In an objective comparison it is shown that for the squared error distortion measure, the rate-distortion performance of the proposed method is very close to that of the theoretically optimal entropy-constrained vector quantization. Furthermore, for the perceptual distortion measure, the proposed scheme is shown to objectively outperform an existing sinusoidal quantization scheme, where frequency quantization is done independently. Finally, a subjective listening test, in which the proposed scheme is compared to an existing state-of-the-art sinusoidal quantization scheme with fixed quantizers for all input signals, indicates that the proposed scheme leads to an average bit rate reduction of 20%, at the same subjective quality level as the existing scheme

Application of the maximum entropy technique in tomographic reconstruction from laser diffraction data to determine local spray drop size distribution

This work proposes a new deconvolution technique to obtain local drop size distributions from line-of-sight intensity data measured by laser diffraction technique. The tomographic reconstruction, based on the maximum entropy (ME) technique, is applied to forward scattered light signal from a laser beam scanning horizontally through the spray on each plane from the center to the edge of spray, resulting in the reconstructed scattered light intensities at particular points in the spray. These reconstructed intensities are in turn converted to local drop size distributions. Unlike the classical method of the onion peeling technique or other mathematical transformation techniques that yield unrealistic negative scattered light intensity solutions, the maximum entropy constraints ensure positive light intensity. Experimental validations to the reconstructed results are achieved by using phase Doppler particle analyzer (PDPA). The results from the PDPA measurements agree very well with the proposed ME tomographic reconstruction.

Heavy Quark Jet Quenching with Collisional plus Radiative Energy Loss and Path Length Fluctuations

With the QGP opacity computed perturbatively and with the global entropy constraints imposed by the observed d N c h / d y ≈ 1000 , radiative energy loss alone cannot account for the observed suppression of single non-photonic electrons. We show that collisional energy loss, which previously has been neglected, is comparable to radiative loss for both light and heavy jets and may in fact be the dominant mechanism for bottom quarks. Predictions taking into account both radiative and collisional losses significantly reduce the discrepancy with data. In addition to elastic energy loss, it is critical to include jet path length fluctuations to account for the observed pion suppression.

Response to “Comment on ‘Variational approach to the volume viscosity of fluids’” [Phys. Fluids 18, 109101 (2006)

We respond to the Comment of Markus Scholle and therewith revise our material entropy constraint to account for the production of entropy.

LENSVIEW: software for modelling resolved gravitational lens images

We have developed a new software tool, lensview, for modelling resolved gravitational lens images. Based on the lensmem algorithm, the software finds the best-fitting lens mass model and source brightness distribution using a maximum entropy constraint. The method can be used with any point spread function or lens model. We review the algorithm and introduce some significant improvements. We also investigate and discuss issues associated with the statistical uncertainties of models and model parameters and the issues of source plane size and source pixel size. We test the software on simulated optical and radio data to evaluate how well lens models can be recovered and with what accuracy. For optical data, lens model parameters can typically be recovered with better than 1 per cent accuracy, and the degeneracy between mass ellipticity and power law is reduced. For radio data, we find that systematic errors associated with using processed radio maps, rather than the visibilities, are of similar magnitude to the random errors. Hence analysing radio data in image space is still useful and meaningful. The software is applied to the optical arc HST J15433+5352 and the radio ring MG1549+3047 using a simple elliptical isothermal lens model. For HST J15433+5352, the Einstein radius is 0.525 ± 0.015 arcsec which probably includes a substantial convergence contribution from a neighbouring galaxy. For MG1549+3047, the model has Einstein radius 1.105 ± 0.005 arcsec and core radius 0.16 ± 0.03 arcsec. The total mass enclosed in the critical radius is 7.06 × 1010 M⊙ for our best model. We finish by discussing issues relating to modelling of resolved lens images for this method and some alternatives.

A physics‐based diffusion scheme for numerical models

A proper description of the dissipation in a forcing‐dissipative system, such as the atmosphere, is very important when the system is discretized. Atmospheric dissipation is generally treated as diffusion in numerical weather prediction (NWP) models. The commonly used fourth‐order diffusion scheme (linear and nonlinear) generates spurious upgradient mass or heat transport, which violates the entropy constraint posed by non‐equilibrium thermodynamics. The second law of thermodynamics is used in this paper to re‐construct the diffusion scheme to ensure downgradient transport. The new physics‐based scheme is applied to the Penn State/NCAR non‐hydrostatic mesoscale model. A hurricane case simulation demonstrates that the new scheme improves the model accuracy in predicting the strength of the hurricane.

Black hole entropy, universality, and horizon constraints

To ask a question about a black hole in quantum gravity, one must restrict initial or boundary data to ensure that a black hole is actually present. For two-dimensional dilaton gravity, and probably a much wider class of theories, I show that the imposition of a ‘‘stretched horizon’’ constraint modifies the algebra of symmetries at the horizon, allowing the use of conformal field theory techniques to determine the asymptotic density of states. The result reproduces the Bekenstein-Hawking entropy without any need for detailed assumptions about the microscopic theory. Horizon symmetries may thus offer an answer to the problem of universality of black hole entropy.