Kullback-Leibler Divergence Rate Research Articles

Kullback–Leibler Divergence of an Open-Queuing Network of a Cell-Signal-Transduction Cascade

Queuing networks (QNs) are essential models in operations research, with applications in cloud computing and healthcare systems. However, few studies have analyzed the cell's biological signal transduction using QN theory. This study entailed the modeling of signal transduction as an open Jackson's QN (JQN) to theoretically determine cell signal transduction, under the assumption that the signal mediator queues in the cytoplasm, and the mediator is exchanged from one signaling molecule to another through interactions between the signaling molecules. Each signaling molecule was regarded as a network node in the JQN. The JQN Kullback-Leibler divergence (KLD) was defined using the ratio of the queuing time (λ) to the exchange time (μ), λ/μ. The mitogen-activated protein kinase (MAPK) signal-cascade model was applied, and the KLD rate per signal-transduction-period was shown to be conserved when the KLD was maximized. Our experimental study on MAPK cascade supported this conclusion. This result is similar to the entropy-rate conservation of chemical kinetics and entropy coding reported in our previous studies. Thus, JQN can be used as a novel framework to analyze signal transduction.

Intelligent Energy Management Strategy Based on an Improved Reinforcement Learning Algorithm With Exploration Factor for a Plug-in PHEV

An intelligent energy management strategy (EMS) based on an improved Reinforcement Learning (RL) algorithm is developed to enhance the adaptability of the EMS and to further improve the fuel efficiency of a Plug-in Parallel Hybrid Electric Vehicle (PHEV). Both the numerical model and the energy management strategy of a plug-in PHEV are described. The improved RL with Q-learning algorithm is implemented to acquire the optimal control strategies for improving fuel economy. The Markov Chain is employed to calculate the Transition Probability Matrix of the required power. A Kullback-Leibler (KL) divergence rate is designed to activate the update of EMS, when a new corresponding driving cycle is expected. An Exploration Factor (EF) is proposed to overcome the disadvantages of the normal RL algorithm in convergence rate and reward cost evaluation. The diverse KL divergence rates are examined to seek optimal solutions. The normal-RL strategy, rule-based strategy, and dynamic programming strategy are implemented as benchmark strategies to verify the effectiveness of the proposed strategy. The validation results indicate that the improved RL algorithm with EF makes it possible to promote the EMS capable of significantly improving the energy efficiency of a plug-in PHEV.

A real-time energy management strategy for off-road hybrid electric vehicles based on the expected SARSA

To improve the power performance and fuel economy and reduce the online computation time of hybrid electric vehicles (HEVs) in off-road driving environments, a real-time energy management strategy based on the expected state-action-reward-state’-action’ (SARSA) algorithm is proposed. This strategy achieves the control effect through coordinated control of the engine generator set (EGS) and the energy distribution between the battery and the EGS. The driving environment is expressed as a transition probability matrix (TPM) of the electric power demand, and the optimal control problem of energy management is solved offline by the expected SARSA algorithm to obtain the control laws. Then, the Kullback-Leibler (KL) divergence rate is used as the evaluation index of the switching control strategy, and these control laws are used online to realize real-time control. Control strategies based on traditional reinforcement learning (RL), stochastic dynamic programming (SDP), and dynamic programming (DP) are considered benchmark strategies to verify the effectiveness of the proposed energy management strategy based on the expected SARSA. The fuel economy is improved by a maximum of 13.2%, and the elapsed time is reduced by more than 99.8% compared with the values achieved with the benchmark algorithms. These results show that the proposed strategy has flexible adaptability to complex and changeable off-road conditions without prior knowledge of the whole driving cycle.

Asymptotic reversibility of thermal operations for interacting quantum spin systems via generalized quantum Stein’s lemma

For quantum spin systems in any spatial dimension with a local, translation-invariant Hamiltonian, we prove that asymptotic state convertibility from a quantum state to another one by a thermodynamically feasible class of quantum dynamics, called thermal operations, is completely characterized by the Kullback–Leibler (KL) divergence rate, if the state is translation-invariant and spatially ergodic. Our proof consists of two parts and is phrased in terms of a branch of the quantum information theory called the resource theory. First, we prove that any states, for which the min and max Rényi divergences collapse approximately to a single value, can be approximately reversibly converted into one another by thermal operations with the aid of a small source of quantum coherence. Second, we prove that these divergences collapse asymptotically to the KL divergence rate for any translation-invariant ergodic state. We show this via a generalization of the quantum Stein’s lemma for quantum hypothesis testing beyond independent and identically distributed situations. Our result implies that the KL divergence rate serves as a thermodynamic potential that provides a complete characterization of thermodynamic convertibility of ergodic states of quantum many-body systems in the thermodynamic limit, including out-of-equilibrium and fully quantum situations.

Energy Management Strategy for Smart Meter Privacy and Cost Saving

We design optimal privacy-enhancing and cost-efficient energy management strategies for consumers that are equipped with a rechargeable energy storage. The Kullback-Leibler divergence rate is used as privacy measure and the expected cost-saving rate is used as utility measure. The corresponding energy management strategy is designed by optimizing a weighted sum of both privacy and cost measures over a finite time horizon, which is achieved by formulating our problem into a belief-state Markov decision process problem. A computationally efficient approximated Q-learning method is proposed as a generalization to high-dimensional problems over an infinite time horizon. At last, we explicitly characterize a stationary policy that achieves the steady belief state over an infinite time horizon, which greatly simplifies the design of the privacy-preserving energy management strategy. The performance of the practical design approaches are finally illustrated in numerical experiments.

Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning

The energy management approach of hybrid electric vehicles has the potential to overcome the increasing energy crisis and environmental pollution by reducing the fuel consumption. This paper proposes an online updating energy management strategy to improve the fuel economy of hybrid electric tracked vehicles. As the basis of the research, the overall model for the hybrid electric tracked vehicle is built in detail and validated through the field experiment. To accelerate the convergence rate of the control policy calculation, a novel reinforcement learning algorithm called fast Q-learning is applied which improves the computational speed by 16%. The cloud-computation is presented to afford the main computation burden to realize the online updating energy management strategy in hardware-in-loop simulation bench. The Kullback-Leibler divergence rate to trigger the update of the control strategy is designed and realized in hardware-in-loop simulation bench. The simulation results show that the fuel consumption of the fast Q-learning based online updating strategy is 4.6% lower than that of stationary strategy, and is close to that of dynamic programming strategy. Besides, the computation time of the proposed method is only 1.35 s which is much shorter than that of dynamic programming based method. The results indicate that the proposed energy management strategy can greatly improve the fuel economy and have the potential to be applied in the real-time application. Moreover, the adaptability of the online energy management strategy is validated in three realistic driving schedules.

Privacy Against a Hypothesis Testing Adversary

Privacy against an adversary (AD) that tries to detect the underlying privacy-sensitive data distribution is studied. The original data sequence is assumed to come from one of the two known distributions, and the privacy leakage is measured by the probability of error of the binary hypothesis test carried out by the AD. A management unit (MU) is allowed to manipulate the original data sequence in an online fashion while satisfying an average distortion constraint. The goal of the MU is to maximize the minimal type II probability of error subject to a constraint on the type I probability of error assuming an adversarial Neyman–Pearson test, or to maximize the minimal error probability assuming an adversarial Bayesian test. The asymptotic exponents of the maximum minimal type II probability of error and the maximum minimal error probability are shown to be characterized by a Kullback–Leibler divergence rate and a Chernoff information rate, respectively. Privacy performances of particular management policies, the memoryless hypothesis-aware policy and the hypothesis-unaware policy with memory, are compared. The proposed formulation can also model adversarial example generation with minimal data manipulation to fool classifiers. At last, the results are applied to a smart meter privacy problem, where the user’s energy consumption is manipulated by adaptively using a renewable energy source in order to hide user’s activity from the energy provider.

Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle

Power allocation is a crucial issue for hybrid energy storage system (HESS) in a plug-in hybrid electric vehicle (PHEV). To obtain the best power distribution between the battery and the ultracapacitor, the reinforcement learning (RL)-based real-time power-management strategy is raised. Firstly, a long driving cycle, which includes various speed variations, is chosen, and the power transition probability matrices based on stationary Markov chain are calculated. Then, the RL algorithm is employed to obtain a control strategy aiming at minimizing the energy loss of HESS. To reduce the energy loss further, the power transition probability matrices should be updated according to the new application driving cycle and Kullback-Leibler (KL) divergence rate is used to judge when the updating of power management strategy is triggered. The conditions of different forgetting factors and KL divergence rates are discussed to seek the optimal value. A comparison between the RL-based online power management and the rule-based power management shows that the RL-based online power management strategy can lessen the energy loss effectively and the relative decrease of the total energy loss can reach 16.8%. Finally, the strategy is verified in different conditions, such as temperatures, states of health, initials of SoC and driving cycles. The results indicate that not only can the RL-based real-time power-management strategy limit the maximum discharge current and reduce the charging frequency of the battery pack, but also can decrease the energy loss and optimize the system efficiency.

Minimum Entropy Rate Simplification of Stochastic Processes.

We propose minimum entropy rate simplification (MERS), an information-theoretic, parameterization-independent framework for simplifying generative models of stochastic processes. Applications include improving model quality for sampling tasks by concentrating the probability mass on the most characteristic and accurately described behaviors while de-emphasizing the tails, and obtaining clean models from corrupted data (nonparametric denoising). This is the opposite of the smoothing step commonly applied to classification models. Drawing on rate-distortion theory, MERS seeks the minimum entropy-rate process under a constraint on the dissimilarity between the original and simplified processes. We particularly investigate the Kullback-Leibler divergence rate as a dissimilarity measure, where, compatible with our assumption that the starting model is disturbed or inaccurate, the simplification rather than the starting model is used for the reference distribution of the divergence. This leads to analytic solutions for stationary and ergodic Gaussian processes and Markov chains. The same formulas are also valid for maximum-entropy smoothing under the same divergence constraint. In experiments, MERS successfully simplifies and denoises models from audio, text, speech, and meteorology.

Optimal Kullback–Leibler Aggregation via Information Bottleneck

In this paper, we present a method for reducing a regular, discrete-time Markov chain (DTMC) to another DTMC with a given, typically much smaller number of states. The cost of reduction is defined as the Kullback-Leibler divergence rate between a projection of the original process through a partition function and a DTMC on the correspondingly partitioned state space. Finding the reduced model with minimal cost is computationally expensive, as it requires an exhaustive search among all state space partitions, and an exact evaluation of the reduction cost for each candidate partition. Our approach deals with the latter problem by minimizing an upper bound on the reduction cost instead of minimizing the exact cost. The proposed upper bound is easy to compute and it is tight if the original chain is lumpable with respect to the partition. Then, we express the problem in the form of information bottleneck optimization, and propose using the agglomerative information bottleneck algorithm for searching a suboptimal partition greedily, rather than exhaustively. The theory is illustrated with examples and one application scenario in the context of modeling bio-molecular interactions.

Optimal Kullback–Leibler approximation of Markov chains via nuclear norm regularisation

This paper is concerned with model reduction for Markov chain models. The goal is to obtain a low-rank approximationto the original Markov chain. The Kullback–Leibler divergence rate is used to measure the similarity between two Markov chains; the nuclear norm is used to approximate the rank function. A nuclear-norm regularised optimisation problem is formulated to approximately find the optimal low-rank approximation. The proposed regularised problem is analysed and performance bounds are obtained through the convex analysis. An iterative fixed point algorithm is developed based on the proximal splitting technique to compute the optimal solutions. The effectiveness of this approach is illustrated via numerical examples.

A Discrete Queue-Based Model for Capturing Memory and Soft-Decision Information in Correlated Fading Channels

A discrete (binary-input 2q-ary output) communication channel with memory is introduced to judiciously capture both the statistical memory and the soft-decision information of a time-correlated discrete fading channel (DFC) used with antipodal signaling and soft output quantization of resolution q. The discrete channel, which can be explicitly described via its binary input process and a 2q-ary noise process, is shown to be symmetric, thus admitting a simple expression for its capacity when its noise is stationary ergodic. It is observed that considerable capacity gains can be achieved due to the channel's memory and the use of as few as 2 bits for soft-decision over interleaving the channel (to render it memoryless) and hard-decision demodulation (q=1). The 2q-ary noise process is next modeled via a queue-based (QB) ball-sampling mechanism to produce a mathematically tractable stationary ergodic Markovian noise source. The DFC is fitted by the QB noise model via an iterative procedure that minimizes the Kullback-Leibler divergence rate between the DFC and QB noise sources. Modeling results, measured in terms of channel noise correlation function and capacity reveal a good agreement between the two channels for a broad range of fading conditions.

Optimal Kullback-Leibler Aggregation via Spectral Theory of Markov Chains

This paper is concerned with model reduction for complex Markov chain models. The Kullback-Leibler divergence rate is employed as a metric to measure the difference between the Markov model and its approximation. For a certain relaxation of the bi-partition model reduction problem, the solution is shown to be characterized by an associated eigenvalue problem. The form of the eigenvalue problem is closely related to the Markov spectral theory for model reduction. This result is the basis of a heuristic proposed for the <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</i> -ary partition problem, resulting in a practical recursive algorithm. The results are illustrated with examples.

Approximation of stationary processes by hidden Markov models

We aim at the construction of a Hidden Markov Model (HMM) of assigned complexity (number of states of the underlying Markov chain) which best approximates, in Kullback-Leibler divergence rate, a given stationary process. We establish, under mild conditions, the existence of the divergence rate between a stationary process and an HMM. Since in general there is no analytic expression available for this divergence rate, we approximate it with a properly defined, and easily computable, divergence between Hankel matrices, which we use as our approximation criterion. We propose a three-step algorithm, based on the Nonnegative Matrix Factorization technique, which realizes an HMM optimal with respect to the defined approximation criterion. A full theoretical analysis of the algorithm is given in the special case of Markov approximation.

A Binary Communication Channel With Memory Based on a Finite Queue

A model for a binary additive noise communication channel with memory is introduced. The channel noise process, which is generated according to a ball sampling mechanism involving a queue of finite length M, is a stationary ergodic Mth-order Markov source. The channel properties are analyzed and several of its statistical and information-theoretical quantities (e.g., block transition distribution, autocorrelation function (ACF), capacity, and error exponent) are derived in either closed or easily computable form in terms of its four parameters. The capacity of the queue-based channel (QBC) is also analytically and numerically compared for a variety of channel conditions with the capacity of other binary models, such as the well-known Gilbert-Elliott channel (GEC), the Fritchman channel, and the finite-memory contagion channel. We also investigate the modeling of the traditional GEC using this QBC model. The QBC parameters are estimated by minimizing the Kullback-Leibler divergence rate between the probability of noise sequences generated by the GEC and the QBC, while maintaining identical bit-error rates (BER) and correlation coefficients. The accuracy of fitting the GEC via the QBC is evaluated in terms of ACF, channel capacity, and error exponent. Numerical results indicate that the QBC provides a good approximation of the GEC for various channel conditions; it thus offers an interesting alternative to the GEC while remaining mathematically tractable.

The Kullback–Leibler Divergence Rate Between Markov Sources

In this work, we provide a computable expression for the Kullback-Leibler divergence rate lim/sub n/spl rarr//spl infin//1/nD(p/sup (n)//spl par/q/sup (n)/) between two time-invariant finite-alphabet Markov sources of arbitrary order and arbitrary initial distributions described by the probability distributions p/sup (n)/ and q/sup (n)/, respectively. We illustrate it numerically and examine its rate of convergence. The main tools used to obtain the Kullback-Leibler divergence rate and its rate of convergence are the theory of nonnegative matrices and Perron-Frobenius theory. Similarly, we provide a formula for the Shannon entropy rate lim/sub n/spl rarr//spl infin//1/nH(p/sup (n)/) of Markov sources and examine its rate of convergence.