Information-theoretic Perspective Research Articles

Locally optimum image watermark detector based on statistical modeling of SWT-EFMs magnitudes

There are three indispensable, yet contrasting requirements for a watermarking scheme: perceptual transparency, watermark capacity, and robustness against attacks. Therefore, a watermarking scheme should provide a trade-off among these requirements from the information-theoretic perspective. Improving the ability of imperceptibility, watermark capacity, and robustness at the same time has been a challenge for all image watermarking algorithms. The statistical model-based transform domain multiplicative watermarking is an effective way to achieve the tradeoff among imperceptibility, robustness and data payload. In this paper, we propose a locally optimum statistical image watermark detector by incorporating stationary wavelet transform (SWT), exponent-Fourier moments (EFMs), and locally optimum (LO) decision rule. SWT has many characteristics, such as high redundancy, high reconstruction accuracy, low time complexity and multi-resolution representation, and EFMs can provide the excellent robustness. In order to obtain valuable samples, we first decompose the digital image by using SWT, and then perform EFMs calculation in the corresponding frequency domain. These robust frequency-domain EFMs magnitudes are treated as the basis for the following research work. In our scheme, the watermark is hidden into the target location of frequency-domain EFMs magnitudes by using multiplicative embedding method. These magnitudes at the target location are abbreviated as SWT-EFMs magnitudes for the convenience of subsequent description. We analyze the statistical characteristics of SWT-EFMs magnitudes in detail. In order to achieve accurate statistical modeling of SWT-EFMs magnitudes, the Beta Generalized Weibull mixtures-based hidden Markov tree (BGWM-HMT) model is constructed in SWT-EFMs magnitudes. The expectation-conditional maximization either (ECME) and the upward–downward algorithms are used to obtain model parameters. Our proposed statistical model can simultaneously capture the special distribution characteristics of SWT-EFMs magnitudes and the dependencies between scales. At the receiver, a statistical watermark detector based on the LO decision rule and BGWM-HMT model is developed in SWT-EFMs magnitudes. We derive closed-form expressions about the LO statistic and analyze the receiver operating characteristic (ROC) of the proposed detector. In addition, we measure the proposed scheme in terms of robustness, invisibility and time complexity. Experimental results on some test images and comparison with well-known existing methods demonstrate the efficacy and superiority of the proposed statistical image watermark detector.

Interference of Periodic and Spread-Spectrum-Modulated Waveforms with Analog and Digital Communications

In this article, the effectiveness and the limitations of spread-spectrum (SS) modulation techniques employed in switching-mode power converters and in digital systems to mitigate interference with communication equipment are analyzed and discussed under the EMC standard perspective and under an information theoretical perspective, with reference to different real-world scenarios. Substantial difference between potential EMI issues in traditional analog radio/TV broadcasting, digital data lines, and digital links featuring advanced channel coding techniques, e.g. in emerging power line communication (PLC) systems, are highlighted. Practical recommendations on the adoption of SS modulations along with a general reflection on the evolution of EMC requirements are finally given.

Privacy-Preserving Aggregate Mobility Data Release: An Information-Theoretic Deep Reinforcement Learning Approach

It is crucial to protect users’ location traces against inference attacks on aggregate mobility data collected from multiple users in various real-world applications. Most of the existing works on aggregate mobility data are focusing on inference attacks rather than designing privacy-preserving release mechanisms, and a few differential private release mechanisms suffer from poor utility-privacy tradeoffs. In this paper, we propose optimal centralized privacy-preserving aggregate mobility data release mechanisms (PAMDRMs) that minimize the leakage from an information-theoretic perspective by releasing perturbed versions of the raw aggregate location. Specifically, we use mutual information to measure user-level and aggregate-level privacy leakage separately, and formulate leakage minimization problems under utility constraints. As directly solving the optimization problems incur exponential complexity w.r.t. users’ trace length, we transform them into belief state Markov Decision Processes (MDPs), with a focus on the MDP formulation for the user-level privacy problem. We build reinforcement learning (RL) models and leverage the efficient Asynchronous Advantage Actor-Critic RL algorithm to derive the solutions to the MDPs as our optimal PAMDRMs. We compare them with two state-of-the-art privacy protection mechanisms PDPR (context-aware local design) and DMLM (context-free centralized design) in terms of mutual information leakage and adversary’s attack success (evaluated by her expected estimation error and Jensen-Shannon Divergence-based error). Extensive experimental results on both synthetic and real-world datasets demonstrate that the user-level PAMDRM performs the best on both measures thanks to its context-aware property and centralized design. Even though the aggregate-level PAMDRM achieves better privacy-utility tradeoff than the other two, it does not always perform better than them on adversarial success, highlighting the necessity of considering privacy measures from different perspectives to avoid overestimating the level of privacy offered to users. Lastly, we discuss an alternative, fully data-driven approach to derive the optimal PAMDRM by leveraging adversarial training on limited data samples.

Variational Bandwidth Auto-encoder for Hybrid Recommender Systems

Hybrid recommendations have recently attracted a lot of attention where user features are utilized as auxiliary information to address the sparsity problem caused by insufficient user-item interactions. However, extracted user features generally contain rich multimodal information, and most of them are irrelevant to the recommendation purpose. In this article, we propose a variational bandwidth auto-encoder (VBAE) for recommendations, aiming to address the sparsity and noise problems simultaneously. VBAE first encodes user collaborative and feature information into Gaussian latent variables via deep neural networks to capture non-linear user similarities. Moreover, by considering the fusion of collaborative and feature variables as a virtual communication channel from an information-theoretic perspective, we introduce a user-dependent channel to dynamically control the information allowed to be accessed from the feature embeddings. A quantum-inspired uncertainty measurement of the hidden rating embeddings is proposed accordingly to infer the channel bandwidth by disentangling the uncertainty information in the ratings from the semantic information. Through this mechanism, VBAE incorporates adequate auxiliary information from user features if collaborative information is insufficient, while avoiding excessive reliance on noisy user features to improve its generalization ability to new users. Extensive experiments conducted on three datasets demonstrate the effectiveness of the proposed method.

Quantifying Decoherence via Increases in Classicality.

As a direct consequence of the interplay between the superposition principle of quantum mechanics and the dynamics of open systems, decoherence is a recurring theme in both foundational and experimental exploration of the quantum realm. Decoherence is intimately related to information leakage of open systems and is usually formulated in the setup of “system + environment” as information acquisition of the environment (observer) from the system. As such, it has been mainly characterized via correlations (e.g., quantum mutual information, discord, and entanglement). Decoherence combined with redundant proliferation of the system information to multiple fragments of environment yields the scenario of quantum Darwinism, which is now a widely recognized framework for addressing the quantum-to-classical transition: the emergence of the apparent classical reality from the enigmatic quantum substrate. Despite the half-century development of the notion of decoherence, there are still many aspects awaiting investigations. In this work, we introduce two quantifiers of classicality via the Jordan product and uncertainty, respectively, and then employ them to quantify decoherence from an information-theoretic perspective. As a comparison, we also study the influence of the system on the environment.

Statistical image watermark decoder using high-order difference coefficients and bounded generalized Gaussian mixtures-based HMT

Imperceptibility, robustness and data payload are three main requirements of any image watermarking systems to guarantee desired functionalities, but there is a tradeoff among them from the information-theoretic perspective. In order to achieve this balance, in this paper, we propose a new statistical image watermarking scheme in nonsubsampled Shearlet transform (NSST) domain, which is based on high-order difference coefficients and bounded generalized Gaussian mixture model (BGGMM) based hidden Markov tree (HMT). In embedding process, NSST is firstly performed on host image and NSST highpass subband is divided into no-overlapping blocks. Then high-order differences of NSST coefficient blocks are computed. And finally, watermark signal is inserted into robust high-order difference coefficients through non-linear multiplicative approach. In extraction phase, robust high-order difference coefficients are firstly modeled by employing BGGMM based HMT, which can capture accurately both marginal distributions and strong dependencies of high-order difference coefficients. Statistical model parameters are then estimated by combining the approach of minimizing the higher bound on data negative log-likelihood function and upward–downward algorithm. And finally, an image watermark decoder is developed using BGGMM based HMT and maximum likelihood decision. Experimental results on some test images and comparison with well-known existing methods demonstrate the efficacy and superiority of the proposed scheme.

Upper Bounds on the Generalization Error of Private Algorithms for Discrete Data

In this work, we study the generalization capability of algorithms from an information-theoretic perspective. It has been shown that the expected generalization error of an algorithm is bounded from above by a function of the relative entropy between the conditional probability distribution of the algorithm's output hypothesis, given the dataset with which it was trained, and its marginal probability distribution. We build upon this fact and introduce a mathematical formulation to obtain upper bounds on this relative entropy. Assuming that the data is discrete, we then develop a strategy using this formulation, based on the method of types and typicality, to find explicit upper bounds on the generalization error of stable algorithms, i.e., algorithms that produce similar output hypotheses given similar input datasets. In particular, we show the bounds obtained with this strategy for the case of $\epsilon$-DP and $\mu$-GDP algorithms.

Information-Theoretic Analysis of OFDM With Subcarrier Number Modulation

With the prevalence of orthogonal frequency-division multiplexing (OFDM) in many standards, e.g., IEEE 802.11, IEEE 802.16, DVB-T, and DVB-T2, a number of variant modulation schemes based on OFDM have been proposed, which resort to signal sparsity to further enhance spectral efficiency and mitigate the high peak-to-average ratio (PAPR) problem. Among these variants, OFDM with subcarrier number modulation (OFDM-SNM) has been proven to be efficient for simple communication systems with low constellation modulation orders and limited decoding capability. To rigorously verify the performance advantages of OFDM-SNM, we present the study of OFDM-SNM in this paper from the information-theoretic perspective. In particular, we determine an upper bound on the mutual information of OFDM-SNM in closed form by using the log sum inequality. Also, we analyze the optimal pattern utilization probabilities (PUPs) for OFDM-SNM by channel-dependent coding and propose an easy-to-implement iterative algorithm to approach the optimal PUPs. Moreover, considering the practical achievability, we propose a Huffman coding based achievable PUP vector construction scheme to obtain the achievable PUPs and the corresponding achievable rate. We carry out numerical simulations to verify the effectiveness of this study and illustrate the efficiency of the obtained PUPs in comparison with several benchmarks.

Optimal Change-Point Detection With Training Sequences in the Large and Moderate Deviations Regimes

This paper investigates a novel offline change-point detection problem from an information-theoretic perspective. In contrast to most related works, we assume that the knowledge of the underlying pre- and post-change distributions are not known and can only be learned from the training sequences which are available. We further require the probability of the estimation error to decay either exponentially or sub-exponentially fast (corresponding respectively to the large and moderate deviations regimes in information theory parlance). Based on the training sequences as well as the test sequence consisting of a single change-point, we design a change-point estimator and further show that this estimator is optimal by establishing matching (strong) converses. This leads to a full characterization of the optimal confidence width (i.e., half the width of the confidence interval within which the true change-point is located at with high probability) as a function of the undetected error, under both the large and moderate deviations regimes.

Non-Equilibrium Entropy and Irreversibility in Generalized Stochastic Loewner Evolution from an Information-Theoretic Perspective.

In this study, we theoretically investigated a generalized stochastic Loewner evolution (SLE) driven by reversible Langevin dynamics in the context of non-equilibrium statistical mechanics. Using the ability of Loewner evolution, which enables encoding of non-equilibrium systems into equilibrium systems, we formulated the encoding mechanism of the SLE by Gibbs entropy-based information-theoretic approaches to discuss its advantages as a means to better describe non-equilibrium systems. After deriving entropy production and flux for the 2D trajectories of the generalized SLE curves, we reformulated the system’s entropic properties in terms of the Kullback–Leibler (KL) divergence. We demonstrate that this operation leads to alternative expressions of the Jarzynski equality and the second law of thermodynamics, which are consistent with the previously suggested theory of information thermodynamics. The irreversibility of the 2D trajectories is similarly discussed by decomposing the entropy into additive and non-additive parts. We numerically verified the non-equilibrium property of our model by simulating the long-time behavior of the entropic measure suggested by our formulation, referred to as the relative Loewner entropy.

Information-Theoretic Quantification of Dedifferentiation in the Aging of Motor and Executive Functions.

A central account of cognitive aging is the dedifferentiation among functions due to reduced processing resources. Previous reports contrasting trends of aging across cognitive domains mostly relied on transformed scores of heterogeneous measures. By quantifying the computational load with information entropy in tasks probing motor and executive functions, this study uncovered interaction among age, task, and load as well as associations among the parametric estimates of these factors at the individual level. Specifically, the linear functions between computational load and performance time differed significantly between motor and executive tasks in the young group but not in the elderly group and showed stronger associations for parameters within and between tasks in the elderly group than in the young group. These findings are in line with the dedifferentiation hypothesis of cognitive aging and provide a more principled approach in contrasting trends of cognitive aging across different domains from the information-theoretic perspective.

State Aggregations in Markov Chains and Block Models of Networks.

We consider state-aggregation schemes for Markov chains from an information-theoretic perspective. Specifically, we consider aggregating the states of a Markov chain such that the mutual information of the aggregated states separated by T time steps is maximized. We show that for T=1 this recovers the maximum-likelihood estimator of the degree-corrected stochastic block model as a particular case, which enables us to explain certain features of the likelihood landscape of this generative network model from a dynamical lens. We further highlight how we can uncover coherent, long-range dynamical modules for which considering a timescale T≫1 is essential. We demonstrate our results using synthetic flows and real-world ocean currents, where we are able to recover the fundamental features of the surface currents of the oceans.

Aperiodic sleep networks promote memory consolidation.

Hierarchical synchronization of sleep oscillations establishes communication pathways to support memory reactivation, transfer, and consolidation. From an information-theoretical perspective, oscillations constitute highly structured network states that provide limited information-coding capacity. Recent findings indicate that sleep oscillations occur in transient bursts that are interleaved with aperiodic network states, which were previously considered to be random noise. We argue that aperiodic activity exhibits unique and variable spatiotemporal patterns, providing an ideal information-rich neurophysiological substrate for imprinting new mnemonic patterns onto existing circuits. We discuss novel avenues in conceptualizing and quantifying aperiodic network states during sleep to further understand their relevance and interplay with sleep oscillations in support of memory consolidation.

Information-theoretic regularization for learning global features by sequential VAE

Sequential variational autoencoders (VAEs) with a global latent variable z have been studied for disentangling the global features of data, which is useful for several downstream tasks. To further assist the sequential VAEs in obtaining meaningful z, existing approaches introduce a regularization term that maximizes the mutual information (MI) between the observation and z. However, by analyzing the sequential VAEs from the information-theoretic perspective, we claim that simply maximizing the MI encourages the latent variable to have redundant information, thereby preventing the disentanglement of global features. Based on this analysis, we derive a novel regularization method that makes z informative while encouraging disentanglement. Specifically, the proposed method removes redundant information by minimizing the MI between z and the local features by using adversarial training. In the experiments, we trained two sequential VAEs, state-space and autoregressive model variants, using speech and image datasets. The results indicate that the proposed method improves the performance of downstream classification and data generation tasks, thereby supporting our information-theoretic perspective for the learning of global features.

An Information-Theoretic Perspective on Proper Quaternion Variational Autoencoders.

Variational autoencoders are deep generative models that have recently received a great deal of attention due to their ability to model the latent distribution of any kind of input such as images and audio signals, among others. A novel variational autoncoder in the quaternion domain , namely the QVAE, has been recently proposed, leveraging the augmented second order statics of -proper signals. In this paper, we analyze the QVAE under an information-theoretic perspective, studying the ability of the -proper model to approximate improper distributions as well as the built-in -proper ones and the loss of entropy due to the improperness of the input signal. We conduct experiments on a substantial set of quaternion signals, for each of which the QVAE shows the ability of modelling the input distribution, while learning the improperness and increasing the entropy of the latent space. The proposed analysis will prove that proper QVAEs can be employed with a good approximation even when the quaternion input data are improper.

Area Entropy and Quantized Mass of Black Holes from Information Theory

In this paper, we present a derivation of the black hole area entropy with the relationship between entropy and information. The curved space of a black hole allows objects to be imaged in the same way as camera lenses. The maximal information that a black hole can gain is limited by both the Compton wavelength of the object and the diameter of the black hole. When an object falls into a black hole, its information disappears due to the no-hair theorem, and the entropy of the black hole increases correspondingly. The area entropy of a black hole can thus be obtained, which indicates that the Bekenstein–Hawking entropy is information entropy rather than thermodynamic entropy. The quantum corrections of black hole entropy are also obtained according to the limit of Compton wavelength of the captured particles, which makes the mass of a black hole naturally quantized. Our work provides an information-theoretic perspective for understanding the nature of black hole entropy.

Quantum broadcast channels with cooperating decoders: An information-theoretic perspective on quantum repeaters

Communication over a quantum broadcast channel with cooperation between the receivers is considered. The first form of cooperation addressed is classical conferencing, where receiver 1 can send classical messages to receiver 2. Another cooperation setting involves quantum conferencing, where receiver 1 can teleport a quantum state to receiver 2. When receiver 1 is not required to recover information and its sole purpose is to help the transmission to receiver 2, the model reduces to the quantum primitive relay channel. The quantum conferencing setting is intimately related to quantum repeaters as the sender, receiver 1, and receiver 2 can be viewed as the transmitter, the repeater, and the destination receiver, respectively. We develop lower and upper bounds on the capacity region in each setting. In particular, the cutset upper bound and the decode-forward lower bound are derived for the primitive relay channel. Furthermore, we present an entanglement-formation lower bound, where a virtual channel is simulated through the conference link. At last, we show that as opposed to the multiple access channel with entangled encoders, entanglement between decoders does not increase the classical communication rates for the broadcast dual.

Statistical image watermark decoder based on local frequency-domain Exponent-Fourier moments modeling

There are three indispensable, yet contrasting requirements for a watermarking scheme: perceptual transparency, watermark capacity, and robustness against attacks. Therefore, a watermarking scheme should provide a trade-off among these requirements from the information-theoretic perspective. In this paper, we propose a statistical image watermark decoder based on the local frequency-domain Exponent-Fourier moments modeling, which can achieve the tradeoff among imperceptibility, robustness and data payload. The frequency-domain EFMs magnitudes are first generated by combining stationary wavelet transform (SWT) and Exponent-Fourier moments (EFMs). We divide the target region to select local frequency-domain EFMs magnitudes, which are planned to embed watermarks, statistical modeling, and extract watermarks. In order to achieve an accurate modeling process, we conduct the comprehensive statistical analyses of local frequency-domain EFMs magnitudes and establish the powerful Beta Generalized Weibull mixtures-based hidden Markov tree (BGW-HMT) model, which can take into account the non-Gaussian distribution characteristic and the interscale dependency at the same time. The Expectation/Conditional Maximisation Either (ECME) algorithm and upward–downward algorithm are successfully applied to estimate the parameters of BGW-HMT model. At the receiver, the BGW-HMT model is used in the design process of watermark decoder. The decoder structure is developed by using the maximum likelihood decision. In order to prove the effectiveness of proposed watermarking scheme, multi-angle performance tests are performed, including imperceptibility, robustness, watermark capacity and time complexity. The corresponding experimental results from the above four perspectives are inspiring. Compared with the state-of-the-art schemes, our statistical decoder is significantly superior to other statistical decoders.

3D Imaging from Multipath Temporal Echoes

Echo location is a broad approach to imaging and sensing that includes both manmade RADAR, LIDAR, SONAR, and also animal navigation. However, full 3D information based on echo location requires some form of scanning of the scene in order to provide the spatial location of the echo origin-points. Without this spatial information, imaging objects in three-dimensional (3D) is a very challenging task as the inverse retrieval problem is strongly ill-posed. Here, we show that the temporal information encoded in the return echoes that are reflected multiple times within a scene is sufficient to faithfully render an image in 3D. Numerical modeling and an information theoretic perspective prove the concept and provide insight into the role of the multipath information. We experimentally demonstrate the concept by using both radio frequency and acoustic waves for imaging individuals moving in a closed environment.

Entropy excesses as quantifiers of nonclassicality

Any quantum state of a bosonic field can be described by a density operator on the system Hilbert space, or equivalently, by the corresponding Husimi function on the phase space. For the density operator, there is a family of quantum Tsallis entropies (including the von Neumann entropy as a particular instance) quantifying its uncertainty (mixedness), and for the Husimi function, there is a corresponding family of classical Tsallis entropies (including the Wehrl entropy as a particular instance) synthesizing its phase-space uncertainty. We propose to use the entropy excesses, i.e., the differences between the classical Tsallis entropies and the quantum Tsallis entropies, as quantifiers of nonclassicality of bosonic field states. We elaborate on this idea and highlight the entropy excess arising from the difference between the Wehrl (classical) entropy and the von Neumann (quantum) entropy as a significant indicator of nonclassicality. We further illustrate these nonclassicality quantifiers by several widely used states in quantum optics. This entropic approach to nonclassicality sheds light on the nature of nonclassicality from an information-theoretic perspective involving the heterodyne measurement.