Information-theoretic Perspective Research Articles

An Information-Theoretic Perspective on Successive Cancellation List Decoding and Polar Code Design

This work identifies information-theoretic quantities that are closely related to the required list size on average for successive cancellation list (SCL) decoding to implement maximum-likelihood decoding over general binary memoryless symmetric (BMS) channels. It also provides upper and lower bounds for these quantities that can be computed efficiently for very long codes. For the binary erasure channel (BEC), we provide a simple method to estimate the mean accurately via density evolution. The analysis shows how to modify, e.g., Reed-Muller codes, to improve the performance when practical list sizes, e.g., <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L\in {[{8, 1024}]}$ </tex-math></inline-formula> , are adopted. Exemplary constructions with block lengths <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N\in \{128,512\}$ </tex-math></inline-formula> outperform polar codes of 5G over the binary-input additive white Gaussian noise channel. It is further shown that there is a concentration around the mean of the logarithm of the required list size for sufficiently large block lengths, over discrete-output BMS channels. We provide the probability mass functions (p.m.f.s) of this logarithm, over the BEC, for a sequence of the modified RM codes with an increasing block length via simulations, which illustrate that the p.m.f.s concentrate around the estimated mean.

Signatures of topological phase transitions in higher Landau levels of HgTe/CdTe quantum wells from an information theory perspective

We analyze the structure of low energy Hamiltonian eigenstates in zincblende heterostructures (like HgTe/CdTe quantum wells) near the gamma point, under magnetic fields, to characterize topological phase transitions (TPT) under an information-theoretic perspective. Using information markers like entanglement, quantum fluctuations, fidelity susceptibility, participation ration, area in phase space, etc., we realize that higher Landau levels (LL) feel the topological phase transition slightly displaced with regard to the edge state, thus leading to the concept of “higher Landau level |n|>0 TPT”, as “echoes” of the standard edge state n=0 TPT. We compute the critical magnetic field and the critical HgTe layer thickness at which these information measures of higher Landau levels undergo a structural change.

Information Bottleneck Signal Processing and Learning to Maximize Relevant Information for Communication Receivers

Digital communication receivers extract information about the transmitted data from the received signal in subsequent processing steps, such as synchronization, demodulation and channel decoding. Technically, the receiver-side signal processing for conducting these tasks is complex and hence causes bottleneck situations in terms of power, delay and chip area. Typically, many bits per sample are required to represent and process the received signal in the digital receiver hardware accurately. In addition, demanding arithmetical operations are required in the signal processing algorithms. A popular recent trend is designing entire receiver chains or some of their crucial building blocks from an information theoretical perspective. Signal processing blocks with very simple mathematical operations can be designed to directly maximize the relevant information that flows through them. At the same time, a strong quantization reduces the number of bits processed in the receiver to further lower the complexity. The described system design approach follows the principle of the information bottleneck method. Different authors proposed various ideas to design and implement mutual information-maximizing signal processing units. The first important aim of this article is to explain the fundamental similarities between the information bottleneck method and the functionalities of communication receivers. Based on that, we present and investigate new results on an entire receiver chain that is designed following the information bottleneck design principle. Afterwards, we give an overview of different techniques following the information bottleneck design paradigm from the literature, mainly dealing with channel decoding applications. We analyze the similarities of the different approaches for information bottleneck signal processing. This comparison leads to a general view on information bottleneck signal processing which goes back to the learning of parameters of trainable functions that maximize the relevant mutual information under compression.

Improved Receivers for Optical Wireless OFDM: An Information Theoretic Perspective

We consider performance enhancement of asymmetrically-clipped optical orthogonal frequency division multiplexing (ACO-OFDM) and related optical OFDM schemes, which are variations of OFDM in intensity-modulated optical wireless communications. Unlike most existing studies on specific designs of improved receivers, this paper investigates information theoretic limits of all possible receivers. For independent and identically distributed (IID) complex Gaussian inputs, we obtain an exact characterization of information rate of ACO-OFDM with improved receivers for all SNRs. It is proved that the high-SNR gain of improved receivers asymptotically achieve 1/4 bits per channel use, which is equivalent to 3 dB in electrical SNR or 1.5 dB in optical SNR; as the SNR decreases, the maximum achievable SNR gain of improved receivers decreases monotonically to a non-zero low-SNR limit, corresponding to an information rate gain of 36.3%. For practically used constellations, we derive an upper bound on the gain of improved receivers. Numerical results demonstrate that the upper bound can be approached to within 1 dB in optical SNR by combining existing improved receivers and coded modulation. We also show that our information theoretic analyses can be extended to Flip-OFDM and PAM-DMT. Our results imply that, for the considered schemes, improved receivers may reduce the gap to channel capacity significantly at low-to-moderate SNR.

Fusing entropy measures for dynamic feature selection in incomplete approximation spaces

The selection of informative and discriminative features from incomplete dynamic data has emerged as an essential problem because the data preparation in modern applications is a dynamic updating process, and the collected data are often fraught with missing or unobserved values. Using the effectiveness of entropy measures to quantify uncertainty information in incomplete approximation spaces under the framework of tolerance rough sets, a novel incremental feature selection approach is presented in this study from an information-theoretic perspective. Based on the updating patterns of tolerance classification and decision partition induced by conditional and decision features, a novel fused representation of entropy measures in an incomplete approximation space is proposed to accelerate the calculation of feature significance during the heuristic search process. A computationally efficient feature selection algorithm is developed by integrating the incremental fusing strategy of entropy when the data increases dynamically in size. Numerical experiments were performed on several benchmark datasets with data updating for different data sizes and incremental ratios to demonstrate the effectiveness of our method. The superiority of the proposed method is established extensively in terms of computational efficiency, cardinality and accuracy of the selected feature subset.

Finding Influencers in Complex Networks: A Novel Method Based on Information Theory

Key node identification is one of the most fundamental and significant problems in complex network research. This article presents a novel approach called information-theory-based centrality <formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex>$C_{H}$</tex></formula> to distinguish influential nodes in complex networks. From an information-theoretic perspective, nodes involved in many information channels will improve the quality of communication. Thus, the greater the number of information channels a node participates, the more the importance of this node during information transmission. With this motivation, the local and global connectivity are measured according to the resistance distance, which captures the characteristic that multiple available routes reduce the resistance of information transmission on networks. We further model the probability of information diffusion according to the local and global connectivity and put forward an information-theory-based centrality to identify influential nodes. Finally, extensive evaluations are implemented on real-world networks to verify the feasibility and effectiveness of the proposed method. According to the results, our proposed method outperforms classical methods in identifying influential nodes and also indicates the potential for analyzing the influence evolution of networks, which shows a positive and effective impact on locating influencers and predicting potential key players in the future.

Simplification in translated Chinese: An entropy-based approach

For a long time, translation researchers, particularly those working in corpus-based translation studies, have held the presumption that translated texts tend to be simpler in lexical and syntactical features than non-translated native texts. Such claims have led to the formulation of the simplification universal hypothesis in translation studies. However, this line of research which focuses predominantly on the investigation of individual linguistic features has failed to provide sufficient evidence to confirm the existence of the simplification universal. To a large extent, the lack of global quantitative indicators for evaluating the complexity level of the translated and non-translated texts has hindered progress in this field. The current study, using entropy as an indicator, analysed the linguistic complexity between translated and native Chinese from the information-theoretical perspective. Our research found that translational Chinese tends to be simpler than its non-translated counterpart at the lexical level based on unigram entropy, but not the syntactic level based on part-of-speech entropy. Our study has confirmed the use of entropy as a reliable measure for lexical and syntactic complexity in the field of translation studies.

An Information-Theoretic View of Mixed-Delay Traffic in 5G and 6G.

Fifth generation mobile communication systems (5G) have to accommodate both Ultra-Reliable Low-Latency Communication (URLLC) and enhanced Mobile Broadband (eMBB) services. While eMBB applications support high data rates, URLLC services aim at guaranteeing low-latencies and high-reliabilities. eMBB and URLLC services are scheduled on the same frequency band, where the different latency requirements of the communications render their coexistence challenging. In this survey, we review, from an information theoretic perspective, coding schemes that simultaneously accommodate URLLC and eMBB transmissions and show that they outperform traditional scheduling approaches. Various communication scenarios are considered, including point-to-point channels, broadcast channels, interference networks, cellular models, and cloud radio access networks (C-RANs). The main focus is on the set of rate pairs that can simultaneously be achieved for URLLC and eMBB messages, which captures well the tension between the two types of communications. We also discuss finite-blocklength results where the measure of interest is the set of error probability pairs that can simultaneously be achieved in the two communication regimes.

Information Scrambling with Conservation Laws

The delocalization or scrambling of quantum information has emerged as a central ingredient in the understanding of thermalization in isolated quantum many-body systems. Recently, significant progress has been made analytically by modeling non-integrable systems as periodically driven systems, lacking a Hamiltonian picture, while honest Hamiltonian dynamics are frequently limited to small system sizes due to computational constraints. In this paper, we address this by investigating the role of conservation laws (including energy conservation) in the thermalization process from an information-theoretic perspective. For general non-integrable models, we use the equilibrium approximation to show that the maximal amount of information is scrambled (as measured by the tripartite mutual information of the time-evolution operator) at late times even when a system conserves energy. In contrast, we explicate how when a system has additional symmetries that lead to degeneracies in the spectrum, the amount of information scrambled must decrease. This general theory is exemplified in case studies of holographic conformal field theories (CFTs) and the Sachdev-Ye-Kitaev (SYK) model. Due to the large Virasoro symmetry in 1+1D CFTs, we argue that, in a sense, these holographic theories are not maximally chaotic, which is explicitly seen by the non-saturation of the second Rényi tripartite mutual information. The roles of particle-hole and U(1) symmetries in the SYK model are milder due to the degeneracies being only two-fold, which we confirm explicitly at both large- and small-N. We reinterpret the operator entanglement in terms of the growth of local operators, connecting our results with the information scrambling described by out-of-time-ordered correlators, identifying the mechanism for suppressed scrambling from the Heisenberg perspective.

Reconfigurable Intelligent Surface-Aided Multi-User Networks: Interplay Between NOMA and RIS

This article focuses on the exploitation of reconfigurable intelligent surfaces (RISs) in multiuser networks employing orthogonal multiple access (OMA) or non-orthogonal multiple access (NOMA), with an emphasis on investigating the interplay between NOMA and RIS. Depending on whether the RIS reflection coefficients can be adjusted only once or multiple times during one transmission, we distinguish between static and dynamic RIS configurations. In particular, the capacity region of RIS-aided single-antenna NOMA networks is discussed and compared with the OMA rate region from an information-theoretic perspective, revealing that the dynamic RIS configuration is capacity-achieving. Then, the impact of the RIS deployment location on the performance of different multiple access schemes is investigated, which reveals that asymmetric and symmetric deployment strategies are preferable for NOMA and OMA, respectively. Furthermore, for RIS-aided multiple-antenna NOMA networks, three novel joint active and passive beamformer designs are proposed based on both beamformer-based and cluster-based strategies. Finally, open research problems for RIS-NOMA networks are highlighted.

Increasing entropy reduces perceived numerosity throughout the lifespan

Numerical illusions may provide a powerful window into the mechanisms that give rise to our visual number sense. Recent research has shown that similarly oriented elements appear more numerous than randomly oriented elements in an array. Here we examine whether the orientation coherence illusion is a more general byproduct of the effect of entropy on numerical information-processing. Participants engaged in an ordinal numerical comparison task where the color entropy of arrays was manipulated. We found that arrays with low color entropy were perceived as more numerous than arrays with high color entropy (Experiments 1 and 2), suggesting that the coherence illusion on numerosity perception is not specific to a particular visual property (e.g., orientation) but instead that the entropy of visual arrays more generally affects numerical processing. In Experiment 3, we explored the developmental trajectory of the color entropy effect in children aged 5 to 17 and found that the strength of the coherence illusion increases into adulthood, raising intriguing questions as to how perceptual experiences influence the progression of this numerosity illusion. We consider a recently proposed resource-rational model as a framework for understanding the entropy effect on numerosity perception under an information-theoretic perspective.

Joint Communication and Binary State Detection

We study a basic joint communication and sensing setup from an information theoretic perspective. A transmitter sends an encoded message through a pair of noisy channels connected to a receiver and a sensor. The receiver is interested in decoding the message from its noisy observation. The sensor has access to the message as side information, and instead is interested in estimating an unknown yet fixed binary state of its channel. This basic setup models scenarios where a wireless transceiver broadcasts an information bearing waveform, to be decoded by possibly more than one receiver, and then observes a modulated echo of the waveform from which it wishes to detect an element or object in the environment. We consider binary symmetric and Gaussian settings with multiplicative binary states, and establish the fundamental trade-off between reliable message communication and efficient state detection in these settings. This trade-off is captured by the message communication rate against the state detection error exponent. The results give insights into the benefits of carrying out communication and sensing jointly.

Joint Constellation Design and Multiuser Detection for Grant-Free NOMA

As a promising solution for massive machine-type communication, grant-free non-orthogonal multiple access (GF-NOMA) has received considerable attention in recent years. However, the multidimensional constellation design (MCD) and multiuser detection (MUD) in GF-NOMA are usually optimized in a divide and conquer way, leading to local optima and performance degradation. To address this issue, we investigate the joint optimization of MCD and MUD for GF-NOMA. The formulated joint optimization is based on variational inference, which is intractable due to the signal superimposition that makes the optimization variables intricately coupled. Then, we resort to end-to-end deep learning (DL) to obtain the optimal solution. Specifically, we propose a DL-based multi-task variational autoencoder (Mul-VAE) that adopts a variational autoencoder network to optimize the distribution of the constellation points. We further derive the loss function of the proposed network and analyze it from an information-theoretic perspective. On this basis, multi-task learning is employed to deal with mutually conflicting yet related detection processes. Besides, taking heterogeneous transmission rates of users into account, a multi-task prioritizing strategy is designed to balance training performance. Simulation results reveal that the proposed method enables significant gains compared to state-of-the-art techniques.

Privacy in Retrieval, Computing, and Learning

The increasing prevalence of massive datasets makes the outsourcing of storage and computation tasks to distributed servers a necessity. This raises a number of concerns regarding the security and integrity of stored information, the privacy of accessing desired information, the communication overhead of distributed systems, the latency, reliability, and complexity of distributed computing, and privacy in distributed training and learning systems. Recent breakthroughs from coding, communication, and information-theoretic perspectives have opened up exciting new research avenues for these topics. There are many theoretical and practical open problems. This Special Issue is dedicated to communication theory, coding theory, information theory, signal processing, and networking aspects of privacy in information retrieval, privacy in coded computing over distributed servers, and privacy in distributed learning.

Quantum Clocks are More Accurate Than Classical Ones

A clock is, from an information-theoretic perspective, a system that emits information about time. One may therefore ask whether the theory of information imposes any constraints on the maximum precision of clocks. Here we show a quantum-over-classical advantage for clocks or, more precisely, the task of generating information about what time it is. The argument is based on information-theoretic considerations: we analyze how the accuracy of a clock scales with its size, measured in terms of the number of bits that could be stored in it. We find that a quantum clock can achieve a quadratically improved accuracy compared to a purely classical one of the same size.Received 28 January 2020Revised 20 September 2021Accepted 21 December 2021DOI:https://doi.org/10.1103/PRXQuantum.3.010319Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasQuantum channelsQuantum entanglementQuantum walksQuantum Information

Towards an information-theoretic framework of intrusion detection for composed systems and robustness analyses

Network-based Intrusion Detection Systems (NIDSs) are an important mechanism to identify malicious behaviour or policy violations within a network. Such detection systems typically face several challenges, among which are the base-rate fallacy and the resilience against adaptive adversaries. These challenges are often countered in modern NIDSs by combining multiple detection systems to diversify the used feature levels or utilize the advantages of multiple detection methods. However, currently there exists no suitable framework for a detailed analysis of such composed systems. Therefore, the contribution of this work is an evaluation framework for composed systems, which builds on previous information-theoretic approaches and highlights the utility of information-theoretic redundancies for robustness evaluations. This framework enables an attribution of the overall system performance to its individual components, to fine-tune parameters and to study the dynamics between classifiers. The versatility of the framework is demonstrated by designing and evaluating a composed NIDS example based on systems described in the literature and using an open data set. Studying the impact of an evasion attempt with adversarial examples on this system highlighted the importance of robustness against false-alarms as well as detection evasion. Moreover, the framework enables general insights on how to improve the design of composed NIDSs: based on the dynamics between classifiers, it can be shown that optimizing the operation point of each component individually does not necessarily maximize the overall system performance from an information-theoretic perspective. Additionally, it can be shown that existing classification redundancies might not be fully utilized during an attack on the NIDS components, due to a static system design.

BGGMM-HMT based locally optimum image watermark detector in high-order NSST difference domain

Imperceptibility, robustness and data payload are three main requirements of any image watermarking systems to guarantee desired functionalities, but there is a tradeoff among them from the information-theoretic perspective. How to achieve this balance is a major challenge. In this paper, we propose a new statistical image watermarking scheme, which is based on the high-order difference coefficients in nonsubsampled Shearlet transform (NSST) domain and the bounded generalized Gaussian mixture model-based hidden Markov tree (BGGMM-HMT). In the watermark embedding process, we use a nonlinear embedding approach to hide the digital watermark into the robust high-order difference coefficients, which can achieve better imperceptibility. In the watermark detection process, high-order difference coefficients are accurately modeled by using BGGMM-HMT, where the distribution characteristics of high-order difference coefficients can be captured through BGGMM, and the scale dependencies of high-order difference coefficients can be captured through HMT. Statistical model parameters are then estimated by combining the approach of minimizing the higher bound on data negative log-likelihood function and upward–downward algorithm. Finally, an image watermark detector based on BGGMM-HMT is developed using the locally optimum (LO) decision rule. For the proposed detector, the receiver operating characteristic (ROC) expression is derived in detail. We evaluate the proposed scheme from different aspects and compare it with the state-of-the-art schemes. After a large number of experimental tests, the encouraging results obtained prove the effectiveness of our watermarking scheme.

Design and Analysis of Coded Caching Schemes in Stochastic Wireless Networks

Coded caching is a technique that promises significant reductions in network traffic by exploiting the multicast opportunities among multiple cache-enabled users. Most works in this area investigate the fundamental performance limits of coded caching from an information-theoretic perspective. In this paper, from a practical perspective, we focus on the design and analysis of the coded caching scheme in large-scale stochastic networks with dynamic traffic. Aiming at the challenges of the large number of users, dynamic packet requests, and distance-dependent interference, the packet caching scheme and packet delivery scheme are jointly designed to ensure the availability of the coded caching gain. To characterize the coded caching performance, the queue dynamic of the packet requests and channel state are first analyzed. Then by combining the successive group decoding theory and stochastic geometry approach, the network interference is characterized and the successful transmission probability of the multicast coded signal is further derived. Moreover, the coded caching scheme is optimized to minimize the average delay of the packet request, which is proved to ensure service fairness and reduce the complexities of network performance analysis. The analytical and numerical results provide useful insights for the provisioning and planning of the coded caching network.

Adaptive Data Dimensionality Reduction for Chemical Process Modeling Based on the Information Criterion Related to Data Association and Redundancy

Chemical process modeling is the basis for research and applications in related fields. With the development of industrial informatization, data-driven process modeling techniques are increasingly applied in chemical processes, helping to obtain more accurate results with less model development costs. However, due to the high-dimensional nonlinear characteristics of most chemical processes, problems such as the “curse of dimensionality” and information redundancy will render the models more prone to overfitting with reduced accuracies and weaker generalization abilities. Many data dimensionality reduction methods are adopted to mitigate the above problems, but most are limited by inaccurate association measurements and weak redundancy exclusion. In this paper, the extensive existence of data associations and information redundancies is first revealed through the analysis from an information-theoretic perspective. Then, a feature selection method based on conditional refined maximal information coefficient maximization (CRMICM) is proposed to improve the consistency of association measurement and the accuracy of redundancy estimation with limited samples. The final prediction modeling test for an actual fluidized catalytic cracking (FCC) process proves the extensive association between the variables and targets. Only a few variables are essential for the modeling, while the rest are redundant. Compared with other methods, CRMICM achieves the best dimensionality reduction effects on the FCC process data regarding the number of features and model prediction accuracy, showing its good applicability for chemical processes.

The Structure and Dynamics of Modern United States Federal Case Law

The structure and dynamics of modern United States Federal Case Law are examined here. The analyses utilize large-scale network analysis tools, natural language processing techniques, and information theory to examine all the federal opinions in the Court Listener database, containing approximately 1.3 million judicial opinions and 11.4 million citations. The analyses are focused on modern United States Federal Case Law, as cases in the Court Listener database range from approximately 1926–2020 and include most Federal jurisdictions. We examine the data set from a structural perspective using the citation network, overall and by time and space (jurisdiction). In addition to citation structure, we examine the dataset from a topical and information theoretic perspective, again, overall and by time and space.