Information-theoretic Properties Research Articles

Language prediction in monolingual and bilingual speakers: an EEG study

Prediction of upcoming words is thought to be crucial for language comprehension. Here, we are asking whether bilingualism entails changes to the electrophysiological substrates of prediction. Prior findings leave it open whether monolingual and bilingual speakers predict upcoming words to the same extent and in the same manner. We address this issue with a naturalistic approach, employing an information-theoretic metric, surprisal, to predict and contrast the N400 brain potential in monolingual and bilingual speakers. We recruited 18 Iranian Azeri-Persian bilingual speakers and 22 Persian monolingual speakers. Subjects listened to a story in Persian while their electroencephalogram (EEG) was recorded. Bayesian item-level analysis was used. While in monolingual speakers N400 was sensitive to information-theoretic properties of both the current and previous words, in bilingual speakers N400 reflected the properties of the previous word only. Our findings show evidence for a processing delay in bilingual speakers which is consistent with prior research.

Deterministic K-Identification for Future Communication Networks: The Binary Symmetric Channel Results

Numerous applications of the Internet of Things (IoT) feature an event recognition behavior where the established Shannon capacity is not authorized to be the central performance measure. Instead, the identification capacity for such systems is considered to be an alternative metric, and has been developed in the literature. In this paper, we develop deterministic K-identification (DKI) for the binary symmetric channel (BSC) with and without a Hamming weight constraint imposed on the codewords. This channel may be of use for IoT in the context of smart system technologies, where sophisticated communication models can be reduced to a BSC for the aim of studying basic information theoretical properties. We derive inner and outer bounds on the DKI capacity of the BSC when the size of the goal message set K may grow in the codeword length n. As a major observation, we find that, for deterministic encoding, assuming that K grows exponentially in n, i.e., K=2nκ, where κ is the identification goal rate, then the number of messages that can be accurately identified grows exponentially in n, i.e., 2nR, where R is the DKI coding rate. Furthermore, the established inner and outer bound regions reflects impact of the input constraint (Hamming weight) and the channel statistics, i.e., the cross-over probability.

The reality of backdoored S-Boxes—An eye opener

The real-life incidents researched in academia have revealed that (possibly) state-level efforts are made to camouflage the intentional flaws in the mathematical layer of an S-Box for exploiting the information-theoretic properties, i.e., Kuznyechik. To investigate the common features in the intentionally weakened mappings, this research thoroughly examines the backdoored structures from the perspective of 24 cryptanalytic attack vectors available in the open literature. We have debunked the earlier claims by the backdoor engineers that the proposed designs are stealthy against statistical distinguishers. Although a backdoored architecture fulfils the notions of randomness, it lacks the strength to resist sophisticated cryptanalytic attacks. Our analysis has revealed that during the backdoor insertion phase, a malicious designer compromises vital cryptographic properties, prominently the algebraic degree, differential trails, avalanche characteristics and leaving the open ground for hybrid cryptanalytic attacks. It is also observed that these mappings attain the upper bound of BCT, FBCT and DLCT, paving the way for hybrid attacks with high probability and lower data complexity.

Information from Noise: Measuring Dyslexia Risk Using Rasch-like Matrix Factorization with a Procedure for Equating Instruments.

This study examines the psychometric properties of a screening protocol for dyslexia and demonstrates a special form of matrix factorization called Nous based on the Alternating Least Squares algorithm. Dyslexia presents an intrinsically multidimensional complex of cognitive loads. By building and enforcing a common 6-dimensional space, Nous extracts a multidimensional signal for each person and item from test data that increases the Shannon entropy of the dataset while at the same time being constrained to meet the special objectivity requirements of the Rasch model. The resulting Dyslexia Risk Scale (DRS) yields linear equal-interval measures that are comparable regardless of the subset of items taken by the examinee. Each measure and cell estimate is accompanied by an efficiently calculated standard error. By incorporating examinee age into the calibration process, the DRS can be generalized to all age groups to allow the tracking of individual dyslexia risk over time. The methodology was implemented using a 2019 calibration sample of 828 persons aged 7 to 82 with varying degrees of dyslexia risk. The analysis yielded high reliability (0.95) and excellent receiver operating characteristics (AUC = 0.96). The analysis is accompanied by a discussion of the information-theoretic properties of matrix factorization.

Continuous majorization in quantum phase space

We explore the role of majorization theory in quantum phase space. To this purpose, we restrict ourselves to quantum states with positive Wigner functions and show that the continuous version of majorization theory provides an elegant and very natural approach to exploring the information-theoretic properties of Wigner functions in phase space. After identifying all Gaussian pure states as equivalent in the precise sense of continuous majorization, which can be understood in light of Hudson's theorem, we conjecture a fundamental majorization relation: any positive Wigner function is majorized by the Wigner function of a Gaussian pure state (especially, the bosonic vacuum state or ground state of the harmonic oscillator). As a consequence, any Schur-concave function of the Wigner function is lower bounded by the value it takes for the vacuum state. This implies in turn that the Wigner entropy is lower bounded by its value for the vacuum state, while the converse is notably not true. Our main result is then to prove this fundamental majorization relation for a relevant subset of Wigner-positive quantum states which are mixtures of the three lowest eigenstates of the harmonic oscillator. Beyond that, the conjecture is also supported by numerical evidence. We conclude by discussing some implications of this conjecture in the context of entropic uncertainty relations in phase space.

Observational entropy, coarse-grained states, and the Petz recovery map: information-theoretic properties and bounds

Observational entropy provides a general notion of quantum entropy that appropriately interpolates between Boltzmann’s and Gibbs’ entropies, and has recently been argued to provide a useful measure of out-of-equilibrium thermodynamic entropy. Here we study the mathematical properties of observational entropy from an information-theoretic viewpoint, making use of recently strengthened forms of the monotonicity property of quantum relative entropy. We present new bounds on observational entropy applying in general, as well as bounds and identities related to sequential and post-processed measurements. A central role in this work is played by what we call the ‘coarse-grained’ state, which emerges from the measurement’s statistics by Bayesian retrodiction, without presuming any knowledge about the ‘true’ underlying state being measured. The degree of distinguishability between such a coarse-grained state and the true (but generally unobservable) one is shown to provide upper and lower bounds on the difference between observational and von Neumann entropies.

Emergent Quantum State Designs from Individual Many-Body Wave Functions

A new perspective on the emergence of statistical behavior in chaotic quantum many-body systems is presented, establishing projected ensembles as a relevant tool for the characterization of information-theoretic properties.

Information-transfer characteristics in network motifs

Information processing in biological systems is realized by the appropriate transmission of information flows over complex networks, such as gene regulatory, signal transduction, and neural networks. These information flows are affected by the input-signal characteristics and structural properties of network systems, such as the network topology, regulation rules, and intrinsic and environmental noise. Many biological networks frequently include several typical patterns called network motifs, which are considered to play important roles in biological functions. However, their information-theoretic properties, particularly the dependence of the information flows in each network on the input signal, remain poorly understood. In our previous study [Mori and Okada, Phys. Rev. Res. 2, 043432 (2020)], we developed a graphical expansion method to describe transfer entropy (TE), a measure of information flow, in Boolean networks in terms of multiple information pathways. There, the input signal was limited to a simple case, and the effect of the input-signal characteristics on TE was not clarified. In this paper, we improve our method to render it applicable to Boolean networks that receive input signals with arbitrary stochastic characteristics. Our formula expresses how TE is determined by the input-signal characteristics, the assignment of Boolean functions, and the noise magnitude. We find that, in both positive and negative feedback loops, TE hardly depends on the signal timescale. In contrast, coherent and incoherent feedforward loops show low- and high-pass filtering properties, respectively, for a time-varying signal, which is consistent with previous reports. The emergence of either low- or high-pass filtering is determined by the Fourier components of the Boolean functions on specific pathways transmitting information flows. Thus our formula reveals the mechanism of information transfer in network motifs and provides insights into the origin of information processing in biological networks.

Information theoretical properties of a spiking neuron trained with Hebbian and STDP learning rules

AbstractUsing formal methods complemented by large-scale simulations we investigate information theoretical properties of spiking neurons trained using Hebbian and STDP learning rules. It is shown that weight space contains meta-stable states, which are points where the average weight change under the learning rule vanishes. These points may capture the random walker transiently. The dwell time in the vicinity of the meta-stable state is either quasi-infinite or very short and depends on the level of noise in the system. Moreover, important information theoretic quantities, such as the amount of information the neuron transmits are determined by the meta-stable state. While the Hebbian learning rule reliably leads to meta-stable states, the STDP rule tends to be unstable in the sense that for most choices of hyper-parameters the weights are not captured by meta-stable states, except for a restricted set of choices. It emerges that stochastic fluctuations play an important role in determining which meta-stable state the neuron takes. To understand this, we model the trajectory of the neuron through weight space as an inhomogeneous Markovian random walk, where the transition probabilities between states are determined by the statistics of the input signal.

Bounding Quantum Capacities via Partial Orders and Complementarity

Quantum capacities are fundamental quantities that are notoriously hard to compute and can exhibit surprising properties such as superadditivity. Thus, a vast amount of literature is devoted to finding tight and computable bounds on these capacities. We add a new viewpoint by giving operationally motivated bounds on several capacities, including the quantum capacity and private capacity of a quantum channel and the one-way distillable entanglement and private key of a quantum state. These bounds are generally phrased in terms of capacity quantities involving the complementary channel or state. As a tool to obtain these bounds, we discuss partial orders on quantum channels and states, such as the less noisy and the more capable order. Our bounds help to further understand the interplay between different capacities, as they give operational limitations on superadditivity and the difference between capacities in terms of the information-theoretic properties of the complementary channel or state. They can also be used as a new approach towards numerically bounding capacities, as discussed with some examples.

Bounds on gravitational brane couplings and tomography in AdS3 black hole microstates

We study information theoretic properties of planar black hole microstates in 2 + 1 dimensional asymptotically anti-de Sitter spacetime, modeled by black holes with an end-of-the-world brane behind the horizon. The von Neumann entropy of sufficiently large subregions in the dual CFT exhibits a time-dependent phase, which from a doubly-holographic perspective corresponds to the appearance of quantum extremal islands in the brane description. Considering the case where dilaton gravity is added to the brane, we show that tuning the associated couplings affects the propagation of information in the dual CFT state. By requiring that information theoretic bounds on the growth of entanglement entropy are satisfied in the dual CFT, we can place bounds on the allowed values of the couplings on the brane. Furthermore, we initiate the study of brane tomography, by showing how subleading corrections to the entanglement velocity can be used to learn about the properties of the brane as well as any gravitational dynamics localized on it.

Parameter andqasymptotics of Lq‐norms of hypergeometric orthogonal polynomials

AbstractThe three canonical families of the hypergeometric orthogonal polynomials in a continuous real variable (Hermite, Laguerre, and Jacobi) control the physical wavefunctions of the bound stationary states of a great number of quantum systems [Correction added after first online publication on 21 December, 2022. The sentence has been modified.]. The algebraic ‐norms of these polynomials describe many chemical, physical, and information theoretical properties of these systems, such as, for example, the kinetic and Weizsäcker energies, the position and momentum expectation values, the Rényi and Shannon entropies and the Cramér‐Rao, the Fisher‐Shannon and LMC measures of complexity. In this work, we examine review and solve theq‐asymptotics and the parameter asymptotics (i.e., when the weight function's parameter tends towards infinity) of the unweighted and weighted ‐norms for these orthogonal polynomials. This study has been motivated by the application of these algebraic norms to the energetic, entropic, and complexity‐like properties of the highly excited Rydberg and high‐dimensional pseudo‐classical states of harmonic (oscillator‐like) and Coulomb (hydrogenic) systems, and other quantum systems subject to central potentials of anharmonic type (such as, e.g., some molecular systems) [Correction added after first online publication on 21 December, 2022. Oscillatorlike has been changed to oscillator‐like.].

Irreducibly SU(2)-covariant quantum channels of low rank

We investigate information theoretic properties of low rank (less than or equal to 3) quantum channels with [Formula: see text]-symmetry, where we have a complete description. We prove that PPT property coincides with entanglement-breaking property and that degradability seldomly holds in this class. In connection with these results, we will demonstrate how we can compute Holevo and coherent information of those channels. In particular, we exhibit a strong form of additivity violation of coherent information, which resembles the superactivation of coherent information of depolarizing channels.

How to build high quality L2R training data: Unsupervised compression-based selective sampling for learning to rank

Learning to Rank (L2R) improves ranking quality but relies on the existence of manually labeled training sets, which are expensive and cumbersome to generate. Using automated labeling (e.g., clickthrough data) imposes its own challenges. Active learning (AL) can be used to gather high-quality training data by producing very informative yet small training sets. Cover, a method we have previously developed, allows for unsupervised sampling of training sets as good as those created using AL. In this paper we provide an extensive analysis of how and why Cover works. We revisit the method in a more formal way, with theorems and proofs, and provide additional empirical evidence of its practicality. We answer questions related to why Cover works so well and how its properties are related to AL methods. We show how certain characteristics of Cover’s clustering step allows it to more thoroughly explore the feature space by selecting query-document pairs that are representative and diverse, allowing L2R methods to produce effective models. The main novel contribution is a detailed analysis of the method’s inner workings and information-theoretic properties, allowing us to advance the understanding of L2R fundamentals through the lens of training set building.

Improved Bounds for Noisy Group Testing With Constant Tests per Item

The group testing problem is concerned with identifying a small set of infected individuals in a large population. At our disposal is a testing procedure that allows us to test several individuals together. In an idealized setting, a test is positive if and only if at least one infected individual is included and negative otherwise. Significant progress was made in recent years towards understanding the information-theoretic and algorithmic properties in this noiseless setting. In this paper, we consider a noisy variant of group testing where test results are flipped with certain probability, including the realistic scenario where sensitivity and specificity can take arbitrary values. Using a test design where each individual is assigned to a fixed number of tests, we derive explicit algorithmic bounds for two commonly considered inference algorithms and thereby naturally extend the results of Scarlett & Cevher (2016) and Scarlett & Johnson (2020). We provide improved performance guarantees for the efficient algorithms in these noisy group testing models – indeed, for a large set of parameter choices the bounds provided in the paper are the strongest currently proved.

Calculation of Information-Theoretic Properties of a Communication Channel with Real Signals and Non-Gaussian Noise

The paper proposes numerical methods to calculate the data transfer rate over a communication channel with real signals in the presence of a mixture of Gaussian and non-Gaussian correlated noise. This paper considers the transmission of discrete and continuous messages. The estimate of the data transmission rate given by the proposed methods is much more accurate than the estimation given by the Shannon limit. The proposed methods can be used for the information-theoretical analysis of the wide range of systems: systems of digital and analogous communication, radiolocation and radio navigation.

Quantifying the Autonomy of Structurally Diverse Automata: A Comparison of Candidate Measures.

Should the internal structure of a system matter when it comes to autonomy? While there is still no consensus on a rigorous, quantifiable definition of autonomy, multiple candidate measures and related quantities have been proposed across various disciplines, including graph-theory, information-theory, and complex system science. Here, I review and compare a range of measures related to autonomy and intelligent behavior. To that end, I analyzed the structural, information-theoretical, causal, and dynamical properties of simple artificial agents evolved to solve a spatial navigation task, with or without a need for associative memory. By contrast to standard artificial neural networks with fixed architectures and node functions, here, independent evolution simulations produced successful agents with diverse neural architectures and functions. This makes it possible to distinguish quantities that characterize task demands and input-output behavior, from those that capture intrinsic differences between substrates, which may help to determine more stringent requisites for autonomous behavior and the means to measure it.

Information-theoretic signal detection theory.

Signal detection theory (SDT), the standard mathematical framework by which we understand how stimuli are classified into distributions such as signal or noise, is an essential part of the modern psychologist's toolkit. This article introduces some mathematical tools derived from information theory which allow surprisingly simple approximations to key quantities in SDT. The main idea is a lower bound on the probability of correct classification of a stimulus, as a function of information-theoretic properties of the generating distribution. This bound depends on three distinct factors, each of which can be quantified information-theoretically: (a) The prior uncertainty in the choice of generating distribution; (b) the inherent separability of the classes; and (c) the discrepancy between the observer's model of the class distributions and the "true" model. The bound is only a loose substitute for the conventional method for computing proportion correct (via integration) but generalizes readily to multiple dimensions and larger numbers of stimulus categories, where direct integration is computationally difficult. Moreover, unlike most conventional SDT formulae, this bound does not require Gaussian distributions. Most importantly, the information-theoretic signal detection theory (IT-SDT) framework sheds light on the way classification performance depends on the discrepancy between the observer's assumptions and those actually governing the environment. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Asymptotic analysis for $$O_N^+$$-Temperley–Lieb quantum channels

Studies on conservation of quantum symmetries have been initiated by recent papers (Brannan et al. in Commun Math Phys 376(2):795–839, 2020; Lee et al. in Quantum channels with quantum group symmetry, 2020). We, in this paper, investigate how quantum symmetries can be applied to study additivity questions for covariant quantum channels, particularly for the so-called \(O_N^+\)-Temperley–Lieb channels. Their information-theoretic properties were analyzed in Brannan et al. (Commun Math Phys 376(2):795–839, 2020), but the associated additivity questions remained open due to highly nontrivial structures. We present a new approach to analyze \(O_N^+\)-Temperley–Lieb channels with small perturbations in terms of Bures distance within the asymptotic regime \(N\rightarrow \infty \) and apply continuity bounds of classical capacity, private classical capacity and quantum capacity to study strong additivity questions.

Multi-Domain Communication Systems and Networks: A Tensor-Based Approach

Most modern communication systems, such as those intended for deployment in IoT applications or 5G and beyond networks, utilize multiple domains for transmission and reception at the physical layer. Depending on the application, these domains can include space, time, frequency, users, code sequences, and transmission media, to name a few. As such, the design criteria of future communication systems must be cognizant of the opportunities and the challenges that exist in exploiting the multi-domain nature of the signals and systems involved for information transmission. Focussing on the Physical Layer, this paper presents a novel mathematical framework using tensors, to represent, design, and analyze multi-domain systems. Various domains can be integrated into the transceiver design scheme using tensors. Tools from multi-linear algebra can be used to develop simultaneous signal processing techniques across all the domains. In particular, we present tensor partial response signaling (TPRS) which allows the introduction of controlled interference within elements of a domain and also across domains. We develop the TPRS system using the tensor contracted convolution to generate a multi-domain signal with desired spectral and cross-spectral properties across domains. In addition, by studying the information theoretic properties of the multi-domain tensor channel, we present the trade-off between different domains that can be harnessed using this framework. Numerical examples for capacity and mean square error are presented to highlight the domain trade-off revealed by the tensor formulation. Furthermore, an application of the tensor framework to MIMO Generalized Frequency Division Multiplexing (GFDM) is also presented.