Information-theoretic Methods Research Articles

A Bayesian Tensor Decomposition Method for Joint Estimation of Channel and Interference Parameters.

Bayesian tensor decomposition has been widely applied in channel parameter estimations, particularly in cases with the presence of interference. However, the types of interference are not considered in Bayesian tensor decomposition, making it difficult to accurately estimate the interference parameters. In this paper, we present a robust tensor variational method using a CANDECOMP/PARAFAC (CP)-based additive interference model for multiple input-multiple output (MIMO) with orthogonal frequency division multiplexing (OFDM) systems. A more realistic interference model compared to traditional colored noise is considered in terms of co-channel interference (CCI) and front-end interference (FEI). In contrast to conventional algorithms that filter out interference, the proposed method jointly estimates the channel and interference parameters in the time-frequency domain. Simulation results validate the correctness of the proposed method by the evidence lower bound (ELBO) and reveal the fact that the proposed method outperforms traditional information-theoretic methods, tensor decomposition models, and robust model based on CP (RCP) in terms of estimation accuracy. Further, the interference parameter estimation technique has profound implications for anti-interference applications and dynamic spectrum allocation.

Information-theoretical limit on the estimates of dissipation by molecular machines using single-molecule fluorescence resonance energy transfer experiments.

Single-molecule fluorescence resonance energy transfer (FRET) experiments are commonly used to study the dynamics of molecular machines. While in vivo molecular processes often break time-reversal symmetry, the temporal directionality of cyclically operating molecular machines is often not evident from single-molecule FRET trajectories, especially in the most common two-color FRET studies. Solving a more quantitative problem of estimating the energy dissipation/entropy production by a molecular machine from single-molecule data is even more challenging. Here, we present a critical assessment of several practical methods of doing so, including Markov-model-based methods and a model-free approach based on an information-theoretical measure of entropy production that quantifies how (statistically) dissimilar observed photon sequences are from their time reverses. The Markov model approach is computationally feasible and may outperform model free approaches, but its performance strongly depends on how well the assumed model approximates the true microscopic dynamics. Markov models are also not guaranteed to give a lower bound on dissipation. Meanwhile, model-free, information-theoretical methods systematically underestimate entropy production at low photoemission rates, and long memory effects in the photon sequences make these methods demanding computationally. There is no clear winner among the approaches studied here, and all methods deserve to belong to a comprehensive data analysis toolkit.

Clustering with Minimum Spanning Trees: How Good Can It Be?

AbstractMinimum spanning trees (MSTs) provide a convenient representation of datasets in numerous pattern recognition activities. Moreover, they are relatively fast to compute. In this paper, we quantify the extent to which they are meaningful in low-dimensional partitional data clustering tasks. By identifying the upper bounds for the agreement between the best (oracle) algorithm and the expert labels from a large battery of benchmark data, we discover that MST methods can be very competitive. Next, we review, study, extend, and generalise a few existing, state-of-the-art MST-based partitioning schemes. This leads to some new noteworthy approaches. Overall, the Genie and the information-theoretic methods often outperform the non-MST algorithms such as K-means, Gaussian mixtures, spectral clustering, Birch, density-based, and classical hierarchical agglomerative procedures. Nevertheless, we identify that there is still some room for improvement, and thus the development of novel algorithms is encouraged.

Association of State Firearm Laws With Firearm Ownership and Mortality

IntroductionFirearm injury is a leading cause of death among Americans. As the right to bear arms is protected by the Second Amendment, policy-makers must consider the impact of legislation on both firearm ownership and firearm harms. Current state of knowledge in firearm research majorly examines the impact of firearm legislation on firearm injuries and fatalities alone, and it relies on correlational analyses. The few studies that consider causal effects employ counterfactual-based inference. This study introduces information-theoretic tools to explore the role of firearm laws in mitigating firearm harms while maintaining citizens’ right to bear arms MethodsWe study monthly time series from January 2000 to October 2019 for the implementation of firearm laws from RAND's State Firearm Law Database, firearm deaths by intent from the Centers for Disease Control and Prevention databases, and firearm ownership from an econometric model. We employ transfer entropy, an information-theoretic method that relies on Granger causality, to infer relationships from time series. Specifically, we examine transfer entropy from firearm restrictiveness to deaths per firearm owner, firearm ownership, and firearm deaths, independently ResultsOn a national level, we uncover a negative association from firearm restrictiveness to deaths per firearm owner and a positive association from firearm restrictiveness to firearm ownership. On a regional level, we identify a negative association from firearm restrictiveness to deaths per firearm owner in the Northeast, a negative association from firearm restrictiveness to firearm ownership in the Midwest, and a negative association from firearm restrictiveness to firearm suicides in the South ConclusionsWe present an information-theoretic approach to study relationships in firearm research. Our method provides preliminary evidence for the role of restrictive legislation in promoting safe firearm ownership. We find that firearm acquisition considerably increases following the implementation of restrictive firearm laws and, simultaneously, firearm deaths decrease. These effects vary with respect to death by intent and the geographical region laws were implemented in.

Simulation testing performance of ensemble models when catch data are underreported

Abstract Ensemble model use in stock assessment is increasing, yet guidance on construction and an evaluation of performance relative to single models is lacking. Ensemble models can characterize structural uncertainty and avoid the conundrum of selecting a “best” assessment model when alternative models explain observed data equally well. Through simulation, we explore the importance of identifying candidate models for both assessment and short-term forecasts and the consequences of different ensemble weighting methods on estimated quantities. Ensemble performance exceeded a single best model only when the set of candidate models spanned the true model configuration. Accuracy and precision depended on the model weighting scheme, and varied between two case studies investigating the impact of catch accuracy. Information theoretic weighting methods performed well in the case study with accurate catch, while equal weighting performed best when catch was underreported. In both cases, equal weighting produced multimodality. Ensuring that an ensemble spans the true state of nature will be challenging, but we observed that a change in sign of Mohn’s rho across candidate models coincided with the true OM being bounded. Further development of protocols to select an objective and balanced set of candidate models, and diagnostics to assess adequacy of candidates are recommended.

Using Sequence Analyses to Quantitatively Measure Oropharyngeal Swallowing Temporality in Point-of-Care Ultrasound Examinations: A Pilot Study.

(1) Background: Swallowing is a complex process that comprises well-timed control of oropharyngeal and laryngeal structures to achieve airway protection and swallowing efficiency. To understand its temporality, previous research adopted adherence measures and revealed obligatory pairs in healthy swallows and the effect of aging and bolus type on the variability of event timing and order. This study aimed to (i) propose a systemic conceptualization of swallowing physiology, (ii) apply sequence analyses, a set of information-theoretic and bioinformatic methods, to quantify and characterize swallowing temporality, and (iii) investigate the effect of aging and dysphagia on the quantified variables using sequence analyses measures. (2) Method: Forty-three participants (17 young adults, 15 older adults, and 11 dysphagic adults) underwent B-mode ultrasound swallowing examinations at the mid-sagittal plane of the submental region. The onset, maximum, and offset states of hyoid bone displacement, geniohyoid muscle contraction, and tongue base retraction were identified and sorted to form sequences which were analyzed using an inventory of sequence analytic techniques; namely, overlap coefficients, Shannon entropy, and longest common subsequence algorithms. (3) Results: The concurrency of movement sequence was found to be significantly impacted by aging and dysphagia. Swallowing sequence variability was also found to be reduced with age and the presence of dysphagia (H(2) = 52.253, p < 0.001, η2 = 0.260). Four obligatory sequences were identified, and high adherence was also indicated in two previously reported pairs. These results provided preliminary support for the validity of sequence analyses for quantifying swallowing sequence temporality. (4) Conclusions: A systemic conceptualization of human deglutition permits a multi-level quantitative analysis of swallowing physiology. Sequence analyses are a set of promising quantitative measurement techniques for point-of-care ultrasound (POCUS) swallowing examinations and outcome measures for swallowing rehabilitation and evaluation of associated physiological conditions, such as sarcopenia. Findings in the current study revealed physiological differences among healthy young, healthy older, and dysphagic adults. They also helped lay the groundwork for future AI-assisted dysphagia assessment and outcome measures using POCUSs. Arguably, the proposed conceptualization and analyses are also modality-independent measures that can potentially be generalized for other instrumental swallowing assessment modalities.

Understanding the Distributions of Aggregation Layers in Deep Neural Networks.

The process of aggregation is ubiquitous in almost all the deep nets' models. It functions as an important mechanism for consolidating deep features into a more compact representation while increasing the robustness to overfitting and providing spatial invariance in deep nets. In particular, the proximity of global aggregation layers to the output layers of DNNs means that aggregated features directly influence the performance of a deep net. A better understanding of this relationship can be obtained using information theoretic methods. However, this requires knowledge of the distributions of the activations of aggregation layers. To achieve this, we propose a novel mathematical formulation for analytically modeling the probability distributions of output values of layers involved with deep feature aggregation. An important outcome is our ability to analytically predict the Kullback-Leibler (KL)-divergence of output nodes in a DNN. We also experimentally verify our theoretical predictions against empirical observations across a broad range of different classification tasks and datasets.

Causality between streaks and bursts in wall-bounded turbulence

We use an information-theoretic method, referred to as information flux, to quantify the causal relationships between streaks and bursts in a non-intrusive manner. Within this framework, causality is quantified as the flux of Shannon information from the present of the quantities of interest to their future. We also use the so-called information leak to measure the information that is not accounted for due to unobserved variables. We investigate data from a direct numerical simulation of turbulent channel flow at the friction Reynolds number Re τ ≈ 180. The spatial distribution of causality is investigated as two time scales based on 50% value of the information leak and maximum of the normalized cross-induced information flux. It is found that the most causal spatial configuration for streaks and bursts is always streamwise-aligned. Furthermore, four dominant causal spatial configurations between streaks and bursts are identified. At the short time scale around 10 viscous units (when information flux is 50%), streaks and bursts have comparable causality to one another. At the longer time scale (corresponding to the maximum cross-induced causality), there is greater causality from streaks to bursts.

Practical forecasting of risk boundaries for industrial metals and critical minerals via statistical machine learning techniques

This study examines the application of machine learning in predicting price risk boundaries for industrial metals and critical minerals, emphasizing the role of statistical dependencies among their yields. Given these commodities’ pivotal role in various industries and their influence on the global economy, accurate forecasting of their price boundaries is critical. Our research employs statistical dependencies as key features to uncover meaningful correlations essential for effective forecasting, underscoring the value of analyzing rate-based statistical dependencies alongside price fluctuations. Applying explainable artificial intelligence (xAI) techniques validates the effectiveness of combining these approaches for price boundary prediction. Our findings indicate variations in effectiveness between the Pearson correlation coefficient and normalized mutual information, challenging the Pearson coefficient’s dominance in financial analysis. The study demonstrates that the relevance of each statistical dependency varies across different machine learning models, necessitating a comprehensive analytical approach incorporating information-theoretic methods. Our analysis improves the accuracy and interpretability of price predictions by pinpointing critical relationships among selected industrial metals and critical minerals.

Weighted multi-error information entropy based you only look once network for underwater object detection

Underwater object detection is considered as one of the most challenging issues in computer vision. In this paper, a weighted multi-error information entropy based YOLO (You Only Look Once) network is proposed to address underwater illumination noise affecting the detection accuracy. First, underwater illumination is essentially structural and non-uniform, and it is modeled as an independent and piecewise identical distribution, which is a generic noise model to describe the complex underwater illuminating environment and accommodates the traditional Gaussian distribution as a special case. Second, assisted by the proposed illumination noise model, a minimum weighted error entropy criterion, which is an information-theoretic learning method, is introduced into the loss function of YOLO network, and then the network parameters are trained and optimized to improve the detection performance. Furthermore, a multi-error processing strategy is simultaneously used to handle vector errors during information back-propagation in order to accelerate convergence. Experiments on underwater object detection datasets including URPC2018, URPC2019 and Enhanced dataset, show the proposed weighted multi-error information entropy based YOLOv8 network gets mean average precision (MAP) of 88.7%, 91.8% and 96.7% respectively, and average frames per second (FPS) of 116.6. These two evaluation metrics are better than the baseline YOLOv8 and the existing advanced non-YOLO approaches by at least 5.2% and 5.3% respectively. The results verify the effectiveness and superiority of the proposed network for underwater object detection in complex underwater environment.

Fallen Sanctuary: A Higher-Order and Leakage-Resilient Rekeying Scheme

This paper presents a provably secure, higher-order, and leakage-resilient (LR) rekeying scheme named LR Rekeying with Random oracle Repetition (LR4), along with a quantitative security evaluation methodology. Many existing LR primitives are based on a concept of leveled implementation, which still essentially require a leak-free sanctuary (i.e., differential power analysis (DPA)-resistant component(s)) for some parts. In addition, although several LR pseudorandom functions (PRFs) based on only bounded DPA-resistant components have been developed, their validity and effectiveness for rekeying usage still need to be determined. In contrast, LR4 is formally proven under a leakage model that captures the practical goal of side-channel attack (SCA) protection (e.g., masking with a practical order) and assumes no unbounded DPA-resistant sanctuary. This proof suggests that LR4 resists exponential invocations (up to the birthday bound of key size) without using any unbounded leak-free component, which is the first of its kind. Moreover, we present a quantitative SCA success rate evaluation methodology for LR4 that combines the bounded leakage models for LR cryptography and a state-of-the-art information-theoretical SCA evaluation method. We validate its soundness and effectiveness as a DPA countermeasure through a numerical evaluation; that is, the number of secure calls of a symmetric primitive increases exponentially by increasing a security parameter under practical conditions.

Fast Attack Detection Method for Imbalanced Data in Industrial Cyber-Physical Systems

Abstract Integrating industrial cyber-physical systems (ICPSs) with modern information technologies (5G, artificial intelligence, and big data analytics) has led to the development of industrial intelligence. Still, it has increased the vulnerability of such systems regarding cybersecurity. Traditional network intrusion detection methods for ICPSs are limited in identifying minority attack categories and suffer from high time complexity. To address these issues, this paper proposes a network intrusion detection scheme, which includes an information-theoretic hybrid feature selection method to reduce data dimensionality and the ALLKNN-LightGBM intrusion detection framework. Experimental results on three industrial datasets demonstrate that the proposed method outperforms four mainstream machine learning methods and other advanced intrusion detection techniques regarding accuracy, F-score, and run time complexity.

An Information-Theoretical Method for Comparing Completions of Contrapunctus XIV from Bach’s Art of Fugue

The unfinished final Contrapunctus from Bach’s posthumously-published Kunst der Fuge has received several dozen completions over the past century. Although the majority of these have been published and recorded, none has yet established themselves as the leading solution among performers, and this may have more to do with their perceived ‘Bach-ness’ than the undoubted contrapuntal skill and ingenuity that they each demonstrate. This study uses an established information-theoretical method drawn from mathematics to explore the extent to which it is possible to ‘measure’ the similarity of these reconstructions to the original, and so determine which are closest to the musical lines of Bach’s existing opening of Contrapunctus XIV.

Commentary on “An Information-Theoretical Method for Comparing Completions of Contrapunctus XIV from Bach’s Art of Fugue”

Information entropy can be a powerful tool for analyzing differences in musical style as suggested in Paz et al. (2022). The application of such a tool, however, requires unambiguous operationalization of the variables being measured, and establishment of clear relations between the variables and the distinguishing features of the style. Without these, interpreting entropy, or any other relative measure of difference, leads to no useful conclusions regarding the similarity between style.

ClearF++: Improved Supervised Feature Scoring Using Feature Clustering in Class-Wise Embedding and Reconstruction.

Feature selection methods are essential for accurate disease classification and identifying informative biomarkers. While information-theoretic methods have been widely used, they often exhibit limitations such as high computational costs. Our previously proposed method, ClearF, addresses these issues by using reconstruction error from low-dimensional embeddings as a proxy for the entropy term in the mutual information. However, ClearF still has limitations, including a nontransparent bottleneck layer selection process, which can result in unstable feature selection. To address these limitations, we propose ClearF++, which simplifies the bottleneck layer selection and incorporates feature-wise clustering to enhance biomarker detection. We compare its performance with other commonly used methods such as MultiSURF and IFS, as well as ClearF, across multiple benchmark datasets. Our results demonstrate that ClearF++ consistently outperforms these methods in terms of prediction accuracy and stability, even with limited samples. We also observe that employing the Deep Embedded Clustering (DEC) algorithm for feature-wise clustering improves performance, indicating its suitability for handling complex data structures with limited samples. ClearF++ offers an improved biomarker prioritization approach with enhanced prediction performance and faster execution. Its stability and effectiveness with limited samples make it particularly valuable for biomedical data analysis.

Irreversibility, QNEC, and defects

We first present an analysis of infinitesimal null deformations for the entanglement entropy, which leads to a major simplification of the proof of the C, F and A-theorems in quantum field theory. Next, we study the quantum null energy condition (QNEC) on the light-cone for a CFT. Finally, we combine these tools in order to establish the irreversibility of renormalization group flows on planar d-dimensional defects, embedded in D-dimensional conformal field theories. This proof completes and unifies all known defect irreversibility theorems for defect dimensions d ≤ 4. The F-theorem on defects (d = 3) is a new result using information-theoretic methods. For d ≥ 4 we also establish the monotonicity of the relative entropy coefficient proportional to Rd−4. The geometric construction connects the proof of irreversibility with and without defects through the QNEC inequality in the bulk, and makes contact with the proof of strong subadditivity of holographic entropy taking into account quantum corrections.

Patient-independent epileptic seizure detection by stable feature selection

The detection of epileptic seizures in EEG signals is a challenging task because it requires careful review of multi-channel EEG recordings over a lengthy time interval. In general, EEG-based seizure detection is strongly dependent on the ability to select descriptive features that are also stable in the sense that they are not sensitive to changes in the training data. This study proposes and investigates a patient-independent seizure detection model that uses stable EEG-based features obtained by comparing multiple feature selection methods. The schemes considered can be divided into five categories, often referred to as similarity, information theoretic, sparse learning, statistical, and graph centrality feature selection methods. The stability of the features these schemes produce was evaluated by random forest classification, using the intersection, frequency, correlation, and similarity measures. In experiments with Temple University Hospital’s Seizure database of EEG records of 341 patients, unsupervised graph centrality feature selection was the most effective method, with a correct classification rate of 91.5%.

An information theoretic method to resolve millisecond-scale spike timing precision in a comprehensive motor program.

Sensory inputs in nervous systems are often encoded at the millisecond scale in a precise spike timing code. There is now growing evidence in behaviors ranging from slow breathing to rapid flight for the prevalence of precise timing encoding in motor systems. Despite this, we largely do not know at what scale timing matters in these circuits due to the difficulty of recording a complete set of spike-resolved motor signals and assessing spike timing precision for encoding continuous motor signals. We also do not know if the precision scale varies depending on the functional role of different motor units. We introduce a method to estimate spike timing precision in motor circuits using continuous MI estimation at increasing levels of added uniform noise. This method can assess spike timing precision at fine scales for encoding rich motor output variation. We demonstrate the advantages of this approach compared to a previously established discrete information theoretic method of assessing spike timing precision. We use this method to analyze the precision in a nearly complete, spike resolved recording of the 10 primary wing muscles control flight in an agile hawk moth, Manduca sexta. Tethered moths visually tracked a robotic flower producing a range of turning (yaw) torques. We know that all 10 muscles in this motor program encode the majority of information about yaw torque in spike timings, but we do not know whether individual muscles encode motor information at different levels of precision. We demonstrate that the scale of temporal precision in all motor units in this insect flight circuit is at the sub-millisecond or millisecond-scale, with variation in precision scale present between muscle types. This method can be applied broadly to estimate spike timing precision in sensory and motor circuits in both invertebrates and vertebrates.

Genome entropy and network centrality contrast exploration and exploitation in evolution of foodborne pathogens

Modelling evolution of foodborne pathogens is crucial for mitigation and prevention of outbreaks. We apply network-theoretic and information-theoretic methods to trace evolutionary pathways of Salmonella Typhimurium in New South Wales, Australia, by studying whole genome sequencing surveillance data over a five-year period which included several outbreaks. The study derives both undirected and directed genotype networks based on genetic proximity, and relates the network’s structural property (centrality) to its functional property (prevalence). The centrality-prevalence space derived for the undirected network reveals a salient exploration-exploitation distinction across the pathogens, further quantified by the normalised Shannon entropy and the Fisher information of the corresponding shell genome. This distinction is also analysed by tracing the probability density along evolutionary paths in the centrality-prevalence space. We quantify the evolutionary pathways, and show that pathogens exploring the evolutionary search-space during the considered period begin to exploit their environment (their prevalence increases resulting in outbreaks), but eventually encounter a bottleneck formed by epidemic containment measures.

Empirical Validation of Automated Vulnerability Curation and Characterization

Prior research has shown that public vulnerability systems such as US National Vulnerability Database (NVD) rely on a manual, time-consuming, and error-prone process which has led to inconsistencies and delays in releasing final vulnerability results. This work provides an approach to curate vulnerability reports in real-time and map textual vulnerability reports to machine readable structured vulnerability attribute data. Designed to support the time consuming human analysis done by vulnerability databases, the system leverages the Common Vulnerabilities and Exposures (CVE) list of vulnerabilities and the vulnerability attributes described by the National Institute of Standards and Technology (NIST) Vulnerability Description Ontology (VDO) framework. Our work uses Natural Language Processing (NLP), Machine Learning (ML) and novel Information Theoretical (IT) methods to provide automated techniques for near real-time publishing, and characterization of vulnerabilities using 28 attributes in 5 domains. Experiment results indicate that vulnerabilities can be evaluated up to 95 hours earlier than using manual methods, they can be characterized with F-Measure values over 0.9, and the proposed automated approach could save up to 47% of the time spent for CVE characterization.