Information-theoretic Methods Research Articles

Information Theory in a Darwinian Evolution Population Dynamics Model

Since Darwin, evolutionary population dynamics has captivated scientists and has applications beyond biology, such as in game theory where economists use it to explore evolution in new ways. This approach has renewed interest in dynamic evolutionary systems. In this paper, we propose an information-theoretic method to estimate trait parameters in a Darwinian model for species with single or multiple traits. Using Fisher information, we assess estimation errors and demonstrate the method through simulations.

Information-Theoretic Modeling of Categorical Spatiotemporal GIS Data.

An information-theoretic data mining method is employed to analyze categorical spatiotemporal Geographic Information System land use data. Reconstructability Analysis (RA) is a maximum-entropy-based data modeling methodology that works exclusively with discrete data such as those in the National Land Cover Database (NLCD). The NLCD is organized into a spatial (raster) grid and data are available in a consistent format for every five years from 2001 to 2021. An NLCD tool reports how much change occurred for each category of land use; for the study area examined, the most dynamic class is Evergreen Forest (EFO), so the presence or absence of EFO in 2021 was chosen as the dependent variable that our data modeling attempts to predict. RA predicts the outcome with approximately 80% accuracy using a sparse set of cells from a spacetime data cube consisting of neighboring lagged-time cells. When the predicting cells are all Shrubs and Grasses, there is a high probability for a 2021 state of EFO, while when the predicting cells are all EFO, there is a high probability that the 2021 state will not be EFO. These findings are interpreted as detecting forest clear-cut cycles that show up in the data and explain why this class is so dynamic. This study introduces a new approach to analyzing GIS categorical data and expands the range of applications that this entropy-based methodology can successfully model.

A Bayesian Tensor Decomposition Method for Joint Estimation of Channel and Interference Parameters.

Bayesian tensor decomposition has been widely applied in channel parameter estimations, particularly in cases with the presence of interference. However, the types of interference are not considered in Bayesian tensor decomposition, making it difficult to accurately estimate the interference parameters. In this paper, we present a robust tensor variational method using a CANDECOMP/PARAFAC (CP)-based additive interference model for multiple input-multiple output (MIMO) with orthogonal frequency division multiplexing (OFDM) systems. A more realistic interference model compared to traditional colored noise is considered in terms of co-channel interference (CCI) and front-end interference (FEI). In contrast to conventional algorithms that filter out interference, the proposed method jointly estimates the channel and interference parameters in the time-frequency domain. Simulation results validate the correctness of the proposed method by the evidence lower bound (ELBO) and reveal the fact that the proposed method outperforms traditional information-theoretic methods, tensor decomposition models, and robust model based on CP (RCP) in terms of estimation accuracy. Further, the interference parameter estimation technique has profound implications for anti-interference applications and dynamic spectrum allocation.

An Information-Theoretic Method for Detecting Edits in AI-Generated Text

An Information-Theoretic Method for Detecting Edits in AI-Generated Text

Information-theoretical limit on the estimates of dissipation by molecular machines using single-molecule fluorescence resonance energy transfer experiments.

Single-molecule fluorescence resonance energy transfer (FRET) experiments are commonly used to study the dynamics of molecular machines. While in vivo molecular processes often break time-reversal symmetry, the temporal directionality of cyclically operating molecular machines is often not evident from single-molecule FRET trajectories, especially in the most common two-color FRET studies. Solving a more quantitative problem of estimating the energy dissipation/entropy production by a molecular machine from single-molecule data is even more challenging. Here, we present a critical assessment of several practical methods of doing so, including Markov-model-based methods and a model-free approach based on an information-theoretical measure of entropy production that quantifies how (statistically) dissimilar observed photon sequences are from their time reverses. The Markov model approach is computationally feasible and may outperform model free approaches, but its performance strongly depends on how well the assumed model approximates the true microscopic dynamics. Markov models are also not guaranteed to give a lower bound on dissipation. Meanwhile, model-free, information-theoretical methods systematically underestimate entropy production at low photoemission rates, and long memory effects in the photon sequences make these methods demanding computationally. There is no clear winner among the approaches studied here, and all methods deserve to belong to a comprehensive data analysis toolkit.

Clustering with Minimum Spanning Trees: How Good Can It Be?

AbstractMinimum spanning trees (MSTs) provide a convenient representation of datasets in numerous pattern recognition activities. Moreover, they are relatively fast to compute. In this paper, we quantify the extent to which they are meaningful in low-dimensional partitional data clustering tasks. By identifying the upper bounds for the agreement between the best (oracle) algorithm and the expert labels from a large battery of benchmark data, we discover that MST methods can be very competitive. Next, we review, study, extend, and generalise a few existing, state-of-the-art MST-based partitioning schemes. This leads to some new noteworthy approaches. Overall, the Genie and the information-theoretic methods often outperform the non-MST algorithms such as K-means, Gaussian mixtures, spectral clustering, Birch, density-based, and classical hierarchical agglomerative procedures. Nevertheless, we identify that there is still some room for improvement, and thus the development of novel algorithms is encouraged.

Information-Theoretic Security in Wireless Sensor Networks: Mathematical Models for Secure and Energy-Efficient Communication

Wireless Sensor Networks (WSNs) are used for a wide range of things these days, from watching the environment to improving healthcare systems. But because these networks are used everywhere, they are also open to security risks like spying, data corruption, and node capture. It is very important to address these issues in order to protect the accuracy, privacy, and access of data being sent. Information-theoretic security looks like a good way to deal with these problems because it uses mathematical models to make sure that transmission in WSNs is safe and uses little energy. This essay goes into detail about information-theoretic security in WSNs. It gives a full rundown of the mathematical basics and real-world implications for getting strong security in places with limited resources. The idea of "secrecy capacity" is at the heart of this method. It measures the highest rate at which authorized nodes can communicate privately while preventing spying foes. Utilizing the chance that exists in wireless channels, encrypted keys can be generated without using shared secrets, making them resistant to attempts that try to steal them. Because sensor nodes only have a short power life, energy economy is very important in WSNs. Information-theoretic security methods, like physical layer security and joint communication, are designed to use as little energy as possible while still providing strong security. Nodes can work together to improve signal strength and fight channel fading using cooperative methods. This makes the network last longer without compromising security. This article also talks about the trade-offs between security, energy savings, and communication performance in WSNs. This helps us figure out how to make the best transfer schemes.

Association of State Firearm Laws With Firearm Ownership and Mortality

IntroductionFirearm injury is a leading cause of death among Americans. As the right to bear arms is protected by the Second Amendment, policy-makers must consider the impact of legislation on both firearm ownership and firearm harms. Current state of knowledge in firearm research majorly examines the impact of firearm legislation on firearm injuries and fatalities alone, and it relies on correlational analyses. The few studies that consider causal effects employ counterfactual-based inference. This study introduces information-theoretic tools to explore the role of firearm laws in mitigating firearm harms while maintaining citizens’ right to bear arms MethodsWe study monthly time series from January 2000 to October 2019 for the implementation of firearm laws from RAND's State Firearm Law Database, firearm deaths by intent from the Centers for Disease Control and Prevention databases, and firearm ownership from an econometric model. We employ transfer entropy, an information-theoretic method that relies on Granger causality, to infer relationships from time series. Specifically, we examine transfer entropy from firearm restrictiveness to deaths per firearm owner, firearm ownership, and firearm deaths, independently ResultsOn a national level, we uncover a negative association from firearm restrictiveness to deaths per firearm owner and a positive association from firearm restrictiveness to firearm ownership. On a regional level, we identify a negative association from firearm restrictiveness to deaths per firearm owner in the Northeast, a negative association from firearm restrictiveness to firearm ownership in the Midwest, and a negative association from firearm restrictiveness to firearm suicides in the South ConclusionsWe present an information-theoretic approach to study relationships in firearm research. Our method provides preliminary evidence for the role of restrictive legislation in promoting safe firearm ownership. We find that firearm acquisition considerably increases following the implementation of restrictive firearm laws and, simultaneously, firearm deaths decrease. These effects vary with respect to death by intent and the geographical region laws were implemented in.

Progressive Feature Extraction with a Greedy Network-growing Algorithm

In this paper, a new information theoretic method called the greedy network-growing algorithm is proposed. The method is called "greedy," because a network with this algorithm grows while absorbing as much information as possible from outside. The method is based upon information theoretic competitive learning and can solve the fundamental problems inherent in competitive learning, such as the dead neurons and inappropriate number of neurons problems. The new model can grow networks by repeatedly maximizing information content and by gradually extracting salient features from input patterns. Because the new model can cope with inappropriate feature detection in the early stage of learning, extracted features should cover most of the input patterns. The new method is applied to political data analysis, medical data analysis, and information science education analysis. Experimental results confirm that the new method can acquire significant information and that more explicit features can be extracted.

Simulation testing performance of ensemble models when catch data are underreported

Abstract Ensemble model use in stock assessment is increasing, yet guidance on construction and an evaluation of performance relative to single models is lacking. Ensemble models can characterize structural uncertainty and avoid the conundrum of selecting a “best” assessment model when alternative models explain observed data equally well. Through simulation, we explore the importance of identifying candidate models for both assessment and short-term forecasts and the consequences of different ensemble weighting methods on estimated quantities. Ensemble performance exceeded a single best model only when the set of candidate models spanned the true model configuration. Accuracy and precision depended on the model weighting scheme, and varied between two case studies investigating the impact of catch accuracy. Information theoretic weighting methods performed well in the case study with accurate catch, while equal weighting performed best when catch was underreported. In both cases, equal weighting produced multimodality. Ensuring that an ensemble spans the true state of nature will be challenging, but we observed that a change in sign of Mohn’s rho across candidate models coincided with the true OM being bounded. Further development of protocols to select an objective and balanced set of candidate models, and diagnostics to assess adequacy of candidates are recommended.

Using Sequence Analyses to Quantitatively Measure Oropharyngeal Swallowing Temporality in Point-of-Care Ultrasound Examinations: A Pilot Study.

(1) Background: Swallowing is a complex process that comprises well-timed control of oropharyngeal and laryngeal structures to achieve airway protection and swallowing efficiency. To understand its temporality, previous research adopted adherence measures and revealed obligatory pairs in healthy swallows and the effect of aging and bolus type on the variability of event timing and order. This study aimed to (i) propose a systemic conceptualization of swallowing physiology, (ii) apply sequence analyses, a set of information-theoretic and bioinformatic methods, to quantify and characterize swallowing temporality, and (iii) investigate the effect of aging and dysphagia on the quantified variables using sequence analyses measures. (2) Method: Forty-three participants (17 young adults, 15 older adults, and 11 dysphagic adults) underwent B-mode ultrasound swallowing examinations at the mid-sagittal plane of the submental region. The onset, maximum, and offset states of hyoid bone displacement, geniohyoid muscle contraction, and tongue base retraction were identified and sorted to form sequences which were analyzed using an inventory of sequence analytic techniques; namely, overlap coefficients, Shannon entropy, and longest common subsequence algorithms. (3) Results: The concurrency of movement sequence was found to be significantly impacted by aging and dysphagia. Swallowing sequence variability was also found to be reduced with age and the presence of dysphagia (H(2) = 52.253, p < 0.001, η2 = 0.260). Four obligatory sequences were identified, and high adherence was also indicated in two previously reported pairs. These results provided preliminary support for the validity of sequence analyses for quantifying swallowing sequence temporality. (4) Conclusions: A systemic conceptualization of human deglutition permits a multi-level quantitative analysis of swallowing physiology. Sequence analyses are a set of promising quantitative measurement techniques for point-of-care ultrasound (POCUS) swallowing examinations and outcome measures for swallowing rehabilitation and evaluation of associated physiological conditions, such as sarcopenia. Findings in the current study revealed physiological differences among healthy young, healthy older, and dysphagic adults. They also helped lay the groundwork for future AI-assisted dysphagia assessment and outcome measures using POCUSs. Arguably, the proposed conceptualization and analyses are also modality-independent measures that can potentially be generalized for other instrumental swallowing assessment modalities.

Understanding the Distributions of Aggregation Layers in Deep Neural Networks.

The process of aggregation is ubiquitous in almost all the deep nets' models. It functions as an important mechanism for consolidating deep features into a more compact representation while increasing the robustness to overfitting and providing spatial invariance in deep nets. In particular, the proximity of global aggregation layers to the output layers of DNNs means that aggregated features directly influence the performance of a deep net. A better understanding of this relationship can be obtained using information theoretic methods. However, this requires knowledge of the distributions of the activations of aggregation layers. To achieve this, we propose a novel mathematical formulation for analytically modeling the probability distributions of output values of layers involved with deep feature aggregation. An important outcome is our ability to analytically predict the Kullback-Leibler (KL)-divergence of output nodes in a DNN. We also experimentally verify our theoretical predictions against empirical observations across a broad range of different classification tasks and datasets.

Causality between streaks and bursts in wall-bounded turbulence

We use an information-theoretic method, referred to as information flux, to quantify the causal relationships between streaks and bursts in a non-intrusive manner. Within this framework, causality is quantified as the flux of Shannon information from the present of the quantities of interest to their future. We also use the so-called information leak to measure the information that is not accounted for due to unobserved variables. We investigate data from a direct numerical simulation of turbulent channel flow at the friction Reynolds number Re τ ≈ 180. The spatial distribution of causality is investigated as two time scales based on 50% value of the information leak and maximum of the normalized cross-induced information flux. It is found that the most causal spatial configuration for streaks and bursts is always streamwise-aligned. Furthermore, four dominant causal spatial configurations between streaks and bursts are identified. At the short time scale around 10 viscous units (when information flux is 50%), streaks and bursts have comparable causality to one another. At the longer time scale (corresponding to the maximum cross-induced causality), there is greater causality from streaks to bursts.

Practical forecasting of risk boundaries for industrial metals and critical minerals via statistical machine learning techniques

This study examines the application of machine learning in predicting price risk boundaries for industrial metals and critical minerals, emphasizing the role of statistical dependencies among their yields. Given these commodities’ pivotal role in various industries and their influence on the global economy, accurate forecasting of their price boundaries is critical. Our research employs statistical dependencies as key features to uncover meaningful correlations essential for effective forecasting, underscoring the value of analyzing rate-based statistical dependencies alongside price fluctuations. Applying explainable artificial intelligence (xAI) techniques validates the effectiveness of combining these approaches for price boundary prediction. Our findings indicate variations in effectiveness between the Pearson correlation coefficient and normalized mutual information, challenging the Pearson coefficient’s dominance in financial analysis. The study demonstrates that the relevance of each statistical dependency varies across different machine learning models, necessitating a comprehensive analytical approach incorporating information-theoretic methods. Our analysis improves the accuracy and interpretability of price predictions by pinpointing critical relationships among selected industrial metals and critical minerals.

Weighted multi-error information entropy based you only look once network for underwater object detection

Underwater object detection is considered as one of the most challenging issues in computer vision. In this paper, a weighted multi-error information entropy based YOLO (You Only Look Once) network is proposed to address underwater illumination noise affecting the detection accuracy. First, underwater illumination is essentially structural and non-uniform, and it is modeled as an independent and piecewise identical distribution, which is a generic noise model to describe the complex underwater illuminating environment and accommodates the traditional Gaussian distribution as a special case. Second, assisted by the proposed illumination noise model, a minimum weighted error entropy criterion, which is an information-theoretic learning method, is introduced into the loss function of YOLO network, and then the network parameters are trained and optimized to improve the detection performance. Furthermore, a multi-error processing strategy is simultaneously used to handle vector errors during information back-propagation in order to accelerate convergence. Experiments on underwater object detection datasets including URPC2018, URPC2019 and Enhanced dataset, show the proposed weighted multi-error information entropy based YOLOv8 network gets mean average precision (MAP) of 88.7%, 91.8% and 96.7% respectively, and average frames per second (FPS) of 116.6. These two evaluation metrics are better than the baseline YOLOv8 and the existing advanced non-YOLO approaches by at least 5.2% and 5.3% respectively. The results verify the effectiveness and superiority of the proposed network for underwater object detection in complex underwater environment.

Fallen Sanctuary: A Higher-Order and Leakage-Resilient Rekeying Scheme

This paper presents a provably secure, higher-order, and leakage-resilient (LR) rekeying scheme named LR Rekeying with Random oracle Repetition (LR4), along with a quantitative security evaluation methodology. Many existing LR primitives are based on a concept of leveled implementation, which still essentially require a leak-free sanctuary (i.e., differential power analysis (DPA)-resistant component(s)) for some parts. In addition, although several LR pseudorandom functions (PRFs) based on only bounded DPA-resistant components have been developed, their validity and effectiveness for rekeying usage still need to be determined. In contrast, LR4 is formally proven under a leakage model that captures the practical goal of side-channel attack (SCA) protection (e.g., masking with a practical order) and assumes no unbounded DPA-resistant sanctuary. This proof suggests that LR4 resists exponential invocations (up to the birthday bound of key size) without using any unbounded leak-free component, which is the first of its kind. Moreover, we present a quantitative SCA success rate evaluation methodology for LR4 that combines the bounded leakage models for LR cryptography and a state-of-the-art information-theoretical SCA evaluation method. We validate its soundness and effectiveness as a DPA countermeasure through a numerical evaluation; that is, the number of secure calls of a symmetric primitive increases exponentially by increasing a security parameter under practical conditions.

Fast Attack Detection Method for Imbalanced Data in Industrial Cyber-Physical Systems

Abstract Integrating industrial cyber-physical systems (ICPSs) with modern information technologies (5G, artificial intelligence, and big data analytics) has led to the development of industrial intelligence. Still, it has increased the vulnerability of such systems regarding cybersecurity. Traditional network intrusion detection methods for ICPSs are limited in identifying minority attack categories and suffer from high time complexity. To address these issues, this paper proposes a network intrusion detection scheme, which includes an information-theoretic hybrid feature selection method to reduce data dimensionality and the ALLKNN-LightGBM intrusion detection framework. Experimental results on three industrial datasets demonstrate that the proposed method outperforms four mainstream machine learning methods and other advanced intrusion detection techniques regarding accuracy, F-score, and run time complexity.

An Information-Theoretical Method for Comparing Completions of Contrapunctus XIV from Bach’s Art of Fugue

The unfinished final Contrapunctus from Bach’s posthumously-published Kunst der Fuge has received several dozen completions over the past century. Although the majority of these have been published and recorded, none has yet established themselves as the leading solution among performers, and this may have more to do with their perceived ‘Bach-ness’ than the undoubted contrapuntal skill and ingenuity that they each demonstrate. This study uses an established information-theoretical method drawn from mathematics to explore the extent to which it is possible to ‘measure’ the similarity of these reconstructions to the original, and so determine which are closest to the musical lines of Bach’s existing opening of Contrapunctus XIV.

Commentary on “An Information-Theoretical Method for Comparing Completions of Contrapunctus XIV from Bach’s Art of Fugue”

Information entropy can be a powerful tool for analyzing differences in musical style as suggested in Paz et al. (2022). The application of such a tool, however, requires unambiguous operationalization of the variables being measured, and establishment of clear relations between the variables and the distinguishing features of the style. Without these, interpreting entropy, or any other relative measure of difference, leads to no useful conclusions regarding the similarity between style.

ClearF++: Improved Supervised Feature Scoring Using Feature Clustering in Class-Wise Embedding and Reconstruction.

Feature selection methods are essential for accurate disease classification and identifying informative biomarkers. While information-theoretic methods have been widely used, they often exhibit limitations such as high computational costs. Our previously proposed method, ClearF, addresses these issues by using reconstruction error from low-dimensional embeddings as a proxy for the entropy term in the mutual information. However, ClearF still has limitations, including a nontransparent bottleneck layer selection process, which can result in unstable feature selection. To address these limitations, we propose ClearF++, which simplifies the bottleneck layer selection and incorporates feature-wise clustering to enhance biomarker detection. We compare its performance with other commonly used methods such as MultiSURF and IFS, as well as ClearF, across multiple benchmark datasets. Our results demonstrate that ClearF++ consistently outperforms these methods in terms of prediction accuracy and stability, even with limited samples. We also observe that employing the Deep Embedded Clustering (DEC) algorithm for feature-wise clustering improves performance, indicating its suitability for handling complex data structures with limited samples. ClearF++ offers an improved biomarker prioritization approach with enhanced prediction performance and faster execution. Its stability and effectiveness with limited samples make it particularly valuable for biomedical data analysis.