Information Theory Concepts Research Articles

Geotechnical engineering surveys and soil testing in the construction of buildings and structures

The article suggests a way to increase the efficiency of geotechnical engineering surveys through the rational use of fast approximate methods of soil testing (“express methods”). Such tests are recommended to be carried out in large volume and considered as a means of assessing the geotechnical structure of the site as a whole. In this paper, these issues are considered by the example of determining the bearing capacity of piles according to static sounding data and static tests of full-scale piles. A mathematical model for assessing the information content of the test complex based on the concepts of information theory is proposed. Calculations based on this model show that the informativeness of a large number of approximate tests may (due to the heterogeneity of the soil) exceed the informativeness of a small number of accurate tests.

Robust rateless space time block coding for mmWave massive MIMO system

In the vast growing wireless communication system creating robust encoding mechanisms using space-time block codes (STBC) for mmWave massive MIMO to overcome uncertainty and ensure reliability is a critical concept to be covered. This article reviews the core concepts behind MIMO communication, Massive MIMO communication, space-time block codes, and rateless codes in the context of wireless communication systems. Building on the foundational concepts of information theory and mmWave massive MIMO, we developed space-time block codes that maintain orthogonality for real-valued symbols. Following a thorough analysis of these codes, we successfully extended them into rateless orthogonal space-time block codes (ROSTBC) for massive MIMO. This extension enables the codes to adapt their rates dynamically based on the unknown channel conditions at the transmitter. The research work was concluded by comparing static G4 encoded Tarokah work and other Orthogonal Space Time Block Codes (OSTBCs) with Rateless Space Time Block Codes (ROSTBCs). The results show that as the number of blocks used to encode the message in Rateless Space Time Block Code (ROSTBC) increases, this coding scheme can outperform static Rateless Space Time Block Codes (OSTBC) in very low SNR values by a minimum of 8.5 %. This work can enhance wireless communication by creating reliable communication over inherently unreliable systems.

The Application of Information Theory to Interpret Shore Platform Erosion Rates

Advancements in information physics have recently introduced the application of information theory to investigate physical systems. The behaviour of erosion at the granular scale is to date still a complex system to unpack, and therefore geomorphology research requires novel approaches to better inform the interpretation of temporal and spatial erosion patterns at different scales. This paper applies information theory concepts to re-evaluate erosional data that were measured on limestone surfaces of two shore platforms in Malta with a traversing micro-erosion meter (TMEM). By representing erosion rates through their information content using a Box-Cox style transformation of the raw data (application of an inverse normal distribution function to fractionally ranked data), it is possible to identify points and measurement periods that contribute to a disproportionately large share of unexpected erosion rates that could provide more insight into the causes of erosion rates. Despite the variations in the information content from erosion rates at individual measurement points, most points consistently contribute to a similar amount of information. These findings illuminate the importance of considering the informational value of erosion data to further understand the underlying physical processes and potentially improve predictive models.

Interaction of acoustic pressure and heat release rate fluctuations in a model rocket engine combustor.

We experimentally clarify the interaction of acoustic pressure and heat release rate fluctuations during a transition to high-frequency combustion instability in a model rocket engine combustor. The dynamical state of acoustic pressure fluctuations undergoes a transition from high-dimensional chaotic oscillations to strongly correlated limit cycle oscillations. The coherent structure in the heat release rate field emerges with the initiation of weakly correlated limit cycle oscillations. The effect of the heat release rate on acoustic pressure fluctuations predominates during high-dimensional chaotic oscillations. In contrast, the effect of acoustic pressure on the heat release rate fluctuations markedly increases during the correlated limit cycle oscillations. These are reasonably shown by an ordinal pattern-based analysis involving the concepts of information theory, synchronization, and complex networks.

Harnessing Maxwell’s demon to establish a macroscale concentration gradient

Maxwell’s demon describes a thought experiment in which a ‘demon’ regulates the flow of particles between two adjoining spaces, establishing a potential gradient without appearing to do work. This seeming paradox led to the understanding that sorting entails thermodynamic work, a foundational concept of information theory. In the past centuries, many systems analogous to Maxwell’s demon have been introduced in the form of molecular information, molecular pumps and ratchets. Here we report a functional example of a Maxwell’s demon that pumps material over centimetres, whereas previous examples operated on a molecular scale. In our system, this demon drives directional transport of o-fluoroazobenzene between the arms of a U-tube apparatus upon light irradiation, transiting through an aqueous membrane containing a coordination cage. The concentration gradient thus obtained is further harnessed to drive naphthalene transport in the opposite direction.

Entropy based network anomaly detection

an era of relentless cyber threats, the increasing complexity and volume of data significantly intensify the risk and impact of cybersecurity breaches. As organizations generate and store more data, the potential attack surface grows, providing more opportunities for malicious actors to exploit vulnerabilities. Consequently, there is a growing necessity for more advanced analytical techniques to effectively detect and mitigate these evolving threats. Shannon entropy, introduced by Claude Shannon in 1948, is a fundamental concept in information theory that measures the unpredictability or randomness of information. It serves as a primary tool for identifying unusual patterns within extensive datasets, offering a quantitative approach to detect anomalies This paper explores the application of Shannon’s Entropy to detect and prevent distributed denial-of-service (DDoS) attacks. Unlike traditional motif identification tools, which focus on recurring patterns within data, Shannon entropy provides a broader measure of randomness and can detect subtle variations that may indicate a security breach. By leveraging the entropy measure, cybersecurity systems can identify and respond to abnormal traffic patterns that signify a potential DDoS attack, thereby enhancing the robustness and reliability of data protection mechanisms

The Separability Problem in Molecular Quantum Systems: Information-Theoretic Framework for Atoms in Molecules.

Even though molecules are fundamentally quantum entities, the concept of a molecule retains certain classical attributes concerning its constituents. This includes the empirical separability of a molecule into its three-dimensional, rigid structure in Euclidean space, a framework often obtained through experimental methods like X-Ray crystallography. In this work, we delve into the mathematical implications of partitioning a molecule into its constituent parts using the widely recognized Atoms-In-Molecules (AIM) schemes, aiming to establish their validity within the framework of Information Theory concepts. We have uncovered information-theoretical justifications for employing some of the most prevalent AIM schemes in the field of Chemistry, including Hirshfeld (stockholder partitioning), Bader's (topological dissection), and the quantum approach (Hilbert's space definition). In the first approach we have applied the generalized principle of minimum relative entropy derived from the Sharma-Mittal two-parameter functional, avoiding the need for an arbitrary selection of reference promolecular atoms. Within the ambit of topological-information partitioning, we have demonstrated that the Fisher information of Bader's atoms conform to a comprehensive theory based on the Principle of Extreme Physical Information avoiding the need of employing the Schwinger's principle, which has been proven to be problematic. For the quantum approach we have presented information-theoretic justifications for conducting Löwdin symmetric transformations on the density matrix to form atomic Hilbert spaces generating orthonormal atomic orbitals with maximum occupancy for a given wavefunction.

Machine Learning and Information Theory Concepts towards an AI Mathematician

The current state of the art in artificial intelligence is impressive, especially in terms of mastery of language, but not so much in terms of mathematical reasoning. What could be missing? Can we learn something useful about that gap from how the brains of mathematicians go about their craft? This essay builds on the idea that current deep learning mostly succeeds at system 1 abilities—which correspond to our intuition and habitual behaviors—but still lacks something important regarding system 2 abilities—which include reasoning and robust uncertainty estimation. It takes an information-theoretical posture to ask questions about what constitutes an interesting mathematical statement, which could guide future work in crafting an AI mathematician. The focus is not on proving a given theorem but on discovering new and interesting conjectures. The central hypothesis is that a desirable body of theorems better summarizes the set of all provable statements, for example, by having a small description length while at the same time being close (in terms of number of derivation steps) to many provable statements.

Logical-semantic models and methods of knowledge representation: cases for energy management systems and SMR digital infrastructures

The subject of this article is the development of information technologies at the end of the 20th and the beginning of the 21st century for the Fourth Industrial Revolution in the form of Industry 4.0, i.e., Internet of Things (IoT) technologies. Successes in the implementation of this technology have led to the development of new applications in the energy sector, such as energy management systems, small modular reactor (SMR) management systems, and alternative power supply systems based on renewable energy sources. The digital infrastructure of these management systems is characterized by a high "density of knowledge", which requires clarification of fundamental concepts in information theory, namely the content of the concepts "data", "information", "knowledge" and "meaning of knowledge". Special attention was given to defining the role of knowledge. The aim of this study is to further develop methods and models of semiotic theory by determining the role and place of logical as well as eight- and four-factor logical-semantic models of knowledge bases in semiotic space. The tasks: comparing existing knowledge representation methods and models. Research results: We found that the logical models used in the development of knowledge bases are based on the principles of artificial intelligence theory, which relies on sign system hypotheses and formal logic theory. The main drawback is the complexity of practical implementation in the form of expert systems. For logical-semantic models in the form of eight-vector graphic models, it was found that there is currently no theoretical justification for defining the vectors that form the coordinate axes, making these models unique to specific subject areas. It was determined that the advantage of using these methods is that an expert can independently form such a knowledge model. For logical-semantic models in the form of four-factor graphic models, there is a theoretical justification for defining the factors of the model that form the coordinate axes, making these models universal for specific subject areas. It was established that the advantage of these models is that they can be developed by experts without the involvement of a knowledge engineer. Therefore, it is proposed to use four-factor logical-semantic knowledge representation models for further application. It is also proposed to split the element "logical-semantic models" into two elements in the semiotic spatial model in vector K8 "Knowledge Representation Models", namely: "logical-semantic eight-vector models" and "logical-semantic four-factor models". Additionally, it is proposed to add the element "post-Cartesian representation of meta-knowledge" to the element "geometric" in vector K5 "Ideal Models". Conclusions: The theoretical basis for developing eight-factor logical-semantic knowledge representation models is the form of connections between adjacent vectors in the form of Cartesian products on elements of the corresponding inter-coordinate matrices. The theoretical basis for the methodology of developing four-factor logical-semantic knowledge representation models is the form of connections between adjacent vectors in the form of Cartesian products for elements of the corresponding inter-coordinate matrices, as well as for diametrically opposite pairs of factors in the form of dialectical unity of the concepts "general" and "particular." The application of logical-semantic knowledge representation models for alternative energy-source management systems will ensure increased energy efficiency. Other cases related to the development of databases for SMR digital infrastructure are discussed.

General or Situational? Exploring Cultural Identification Patterns Using Entropy Among Maghrebi Immigrants to Canada

Migrants and members of cultural minorities must negotiate their identification with multiple cultural groups. Many studies have investigated associations between general questionnaire–based cultural identity patterns and psychological adjustment. Research on situational cultural identity patterns—context-bound, momentary identification with a given cultural group—is scarcer. Furthermore, we know little about how variability in identification across contexts and situations may be associated with psychological adjustment. This study addresses these issues by (a) comparing the relative ability of general questionnaire–based and situational diary–based cultural identity patterns in statistically predicting psychological adjustment among Maghrebi migrants to Canada, and (b) introducing and testing cultural identity entropy, a novel approach to characterizing variability in a person’s multiple cultural identities during daily interactions. Drawing on concepts in thermodynamics and information theory, cultural identity entropy indexes greater balance in one’s multiple identifications during an interaction and reflects greater flexibility in cultural ways in that moment. Participants were 93 Maghrebi migrants to Canada who completed baseline questionnaires and daily diaries on situational identification during interactions for 7 days. Results show that situational diary–based cultural identity patterns accounted for substantial variance in psychological adjustment, above and beyond general questionnaire–based patterns, and that greater entropy in heritage cultural contexts was associated with greater psychological adjustment. These results underscore the importance of going beyond general characterizations of multicultural identity by investigating the shifting and contextual ways in which migrants mobilize and negotiate their cultural identities in daily life.

On Estimation of Shannon’s Entropy of Maxwell Distribution Based on Progressively First-Failure Censored Data

Shannon’s entropy is a fundamental concept in information theory that quantifies the uncertainty or information in a random variable or data set. This article addresses the estimation of Shannon’s entropy for the Maxwell lifetime model based on progressively first-failure-censored data from both classical and Bayesian points of view. In the classical perspective, the entropy is estimated using maximum likelihood estimation and bootstrap methods. For Bayesian estimation, two approximation techniques, including the Tierney-Kadane (T-K) approximation and the Markov Chain Monte Carlo (MCMC) method, are used to compute the Bayes estimate of Shannon’s entropy under the linear exponential (LINEX) loss function. We also obtained the highest posterior density (HPD) credible interval of Shannon’s entropy using the MCMC technique. A Monte Carlo simulation study is performed to investigate the performance of the estimation procedures and methodologies studied in this manuscript. A numerical example is used to illustrate the methodologies. This paper aims to provide practical values in applied statistics, especially in the areas of reliability and lifetime data analysis.

Entanglement Monogamy via Multivariate Trace Inequalities

Entropy is a fundamental concept in quantum information theory that allows to quantify entanglement and investigate its properties, for example its monogamy over multipartite systems. Here, we derive variational formulas for relative entropies based on restricted measurements of multipartite quantum systems. By combining these with multivariate matrix trace inequalities, we recover and sometimes strengthen various existing entanglement monogamy inequalities. In particular, we give direct, matrix-analysis-based proofs for the faithfulness of squashed entanglement by relating it to the relative entropy of entanglement measured with one-way local operations and classical communication, as well as for the faithfulness of conditional entanglement of mutual information by relating it to the separably measured relative entropy of entanglement. We discuss variations of these results using the relative entropy to states with positive partial transpose, and multipartite setups. Our results simplify and generalize previous derivations in the literature that employed operational arguments about the asymptotic achievability of information-theoretic tasks.

Entropy removal of medical diagnostics

Shannon entropy is a core concept in machine learning and information theory, particularly in decision tree modeling. To date, no studies have extensively and quantitatively applied Shannon entropy in a systematic way to quantify the entropy of clinical situations using diagnostic variables (true and false positives and negatives, respectively). Decision tree representations of medical decision-making tools can be generated using diagnostic variables found in literature and entropy removal can be calculated for these tools. This concept of clinical entropy removal has significant potential for further use to bring forth healthcare innovation, such as quantifying the impact of clinical guidelines and value of care and applications to Emergency Medicine scenarios where diagnostic accuracy in a limited time window is paramount. This analysis was done for 623 diagnostic tools and provided unique insights into their utility. For studies that provided detailed data on medical decision-making algorithms, bootstrapped datasets were generated from source data to perform comprehensive machine learning analysis on these algorithms and their constituent steps, which revealed a novel and thorough evaluation of medical diagnostic algorithms.

Functional Interactions of Low‐density Electroencephalography During Sleep Cycles to Detect Alzheimer’s Disease

AbstractBackgroundAlzheimer’s disease (AD) causes disruption in the collective dynamics of the brain activities which can be captured by analyzing the functional interactions between electroencephalographic (EEG) signals in different regions of the brain. To better assess the AD‐related changes in the interactions, one possible approach is to increase the EEG electrode density for better spatial information. Alternatively, high order interactions within the fluctuations of low‐density EEG — a concept of the multivariate information theory — may better reveal complex EEG interactions and can be used to detect AD‐related changes in brain activities. This study is aimed to demonstrate this conceptual approach using machine learning.MethodWe analyzed overnight 4‐channel EEG recordings (left central, right central, left mastoid, and right mastoid) in the polysomnography (PSG) data collected from 28 older men in the Osteoporotic Fractures in Men Study (MrOS). 14 participants reported a history of clinical diagnosis of AD while the other 14 did not. Groups were matched in age and education. Each of 30‐second epochs of the PSG recording was scored as wake, N1, N2, N3, or rapid eye movement sleep according to the established guideline. Features based on low (2) and high order (3‐4) interactions between electrodes, i.e., total correlation (TC), dual total correlation (DTC), O‐information (O) and S‐information (S), were extracted from raw and band‐passed filtered data (δ: 0.5‐4 Hz, θ: 4‐8 Hz, α: 8‐12 Hz, β: 12‐30 Hz, γl: 30‐40 Hz, γh: 40‐100 Hz) within each wake/sleep stage. The optimal set of features was identified using a cross‐validated random forest classifier to maximize the group discrimination.ResultAD was discriminated from control with an average area under the receiver operating characteristic curve of 0.90±0.05 using just 3 features: O (triplet) measured in the γh band within wake stage, DTC (triplet) of raw EEG within N1 stage, and TC (pairwise) in the δ band within N1 stage.ConclusionThis proof‐of‐concept study shows the possibility of using low‐density sleep EEG to detect AD. This method might be useful for screening people with risk of AD in ambulatory setting.

FastMI: A fast and consistent copula-based nonparametric estimator of mutual information

As a fundamental concept in information theory, mutual information (MI) has been commonly applied to quantify association between random vectors. Most existing nonparametric estimators of MI have unstable statistical performance since they involve parameter tuning. We develop a consistent and powerful estimator, called fastMI, that does not incur any parameter tuning. Based on a copula formulation, fastMI estimates MI by leveraging Fast Fourier transform-based estimation of the underlying density. Extensive simulation studies reveal that fastMI outperforms state-of-the-art estimators with improved estimation accuracy and reduced run time for large data sets. fastMI provides a powerful test for independence that exhibits satisfactory type I error control. Anticipating that it will be a powerful tool in estimating mutual information in a broad range of data, we develop an R package fastMI for broader dissemination.

Griffith theory of physical fractures, statistical procedures and entropy production: Rosetta stone's legacy.

A physical model, based on energy balances, is proposed to describe the fractures in solid structures such as stelae, tiles, glass, and others. We applied the model to investigate the transition of the Rosetta Stone from the original state to the final state with three major fractures. We consider a statistical corner-breaking model with cutting rules. We obtain a probability distribution as a function of the area and the number of vertices. Our generic results are consistent with the current state of the Rosetta Stone and, additionally, predictions related to a fourth fracture are declared. The loss of information on such heritage pieces is considered through entropy production. The explicit quantification of this concept in information theory stays examined.

Prediction of prokaryotic and eukaryotic promoters based on information-theoretic features

Promoters are DNA regulatory elements located near the transcription start site and are responsible for regulating the transcription of genes. DNA fragments arranged in a certain order form specific functional regions with different information contents. Information theory is the science that studies the extraction, measurement and transmission of information. The genetic information contained in DNA follows the general laws of information storage. Therefore, method in information theory can be used for the analysis of promoters carrying genetic information.In this study, we introduced the concept of information theory to the study of promoter prediction. We used 107 features extracted based on information theory methods and a backpropagation neural network to build a classifier. Then, the trained classifier was applied to predict the promoters of 6 organisms. The average AUCs of the 6 organisms obtained by using hold-out validation and ten-fold cross-validation were 0.885 and 0.886, respectively. The results verified the effectiveness of information-theoretic features in promoter prediction. Considering the possible redundancy in the feature set, we performed feature selection and obtained key feature subsets related to promoter characteristics. The results indicate the potential utility of information-theoretic features in promoter prediction.

A: Meir Feder on Using Information Theory's Concepts to Solve Complex Problems Across Disciplines

A: Meir Feder on Using Information Theory's Concepts to Solve Complex Problems Across Disciplines

A Generic Approach towards Enhancing Utility and Privacy in Person-Specific Data Publishing Based on Attribute Usefulness and Uncertainty

This paper proposes a generic anonymization approach for person-specific data, which retains more information for data mining and analytical purposes while providing considerable privacy. The proposed approach takes into account the usefulness and uncertainty of attributes while anonymizing the data to significantly enhance data utility. We devised a method for determining the usefulness weight for each attribute item in a dataset, rather than manually deciding (or assuming based on domain knowledge) that a certain attribute might be more useful than another. We employed an information theory concept for measuring the uncertainty regarding sensitive attribute’s value in equivalence classes to prevent unnecessary generalization of data. A flexible generalization scheme that simultaneously considers both attribute usefulness and uncertainty is suggested to anonymize person-specific data. The proposed methodology involves six steps: primitive analysis of the dataset, such as analyzing attribute availability in the data, arranging the attributes into relevant categories, and sophisticated pre-processing, computing usefulness weights of attributes, ranking users based on similarities, computing uncertainty in sensitive attributes (SAs), and flexible data generalization. Our methodology offers the advantage of retaining higher truthfulness in data without losing guarantees of privacy. Experimental analysis on two real-life benchmark datasets with varying scales, and comparisons with prior state-of-the-art methods, demonstrate the potency of our anonymization approach. Specifically, our approach yielded better performance on three metrics, namely accuracy, information loss, and disclosure risk. The accuracy and information loss were improved by restraining heavier anonymization of data, and disclosure risk was improved by preserving higher uncertainty in the SA column. Lastly, our approach is generic and can be applied to any real-world person-specific tabular datasets encompassing both demographics and SAs of individuals.

Algorithm 1033: Parallel Implementations for Computing the Minimum Distance of a Random Linear Code on Distributed-memory Architectures

The minimum distance of a linear code is a key concept in information theory. Therefore, the time required by its computation is very important to many problems in this area. In this article, we introduce a family of implementations of the Brouwer–Zimmermann algorithm for distributed-memory architectures for computing the minimum distance of a random linear code over 𝔽 2 . Both current commercial and public-domain software only work on either unicore architectures or shared-memory architectures, which are limited in the number of cores/processors employed in the computation. Our implementations focus on distributed-memory architectures, thus being able to employ hundreds or even thousands of cores in the computation of the minimum distance. Our experimental results show that our implementations are much faster, even up to several orders of magnitude, than current implementations widely used nowadays.