Conditional Entropy Research Articles

The natural stability of autonomous morphology: how an attraction–repulsion dynamic emerges from paradigm cell filling

AbstractAutonomous morphology, such as inflection class systems and paradigmatic distribution patterns, is widespread and diachronically resilient in natural language. Why this should be so has remained unclear given that autonomous morphology imposes learning costs, offers no clear benefit relative to its absence and could easily be removed by the analogical forces which are constantly reshaping it. Here we propose an explanation for the resilience of autonomous morphology, in terms of a diachronic dynamic of attraction and repulsion between morphomic categories, which emerges spontaneously from a simple paradigm cell filling process. Employing computational evolutionary models, our key innovation is to bring to light the role of ‘dissociative evidence’, i.e., evidence for inflectional distinctiveness which a rational reasoner will have access to during analogical inference. Dissociative evidence creates a repulsion dynamic which prevents morphomic classes from collapsing together entirely, i.e., undergoing complete levelling. As we probe alternative models, we reveal the limits of conditional entropy as a measure for predictability in systems that are undergoing change. Finally, we demonstrate that autonomous morphology, far from being ‘unnatural’, is rather the natural (emergent) consequence of a natural (rational) process of inference applied to inflectional systems.

Adaptive fuzzy neighborhood decision tree

Decision tree algorithms have gained widespread acceptance in machine learning, with the central challenge lying in devising an optimal splitting strategy for node sample subspaces. In the context of continuous data, conventional approaches typically involve fuzzifying data or adopting a dichotomous scheme akin to the CART tree. Nevertheless, fuzzifying continuous features often entails information loss, whereas the dichotomous approach can generate an excessive number of classification rules, potentially leading to overfitting. To address these limitations, this study introduces an adaptive growth decision tree framework, termed the fuzzy neighborhood decision tree (FNDT). Initially, we establish a fuzzy neighborhood decision model by leveraging the concept of fuzzy inclusion degree. Furthermore, we delve into the topological structure of misclassified samples under the proposed decision model, providing a theoretical foundation for the construction of FNDT. Subsequently, we utilize conditional information entropy to sift through original features, prioritizing those that offer the maximum information gain for decision tree nodes. By leveraging the conditional decision partitions derived from the fuzzy neighborhood decision model, we achieve an adaptive splitting method for optimal features, culminating in an adaptive growth decision tree algorithm that relies solely on the inherent structure of real-valued data. Experimental evaluations reveal that, compared with advanced decision tree algorithms, FNDT exhibits a simple tree structure, stronger generalization capabilities, and superior performance in classifying continuous data.

Quantifying the informativity of emission lines to infer physical conditions in giant molecular clouds

Context. Observations of ionic, atomic, or molecular lines are performed to improve our understanding of the interstellar medium (ISM). However, the potential of a line to constrain the physical conditions of the ISM is difficult to assess quantitatively, because of the complexity of the ISM physics. The situation is even more complex when trying to assess which combinations of lines are the most useful. Therefore, observation campaigns usually try to observe as many lines as possible for as much time as possible. Aims. We have searched for a quantitative statistical criterion to evaluate the full constraining power of a (combination of) tracer(s) with respect to physical conditions. Our goal with such a criterion is twofold. First, we want to improve our understanding of the statistical relationships between ISM tracers and physical conditions. Secondly, by exploiting this criterion, we aim to propose a method that helps observers to make their observation proposals; for example, by choosing to observe the lines with the highest constraining power given limited resources and time. Methods. We propose an approach based on information theory, in particular the concepts of conditional differential entropy and mutual information. The best (combination of) tracer(s) is obtained by comparing the mutual information between a physical parameter and different sets of lines. The presented analysis is independent of the choice of the estimation algorithm (e.g., neural network or χ2 minimization). We applied this method to simulations of radio molecular lines emitted by a photodissociation region similar to the Horsehead Nebula. In this simulated data, we considered the noise properties of a state-of-the-art single dish telescope such as the IRAM 30m telescope. We searched for the best lines to constrain the visual extinction, AVtot, or the ultraviolet illumination field, G0. We ran this search for different gas regimes, namely translucent gas, filamentary gas, and dense cores. Results. The most informative lines change with the physical regime (e.g., cloud extinction). However, the determination of the optimal (combination of) line(s) to constrain a physical parameter such as the visual extinction depends not only on the radiative transfer of the lines and chemistry of the associated species, but also on the achieved mean signal-to-noise ratio. The short integration time of the CO isotopologue J = 1 − 0 lines already yields much information on the total column density for a large range of (AVtot, G0) space. The best set of lines to constrain the visual extinction does not necessarily combine the most informative individual lines. Precise constraints on the radiation field are more difficult to achieve with molecular lines. They require spectral lines emitted at the cloud surface (e.g., [CII] and [CI] lines). Conclusions. This approach allows one to better explore the knowledge provided by ISM codes, and to guide future observation campaigns.

Bridging chemical structure and conceptual knowledge enables accurate prediction of compound-protein interaction

BackgroundAccurate prediction of compound-protein interaction (CPI) plays a crucial role in drug discovery. Existing data-driven methods aim to learn from the chemical structures of compounds and proteins yet ignore the conceptual knowledge that is the interrelationships among the fundamental elements in the biomedical knowledge graph (KG). Knowledge graphs provide a comprehensive view of entities and relationships beyond individual compounds and proteins. They encompass a wealth of information like pathways, diseases, and biological processes, offering a richer context for CPI prediction. This contextual information can be used to identify indirect interactions, infer potential relationships, and improve prediction accuracy. In real-world applications, the prevalence of knowledge-missing compounds and proteins is a critical barrier for injecting knowledge into data-driven models.ResultsHere, we propose BEACON, a data and knowledge dual-driven framework that bridges chemical structure and conceptual knowledge for CPI prediction. The proposed BEACON learns the consistent representations by maximizing the mutual information between chemical structure and conceptual knowledge and predicts the missing representations by minimizing their conditional entropy. BEACON achieves state-of-the-art performance on multiple datasets compared to competing methods, notably with 5.1% and 6.6% performance gain on the BIOSNAP and DrugBank datasets, respectively. Moreover, BEACON is the only approach capable of effectively predicting knowledge representations for knowledge-lacking compounds and proteins.ConclusionsOverall, our work provides a general approach for directly injecting conceptual knowledge to enhance the performance of CPI prediction.

Neighborhood entropy guided by a decision attribute and its applications in multi-source information fusion and attribute selection

The existing methods for information fusion and attribute selection typically require a computationally intensive grid search to determine the optimal parameter. This grid search can significantly impact the efficiency of these processes. To tackle this issue, this paper introduces the concept of neighborhood entropy, which incorporates both the neighborhood and decision classes, and delves into its applications in multi-source information fusion and attribute selection. First, the concept of a δ-neighborhood based on a predefined distance is introduced. Then the neighborhood entropy based on the δ-neighborhood is defined. The optimal parameters for information fusion and attribute selection are determined by minimizing the neighborhood entropy. Furthermore, an information fusion method and an attribute selection algorithm based on the neighborhood entropy, conditional information entropy (CIE), and belief functions are developed. Finally, we demonstrate the effectiveness of our approach through experiments and Wilcoxon tests on 12 datasets, including a large-scale gene dataset. The experimental results show that the optimal parameter determined by the neighborhood entropy aligns with that obtained through grid search and the neighborhood entropy reduces the computational burden of information fusion and attribute selection compared to grid search. When compared to other state-of-the-art algorithms, our approach based on the neighborhood entropy and CIE demonstrates superior classification performance and time efficiency. These findings demonstrate the efficiency and effectiveness of neighborhood entropy and pave the way for applying granular-computing-based information fusion methods to large-scale datasets.

A Heuristic Attribute-Reduction Algorithm Based on Conditional Entropy for Incomplete Information Systems

With the continuous expansion of databases, the extraction of information has been an urgent research topic in many fields. As an effective method to remove redundant attributes, attribute reduction demonstrates extraordinary ability in simplifying information systems. This paper applies a novel form of conditional entropy to investigate the attribute reduction in incomplete information systems. Firstly, a novel definition of conditional entropy is introduced based on tolerance relation. Additionally, in order to reduce time complexity, we propose a binsearch heuristic attribute-reduction algorithm with conditional entropy as heuristic knowledge. Furthermore, two examples are used to illustrate the feasibility and validity of the reduction algorithm.

Quantum channels that destroy negative conditional entropy

Quantum channels that destroy negative conditional entropy

An elementary proof of the inequality $\chi\leq\chi^{*}$ for conditional free entropy

Through the study of large deviations theory for matrix Brownian motion, Biane, Capitaine, and Guionnet proved the inequality \chi(X)\leq\chi^{*}(X) that relates two analogs of entropy in free probability defined by Voiculescu. We give a new proof of \chi\leq\chi^{*} that is elementary in the sense that it does not rely on stochastic differential equations and large deviations theory. Moreover, we generalize the result to conditional microstates and non-microstates free entropy.

Optimal scale combination selection in generalized multi-scale hybrid decision systems

When processing multi-scale hybrid data, the existing methods are insufficient to directly select the optimal scale for different types of data. To deal with this issue, the concept of generalized multi-scale hybrid decision systems (GMHDSs) consisting of numerical, nominal and set-valued values is first introduced. Then, under a scale combination, the information granules of different types of attributes in a GMHDS are constructed and they further fused as the granules of the system. Based on these granules of a given scale combination, positive region and conditional entropy are specified in a GMHDS. To select appropriate single scale systems to keep the positive region and the conditional entropy unchanged for final decision, concepts of positive region optimal scale combinations (POSCs) and conditional entropy optimal scale combinations (CEOSCs) in GMHDSs are defined, respectively. Furthermore, algorithms for calculating a POSC and a CEOSC in a GMHDS are also formulated. Finally, a comparative study of our methods with these based on original data under KNN and NB classifiers is conducted on twelve UCI datasets for the assessment of their classification performance. The experimental results indicate that the proposed methods are of better classification performance than the original data based methods in most cases.

A temporal information transfer network approach considering federal funds rate for an interpretable asset fluctuation prediction framework

This study explores the complex interdependencies among five major financial assets—S&P 500, Bitcoin, Crude Oil, Gold, and USD/EUR—from April 2015 to September 2022, emphasizing the importance of understanding global financial dynamics for robust financial management. We quantitatively analyze daily causal relationships using conditional transfer entropy, a method that surpasses traditional correlation analyses. By incorporating the effective federal funds rate into our models, we enhance predictive accuracy and account for monetary policy impacts, ensuring our findings are relevant to current economic conditions. Our results reveal significant causal networks, providing key insights into asset interdependencies that support advanced hedging strategies and effective diversification. This research improves prediction models through the innovative use of network-based features and offers practical strategies for managing multinational financial assets, with relevance across various economic scenarios.

Inferring Dealer Networks in the Foreign Exchange Market Using Conditional Transfer Entropy: Analysis of a Central Bank Announcement.

The foreign exchange (FX) market has evolved into a complex system where locally generated information percolates through the dealer network via high-frequency interactions. Information related to major events, such as economic announcements, spreads rapidly through this network, potentially inducing volatility, liquidity disruptions, and contagion effects across financial markets. Yet, research on the mechanics of information flows in the FX market is limited. In this paper, we introduce a novel approach employing conditional transfer entropy to construct networks of information flows. Leveraging a unique, high-resolution dataset of bid and ask prices, we investigate the impact of an announcement by the European Central Bank on the information transfer within the market. During the announcement, we identify key dealers as information sources, conduits, and sinks, and, through comparison to a baseline, uncover shifts in the network topology.

Device-independent lower bounds on the conditional von Neumann entropy

The rates of several device-independent (DI) protocols, including quantum key-distribution (QKD) and randomness expansion (RE), can be computed via an optimization of the conditional von Neumann entropy over a particular class of quantum states. In this work we introduce a numerical method to compute lower bounds on such rates. We derive a sequence of optimization problems that converge to the conditional von Neumann entropy of systems defined on general separable Hilbert spaces. Using the Navascués-Pironio-Acín hierarchy we can then relax these problems to semidefinite programs, giving a computationally tractable method to compute lower bounds on the rates of DI protocols. Applying our method to compute the rates of DI-RE and DI-QKD protocols we find substantial improvements over all previous numerical techniques, demonstrating significantly higher rates for both DI-RE and DI-QKD. In particular, for DI-QKD we show a minimal detection efficiency threshold which is within the realm of current capabilities. Moreover, we demonstrate that our method is capable of converging rapidly by recovering all known tight analytical bounds up to several decimal places. Finally, we note that our method is compatible with the entropy accumulation theorem and can thus be used to compute rates of finite round protocols and subsequently prove their security.

An overview of the development and applications of information entropy

With information entropy gradually taking the lead in modern information theory development, it begins to hold greater influence over multiple research areas as well as technology innovation. This paper aims to clarify peoples confusion with the development of entropy theory and provide a brief overview of the origin of entropy theory, including the original Shannons proposal, variants such as relative entropy and conditional entropy, and entropy concepts proposed by other scientists, such as Rnyi Tsallis entropy. The paper also includes the current application of entropy, studies hotspots, and predicts future entropy development trends. This research paper is able to add more coherence and consistency to information entropys development, helping more people to better understand the concept of entropy and its derivation. At the same time, with hotspots of entropy fields of study, this paper hopes to attract more people to devote themselves to studying entropy-related fields, and boost technological development.

Largest Lyapunov exponent and Shannon entropy: Two indices to analyze mixing in fluidized beds

The quality of mixing in fluidized beds greatly influences performance in many applications. Assessing quality of mixing involves measuring the mixing rate and evaluating the bed mixedness. Quantifying the bed mixedness is typically done using mixing indices. However, the application of existing mixing indices to fluidized beds can be problematic due to aeration and complications from the particle phase in the Two-Fluid Model (TFM).The objectives of this study are twofold. First, the largest Lyapunov exponent is proposed to quantify mixing in fluidized beds. Its effectiveness is shown on a mono-disperse bed with varying gas velocities. The increase in mixing rate with higher gas velocity is accurately represented by the largest Lyapunov exponent. Second, the Shannon entropy mixing index is adopted to quantify the bed mixedness for TFM results. This index is tested on a bi-disperse bed to predict segregation and evaluate the effect of bed composition and superficial gas velocity on this process. The effect on segregation is reflected in the entropy components: distributional entropy showed minimal variation, whereas conditional entropy was significantly affected. Evaluating bed mixedness at different length scales showed that increasing bin spatial resolution slightly reduced conditional entropy. The results are validated against experimental data.

Supervised spectral feature selection with neighborhood rough set

Spectral feature selection, an excellent dimensionality reduction method, is extensively used in knowledge mining and protein sequence analysis. However, the graph representation derived from data with potential noises significantly impacts feature selection performance. Similarly, its performance deteriorates when label information is ignored, as the selected features have poor discrimination ability. To overcome these drawbacks, this article optimizes neighborhood structure and further constructs the conditional entropy based on neighborhood purity to generate a precise graph representation incorporating label information to enhance the compactness of intra-class samples and increase the dispersion between inter-class samples. Then this article proposes a novel spectral feature selection method. Specifically, the method dynamically learns the precise graph representation from low-dimension space, which can effectively preserve the local structure of data, and the global structure is preserved by introducing a constraint term that maintains the graph representation before and after dimension reduction. Additionally, ℓ2,1-norm is introduced to ensure row sparsity and select the valuable features. The alternative optimization algorithm is employed to solve the optimization problem and its convergence is theoretically analyzed. Finally, to assess the performance of the proposed method, a series of comprehensive experiments are performed on several real-world datasets.

A quantum information theoretic analysis of reinforcement learning-assisted quantum architecture search

In the field of quantum computing, variational quantum algorithms (VQAs) represent a pivotal category of quantum solutions across a broad spectrum of applications. These algorithms demonstrate significant potential for realising quantum computational advantage. A fundamental aspect of VQAs involves formulating expressive and efficient quantum circuits (namely ansatz), and automating the search of such ansatz is known as quantum architecture search (QAS). Recently reinforcement learning (RL) techniques is utilized to automate the search for ansatzes, know as RL-QAS. This study investigates RL-QAS for crafting ansatz tailored to the variational quantum state diagonalisation problem. Our investigation includes a comprehensive analysis of various dimensions, such as the entanglement thresholds of the resultant states, the impact of initial conditions on the performance of RL-agent, the phase transition behaviour of correlation in concurrence bounds, and the discrete contributions of qubits in deducing eigenvalues through conditional entropy metrics. We leverage these insights to devise an entanglement-guided admissible ansatz in QAS to diagonalise random quantum states using optimal resources. Furthermore, the methodologies presented herein offer a generalised framework for constructing reward functions within RL-QAS applicable to variational quantum algorithms.

Quantum conditional entropy and classical-quantum conditional entropy with localization characteristics

In the era of rapid development of quantum mechanics, some issues in the field of quantum information have become increasingly important. Local quantum Bernoulli noises (LQBNs) is a localized Bernoulli functional, and it is the localization of quantum Bernoulli noises (QBNs). The distinctive feature of LQBNs lies in its localization properties, which make it highly valuable in the field of quantum information. In this paper, we study three new types of quantum conditional entropy with localization characteristics from three dimensions: local quantum conditional entropy, local quantum conditional entropy at the same position, and classical-quantum conditional entropy. Based on the locality of LQBNs, we prove that the local quantum conditional entropy is concave and unitary invariant. Surprisingly, we find that the local quantum mutual information (entropy) corresponding to the same position is 0. The local quantum conditional entropy defined based on the local quantum state at a specific position, proposed in this paper, provides new insights into understanding the impact of local quantum noise on the information transmission in quantum channels. This discovery not only deepens our understanding of the internal structure of local quantum entropy, but also reveals its influence on the performance of quantum communication.

Reliability of a probabilistic knowledge structure

Indexes for estimating the overall reliability of a test in the framework of knowledge space theory (KST) are proposed and analyzed. First, the possibility of applying in KST the existing classical test theory (CTT) methods, based on the ratio between the true score variance and the total variance of the measure, has been explored. However, these methods are not suitable because in KST error and true score are not independent. Therefore, two new indexes based on the concepts of entropy and conditional entropy are developed. One index is used to estimate the reliability of the response pattern given the knowledge state, while the second one refers to the reliability of the estimated knowledge state of a person. Some theoretical considerations as well as simulations and an empirical example on real data are provided within a study of the behavior of these indexes under a certain number of different conditions.

Comprehensive Evaluation of the Development Level of China’s Characteristic Towns under the Perspective of an Urban–Rural Integration Development Strategy

With the advancement of urbanization and the continuous deepening of reforms in urban–rural systems, China’s urbanization process has entered a new era of integrated urban–rural integration. Currently, as a global “new green revolution” gains momentum, numerous countries are deeply integrating the concept of sustainable development into new urban planning. Against this backdrop, urban planners worldwide are committed to building green, livable, and smart cities that can meet the needs of the present generation without compromising the ability of future generations to meet their needs, thus achieving the vision of harmonious coexistence between humanity and nature. Characteristic towns, leveraging their resource advantages, play a significant role in achieving sustainable regional economic development. They serve as valuable references for China’s urban transformation and upgrading, as well as for promoting rural urbanization, and are crucial avenues for advancing China’s urban–rural integration development strategy. The evaluation of the development level of characteristic towns is a necessary step in their progress and a strong guarantee for promoting their construction and development. Therefore, effectively evaluating the social benefits of characteristic towns is paramount. This study constructs an evaluation model based on the grey rough set theory and Technique for Order Preference by Similarity to Ideal Solution of TOPSIS. Firstly, an evaluation index system for the development level of characteristic towns is established. Then, the grey relational analysis method and rough set theory are used to reduce the index attributes, while the conditional information entropy theory is introduced to determine the weights of the reduced indicators. Finally, the TOPSIS model is applied to evaluate the development level of characteristic towns. Through empirical research, eight characteristic towns in Zhejiang Province, China, were assessed and ranked, verifying the effectiveness and feasibility of the proposed model.

General Adaptable Design and Evaluation Using Markov Processes

Abstract Facing the challenges posed by increasingly complex, dynamic, and unforeseen requirements, the design process is grappling with the critical issue of ensuring sustained product satisfaction amid changing demands. This paper introduces an approach for evaluating design adaptability, considering potential future requirements. Entropy serves as a crucial indicator to quantify design effort and the Markov process is employed to simulate potential requirement changes. The information contents of design requirements and design solutions are defined based on information entropy theory, and the design adaptability of a design candidate is evaluated by calculating the extra design effort for satisfying the design requirements, which is the difference in information content between the design candidate and design requirements. Moreover, a simulation method for requirement evolution is proposed, which integrates information entropy theory and the Markov process to accommodate potential future requirements. The general design adaptability of design solutions is then calculated based on conditional entropy, taking into account the evolving design requirements. Finally, the effectiveness of the proposed approach is validated through a case study involving the design and evaluation of a hybrid additive manufacturing device.