Information Entropy Research Articles

Prediction of Protein Allosteric Sites with Transfer Entropy and Spatial Neighbor-Based Evolutionary Information Learned by an Ensemble Model.

Allostery is one of the most direct and efficient ways to regulate protein functions. The diverse allosteric sites make it possible to design allosteric modulators of differential selectivity and improved safety compared with those of orthosteric drugs targeting conserved orthosteric sites. Here, we develop an ensemble machine learning method AllosES to predict protein allosteric sites in which the new and effective features are utilized, including the entropy transfer-based dynamic property, secondary structure features, and our previously proposed spatial neighbor-based evolutionary information besides the traditional physicochemical properties. To overcome the class imbalance problem, the multiple grouping strategy is proposed, which is applied to feature selection and model construction. The ensemble model is constructed where multiple submodels are trained on multiple training subsets, respectively, and their results are then integrated to be the final output. AllosES achieves a prediction performance of 0.556 MCC on the independent test set D24, and additionally, AllosES can rank the real allosteric sites in the top three for 83.3/89.3% of allosteric proteins from the test set D24/D28, outperforming the state-of-the-art peer methods. The comprehensive results demonstrate that AllosES is a promising method for protein allosteric site prediction. The source code is available at https://github.com/ChunhuaLab/AllosES.

3D Industrial anomaly detection via dual reconstruction network

Currently, 2D anomaly detection has demonstrated outstanding performance. However, 2D images limit the improvement of anomaly detection accuracy without utilizing depth information. Therefore, this paper proposes a Dual Reconstruction viAInpainting Network for 3D industrial anomaly detection (DRAIN). Firstly, we design a 3D reconstruction network using an encoder-decoder-based U-shaped network for processing RGB images and depth images. Subsequently, accurate anomaly segmentation is implemented through a 3D segmentation network. We introduce a lightweight MLP module to enhance segmentation performance to capture long-range dependencies in the reconstructed images. Furthermore, we propose a dual attention-based information entropy fusion module to expedite feature fusion in the inference process, aiming for enhanced deployment in the industry. Extensive experiments demonstrate that DRAIN achieves a 94.3% AUROC on the 3D anomaly detection dataset MVTec 3D-AD, surpassing other research methods.Graphical abstractOverall architecture for 3D industrial anomaly detection via dual reconstruction network

ФИЗИКАНЫ ЖАНА АСТРОНОМИЯНЫ ОКУТУУДА ЖАНА ИЗИЛДӨӨДӨ МАТЕМАТИКАНЫН ОРДУ

This scientific article is devoted to the ways of using mathematics in teaching and researching the subjects of physics and astronomy. Newton's first polynomial was used in mathematical modeling of the emission of electron and hole color centers in solid state physics. Shannon's formula was used to determine information entropy, and the logarithmic function of Boltzmann's formula was used to determine statistical entropy. In astronomy, trigonometric functions and stereographic transformations are used to determine celestial coordinates and transform very large angles. This research paper will help students in their study of these disciplines. Particular attention is paid to the close connection between mathematics and astronomy and physics.

Quantum conditional entropy and classical-quantum conditional entropy with localization characteristics

In the era of rapid development of quantum mechanics, some issues in the field of quantum information have become increasingly important. Local quantum Bernoulli noises (LQBNs) is a localized Bernoulli functional, and it is the localization of quantum Bernoulli noises (QBNs). The distinctive feature of LQBNs lies in its localization properties, which make it highly valuable in the field of quantum information. In this paper, we study three new types of quantum conditional entropy with localization characteristics from three dimensions: local quantum conditional entropy, local quantum conditional entropy at the same position, and classical-quantum conditional entropy. Based on the locality of LQBNs, we prove that the local quantum conditional entropy is concave and unitary invariant. Surprisingly, we find that the local quantum mutual information (entropy) corresponding to the same position is 0. The local quantum conditional entropy defined based on the local quantum state at a specific position, proposed in this paper, provides new insights into understanding the impact of local quantum noise on the information transmission in quantum channels. This discovery not only deepens our understanding of the internal structure of local quantum entropy, but also reveals its influence on the performance of quantum communication.

Uncertainty-Aware and Explainable Human Error Detection in the Operation of Nuclear Power Plants

The timely and accurate identification of incidents, such as human factor error, is important to restore nuclear power plants (NPPs) to a stable state. However, the identification of abnormal operating conditions is difficult because of the existence of multiple scenarios. In addition, to implement mitigation actions rapidly after an incident occurs, operators must accurately identify an incident by monitoring the trends of many variables. The mental burden posed by this can increase human error and cause failure in identifying incidents. Failure to identify incidents directly results in erroneous mitigation measures, which are detrimental to NPPs. In this study, we leverage uncertainty-aware models to identify such errors and thereby increase the chances of mitigating them. We use the data collected from a physical test bed. The goal is to identify both certain and accurate models. For this, the two main aspects of focus in this study are explainable artificial intelligence (XAI) and uncertainty quantification (UQ). While XAI elucidates the decision pathway, UQ evaluates decision reliability. Their integration paints a comprehensive picture, signifying that understanding decisions and their confidence should be interlinked. Thus, in this study we leverage UQ measures (e.g. entropy and mutual information) along with Shapley additive explanations to gain insights into the features contributing to both accuracy and uncertainty in error identification. Our results show that uncertainty-aware models combined with XAI tools can explain the artificial intelligence–prescribed decisions, with the potential of better explaining errors for the operators.

Energetic Information from Information-Theoretic Approach in Density Functional Theory as Quantitative Measures of Physicochemical Properties.

The Hohenberg-Kohn theorem of density functional theory (DFT) stipulates that energy is a universal functional of electron density in the ground state, so energy can be thought of having encoded essential information for the density. Based on this, we recently proposed to quantify energetic information within the framework of information-theoretic approach (ITA) of DFT (J. Chem. Phys. 2022, 157, 101103). In this study, we systematically apply energetic information to a variety of chemical phenomena to validate the use of energetic information as quantitative measures of physicochemical properties. To that end, we employed six ITA quantities such as Shannon entropy and Fisher information for five energetic densities, yielding twenty-six viable energetic information quantities. Then, they are applied to correlate with physicochemical properties of molecular systems, including chemical bonding, conformational stability, intermolecular interactions, acidity, aromaticity, cooperativity, electrophilicity, nucleophilicity, and reactivity. Our results show that different quantities of energetic information often behave differently for different properties but a few of them, such as Shannon entropy of the total kinetic energy density and information gain of the Pauli energy density, stand out and strongly correlate with several properties across different categories of molecular systems. These results suggest that they can be employed as quantitative measures of physicochemical properties. This work not only enriches the body of our knowledge about the relationship between energy and information, but also provides scores of newly introduced explicit density functionals to quantify physicochemical properties, which can serve as robust features for building machine learning models in future studies.

Risk analysis and assessment method for infectious diseases based on information entropy theory

Infectious diseases risk is directly related to human life safety. After the COVID-19 pandemic, people have paid unprecedented attention to the risk of infectious diseases. Compared with treatment after the outbreak of the epidemic, identifying the influencing factors of infectious disease risk and quantitatively analyzing and assessing infectious disease risk before the outbreak of the epidemic plays an equally important role. This article focuses on the risk of irregular outbreaks of infectious diseases. On the one hand, a method based on information gain is proposed to calculate the weight of environmental factors directly related to infectious disease risk, to clarify the correlation between environmental factors and infectious disease risk. On the other hand, the risk calculation method based on risk weight number is proposed to calculate the risk level of different infectious diseases under the influence of specific environmental factors. Finally, the effectiveness and feasibility of the proposed method are verified through case analysis and discussion. By comparing it with other risk assessment methods, the advantages and disadvantages of the proposed method are demonstrated.

A coarse and fine-grained deep multi view subspace clustering method for unsupervised fault diagnosis of rolling bearings

Abstract To address the issue of insufficient characterization of fault features in inherent vibration data that affects the performance of unsupervised learning-based fault diagnosis, a coarse and fine-grained deep multi view subspace clustering method (CFG-DMVSC) for unsupervised fault diagnosis of rolling bearings is proposed. The proposed method designs a convolutional autoencoder network based on the Gramian angular field transformation for multi-signal analysis domains. A multi-view coarse-grained self-expressive method based on information entropy is designed to handle differences in information across different views. Furthermore, a fine-grained common and independent information separation loss function based on mutual information is proposed to ensure compactness among multiple views. Both the Case Western Reserve University rolling bearing dataset and privately built bearing fault test bench data demonstrate that, compared to existing methods, the proposed method can perform coarse and fine-grained division in multi-view subspaces, achieving better clustering diagnosis performance on the extracted common information among views.

DBMKA-Net:Dual branch multi-perception kernel adaptation for underwater image enhancement

In recent years, due to the dependence on wavelength-based light absorption and scattering, underwater photographs captured by devices often exhibit characteristics such as blurriness, faded color tones, and low contrast. To address these challenges, convolutional neural networks (CNNs) with their robust feature-capturing capabilities and adaptable structures have been employed for underwater image enhancement. However, most CNN-based studies on underwater image enhancement have not taken into account color space kernel convolution adaptability, which can significantly enhance the model’s expressive capacity. Building upon current academic research on adjusting the color space size for each perceptual field, this paper introduces a Double-Branch Multi-Perception Kernel Adaptive (DBMKA) model. A DBMKA module is constructed through two perceptual branches that adapt the kernels: channel features and local image entropy. Additionally, considering the pronounced attenuation of the red channel in underwater images, a Dependency-Capturing Feature Jump Connection module (DCFJC) has been designed to capture the red channel’s dependence on the blue and green channels for compensation. Its skip mechanism effectively preserves color contextual information. To better utilize the extracted features for enhancing underwater images, a Cross-Level Attention Feature Fusion (CLAFF) module has been designed. With the Double-Branch Multi-Perception Kernel Adaptive model, Dependency-Capturing Skip Connection module, and Cross-Level Adaptive Feature Fusion module, this network can effectively enhance various types of underwater images. Qualitative and quantitative evaluations were conducted on the UIEB and EUVP datasets. In the color correction comparison experiments, our method demonstrated a more uniform red channel distribution across all gray levels, maintaining color consistency and naturalness. Regarding image information entropy (IIE) and average gradient (AG), the data confirmed our method’s superiority in preserving image details. Furthermore, our proposed method showed performance improvements exceeding 10% on other metrics like MSE and UCIQE, further validating its effectiveness and accuracy.

Autonomous three-dimensional oscillator with two and four wings attractors embedded in the microcontroller: analysis, amplitude controls, random number generator, and image encryption application

Robust chaotic systems offer unpredictability, complex dynamics, noise-like properties, efficient bifurcation behavior, and the ability to model real-world phenomena, making them valuable in diverse scientific and engineering applications. This paper details on the dynamical appraisal, amplitude controls, microcontroller execution, Random number generator (RNG) of an autonomous three-dimensional (3D) oscillator with two and four wings attractors (ATDOTFWA), and its image encryption application. Thanks to the Routh-Hurwitz criteria, five steady states found in the ATDOTFWA are classified as stable or unstable, depending on its two control parameters. During the numerical simulations employing the Runge–Kutta scheme, the ATDOTFWA exhibit a wide range of dynamic behaviors, including no oscillations, Hopf bifurcation, limit cycle, five distinct presentations of two wings chaotic structures, monostable and bistable two wings chaotic structures, bistable and monostable regular oscillations, chaotic bursting characteristics, coexistence of period-2-oscillations and four wings chaotic structure, and four wings chaotic attractor which were validated experimentally by the microcontroller implementation. The total and partial controls of the amplitude are achieved in the ATDOTFWA. A RNG is designed based on the ATDOTFWA, and the generated random numbers are successfully tested using the ENT and NIST 800–22 statistical test suites, demonstrating the reliability of the ATDOTFWA-based RNG. This reliability is further confirmed through the application of the ATDOTFWA-based RNG in an efficient and secure image encryption process, where the generated random numbers are used as the encryption key. The effectiveness of the image encryption process is validated through comprehensive cryptanalysis, with an encryption time of 0.1923 s for a 512×512 image, an average normalized pixel change rate (NPCR) of 99.6126%, an average unified average changing intensity (UACI) of 33.4578%, and an average information entropy of 7.9994.

A hierarchical reinforcement learning-aware hyper-heuristic algorithm with fitness landscape analysis

The automation of meta-heuristic algorithm configuration holds the utmost significance in evolutionary computation. A hierarchical reinforcement learning-aware hyper-heuristic algorithm with fitness landscape analysis (HRLHH) is proposed to flexibly configure the suitable algorithms under various optimization scenarios. Two kinds of fitness landscape analysis techniques improved based on specific problem characteristics construct the state spaces for hierarchical reinforcement learning. Among them, an adaptive classification based on dynamic ruggedness of information entropy is designed to discern the complexity of problems, which serves as the basis for decision-making actions in upper-layer space. Additionally, an online dispersion metric based on knowledge is further presented to distinguish the precise landscape features in lower-layer space. In light of the characteristics of the state spaces, the hierarchical action spaces composed of meta-heuristics with disparate exploration and exploitation are designed, and various action selection strategies are introduced. Taking into account the real-time environment and algorithm evolution behavior, dynamic reward mechanisms based on evolutionary success rate and population convergence rate are utilized to enhance search efficiency. The experimental results on the IEEE Congress on Evolutionary Computation (CEC) 2017, CEC 2014, and large-scale CEC 2013 test suites demonstrate that the proposed HRLHH exhibits superiority in terms of accuracy, stability, and convergence speed, and possesses strong generalization.

Evaluating the Attraction of Scenic Spots Based on Tourism Trajectory Entropy.

With the development of positioning technology and the widespread application of mobile positioning terminal devices, the acquisition of trajectory data has become increasingly convenient. Furthermore, mining information related to scenic spots and tourists from trajectory data has also become increasingly convenient. This study used the normalization results of information entropy to evaluate the attraction of scenic spots and the experience index of tourists. Tourists and scenic spots were chosen as the probability variables to calculate information entropy, and the probability values of each variable were calculated according to certain methods. There is a certain competitive relationship between scenic spots of the same type. When the distance between various scenic spots is relatively close (less than 8 km), a strong cooperative relationship can be established. Scenic spots with various levels of attraction can generally be classified as follows: cultural heritage, natural landscape, and leisure and entertainment. Scenic spots with higher attraction are usually those with a higher A-level and convenient transportation. A considerable number of tourists do not choose to visit crowded scenic destinations but choose some spots that they are more interested in according to personal preferences and based on access to free travel.

Multidimensional information fusion and broad learning system-based condition recognition for energy pipeline safety

Mechanical activities near energy pipelines pose a significant threat to energy transportation safety and energy system supply. The distributed acoustic sensing (DAS) system, which is an emerging intelligent sensing technology, can help to identify threat activities outside the pipeline. However, existing research has not distinguished fine-grained threat conditions because of insufficient information utilization and real-time requirements in the DAS system dataset, resulting in a large number of false alarms and time-consuming training in practice. To address these issues, this study proposes a pipeline radial threat condition recognition model based on multidimensional information fusion and a broad learning system (MIFBLS). First, the signal is preprocessed to improve the signal-to-noise ratio and generate the original features. Then, the extracted multidimensional temporal features are dimensionally reduced through information entropy, and the time–frequency map is fused and extracted based on a pretrained deep module, thereby achieving information fusion. Finally, the BLS incremental learning strategy is adopted to enhance the updating ability of the model and reduce the burden of data increments on the model training. Comparative experiments on a real-world natural gas pipeline dataset demonstrate the effectiveness and efficiency of the proposed method, indicating the potential for real-time monitoring and intelligent identification in pipeline transportation scenarios.

Spatial entropy as an inductive bias for vision transformers

Recent work on Vision Transformers (VTs) showed that introducing a local inductive bias in the VT architecture helps reducing the number of samples necessary for training. However, the architecture modifications lead to a loss of generality of the Transformer backbone, partially contradicting the push towards the development of uniform architectures, shared, e.g., by both the Computer Vision and the Natural Language Processing areas. In this work, we propose a different and complementary direction, in which a local bias is introduced using an auxiliary self-supervised task, performed jointly with standard supervised training. Specifically, we exploit the observation that the attention maps of VTs, when trained with self-supervision, can contain a semantic segmentation structure which does not spontaneously emerge when training is supervised. Thus, we explicitly encourage the emergence of this spatial clustering as a form of training regularization. In more detail, we exploit the assumption that, in a given image, objects usually correspond to few connected regions, and we propose a spatial formulation of the information entropy to quantify this object-based inductive bias. By minimizing the proposed spatial entropy, we include an additional self-supervised signal during training. Using extensive experiments, we show that the proposed regularization leads to equivalent or better results than other VT proposals which include a local bias by changing the basic Transformer architecture, and it can drastically boost the VT final accuracy when using small-medium training sets. The code is available at https://github.com/helia95/SAR.

A Secure Image Encryption Scheme Based on a New Hyperchaotic System and 2D Compressed Sensing.

In insecure communication environments where the communication bandwidth is limited, important image data must be compressed and encrypted for transmission. However, existing image compression and encryption algorithms suffer from poor image reconstruction quality and insufficient image encryption security. To address these problems, this paper proposes an image-compression and encryption scheme based on a newly designed hyperchaotic system and two-dimensional compressed sensing (2DCS) technique. In this paper, the chaotic performance of this hyperchaotic system is verified by bifurcation diagrams, Lyapunov diagrams, approximate entropy, and permutation entropy, which have certain advantages over the traditional 2D chaotic system. The new 2D chaotic system as a pseudo-random number generator can completely pass all the test items of NIST. Meanwhile, this paper improves on the existing 2D projected gradient (2DPG) algorithm, which improves the quality of image compression and reconstruction, and can effectively reduce the transmission pressure of image data confidential communication. In addition, a new image encryption algorithm is designed for the new 2D chaotic system, and the security of the algorithm is verified by experiments such as key space size analysis and encrypted image information entropy.

Intelligent fault diagnosis of bearings driven by double-level data fusion based on multichannel sample fusion and feature fusion under time-varying speed conditions

Bearing fault diagnosis holds paramount importance in guaranteeing the smooth operation of mechanical systems. However, single-channel vibration sensor data poses limitations in providing a comprehensive representation of bearing fault characteristics, and the fault diagnosis model extracts fault features with insufficient representativeness, resulting in low reliability and poor generalization capability of the model in the face of variable operating conditions. Therefore, a novel intelligent fault diagnosis model of bearings driven by double-level data fusion (DLDF) is developed in this paper. Initially, the fusion weights of each channel's data are obtained using the information entropy method. Then, a novel multi-modal image fusion strategy is proposed to integrate the time-frequency representations of vibration data from channels X, Y, and Z at a specific measurement point. Finally, the shallow and deep feature components extracted from the CNN model and the feature components obtained through different mapping methods are fused to extract more representative fault features, achieving a reliable diagnosis of bearing faults under time-varying speed conditions. The efficacy and reliability of the proposed approach are demonstrated through the fault diagnosis outcomes of two sets of bearing experimental data under time-varying speed conditions.

Statistical Complexity Analysis of Neurovascular Coupling with Cognitive Stimulation in Healthy Participants.

Neurovascular coupling (NVC) is the tight relationship between changes in cerebral blood flow and neural activation. NVC can be evaluated non-invasively using transcranial Doppler ultrasound (TCD)-measured changes in brain activation (cerebral blood velocity [CBv]) using different cognitive tasks and stimuli. This study used a novel approach to analyzing CBv changes occurring in response to 20 tasks from the Addenbrooke's Cognitive Examination III in 40 healthy individuals. The novel approach compared various information entropy families (permutation, Tsallis, and Rényi entropy) and statistical complexity measures based on disequilibrium. Using this approach, we found the majority of the attention, visuospatial, and memory tasks from the Addenbrooke's Cognitive Examination III that showed lower statistical complexity values when compared with the resting state. On the entropy-complexity (HC) plane, a receiver operating characteristic curve was used to distinguish between baseline and cognitive tasks using the area under the curve. Best area under the curve values were 0.91 ± 0.04, p = .001, to distinguish between resting and cognitively active states. Our findings show that brain hemodynamic signals captured with TCD can be used to distinguish between resting state (baseline) and cognitive effort (stimulation paradigms) using entropy and statistical complexity as an alternative method to traditional techniques such as coherent averaging of CBv signals. Further work should directly compare these analysis methods to identify the optimal method for analyzing TCD-measured changes in NVC.

Impact of Quantum Non-Locality and Electronic Non-Ideality on the Shannon Entropy for Atomic States in Dense Plasma.

The influence of the collective and quantum effects on the Shannon information entropy for atomic states in dense nonideal plasma was investigated. The interaction potential, which takes into account the effect of quantum non-locality as well as electronic correlations, was used to solve the Schrödinger equation for the hydrogen atom. It is shown that taking into account ionic screening leads to an increase in entropy, while taking into account only electronic screening does not lead to significant changes.

Comprehensive Evaluation of the Development Level of China’s Characteristic Towns under the Perspective of an Urban–Rural Integration Development Strategy

With the advancement of urbanization and the continuous deepening of reforms in urban–rural systems, China’s urbanization process has entered a new era of integrated urban–rural integration. Currently, as a global “new green revolution” gains momentum, numerous countries are deeply integrating the concept of sustainable development into new urban planning. Against this backdrop, urban planners worldwide are committed to building green, livable, and smart cities that can meet the needs of the present generation without compromising the ability of future generations to meet their needs, thus achieving the vision of harmonious coexistence between humanity and nature. Characteristic towns, leveraging their resource advantages, play a significant role in achieving sustainable regional economic development. They serve as valuable references for China’s urban transformation and upgrading, as well as for promoting rural urbanization, and are crucial avenues for advancing China’s urban–rural integration development strategy. The evaluation of the development level of characteristic towns is a necessary step in their progress and a strong guarantee for promoting their construction and development. Therefore, effectively evaluating the social benefits of characteristic towns is paramount. This study constructs an evaluation model based on the grey rough set theory and Technique for Order Preference by Similarity to Ideal Solution of TOPSIS. Firstly, an evaluation index system for the development level of characteristic towns is established. Then, the grey relational analysis method and rough set theory are used to reduce the index attributes, while the conditional information entropy theory is introduced to determine the weights of the reduced indicators. Finally, the TOPSIS model is applied to evaluate the development level of characteristic towns. Through empirical research, eight characteristic towns in Zhejiang Province, China, were assessed and ranked, verifying the effectiveness and feasibility of the proposed model.

Rényi entropy-complexity causality space: a novel neurocomputational tool for detecting scale-free features in EEG/iEEG data.

Scale-free brain activity, linked with learning, the integration of different time scales, and the formation of mental models, is correlated with a metastable cognitive basis. The spectral slope, a key aspect of scale-free dynamics, was proposed as a potential indicator to distinguish between different sleep stages. Studies suggest that brain networks maintain a consistent scale-free structure across wakefulness, anesthesia, and recovery. Although differences in anesthetic sensitivity between the sexes are recognized, these variations are not evident in clinical electroencephalographic recordings of the cortex. Recently, changes in the slope of the power law exponent of neural activity were found to correlate with changes in Rényi entropy, an extended concept of Shannon's information entropy. These findings establish quantifiers as a promising tool for the study of scale-free dynamics in the brain. Our study presents a novel visual representation called the Rényi entropy-complexity causality space, which encapsulates complexity, permutation entropy, and the Rényi parameter q. The main goal of this study is to define this space for classical dynamical systems within theoretical bounds. In addition, the study aims to investigate how well different time series mimicking scale-free activity can be discriminated. Finally, this tool is used to detect dynamic features in intracranial electroencephalography (iEEG) signals. To achieve these goals, the study implementse the Bandt and Pompe method for ordinal patterns. In this process, each signal is associated with a probability distribution, and the causal measures of Rényi entropy and complexity are computed based on the parameter q. This method is a valuable tool for analyzing simulated time series. It effectively distinguishes elements of correlated noise and provides a straightforward means of examining differences in behaviors, characteristics, and classifications. For the iEEG experimental data, the REM state showed a greater number of significant sex-based differences, while the supramarginal gyrus region showed the most variation across different modes and analyzes. Exploring scale-free brain activity with this framework could provide valuable insights into cognition and neurological disorders. The results may have implications for understanding differences in brain function between the sexes and their possible relevance to neurological disorders.