Minimum Conditional Entropy Research Articles

A novel method of nonuniform phase space reconstruction for multivariate prediction of daily runoff

Phase space reconstruction is crucial for predicting chaotic hydrological time series. However, traditional multivariate phase space reconstruction methods, such as high-dimensional and low-dimensional phase space reconstruction (HDPSR and LDPSR) neglect variable interdependence, resulting in redundant information in the reconstructed phase space and affecting the model prediction accuracy. To address this limitation, this study proposes a new nonuniform phase space reconstruction (NUPSR) method that combines the minimum redundancy maximum relevance criterion and conditional entropy. Then, using HDPSR, LDPSR, NUPSR, particle swarm optimization (PSO), support vector regression (SVR), and bidirectional long- and short-term memory neural networks (BiLSTM), the models HDPSR-PSO-SVR, LDPSR-PSO-SVR, NUPSR-PSO-SVR, HDPSR-PSO-BiLSTM, LDPSR-PSO-BiLSTM, and NUPSR-PSO-BiLSTM are built to evaluate the performance of the NUPSR method. To demonstrate the effectiveness of the NUPSR method in multistep daily runoff prediction, the models PSO-NUPSR-BiLSTM, LDPSR-PSO-BiLSTM, and HDPSR-PSO-BiLSTM are also built. The multifactor daily runoff data from five hydrological stations in the Weihe River Basin are used to evaluate the prediction performance of all models. The results show that the state variables obtained by NUPSR are more independent and less redundant than those obtained by HDPSR and LDPSR, as evidenced by the embedded state variables and lower embedding dimension of NUPSR and the performance of the model based on NUPSR outperforms based on HDPSR and LDPSR. Particularly, the PSO-NUPSR-BiLSTM model exhibits a slower decrease in prediction accuracy compared to the other two models as the number of prediction steps increases. In summary, the NUPSR method effectively selects and reconstructs predictors, significantly improves the prediction accuracy of the coupled models, and has broad application prospects in multivariate prediction in fields such as hydrology.

T-TIME: Test-Time Information Maximization Ensemble for Plug-and-Play BCIs.

An electroencephalogram (EEG)-based brain-computer interface (BCI) enables direct communication between the human brain and a computer. Due to individual differences and non-stationarity of EEG signals, such BCIs usually require a subject-specific calibration session before each use, which is time-consuming and user-unfriendly. Transfer learning (TL) has been proposed to shorten or eliminate this calibration, but existing TL approaches mainly consider offline settings, where all unlabeled EEG trials from the new user are available. This article proposes Test-Time Information Maximization Ensemble (T-TIME) to accommodate the most challenging online TL scenario, where unlabeled EEG data from the new user arrive in a stream, and immediate classification is performed. T-TIME initializes multiple classifiers from the aligned source data. When an unlabeled test EEG trial arrives, T-TIME first predicts its labels using ensemble learning, and then updates each classifier by conditional entropy minimization and adaptive marginal distribution regularization. Our code is publicized. Extensive experiments on three public motor imagery based BCI datasets demonstrated that T-TIME outperformed about 20 classical and state-of-the-art TL approaches. To our knowledge, this is the first work on test time adaptation for calibration-free EEG-based BCIs, making plug-and-play BCIs possible.

A-unital Operations and Quantum Conditional Entropy

Negative quantum conditional entropy states are key ingredients for information theoretic tasks such as superdense coding, state merging and one-way entanglement distillation. In this work, we ask: how does one detect if a channel is useful in preparing negative conditional entropy states? We answer this question by introducing the class of A-unital channels, which we show are the largest class of conditional entropy non-decreasing channels. We also prove that A-unital channels are precisely the completely free operations for the class of states with non-negative conditional entropy. Furthermore, we study the relationship between A-unital channels and other classes of channels pertinent to the resource theory of entanglement. We then prove similar results for ACVENN: a previously defined, relevant class of states and also relate the maximum and minimum conditional entropy of a state with its von Neumann entropy. The definition of A-unital channels naturally lends itself to a procedure for determining membership of channels in this class. Thus, our work is valuable for the detection of resourceful channels in the context of conditional entropy.

An effective health indicator for bearing using corrected conditional entropy through diversity-driven multi-parent evolutionary algorithm

The bearing vibration signal possesses nonlinear and non-stationary characteristics; hence; it is difficult to diagnosis the faults in the bearing under different working conditions. In this article, a new scheme has been proposed based on complete ensemble empirical mode decomposition with adaptive noise and corrected conditional entropy to recognize the different class of faults in bearing. The mode with minimum corrected conditional entropy is treated as a prominent mode from which sensitive features are extracted. A filter-based feature selection scheme is used for the same and for ranking the features based on variance to reduce the redundancy of data set. This data set is made input to support vector machine. The performance of the support vector machine classifier is improved by optimizing its parameters to obtain maximum classification accuracy. To address this issue, an evolutionary algorithm (diversity-driven multi-parent evolutionary algorithm) is used. With optimized support vector machine parameters, the support vector machine is trained to build a classification model with 10-fold cross-validation. After training, the built model is tested against test data set for fitness evaluation. The support vector machine classifier gives 100% accuracy at regularization and kernel parameter’s value of 1.3343 and of 782.6329, respectively, with 27.93 s of training time for a single iteration.

Multi-Harmonic Source Localization Based on Sparse Component Analysis and Minimum Conditional Entropy.

Aiming at the fact that the independent component analysis algorithm requires more measurement points and cannot solve the problem of harmonic source location under underdetermined conditions, a new method based on sparse component analysis and minimum conditional entropy for identifying multiple harmonic source locations in a distribution system is proposed. Under the condition that the network impedance is unknown and the number of harmonic sources is undetermined, the measurement node configuration algorithm selects the node position to make the separated harmonic current more accurate. Then, using the harmonic voltage data of the selected node as the input, the sparse component analysis is used to solve the harmonic current waveform under underdetermination. Finally, the conditional entropy between the harmonic current and the system node is calculated, and the node corresponding to the minimum condition entropy is the location of the harmonic source. In order to verify the effectiveness and accuracy of the proposed method, the simulation was performed in an IEEE 14-node system. Moreover, compared with the results of independent component analysis algorithms. Simulation results verify the correctness and effectiveness of the proposed algorithm.

A study into mechanisms of attitudinal scale conversion: A randomized stochastic ordering approach

This paper considers the methodological challenge of how to convert categorical attitudinal scores (like satisfaction) measured on one scale to a categorical attitudinal score measured on another scale with a different range. This is becoming a growing issue in marketing consulting and the common available solutions seem too few and too superficial. A new methodology for scale conversion is proposed, and tested in a comprehensive study. This methodology is shown to be both relevant and optimal in fundamental aspects. The new methodology is based on a novel algorithm named minimum conditional entropy, that uses the marginal distributions of the responses on each of the two scales to produce a unique joint bivariate distribution. In this joint distribution, the conditional distributions follow a stochastic order that is monotone in the categories and has the relevant optimal property of maximizing the correlation between the two underlying marginal scales. We show how such a joint distribution can be used to build a mechanism for scale conversion. We use both a frequentist and a Bayesian approach to derive mixture models for conversion mechanisms, and discuss some inferential aspects associated with the underlying models. These models can incorporate background variables of the respondents. A unique observational experiment is conducted that empirically validates the proposed modeling approach. Strong evidence of validation is obtained.

Uncertainty relations with quantum memory for the Wehrl entropy

We prove two new fundamental uncertainty relations with quantum memory for the Wehrl entropy. The first relation applies to the bipartite memory scenario. It determines the minimum conditional Wehrl entropy among all the quantum states with a given conditional von Neumann entropy and proves that this minimum is asymptotically achieved by a suitable sequence of quantum Gaussian states. The second relation applies to the tripartite memory scenario. It determines the minimum of the sum of the Wehrl entropy of a quantum state conditioned on the first memory quantum system with the Wehrl entropy of the same state conditioned on the second memory quantum system and proves that also this minimum is asymptotically achieved by a suitable sequence of quantum Gaussian states. The Wehrl entropy of a quantum state is the Shannon differential entropy of the outcome of a heterodyne measurement performed on the state. The heterodyne measurement is one of the main measurements in quantum optics and lies at the basis of one of the most promising protocols for quantum key distribution. These fundamental entropic uncertainty relations will be a valuable tool in quantum information and will, for example, find application in security proofs of quantum key distribution protocols in the asymptotic regime and in entanglement witnessing in quantum optics.

Modified sequential k‐means clustering by utilizing response: A case study for fashion products

AbstractModified sequential k‐means clustering concerns a k‐means clustering problem in which the clustering machine utilizes output similarity in addition. While conventional clustering methods commonly recognize similar instances at features‐level modified sequential clustering takes advantage of response, too. To this end, the approach we pursue is to enhance the quality of clustering by using some proper information. The information enables the clustering machine to detect more patterns and dependencies that may be relevant. This allows one to determine, for instance, which fashion products exhibit similar behaviour in terms of sales. Unfortunately, conventional clustering methods cannot tackle such cases, because they handle attributes solely at the feature level without considering any response. In this study, we introduce a novel approach underlying minimum conditional entropy clustering and show its advantages in terms of data analytics. In particular, we achieve this by modifying the conventional sequential k‐means algorithm. This modified clustering approach has the ability to reflect the response effect in a consistent manner. To verify the feasibility and the performance of this approach, we conducted several experiments based on real data from the apparel industry.

Harmonic Source Localization Approach Based on Fast Kernel Entropy Optimization ICA and Minimum Conditional Entropy

Based on the fast kernel entropy optimization independent component analysis and the minimum conditional entropy, this paper proposes a harmonic source localization method which aims at accurately estimating harmonic currents and identifying harmonic sources. The injected harmonic currents are estimated by the fast kernel entropy optimization independent component analysis (FKEO-ICA) in the absence of prior knowledge of harmonic impedances. Then, the minimum conditional entropy is applied to locate the harmonic sources based on the estimated harmonic currents. The proposed harmonic source localization method is validated on the IEEE 34-bus system. By applying the correlation coefficient and three error evaluation indicators, comparison has been made among the performances of the FKEO-ICA and three other ICA algorithms. The results show that the FKEO-ICA algorithm could achieve a significantly better accuracy of harmonic current estimation, while the minimum conditional entropy could determine the locations of harmonic sources precisely.

Analysis of Gait Rhythm Fluctuations for Neurodegenerative Diseases by Phase Synchronization and Conditional Entropy.

Previous studies have revealed that gait rhythm fluctuations convey important information, which is useful for understanding certain types of neurodegenerative diseases such as Amyotrophic Lateral Sclerosis (ALS), Huntington's disease (HD) and Parkinson's disease (PD). However, previous investigations only focused on the locomotor patterns of each individual foot rather than the relations between both feet. Therefore, in our study, phase synchronization (the index ρ) and conditional entropy (Hc) were applied to the five types of time series pairs of gait rhythms (stride time, swing time, stance time, % swing time and % stance time). The results revealed that compared with the patients with ALS, HD and PD, gait rhythms of normal subjects have the strongest phase synchronization property and minimum conditional entropy value. In addition, the indices ρ and Hc cannot only significantly differentiate among the four groups of subjects (ALS, HD, PD and control) but also have the ability to discriminate between any two of these subject groups. Finally, three representative classifiers were utilized in order to evaluate the possible capabilities of the indices ρ and Hc to distinguish the patients with neurodegenerative diseases from the healthy subjects, and achieved maximum area under the curve (AUC) values of 0.959, 0.928 and 0.824 for HD, PD and ALS detection, respectively. In summary, our study provides insight into the relational analysis between gait rhythms measured from both feet, and suggests that it should be considered seriously in the future studies investigating the impact of neurodegenerative disease and potential therapeutic intervention.

Conditional density estimation with dimensionality reduction via squared-loss conditional entropy minimization.

Regression aims at estimating the conditional mean of output given input. However, regression is not informative enough if the conditional density is multimodal, heteroskedastic, and asymmetric. In such a case, estimating the conditional density itself is preferable, but conditional density estimation (CDE) is challenging in high-dimensional space. A naive approach to coping with high dimensionality is to first perform dimensionality reduction (DR) and then execute CDE. However, a two-step process does not perform well in practice because the error incurred in the first DR step can be magnified in the second CDE step. In this letter, we propose a novel single-shot procedure that performs CDE and DR simultaneously in an integrated way. Our key idea is to formulate DR as the problem of minimizing a squared-loss variant of conditional entropy, and this is solved using CDE. Thus, an additional CDE step is not needed after DR. We demonstrate the usefulness of the proposed method through extensive experiments on various data sets, including humanoid robot transition and computer art.

A nonparametric clustering algorithm with a quantile-based likelihood estimator.

Clustering is a representative of unsupervised learning and one of the important approaches in exploratory data analysis. By its very nature, clustering without strong assumption on data distribution is desirable. Information-theoretic clustering is a class of clustering methods that optimize information-theoretic quantities such as entropy and mutual information. These quantities can be estimated in a nonparametric manner, and information-theoretic clustering algorithms are capable of capturing various intrinsic data structures. It is also possible to estimate information-theoretic quantities using a data set with sampling weight for each datum. Assuming the data set is sampled from a certain cluster and assigning different sampling weights depending on the clusters, the cluster-conditional information-theoretic quantities are estimated. In this letter, a simple iterative clustering algorithm is proposed based on a nonparametric estimator of the log likelihood for weighted data sets. The clustering algorithm is also derived from the principle of conditional entropy minimization with maximum entropy regularization. The proposed algorithm does not contain a tuning parameter. The algorithm is experimentally shown to be comparable to or outperform conventional nonparametric clustering methods.

Distribution of quantum correlations and conditional entropy in Heisenberg spin chains

The quantum discord, which quantifies the amount of quantum correlation present between parts of a system, is investigated for spin systems. The discord for a pair of spins in the many-spin ground state, computed by minimizing the conditional entropy of the two spins, is related to the diagonal and off-diagonal spin–spin correlation functions and the local magnetization. We show that for isotropic and translationally invariant states, the discord depends only on the diagonal spin correlation function. Thus, near a thermal/quantum critical point, the pairwise discord exhibits long-range behavior, analogously to the correlation function. The discord exhibits a kink singularity as a function of the anisotropy parameter for the ground state of the anisotropic Heisenberg model, for both nearest-neighbor spins as well as for well-separated spins. The preferred measurement basis for the minimum conditional entropy, which determines the discord, changes discontinuously across the critical point. We also investigate, from exact numerical diagonalization, the distribution of conditional entropies as the measurement basis is varied. For the isotropic model, the conditional entropy distribution is a δ-function, whereas it has a twin-peak structure for the anisotropic model. It is shown that the average value and the mean-square fluctuation of the conditional entropy also show a signature of the critical-point behavior.

Affinity Propagation for the Context Quantization

In this paper, the context quantization for I-ary sources based on the affinity propagation algorithm is presented. In purpose of finding the optimal number of classes, the increment of the adaptive code length is suggested to be the similarity measure between two conditional probability distributions, by which the similarity matrix is constructed as the input of the affinity propagation algorithm. After the given number of iterations, the optimal quantizer with the optimal number of classes is achieved and the adaptive code length is minimized at the same time. The simulations indicate that the proposed algorithm produces results that are better than the results obtained by the minimum conditional entropy context quantization implemented by K-means with lower computational complexity.

Entropy-based sliced inverse regression

The importance of dimension reduction has been increasing according to the growth of the size of available data in many fields. An appropriate dimension reduction method of raw data helps to reduce computational time and to expose the intrinsic structure of complex data. Sliced inverse regression is a well-known dimension reduction method for regression, which assumes an elliptical distribution for the explanatory variable, and ingeniously reduces the problem of dimension reduction to a simple eigenvalue problem. Sliced inverse regression is based on the strong assumptions on the data distribution and the form of regression function, and there are a number of methods to relax or remove these assumptions to extend the applicability of the inverse regression method. However, each method is known to have its drawbacks either theoretically or empirically. To alleviate drawbacks in the existing methods, a dimension reduction method for regression based on the notion of conditional entropy minimization is proposed. Using entropy as a measure of dispersion of data, a low dimensional subspace is estimated without assuming any specific distribution nor any regression function. The proposed method is shown to perform comparable or superior to the conventional methods through experiments using artificial and real-world datasets.

Leveraging domain information to restructure biological prediction

BackgroundIt is commonly believed that including domain knowledge in a prediction model is desirable. However, representing and incorporating domain information in the learning process is, in general, a challenging problem. In this research, we consider domain information encoded by discrete or categorical attributes. A discrete or categorical attribute provides a natural partition of the problem domain, and hence divides the original problem into several non-overlapping sub-problems. In this sense, the domain information is useful if the partition simplifies the learning task. The goal of this research is to develop an algorithm to identify discrete or categorical attributes that maximally simplify the learning task.ResultsWe consider restructuring a supervised learning problem via a partition of the problem space using a discrete or categorical attribute. A naive approach exhaustively searches all the possible restructured problems. It is computationally prohibitive when the number of discrete or categorical attributes is large. We propose a metric to rank attributes according to their potential to reduce the uncertainty of a classification task. It is quantified as a conditional entropy achieved using a set of optimal classifiers, each of which is built for a sub-problem defined by the attribute under consideration. To avoid high computational cost, we approximate the solution by the expected minimum conditional entropy with respect to random projections. This approach is tested on three artificial data sets, three cheminformatics data sets, and two leukemia gene expression data sets. Empirical results demonstrate that our method is capable of selecting a proper discrete or categorical attribute to simplify the problem, i.e., the performance of the classifier built for the restructured problem always beats that of the original problem.ConclusionsThe proposed conditional entropy based metric is effective in identifying good partitions of a classification problem, hence enhancing the prediction performance.

Spike-Timing-Dependent Plasticity and Reliability Optimization: The Role of Neuron Dynamics

Plastic changes in synaptic efficacy can depend on the time ordering of presynaptic and postsynaptic spikes. This phenomenon is called spike-timing-dependent plasticity (STDP). One of the most striking aspects of this plasticity mechanism is that the STDP windows display a great variety of forms in different parts of the nervous system. We explore this issue from a theoretical point of view. We choose as the optimization principle the minimization of conditional entropy or maximization of reliability in the transmission of information. We apply this principle to two types of postsynaptic dynamics, designated type I and type II. The first is characterized as being an integrator, while the second is a resonator. We find that, depending on the parameters of the models, the optimization principle can give rise to a wide variety of STDP windows, such as antisymmetric Hebbian, predominantly depressing or symmetric with one positive region and two lateral negative regions. We can relate each of these forms to the dynamical behavior of the different models. We also propose experimental tests to assess the validity of the optimization principle.

A Conditional Entropy Minimization Criterion for Dimensionality Reduction and Multiple Kernel Learning

Reducing the dimensionality of high-dimensional data without losing its essential information is an important task in information processing. When class labels of training data are available, Fisher discriminant analysis (FDA) has been widely used. However, the optimality of FDA is guaranteed only in a very restricted ideal circumstance, and it is often observed that FDA does not provide a good classification surface for many real problems. This letter treats the problem of supervised dimensionality reduction from the viewpoint of information theory and proposes a framework of dimensionality reduction based on class-conditional entropy minimization. The proposed linear dimensionality-reduction technique is validated both theoretically and experimentally. Then, through kernel Fisher discriminant analysis (KFDA), the multiple kernel learning problem is treated in the proposed framework, and a novel algorithm, which iteratively optimizes the parameters of the classification function and kernel combination coefficients, is proposed. The algorithm is experimentally shown to be comparable to or outperforms KFDA for large-scale benchmark data sets, and comparable to other multiple kernel learning techniques on the yeast protein function annotation task.

Minimum conditional entropy principle for irregular firing of cortical neurons

Minimum conditional entropy principle for irregular firing of cortical neurons

Minimum conditional entropy principle inferred from irregular firing of in vivo cortical neurons

A major challenge in biology is to understand how molecular processes determine phenotypic features. We address this fundamental problem in a class of model systems by developing a general mathematical framework that allows the calculation of mesoscopic properties from the knowledge of microscopic Markovian transition probabilities. We show how exact analytic formulae for the first and second moments of resident time distributions in mesostates can be derived from microscopic resident times and transition probabilities even for systems with a large number of microstates. We apply our formalism to models of the inositol trisphosphate receptor, which plays a key role in generating calcium signals triggering a wide variety of cellular responses. We demonstrate how experimentally accessible quantities, such as opening and closing times and the coefficient of variation of inter-spike intervals, and other, more elaborated, quantities can be analytically calculated from the underlying microscopic Markovian dynamics. A virtue of our approach is that we do not need to follow the detailed time evolution of the whole system, as we derive the relevant properties of its steady state without having to take into account the often extremely complicated transient features. We emphasize that our formulae fully agree with results obtained by stochastic simulations and approaches based on a full determination of the microscopic system's time evolution. We also illustrate how experiments can be devised to discriminate between alternative molecular models of the inositol trisphosphate receptor. The developed approach is applicable to any system described by a Markov process and, owing to the analytic nature of the resulting formulae, provides an easy way to characterize also rare events that are of particular importance to understand the intermittency properties of complex dynamic systems.