Error Entropy Research Articles

Displacement Time Series Prediction Model of Landslide Based on Phase Space Reconstruction

In order to fully reveal information about landslide displacement, it was necessary to extend a time series to a higher-dimensional state space for the characteristic of univariate time series. However, in order to control the expansion of noise, an appropriate embedded dimension of phase space reconstruction was not the bigger the better. In this paper, based on the displacement time series of landslide, the phase space theory was used to build displacement time series matrix and the entropy theory was used to get the entropy. The embedded dimension of phase space reconstruction could be adjusted according to the change of entropy and feedback of displacement prediction error and a support vector regression model was created via the support vector machine’s learning. The application on Baijiabao landslide indicates that the proposed method achieves a high accuracy and stability of prediction.

Impact of different saturation encoding modes on object classification using a BP wavelet neural network

Wavelet neural networks have been successfully applied to object classification due to their unique various advantages. The wavelet neural network used in this paper is a type of back-propagation algorithm-learning wavelet neural network. The log-sigmoid function and wavelet basis function satisfying the frame condition are employed as an activation function in the output and hidden layers, respectively, and the entropy error function is also used to accelerate the learning speed. The log-sigmoid function has two saturated values, 0 and 1, which are the value of the function at a point whose value changes slightly as the independent variable changes at a somewhat wide range. Using this property of the saturated values and simplifying the mathematical model of neural network classification, we may mathematically prove that using different saturated values to encode the modes can affect the training error, generalization ability, and anti-noise ability of the wavelet neural network, in turn resulting in differences in classification accuracy. The saturated and unsaturated value-encoding modes will both decrease the generalization ability of the network and reduce the classification accuracy due to excessively strong or weak anti-noise ability. Therefore, we propose a type of moderate saturated-value encoding mode, in which the anti-noise ability, the gradient, and error in training process are more moderate than the other two encodings, so that this kind of encoding mode can facilitate a stronger generalization ability and higher classification accuracy for the wavelet neural network, and which have been affirmed in the classification experiments of CHRIS remote-sensing imagery of the Huanghe estuary coastal wetland and SIR-C remote-sensing image of sea ice in the Labrador Gulf, and reaffirmed in classification experiments where noise was added to the test data.

Minimum Entropy-Based Cascade Control for Governing Hydroelectric Turbines

In this paper, an improved cascade control strategy is presented for hydroturbine speed governors. Different from traditional proportional-integral-derivative (PID) control and model predictive control (MPC) strategies, the performance index of the outer controller is constructed by integrating the entropy and mean value of the tracking error with the constraints on control energy. The inner controller is implemented by a proportional controller. Compared with the conventional PID-P and MPC-P cascade control methods, the proposed cascade control strategy can effectively decrease fluctuations of hydro-turbine speed under non-Gaussian disturbance conditions in practical hydropower plants. Simulation results show the advantages of the proposed cascade control method.

Data Processing with Computation in Bayes Reliability Analysis for Burr Type X Distribution under Different Loss Functions

This paper studies the estimation of the parameter of Burr Type X distribution. Maximum likelihood estimator is first derived, and then the Bayes and Empirical Bayes estimators of the unknown parameter are obtained under three loss functions, which are squared error loss, LINEX loss and entropy loss functions. The prior distribution of parmeter used in this paper is Gamma distribution. Finally, a Monte Carlo simulation is given to illustrate the application of these estimators.

Estimation for the Parameters of Generalized Half-normal Distribution Based on Progressive Type-I Interval Censoring

Based on progressively Type-I interval censored sample, the problem of estimating unknown parameters of a two parameter generalized half-normal(GHN) distribution is considered. Different methods of estimation are discussed. They include the maximum likelihood estimation, midpoint approximation method, approximate maximum likelihood estimation, method of moments, and estimation based on probability plot. Several Bayesian estimates with respect to different symmetric and asymmetric loss functions such as squared error, LINEX, and general entropy is calculated. The Lindley’s approximation method is applied to determine Bayesian estimates. Monte Carlo simulations are performed to compare the performances of the different methods. Finally, analysis is also carried out for a real dataset.

Estimation of atmospheric predictability for multivariable system using information theory in nonlinear error growth dynamics

To estimate atmospheric predictability for multivariable system, based on information theory in nonlinear error growth dynamics, a quantitative method is introduced in this paper using multivariable joint predictability limit (MJPL) and corresponding single variable predictability limit (SVPL). The predictability limit, obtained from the evolutions of nonlinear error entropy and climatological state entropy, is not only used to measure the predictability of dynamical system with the constant climatological state entropy, but also appropriate to the case of climatological state entropy changed with time. With the help of daily NCEP-NCAR reanalysis data, by using a method of local dynamical analog, the nonlinear error entropy, climatological state entropy, and predictability limit are obtained, and the SVPLs and MJPL of the winter 500-hPa temperature field, zonal wind field and meridional wind field are also investigated. The results show that atmospheric predictability is well associated with the analytical variable. For single variable predictability, there exists a big difference for the three variables, with the higher predictability found for the temperature field and zonal wind field and the lower predictability for the meridional wind field. As seen from their spatial distributions, the SVPLs of the three variables appear to have a property of zonal distribution, especially for the meridional wind field, which has three zonal belts with low predictability and four zonal belts with high predictability. For multivariable joint predictability, the MJPL of multivariable system with the three variables is not a simple mean or linear combination of its SVPLs. It presents an obvious regional difference characteristic. Different regions have different results. In some regions, the MJPL is among its SVPLs. However, in other regions, the MJPL is less than its all SVPLs.

An Extended Result on the Optimal Estimation Under the Minimum Error Entropy Criterion

The minimum error entropy (MEE) criterion has been successfully used in fields such as parameter estimation, system identification and the supervised machine learning. There is in general no explicit expression for the optimal MEE estimate unless some constraints on the conditional distribution are imposed. A recent paper has proved that if the conditional density is conditionally symmetric and unimodal (CSUM), then the optimal MEE estimate (with Shannon entropy) equals the conditional median. In this study, we extend this result to the generalized MEE estimation where the optimality criterion is the Renyi entropy or equivalently, the α-order information potential (IP).

A Note on the W-S Lower Bound of the MEE Estimation

The minimum error entropy (MEE) estimation is concerned with the estimation of a certain random variable (unknown variable) based on another random variable (observation), so that the entropy of the estimation error is minimized. This estimation method may outperform the well-known minimum mean square error (MMSE) estimation especially for non-Gaussian situations. There is an important performance bound on the MEE estimation, namely the W-S lower bound, which is computed as the conditional entropy of the unknown variable given observation. Though it has been known in the literature for a considerable time, up to now there is little study on this performance bound. In this paper, we reexamine the W-S lower bound. Some basic properties of the W-S lower bound are presented, and the characterization of Gaussian distribution using the W-S lower bound is investigated.

Two new mathematical models for prediction of early mortality risk in coronary artery bypass graft surgery

The aim of this study was to develop new models for prediction of short-term mortality risk in on-pump coronary artery bypass grafting (CABG) surgery using decision tree (DT) methods. Between September 2005 and April 2006, 948 consecutive patients underwent CABG surgery at Rajaie Heart Center. Potential risk factors were reviewed and univariate and multivariate analysis for short-term mortality were performed. The whole dataset was divided into mutually exclusive subsets. An entropy error fuzzy decision tree (EEFDT) and an entropy error crisp decision tree (EECDT) were implemented using 650 (68.6%) patient data and tested with 298 (31.4%) patient data. Ten times hold-out cross validation was done and the area under the receiver operative characteristic curve (AUC) was reported as model performance. The results were compared with the logistic regression (LR) model and EuroSCORE. The overall short-term mortality rate was 3.8%, and was statistically higher in women than men (P<.001). The final EEFDT selected 19 variables and resulted in a tree with 39 nodes, 20 conditional rules, and AUC of 0.90±0.008. The final EECDT selected 15 variables and resulted in a tree with 35 nodes, 18 conditional rules, and AUC of 0.86±0.008. The LR model selected 10 variables and resulted in an AUC of 0.78±0.008; the AUC for EuroSCORE was 0.77±0.003. There were no differences in the discriminatory power of EEFDT and EECDT (P=.066) and their performance was superior to LR and EuroSCORE. EEFDT, EECDT, LR, and EuroSCORE had clinical acceptance but the performance and accuracy of the DTs were superior to the other models.

Estimation for the generalized half-normal distribution based on progressive type-II censoring

Based on progressively type-II censored data, the problem of estimating unknown parameters and reliability function of a two-parameter generalized half-normal distribution is considered. Maximum likelihood estimates are obtained by applying expectation-maximization algorithm. Since they do not have closed forms, approximate maximum likelihood estimators are proposed. Several Bayesian estimates with respect to different symmetric and asymmetric loss functions such as squared error, LINEX and general entropy are calculated. The Lindley approximation method is applied to determine Bayesian estimates. Monte Carlo simulations are performed to compare the performances of the different methods. Finally, one real data set is analysed.

Invexity of the Minimum Error Entropy Criterion

In this letter, optimization properties of Minimization of Error Entropy (MEE) and Minimization of Error Entropy with Fiducial points (MEEF) are presented. It is proved that by varying the kernel parameter of the MEE and/or MEEF objective function, the resulting problem, in general leads to an invex problem. Furthermore, for certain values of the kernel parameter it is shown that the problems may transform to convex or pseudo-convex problems.

Indirect iterative learning control for robot manipulator with non‐Gaussian disturbances

In this study, a novel indirect iterative learning control (ILC) strategy is presented for a robotic manipulator that performs repeat operation and is also subjected to non-Gaussian disturbances. The performance index about the entropy of tracking error and the related optimisation method are used to update the local parameters of controller between any two adjacent batches. Moreover, the entropy is employed as it is a unified probabilistic measure of uncertainty quantification regardless whether the random disturbances are Gaussian distribution or not. Thus, an innovative performance index about tracking error entropy that represents the relationship between the entropy of error and controller gains is proposed in order to obtain the controller which can drive the uncertainty of output error as small as possible with the increase of the batch number. Then the non-linear stochastic optimal method is presented so as to obtain updated gains for the next batch. Stability of closed-loop system is analysed. Finally, a comparison between a classic ILC and the proposed approach is given. Moreover, the effectiveness and feasibility of the proposed control schemes is verified by some simulation results of robotic trajectory tracking.

Improved single neuron controller for multivariable stochastic systems with non-Gaussianities and unmodeled dynamics

In this paper, a new adaptive control approach is presented for multivariate nonlinear non-Gaussian systems with unknown models. A more general and systematic statistical measure, called (h,ϕ)-entropy, is adopted here to characterize the uncertainty of the considered systems. By using the “sliding window” technique, the non-parameter estimate of the (h,ϕ)-entropy is formulated. Then, the improved neuron based controllers are developed for multivariate nonlinear non-Gaussian systems by minimizing the entropies of the tracking errors in closed loops. The condition to guarantee the strictly decreasing entropy of tracking error is presented. Moreover, the convergence in the mean-square sense has been analyzed for all the weights in the neural controllers. Finally, the comparative simulation results are presented to show that the performance of the proposed algorithm is superior to that of PID control strategy.

Improved Minimum Entropy Filtering for Continuous Nonlinear Non-Gaussian Systems Using a Generalized Density Evolution Equation

This paper investigates the filtering problem for multivariate continuous nonlinear non-Gaussian systems based on an improved minimum error entropy (MEE) criterion. The system is described by a set of nonlinear continuous equations with non-Gaussian system noises and measurement noises. The recently developed generalized density evolution equation is utilized to formulate the joint probability density function (PDF) of the estimation errors. Combining the entropy of the estimation error with the mean squared error, a novel performance index is constructed to ensure the estimation error not only has small uncertainty but also approaches to zero. According to the conjugate gradient method, the optimal filter gain matrix is then obtained by minimizing the improved minimum error entropy criterion. In addition, the condition is proposed to guarantee that the estimation error dynamics is exponentially bounded in the mean square sense. Finally, the comparative simulation results are presented to show that the proposed MEE filter is superior to nonlinear unscented Kalman filter (UKF).

Kernel minimum error entropy algorithm

As an alternative adaptation criterion, the minimum error entropy (MEE) criterion has been receiving increasing attention due to its successful use in, especially, nonlinear and non-Gaussian signal processing. In this paper, we study the application of error entropy minimization to kernel adaptive filtering, a new and promising technique that implements the conventional linear adaptive filters in reproducing kernel Hilbert space (RKHS) and obtains the nonlinear adaptive filters in original input space. The kernel minimum error entropy (KMEE) algorithm is derived, which is essentially a generalized stochastic information gradient (SIG) algorithm in RKHS. The computational complexity of KMEE is just similar to the kernel affine projection algorithm (KAPA). We also utilize the quantization approach to constrain the network size growth, and develop the quantized KMEE (QKMEE) algorithm. Further, we analyze the mean square convergence of KMEE. The energy conservation relation is derived and a sufficient condition that ensures the mean square convergence is obtained. The performance of the new algorithm is demonstrated in nonlinear system identification and short-term chaotic time series prediction.

Complex-Valued Filtering Based on the Minimization of Complex-Error Entropy

In this paper, we consider the training of complex-valued filter based on the information theoretic method. We first generalize the error entropy criterion to complex domain to present the complex error entropy criterion (CEEC). Due to the difficulty in estimating the entropy of complex-valued error directly, the entropy bound minimization (EBM) method is used to compute the upper bounds of the entropy of the complex-valued error, and the tightest bound selected by the EBM algorithm is used as the estimator of the complex-error entropy. Then, based on the minimization of complex-error entropy (MCEE) and the complex gradient descent approach, complex-valued learning algorithms for both the (linear) transverse filter and the (nonlinear) neural network are derived. The algorithms are applied to complex-valued linear filtering and complex-valued nonlinear channel equalization to demonstrate their effectiveness and advantages.

Resilient Minimum Entropy Filter Design for Non-Gaussian Stochastic Systems

In this paper, the resilient minimum entropy filter problem is investigated for the stochastic systems with non-Gaussian disturbances. The goal of designing the filter is to guarantee that the entropy of the estimation error is monotonically decreasing, moreover, the error system is exponentially ultimately bounded in the mean square. Based on the entropy performance function, a filter gain updating algorithm is presented to make the entropy decrease at every sampling instant k. Then the boundedness of the gain updating law is analyzed using the kernel density estimation technique. Furthermore, a suboptimal resilient filter gain is designed in terms of LMI. Finally, a simulation example is given to show the effectiveness of the proposed results.

Learning theory approach to minimum error entropy criterion

We consider the minimum error entropy (MEE) criterion and an empirical risk minimization learning algorithm when an approximation of Renyi's entropy (of order 2) by Parzen windowing is minimized. T...

Shannon's Random-Cipher Result and the GeneralizedR-Norm Entropy of Typeβ

Using the Fano inequality for the generalizedR-norm entropy and Bayess probability of error, a generalized random-cipher result is proved by taking into account the generalizedR-norm entropy of typeβ.

Neural PID Control Strategy for Networked Process Control

A new method with a two-layer hierarchy is presented based on a neural proportional-integral-derivative (PID) iterative learning method over the communication network for the closed-loop automatic tuning of a PID controller. It can enhance the performance of the well-known simple PID feedback control loop in the local field when real networked process control applied to systems with uncertain factors, such as external disturbance or randomly delayed measurements. The proposed PID iterative learning method is implemented by backpropagation neural networks whose weights are updated via minimizing tracking error entropy of closed-loop systems. The convergence in the mean square sense is analysed for closed-loop networked control systems. To demonstrate the potential applications of the proposed strategies, a pressure-tank experiment is provided to show the usefulness and effectiveness of the proposed design method in network process control systems.