non-Gaussian Situations Research Articles

Minimum Error Entropy Algorithms with Sparsity Penalty Constraints

Recently, sparse adaptive learning algorithms have been developed to exploit system sparsity as well as to mitigate various noise disturbances in many applications. In particular, in sparse channel estimation, the parameter vector with sparsity characteristic can be well estimated from noisy measurements through a sparse adaptive filter. In previous studies, most works use the mean square error (MSE) based cost to develop sparse filters, which is rational under the assumption of Gaussian distributions. However, Gaussian assumption does not always hold in real-world environments. To address this issue, we incorporate in this work an l1-norm or a reweighted l1-norm into the minimum error entropy (MEE) criterion to develop new sparse adaptive filters, which may perform much better than the MSE based methods, especially in heavy-tailed non-Gaussian situations, since the error entropy can capture higher-order statistics of the errors. In addition, a new approximator of l0-norm, based on the correntropy induced metric (CIM), is also used as a sparsity penalty term (SPT). We analyze the mean square convergence of the proposed new sparse adaptive filters. An energy conservation relation is derived and a sufficient condition is obtained, which ensures the mean square convergence. Simulation results confirm the superior performance of the new algorithms.

Particle Filtering-Based System Degradation Prediction Applied to Jet Engines


 
 
 This paper investigates a real-time fault detection and degradation prediction scheme for dynamical systems such as jet engines, based on Regularized Particle Filtering (RPF). Particle Filtering is a prognosis method for the prediction of state degradation and remaining useful life (RUL) due to its demonstrated performance in handling non-linear and non-Gaussian situations. RPF overcomes the problem of sample impoverishment among particles over the resampling process. Based on measured data from hybrid sensing and nonlinear models, which link system parameters and degradation state to the measurement, RPF has been applied to establishing a framework for both state and parameter estimation, to achieve prognosis at the component level. In addition, a modified system evolution model is proposed to track both exponential and transient types of system performance degradation. The developed method is evaluated using simulated data created with C- MAPSS, which contains measured parameters associated with engine degradation under nominal and varied fault types (fan, compressor and turbine) during a series of flights. The developed system-parameter estimation method is found effective in state estimation and degradation prediction in jet engines.
 
 

A Note on the W-S Lower Bound of the MEE Estimation

The minimum error entropy (MEE) estimation is concerned with the estimation of a certain random variable (unknown variable) based on another random variable (observation), so that the entropy of the estimation error is minimized. This estimation method may outperform the well-known minimum mean square error (MMSE) estimation especially for non-Gaussian situations. There is an important performance bound on the MEE estimation, namely the W-S lower bound, which is computed as the conditional entropy of the unknown variable given observation. Though it has been known in the literature for a considerable time, up to now there is little study on this performance bound. In this paper, we reexamine the W-S lower bound. Some basic properties of the W-S lower bound are presented, and the characterization of Gaussian distribution using the W-S lower bound is investigated.

PARTICLE FILTER BASED VEHICLE TRACKING APPROACH WITH IMPROVED RESAMPLING STAGE

ICTACT Journals are peer reviewed International Journals published quarterly. With the objective of creating Indian-based Journals of International quality and to build up local research content, ICTACT Journals exists for the purpose of providing a quality peer-reviewed platform for Scientists, Researchers, Academicians and Engineers who carry out their original research. The Journals will highlight the continued growth and new challenges in both basic research and application development.

A particle filter and joint likelihood ratio based error source diagnosing method for IEEE 802.11 networks

As a result of the fast growing scale of IEEE 802.11 networks, problems such as low signal-to-noise ratio, collision, and small-scale fading have seriously impacted the performance of IEEE 802.11 networks. In this work, we describe a novel cross-layer analysis method, using the combination of received channel power sampling at the physical PHY layer and information at the medium access control MAC layer. The proposed method analyzes the causes of error frames by recording samples of received channel power at the physical layer on a small time scale 5 µs and employs the particle filter-based joint likelihood ratio method in order to detect changes in the received channel power and to isolate models of the changes within the time domain. At the same time, it determines the source and the destination addresses of the error frames by decoding packet physical addresses at the MAC layer and then locates the error source. On the basis of the proposed method, optimizations are possible both at the MAC layer and the PHY layer. The simulation and the experimental validation were both carried out for the proposed method. The simulation validation was carried out in order to validate the accuracy of the particle filter-based joint likelihood ratio method for fault detection and for model isolation using the proposed method. We compared the performance of the extended Kalman filter and the particle filter-based likelihood ratio method using the non-Gaussian situation for the proposed method. We then performed several experiments in order to validate the accuracy of the proposed method for error source diagnosis. We also show the applications of the proposed method. The experiments under actual scene showed that different optimizations can be made to optimize the actual wireless local area network by determining the three different causes of the errors. Copyright © 2013 John Wiley & Sons, Ltd.

On the Smoothed Minimum Error Entropy Criterion

Recent studies suggest that the minimum error entropy (MEE) criterion can outperform the traditional mean square error criterion in supervised machine learning, especially in nonlinear and non-Gaussian situations. In practice, however, one has to estimate the error entropy from the samples since in general the analytical evaluation of error entropy is not possible. By the Parzen windowing approach, the estimated error entropy converges asymptotically to the entropy of the error plus an independent random variable whose probability density function (PDF) corresponds to the kernel function in the Parzen method. This quantity of entropy is called the smoothed error entropy, and the corresponding optimality criterion is named the smoothed MEE (SMEE) criterion. In this paper, we study theoretically the SMEE criterion in supervised machine learning where the learning machine is assumed to be nonparametric and universal. Some basic properties are presented. In particular, we show that when the smoothing factor is very small, the smoothed error entropy equals approximately the true error entropy plus a scaled version of the Fisher information of error. We also investigate how the smoothing factor affects the optimal solution. In some special situations, the optimal solution under the SMEE criterion does not change with increasing smoothing factor. In general cases, when the smoothing factor tends to infinity, minimizing the smoothed error entropy will be approximately equivalent to minimizing error variance, regardless of the conditional PDF and the kernel.

Functional equation helps in finding affine-covariant probability distributions

To select variables which provide the most relevant clustering, researchers use a probability density f that depends on the means E and the covariance matrix C of the data. Under the assumption that the distributions are normal, this density becomes proportional to f(C) = |det(C)|α for some real value α, where det(C) is the determinant of the matrix C. This function is affine-covariant in the sense that the ratios of the two density values do not change under an arbitrary affine transformation. In this paper, we show that only the functions f(C) = |det(C)|α satisfy the functional equation $$\frac{f(E,C)}{f(E',C')}= \frac{f(U^TE+S,U^TCU)}{f(U^TE'+S,U^TC'U)}$$ that describes affine covariance. This result justifies the use of the functions f(C) = |det(C)|α in non-Gaussian situations as well.

Rank-dependent moderate deviations of U-empirical measures in strong topologies

We prove a rank-dependent moderate deviation principle for U-empirical measures, where the underlying i.i.d. random variables take values in a measurable (not necessarily Polish) space (S,𝒮). The result can be formulated on a suitable subset of all signed measures on (S m ,𝒮⊗ m ). We endow this space with a topology, which is stronger than the usual τ-topology. A moderate deviation principle for Banach-space valued U-statistics is obtained as a particular application. The advantage of our result is that we obtain in the degenerate case moderate deviations in non-Gaussian situations with non-convex rate functions.

Efficient estimation of general linear mixed effects models

An extension of the linear growth curve model (Biometrics 38 (1982) 963) was proposed by Stukel and Demidenko (Biometrics 53 (1997) 720) to study the effects of population covariates on one or more characteristics of the curve, when the characteristics are expressed as linear combinations of the growth curve parameters. In the present paper, this general growth curve model receives a comprehensive theoretical treatment. A two-stage estimator, consisting of a generalized least squares estimator under constraints for the population parameters and a moment estimator for the variance parameters, is developed for application in the non-Gaussian error situation. Two likelihood based estimators, global maximum likelihood and second-stage maximum likelihood, are also developed. It is shown that all three estimators are consistent, asymptotically normally distributed, and efficient, and are equivalent when the number of individuals tends to infinity. An expression for the bias in the estimator of the population parameters is derived under second stage model misspecification. We show that if parameters that are not of primary interest are incorrectly specified, bias may occur in parameters that are of interest using the standard growth curve model. The general growth curve model does not require specification of such nuisance parameters and is robust in terms of bias. The general linear growth curve model is used to study the effects of host sex on pancreatic tumor growth in rats.

Matrix CRLB scaling due to measurements of uncertain origin

In many estimation situations, measurements are of uncertain origin. This is best exemplified by the target-tracking situation in which at each scan (of a radar, sonar, or electro-optical sensor), a number of measurements are obtained, and it is not known which, if any, of these is target originated. The source of extraneous measurements can be false alarms-especially in low-SNR situations that force the detector at the end of the signal processing chain to operate with a reduced threshold-or spurious targets. In several earlier papers, the surprising observation was made that the Cramer-Rao lower bound (CRLB) for the estimation of a fixed parameter vector (e.g., initial position and velocity) that characterizes the target motion, for the special case of multidimensional measurements in the presence of additive white Gaussian noise, is simply a multiple of that for the case with no uncertainty. That is, there is a scalar information-reduction factor; this is particularly useful as it allows comparison in terms of a scalar. In this paper, we explore this result to determine how wide the class of such problems is. It turns out to include many non-Gaussian situations. Simulations corroborate the analysis.

Moderate deviations for degenerate U-processes

Sufficient conditions for a rank-dependent moderate deviations principle (MDP) for degenerate U-processes are presented. The MDP for VC classes of functions is obtained under exponential moments of the envelope. Among other techniques, randomization, decoupling inequalities and integrability of Gaussian and Rademacher chaos are used to present new Bernstein-type inequalities for U-processes which are the basis of our proofs of the MDP. We present a complete rank-dependent picture. The advantage of our approach is that we obtain in the degenerate case moderate deviations in non-Gaussian situations.

Some Aspects of Asymptotic Theory with Applications to Time Series Models

The primary purpose of this paper is to review a very few results on some basic elements of large sample theory in a restricted structural framework, as described in detail in the recent book by LeCam and Yang (1990, Asymptotics in Statistics: Some Basic Concepts. New York: Springer), and to illustrate how the asymptotic inference problems associated with a wide variety of time series regression models fit into such a structural framework. The models illustrated include many linear time series models, including cointegrated models and autoregressive models with unit roots that are of wide current interest. The general treatment also includes nonlinear models, including what have become known as ARCH models. The possibility of replacing the density of the error variables of such models by an estimate of it (adaptive estimation) based on the observations is also considered.Under the framework in which the asymptotic problems are treated, only the approximating structure of the likelihood ratios of the observations, together with auxiliary estimates of the parameters, will be required. Such approximating structures are available under quite general assumptions, such as that the Fisher information of the common density of the error variables is finite and nonsingular, and the more specific assumptions, such as Gaussianity, are not required. In addition, the construction and the form of inference procedures will not involve any additional complications in the non-Gaussian situations because the approximating quadratic structure actually will reduce the problems to the situations similar to those involved in the Gaussian cases.

Estimation and choice of neighbors in spatial-interaction models of images

Some aspects of statistical inference for a class of spatial-interaction models for finite images are presented: primarily the simultaneous autoregressive (SAR) models and conditional Markov (CM) models. Each of these models is characterized by a set of neighbors, a set of coefficients, and a noise sequence of specified characteristics. We are concerned with two problems: the estimation of the unknown parameters in both SAR and CM models and the choice of an appropriate model from a class of such competing models. Assuming Gaussian-distributed variables, we discuss maximum likelihood (ML) estimation methods. In general, the ML scheme leads to nonlinear optimization problems. To avoid excessive computation, an iterative scheme is given for SAR models, which gives approximate ML estimates in the Gaussian case and reasonably good estimates in some non-Gaussian situations as well. Likewise, for CM models, an easily computable consistent estimate is given. The asymptotic mean-squared error (mse) of this estimate for a four-neighbor CM model is shown tn be substantially less than the mse of the popular coding estimate. Asymptotically consistent decision rules are given for choosing an appropriate SAR or CM model. The usefulness of the estimation scheme and the decision rule for the choice of neighbors is illustrated by using synthetic patterns. Synthetic patterns obeying known SAR and CM models are generated, and the models corresponding to true and several competing neighbor sets are fitted. The estimation scheme yields estimates close to the parameters of the true models, and the decision rule for the choice of neighbors picks up the true model from the class of competing models.

The Minimum Sum of Absolute Errors Regression: A State of the Art Survey

Summary The minimum sum of absolute errors regression is resistant to outliers and error distributions with long tails. With the development of several efficient algorithms, computer codes and some statistical inference procedures, it is fast becoming a viable and desirable alternative to the popular least squares regression. The objective of this paper is to present a state of the art survey for the minimum sum of absolute errors regression. The least squares regression has dominated the statistical literature for a long time. This dominance and popularity of the least squares regression can be ascribed, at least partially, to the fact that the theory is simple, well developed and documented. The computer packages are also easily available. The least squares regression is optimal and results in the maximum likelihood estimators of the unknown parameters of the model if the errors are independent and follow a normal distribution with mean zero and a common (though unknown) variance o2. The least squares regression is very far from the optimal in many non-Gaussian situations, especially when the errors follow distributions with longer tails. For the regression problems, Huber (1973) stated, 'just a single grossly outlying observation may spoil the least squares estimate, and, moreover, outliers are much harder to spot in the regression than in the simple location case'. The outliers occurring with extreme values of the regressor variables can be especially disruptive. Andrews (1974) noted that even when the errors follow a normal distribution, alternatives to least squares may be required; especially if the form of the model is not exactly known. Further, least squares is not very satisfactory if the quadratic loss function is not a satisfactory measure of the loss. Loss denotes the seriousness of the nonzero prediction error to the investigator, where prediction error is the difference between the predicted and the observed value of the response variable. Meyer & Glauber (1964) stated that for at least certain economic problems absolute error may be a more satisfactory measure of loss than the squared error. The minimum sum of absolute errors regression (or for brevity, absolute errors regression) overcomes the aforementioned drawbacks of the least squares regression and provides an attractive alternative. It is less sensitive than least squares regression to the extreme errors and assumes absolute error loss function. Because of its resistance to

Tests of Linear Hypotheses and l"1 Estimation

statistics of a linear hypothesis in the standard linear model. These test statistics, which correspond to Wald, likelihood ratio, and Lagrange multiplier tests, are shown to have the same limiting chi-square behavior under mild regularity conditions on design and the distribution of errors. The asymptotic theory of the tests is derived for a large class of error distributions; thus in Huber's [10] terminology we investigate the behavior of the likelihood ratio test under non-standard conditions. The asymptotic efficiency of the 11 tests involves a modest sacrifice of power compared to classical tests in cases of strictly Gaussian errors but may yield large efficiency gains in non-Gaussian situations. The Lagrange multiplier test seems particularly attractive from a computational standpoint. We derive the asymptotic distribution of the three alternative 11 test statistics for a simple linear exclusion hypothesis. Extension of these results to hypotheses of the form R,8 = r is a straightforward exercise. When the density of the error distribution is strictly positive at the median, all three test statistics have the same limiting central x2 behavior at the null and noncentral x2 behavior for local alternatives to the null. When the variance of the error distribution is bounded, analogous results are well known for classical forms of the Wald, likelihood ratio, and Lagrange multipler tests based on least-squares methods. See, for example, Silvey [18] and the discussion in Section 4 below.

Conditional probability approach to spatial interaction

The merits of a conditional probability (or Markovian) approach to the analysis of local spatial interaction lie in its intuitive appeal and its ability to cope with non-Gaussian situations. Whilst relationships between plausible spatial-temporal models and Markov fields may sometimes be obtained, this does not remove the need for a purely spatial approach to be available. In addition to promoting the above point of view, some Monte Carlo simulation results, due to Mr. D. H. Green, were presented. Analytical progress on coding efficiency was reported.

Data analysis, computation and mathematics

"Data analysis” instead of “statistics” is a name that allows us to use probability where it is needed and avoid it when we should. Data analysis has to analyze real data. Most real data calls for data investigation, while almost all statistical theory is concerned with data processing. This can be borne, in part because large segments of data investigation are, by themselves, data processing. Summarizing a batch of 20 numbers is a convenient paradigm for more complex aims in data analysis. A particular summary, highly competitive among those known and known about in August 1971, is a hybrid between two moderately complex summaries. Data investigation comes in three stages: exploratory data analysis (no probability), rough confirmatory data analysis (sign test procedures and the like), mustering and borrowing strength (the best of modern robust techniques, and an art of knowing when to stop). Exploratory data analysis can be improved by being made more resistant, either with medians or with fancier summaries. Rough confirmatory data analysis can be improved by facing up to the issues surrounding the choice of what is to be confirmed or disaffirmed. Borrowing strength is imbedded in our classical procedures, though we often forget this. Mustering strength calls for the best in robust summaries we can supply. The sampling behavior of such a summary as the hybrid mentioned above is not going to be learned through the mathematics of certainty, at least as we know it today, especially if we are realistic about the diversity of non-Gaussian situations that are studied. The mathematics of simulation, inevitably involving the mathematically sound “swindles” of Monte Carlo, will be our trust and reliance. I illustrate results for a few summaries, including the hybrid mentioned above. Bayesian techniques are still a problem to the author, mainly because there seems to be no agreement on what their essence is. From my own point of view, some statements of their essence are wholly acceptable and others are equally unacceptable. The use of exogeneous information in analyzing a given body of data is a very different thing (a) depending on sample size and (b) depending on just how the exogeneous information is used. It would be a very fine thing if the questions that practical data analysis has to have answered could be answered by the mathematics of certainty. For my own part, I see no escape, for the next decade or so at least, from a dependence on the mathematics of simulation, in which we should heed von Neumann’s aphorism as much as we can.