Class Covariance Matrices Research Articles

Covariance discriminative power of kernel clustering methods

Let x1,⋯,xn be independent observations of size p, each of them belonging to one of c distinct classes. We assume that observations within the class a are characterized by their distribution N(0,1 pCa) where here C1,⋯,Cc are some non-negative definite p×p matrices. This paper studies the asymptotic behavior of the symmetric matrix Φ˜kl=p(x kTx l)2δ k≠l when p and n grow to infinity with n p→c0. Particularly, we prove that, if the class covariance matrices are sufficiently close in a certain sense, the matrix Φ˜ behaves like a low-rank perturbation of a Wigner matrix, presenting possibly some isolated eigenvalues outside the bulk of the semi-circular law. We carry out a careful analysis of some of the isolated eigenvalues of Φ˜ and their associated eigenvectors and illustrate how these results can help understand spectral clustering methods that use Φ˜ as a kernel matrix.

Linear Pooling of Sample Covariance Matrices

We consider the problem of estimating high-dimensional covariance matrices of $K$-populations or classes in the setting where the sample sizes are comparable to the data dimension. We propose estimating each class covariance matrix as a distinct linear combination of all class sample covariance matrices. This approach is shown to reduce the estimation error when the sample sizes are limited, and the true class covariance matrices share a somewhat similar structure. We develop an effective method for estimating the coefficients in the linear combination that minimize the mean squared error under the general assumption that the samples are drawn from (unspecified) elliptically symmetric distributions possessing finite fourth-order moments. To this end, we utilize the spatial sign covariance matrix, which we show (under rather general conditions) to be an asymptotically unbiased estimator of the normalized covariance matrix as the dimension grows to infinity. We also show how the proposed method can be used in choosing the regularization parameters for multiple target matrices in a single class covariance matrix estimation problem. We assess the proposed method via numerical simulation studies including an application in global minimum variance portfolio optimization using real stock data.

Localization of vectors–patterns in the problems of parametric classification with the purpose of increasing its accuracy

The problem of the accuracy of classification, determined by the share of the correctly classified objects with the use of parametric methods of pattern recognition, is examined. In particular, we analyzed the influence of deviation from the normal law of distribution of variables of the space of features – random variables, in essence – and the inequality of covariance matrices of classes on the accuracy of classification. It was found that the inequality of covariance matrices of the divided classes leads to the shift of the dividing surface; however, the magnitude of this displacement may not become the essential factor, which influences the accuracy of classification. It is shown for the space of variables of dimensionality (N×2) that for the randomly selected classes inside the square with the length of the edge, equal to two, the accuracy of classification for classes A and B proves to be different and depends on the position of the straight line, which divides the classes. It is shown that localization of vectors-patterns in the space of features can be selected in accordance with the plans of full factorial experiment (FFE). Due to this localization, the equality of covariance matrices of the classes is ensured and the share of the correctly classified objects increases. We proposed, to overcoming the main shortcoming of the given research, connected to the absence of functional dependencies, which make it possible to quantitatively evaluate the accuracy of classification at various variants of arrangement of FFE plans, to use the methods of experimental optimization. Its purpose is the selection of such coordinates of the centers of plans and their geometric characteristics for classes A and B, which ensure the maximum share of the correctly classified objects based on the obtained decision rules. The realization of this approach may become the reserve for the increase in the accuracy of classification with application of the methods of parametric optimization.

Detecting single-trial EEG evoked potential using a wavelet domain linear mixed model: application to error potentials classification

Objective. The main goal of this work is to develop a model for multisensor signals, such as magnetoencephalography or electroencephalography (EEG) signals that account for inter-trial variability, suitable for corresponding binary classification problems. An important constraint is that the model be simple enough to handle small size and unbalanced datasets, as often encountered in BCI-type experiments. Approach. The method involves the linear mixed effects statistical model, wavelet transform, and spatial filtering, and aims at the characterization of localized discriminant features in multisensor signals. After discrete wavelet transform and spatial filtering, a projection onto the relevant wavelet and spatial channels subspaces is used for dimension reduction. The projected signals are then decomposed as the sum of a signal of interest (i.e., discriminant) and background noise, using a very simple Gaussian linear mixed model. Main results. Thanks to the simplicity of the model, the corresponding parameter estimation problem is simplified. Robust estimates of class-covariance matrices are obtained from small sample sizes and an effective Bayes plug-in classifier is derived. The approach is applied to the detection of error potentials in multichannel EEG data in a very unbalanced situation (detection of rare events). Classification results prove the relevance of the proposed approach in such a context. Significance. The combination of the linear mixed model, wavelet transform and spatial filtering for EEG classification is, to the best of our knowledge, an original approach, which is proven to be effective. This paper improves upon earlier results on similar problems, and the three main ingredients all play an important role.

Robust Common Spatial Filters with a Maxmin Approach

Electroencephalographic signals are known to be nonstationary and easily affected by artifacts; therefore, their analysis requires methods that can deal with noise. In this work, we present a way to robustify the popular common spatial patterns (CSP) algorithm under a maxmin approach. In contrast to standard CSP that maximizes the variance ratio between two conditions based on a single estimate of the class covariance matrices, we propose to robustly compute spatial filters by maximizing the minimum variance ratio within a prefixed set of covariance matrices called the tolerance set. We show that this kind of maxmin optimization makes CSP robust to outliers and reduces its tendency to overfit. We also present a data-driven approach to construct a tolerance set that captures the variability of the covariance matrices over time and shows its ability to reduce the nonstationarity of the extracted features and significantly improve classification accuracy. We test the spatial filters derived with this approach and compare them to standard CSP and a state-of-the-art method on a real-world brain-computer interface (BCI) data set in which we expect substantial fluctuations caused by environmental differences. Finally we investigate the advantages and limitations of the maxmin approach with simulations.

Probabilistic Fisher discriminant analysis: A robust and flexible alternative to Fisher discriminant analysis

Fisher discriminant analysis (FDA) is a popular and powerful method for dimensionality reduction and classification. Unfortunately, the optimality of the dimension reduction provided by FDA is only proved in the homoscedastic case. In addition, FDA is known to have poor performances in the cases of label noise and sparse labeled data. To overcome these limitations, this work proposes a probabilistic framework for FDA which relaxes the homoscedastic assumption on the class covariance matrices and adds a term to explicitly model the non-discriminative information. This allows the proposed method to be robust to label noise and to be used in the semi-supervised context. Experiments on real-world datasets show that the proposed approach works at least as well as FDA in standard situations and outperforms it in the label noise and sparse label cases.

Toward a tight upper bound for the error probability of the binary Gaussian classification problem

It is well known that the error probability, of the binary Gaussian classification problem with different class covariance matrices, cannot be generally evaluated exactly because of the lack of closed-form expression. This fact pointed out the need to find a tight upper bound for the error probability. This issue has been for more than 50 years ago and is still of interest. All derived upper-bounds are not free of flaws. They might be loose, computationally inefficient particularly in highly dimensional situations, or excessively time consuming if high degree of accuracy is desired. In this paper, a new technique is developed to estimate a tight upper bound for the error probability of the well-known binary Gaussian classification problem with different covariance matrices. The basic idea of the proposed technique is to replace the optimal Bayes decision boundary with suboptimal boundaries which provide an easy-to-calculate upper bound for the error probability. In particular, three types of decision boundaries are investigated: planes, elliptic cylinders, and cones. The new decision boundaries are selected in such a way as to provide the tightest possible upper bound. The proposed technique is found to provide an upper bound, tighter than many of the often used bounds such as the Chernoff bound and the Bayesian-distance bound. In addition, the computation time of the proposed bound is much less than that required by the Monte-Carlo simulation technique. When applied to real world classification problems, obtained from the UCI repository [H. Chernoff, A measure for asymptotic efficiency of a hypothesis based on a sum of observations, Ann. Math. Statist. 23 (1952) 493–507.], the proposed bound was found to provide a tight bound for the analytical error probability of the quadratic discriminant analysis (QDA) classifier and a good approximation to its empirical error probability.

Uncorrelated heteroscedastic LDA based on the weighted pairwise Chernoff criterion

We propose an uncorrelated heteroscedastic LDA (UHLDA) technique, which extends the uncorrelated LDA (ULDA) technique by integrating the weighted pairwise Chernoff criterion. The UHLDA can extract discriminatory information present in both the differences between per class means and the differences between per class covariance matrices. Meanwhile, the extracted feature components are statistically uncorrelated the maximum number of which exceeds the limitation of the ULDA. Experimental results demonstrate the promising performance of our proposed technique compared with the ULDA.

Linear dimensionality reduction via a heteroscedastic extension of LDA: the Chernoff criterion

We propose an eigenvector-based heteroscedastic linear dimension reduction (LDR) technique for multiclass data. The technique is based on a heteroscedastic two-class technique which utilizes the so-called Chernoff criterion, and successfully extends the well-known linear discriminant analysis (LDA). The latter, which is based on the Fisher criterion, is incapable of dealing with heteroscedastic data in a proper way. For the two-class case, the between-class scatter is generalized so to capture differences in (co)variances. It is shown that the classical notion of between-class scatter can be associated with Euclidean distances between class means. From this viewpoint, the between-class scatter is generalized by employing the Chernoff distance measure, leading to our proposed heteroscedastic measure. Finally, using the results from the two-class case, a multiclass extension of the Chernoff criterion is proposed. This criterion combines separation information present in the class mean as well as the class covariance matrices. Extensive experiments and a comparison with similar dimension reduction techniques are presented.

LVQNET 1.10 — a program for neural net and statistical pattern recognition

A neural net program for pattern classification is presented. The neural net architecture is based on an improved version of Kohonen's learning vector organization: learning vector quantization with training count. The number of times a neuron is trained by input patterns of each class is stored in newly introduced training counters. This information, together with other which is collected during training, is used at the end of each training epoch for pruning, merging and creating neurons, and also in the classification process to estimate the reliability of the classification. Initially, neurons are automatically allocated to classes in proportion to the volumes and linear sizes of the corresponding distributions of input patterns in the pattern space, as estimated from the class covariance matrices. As an aside, pattern classification according to Mahalanobis distance and Fisher linear discrimination is also provided

On an Extended Fisher Criterion for Feature Selection

This correspondence considers the extraction of features as a task of linear transformation of an initial pattern space into a new space, optimal with respect to discriminating the data. A solution of the feature extraction problem is given for two multivariate normal distributed pattern classes using an extended Fisher criterion as the distance measure. The introduced distance measure consists of two terms. The first term estimates the distance between classes upon the difference of mean vectors of classes and the second one upon the difference of class covariance matrices. The proposed method is compared to some of the more popular alternative methods: Fukunaga-Koontz method and Foley-Sammon method.