Within-class Scatter Matrix Research Articles

Comparison of Matrix Decomposition in Null Space-Based LDA Method

Problems with small sample sizes and high dimensionality are common in pattern recognition. Almost all machine learning algorithms degrade in high-dimensional data, so that singularities in the scatter matrices, the main problem of the Linear Discriminant Analysis (LDA) technique, might result. A null space-based LDA (NLDA) has been conceived to address the singularity issue. NLDA aims to maximize the distance between classes in the null space of the within-class scatter matrix. In the first research, the NLDA method was performed by computing the eigenvalue decomposition and singular value decomposition (SVD). This research led to several new implementations of the NLDA method that use other matrix decompositions. The new implementations include NLDA using Cholesky decomposition and NLDA using QR decomposition. This paper compares the performance of three NLDA methods that use different matrix decompositions, namely, SVD, Cholesky decomposition, and QR decomposition. Two sets of data were used in the experiments that used three different NLDA algorithms. To determine the performance of the NLDA methods, the classification accuracy of the three methods was measured using the confusion matrix. The results show that the NLDA method using SVD has the best performance when compared to the other two methods, achieving a precision of 77.8% accuracy for the Colon dataset and a precision of 98.8% accuracy for the TKI-resistance dataset.

Feature extraction of small underwater sonar targets using neighborhood discriminant analysis

Linear discriminant analysis (LDA) stands as a widely used supervised feature extraction technique that maps data onto a low-dimensional subspace such that the between-class scatter is maximized while within-class scatter is minimized. Despite its utility, LDA faces many challenges, particularly when the within-class scatter matrix becomes singular due to small sample size. Other dimensionality reduction techniques, such as Neighbourhood Component Analysis (NCA), have been proposed as alternatives to LDA. NCA learns a linear transformation that maximizes the likelihood that datapoints of the same class are clustered together in the lower-dimensional space. However, the optimization of NCA relies on a non-convex cost function, making it prone to local minima. To address the challenges faced by both LDA and NCA, we propose a novel dimensionality reduction method named neighborhood discriminant analysis (NDA). Like NCA, NDA learns a linear transformation that aims to cluster datapoints based on class label. However, NDA is framed as an eigendecomposition problem, eliminating the need for non-convex optimization. We demonstrate the new approach on real small target sonar data. [Work funded by the ONR grant numbers N000142112420 and N000142312503, and DoD Navy (NEEC) Grant No. N001742010016.]

Exponential Local Fisher Discriminant Analysis with Sparse Variables Selection: A Novel Fault Diagnosis Scheme for Industry Application

Local Fisher discriminant analysis (LFDA) has been widely applied to dimensionality reduction and fault classification fields. However, it often suffers from small sample size (SSS) problem and incorporates all process variables without emphasizing the key faulty ones, thus leading to degraded fault diagnosis performance and poor model interpretability. To this end, this paper develops the sparse variables selection based exponential local Fisher discriminant analysis (SELFDA) model, which can overcome the two limitations of basic LFDA concurrently. First, the responsible faulty variables are identified automatically through the least absolute shrinkage and selection operator, and the current optimization problem are subsequently recast as an iterative convex optimization problem and solved by the minimization-maximization method. After that, the matrix exponential strategy is implemented on LFDA, it can essentially overcome the SSS problem by ensuring that the within-class scatter matrix is always full-rank, thus more practical in real industrial practices, and the margin between different categories is enlarged due to the distance diffusion mapping, which is benefit for the enhancement of classification accuracy. Finally, the Tennessee Eastman process and a real-world diesel working process are employed to validate the proposed SELFDA method, experimental results prove that the SELFDA framework is more excellent than the other approaches.

Matrix randomized autoencoder

Randomized autoencoder (RAE) has attracted much attention due to its strong capability of representation with fast learning speed. However, the mainstream RAEs are still designed for scalar/vector data, which inevitably destroys the structure information of tensor data. To alleviate this deficiency, a novel convolutions based matrix randomized autoencoder (MRAE) is developed for two-dimensional (2D) data in this paper, including a one-side MRAE (OMRAE) exploiting the row or column information and a double-side MRAE (DMRAE) that simultaneously extracts the row and column information by 2 parallel OMRAEs. To reduce meaningless encoded features, the within-class scatter matrix (WSI) and within-class interaction distance (WID) constraints are added into OMRAE resulting WSI-OMRAE and WID-OMRAE, respectively. To demonstrate the superiority, stacked MRAEs are embedded into hierarchical regularized least squares for one-class classification and comparisons with several state-of-the-art methods are provided. The source code would be available at https://github.com/ML-HDU/MRAE.

Self-Weighted Unsupervised LDA.

As a hot topic in unsupervised learning, clustering methods have been greatly developed. However, the model becomes more and more complex, and the number of parameters becomes more and more with the continuous development of clustering methods. And parameter-tuning in most methods is a laborious work due to its complexity and unpredictability. How to propose a concise and beautiful model in which the parameters can be learned adaptively becomes a very meaningful problem. Aim at tackling this problem, we develop a novel self-weighted unsupervised linear discriminative analysis method, namely SWULDA. The proposed method not only avoids adjusting parameters but also explains the link between k-means and linear discriminant analysis (LDA). To obtain superior structural performance, the idea of minimizing the within-class scatter matrix and maximizing the between-class scatter matrix is embedded in the unsupervised model. Moreover, equipped with the proposed quadratic weighted optimization framework, the parameter can be adaptively learned. The extensive experiments on several datasets are conducted to validate the effectiveness of our method.

A Discriminative Feature Learning Approach With Distinguishable Distance Metrics for Remote Sensing Image Classification and Retrieval

The fast data acquisition rate due to the shorter revisit periods and wider observation coverage of satellites results in large amounts of remote sensing images every day. This brings the challenge of how to accurately search the images with similar visual content as the query image. Content-based image retrieval (CBIR) is a solution to this challenge, its performance heavily depends on the effectiveness of the image representation features and similarity evaluation metrics. Ideal image feature representations have dispersed interclass, compact intraclass distribution. However, the neural networks employed by many CBIR methods are trained with cross entropy loss, which does not directly optimize the metrics that evaluates interclass variance over intraclass variance, hence, their feature representations are suboptimal. Meanwhile, the traditional distance metrics used by many CBIR methods cannot index the similarity of feature representations well in high-dimensional space. For better CBIR performance, we propose a discriminative feature learning approach with distinguishable distance metrics for remote sensing image classification and retrieval. By balancing the diagonal elements and nondiagonal elements of the within-class scatter matrix of deep linear discriminant analysis, our proposed loss function, balanced deep linear discriminant analysis, can better optimize the Rayleigh–Ritz quotient, which measures interclass variance over intraclass variance. In addition, the proposed distance metrics, reciprocal exponential distance (RED), is more capable of maintaining distance contrast in high dimensionality, therefore, it can better index similarity for feature representations in high dimensionality. Both visual interpretations and quantitative metrics of extensive experiments demonstrated the effectiveness of our approach.

Multiparty Data Publishing via Blockchain and Differential Privacy

Data are distributed between different parties. Collecting data from multiple parties for analysis and mining will serve people better. However, it also brings unprecedented privacy threats to the participants. Therefore, safe and reliable data publishing among multiple data owners is an urgent problem to be solved. We mainly study the problem of privacy protection in data publishing. For a centralized scenario, we propose the LDA-DP algorithm. First, the within-class mean vectors and the pooled within-class scatter matrix are perturbed by the Gaussian noise. Second, the optimal projection direction vector with differential privacy is obtained by the Fisher criterion. Finally, the low-dimensional projection data of the original data are obtained. For distributed scenarios, we propose the Mul-LDA-DP algorithm based on a blockchain and differential privacy technology. First, the within-class mean vectors and within-class scatter matrices of local data are perturbed by the Gaussian noise and uploaded to the blockchain network. Second, the projection direction vector is calculated in the blockchain network and returned to the data owner. Finally, the data owner uses the projection direction vector to generate low-dimensional projection data of the original data and upload it to the blockchain network for publishing. Furthermore, in a distributed scenario, we propose a correlated noise generation scheme that uses the additivity of the Gaussian distribution to mitigate the effects of noise and can achieve the same noise level as the centralized scenario. We measure the utility of the published data by the SVM misclassification rate. We conduct comparative experiments with similar algorithms on different real data sets. The experimental results show that the data released by the two algorithms can maintain good utility in SVM classification.

Inverted Sparse Discriminant Preserving Projection for Face Recognition

Image classification and face recognition has been a popular subject matter for the last several decades. Images are usually handled as transformed as vectors which makes their classification a dimensionality reduction task. Some of the well-known algorithms in the area, such as the Sparsity Preserving Projection (SPP), create new theoretical concepts families, while other successfully modify or combine useful properties of the former ones. Compiled algorithms like Sparse Discriminant Preserving Projections (SDPP) employ the properties of the Sparse Representation (SR) as in their objective functions they include a supervised modification of the sparse weight matrix that considers the intra-class relations. By examining the construction of the SDPP algorithm and by providing some arguments on the supervised SR, in this paper we propose a new subspace learning algorithm, called Inverted Sparse Discriminant Preserving Projection (ISDPP). Likewise SDPP, ISDPP integrates supervised SR with the Fisher’s criterion. In contrast to SDPP, ISDPP incorporates a between-class SR with the Fischer’s within-class scatter matrix. A preliminary round of experiments support the initiative and provide an expectation for possible superior performance of the proposed ISDPP that is confirmed in the next round of empirical examinations.

Supervised dimensionality reduction technology of generalized discriminant component analysis and its kernelization forms

Supervised subspace projection technology is a major method for dimensionality reduction in pattern recognition. At present, most supervised subspace projection algorithms are derived from the multi-dimensional extended version of Fisher linear discriminant analysis (FDA), also known as Multi-dimensional Fisher discriminant analysis (MD-FDA). However, MD-FDA needs to be improved further because the projection vectors in the noise-subspace cannot be sorted and the ill-condition of the within-class scatter matrix may cause severe numerical instabilities. Generalized discriminant component analysis (GDCA), the generalization of MD-FDA, together with its kernelization forms are proposed and correspondingly rigorous mathematical proofs are detailed in this paper. By virtue of 5 validation data sets derived from UCI Machine Learning Repository and our laboratory, the theoretical validity and technical advantages of GDCA as well as its kernelization forms are verified, and the effectiveness of the newly proposed method is demonstrated in comparison with 36 kinds of state-of-the-art dimensionality reduction algorithms.

Weighted multi-view common subspace learning method

• A novel multi-view common subspace learning model is proposed. • The model can obtain a unified common subspace by using the discriminant information within and between views. • The discriminant information within and between views can be adjusted by the weighted coefficient. • Our model adopts the maximize scatter difference criterion as the metric between and within the views after projection. How to use multi-view data effectively has become one of the challenging problems in the computer vision community. The existing multi-view learning methods are mainly based on common subspace learning which aim to explore the discriminative information between multi-view data and find its potential common subspace. Most of the existing multi-view subspace learning methods rely on the within-class scatter matrix and between-class scatter matrix while capturing the discriminative information of multiple views. However, these methods just roughly minimize the within-class distance and maximize the between-class distance, and do not make full use of the intra-view and inter-view information. To address this problem, we propose a weighted common subspace learning method, which can effectively adjust the contribution ratio of between-class and within-class information through a weighted parameter, so that an optimized common subspace can be obtained. And we use the maximum scatter difference criterion as the metric of inter-view and intra-view after projection. Extensive experiments on the public data sets show the superiority of this method.

Density-oriented linear discriminant analysis

The conventional Linear Discriminant Analysis (LDA) model has some challenges, such as sensitivity to the outlier, the singularity problem of the within-class scatter matrix, and Gaussian assumption of data within the same class. This paper proposes a robust LDA method that tries to solve the sensitivity to outliers and singularity problems. Specifically, we first use Bayesian risk to design the proposed method optimization problem. Then, the proposed Density-oriented LDA (DLDA) method used the data density as prior knowledge for robustness against outliers. The proposed method can classify non-linear and multi-mode distribution data sets. Furthermore, the proposed method can be employed for big data classification using the AdaBoost approach. Experimental results on synthetic and real data sets demonstrate the proposed DLDA method’s superiority over other competing methods.

Feature Line Embedding Based on Support Vector Machine for Hyperspectral Image Classification

In this paper, a novel feature line embedding (FLE) algorithm based on support vector machine (SVM), referred to as SVMFLE, is proposed for dimension reduction (DR) and for improving the performance of the generative adversarial network (GAN) in hyperspectral image (HSI) classification. The GAN has successfully shown high discriminative capability in many applications. However, owing to the traditional linear-based principal component analysis (PCA) the pre-processing step in the GAN cannot effectively obtain nonlinear information; to overcome this problem, feature line embedding based on support vector machine (SVMFLE) was proposed. The proposed SVMFLE DR scheme is implemented through two stages. In the first scatter matrix calculation stage, FLE within-class scatter matrix, FLE between-scatter matrix, and support vector-based FLE between-class scatter matrix are obtained. Then in the second weight determination stage, the training sample dispersion indices versus the weight of SVM-based FLE between-class matrix are calculated to determine the best weight between-scatter matrices and obtain the final transformation matrix. Since the reduced feature space obtained by the SVMFLE scheme is much more representative and discriminative than that obtained using conventional schemes, the performance of the GAN in HSI classification is higher. The effectiveness of the proposed SVMFLE scheme with GAN or nearest neighbor (NN) classifiers was evaluated by comparing them with state-of-the-art methods and using three benchmark datasets. According to the experimental results, the performance of the proposed SVMFLE scheme with GAN or NN classifiers was higher than that of the state-of-the-art schemes in three performance indices. Accuracies of 96.3%, 89.2%, and 87.0% were obtained for the Salinas, Pavia University, and Indian Pines Site datasets, respectively. Similarly, this scheme with the NN classifier also achieves 89.8%, 86.0%, and 76.2% accuracy rates for these three datasets.

Gallery-sensitive single sample face recognition based on domain adaptation

Taking advantage of labeled auxiliary training data whose distribution is similar to the distribution of the gallery, single sample face recognition (SSFR) has achieved encouraging performance. However, in many real-world applications, it is difficult to collect such an auxiliary training dataset, while it may be easier to collect an unlabeled target training dataset whose distribution is similar to the distribution of the gallery and a labeled source training dataset whose distribution may be different to the distribution of the gallery. How can these three datasets be effectively leveraged to handle SSFR? To address this issue, this paper proposes a new method of Gallery-Sensitive Single Sample Face Recognition based on Domain Adaptation (GS-DA). First, GS-DA employs the method of TSD (targetize the source domain) to construct a common subspace and a targetized source domain. Secondly, it projects each gallery image into the common subspace and obtains the sparse representation of each gallery image in the common subspace. Thirdly, it reconstructs each gallery image from the targetized source domain to estimate the within-class scatter matrix and the between-class scatter matrix of the gallery. Lastly, it learns a discriminant model by maximizing the sum of the traces of the between-class scatter matrix of the gallery and the between-class scatter matrix of the targetized source domain as well as minimizing the sum of the traces of the total scatter matrix of the gallery and the total scatter matrix of the target training data. The experimental results on five datasets illustrate the superiority of GS-DA in leveraging these three datasets for SSFR.

Trace Ratio Optimization for High-Dimensional Multi-Class Discrimination

In multi-class discrimination with high-dimensional data, identifying a lower-dimensional subspace with maximum class separation is crucial. We propose a new optimization criterion for finding such a discriminant subspace, which is the ratio of two traces: the trace of between-class scatter matrix and the trace of within-class scatter matrix. Since this problem is not well-defined for high-dimensional data, we propose to regularize the within trace and maximize the between trace. A careful investigation reveals that this optimization has an innate connection to the eigenvalue decomposition of an indefinite matrix. For the sake of better interpretability of the classifier, we also consider a sparse estimation via a group-wise soft-thresholding. Interesting relationships between the proposed method and some classical methods such as Fisher’s linear discriminant analysis and its variants are discussed. Empirical examples with simulated and real datasets suggest that the proposed method works well and is often better than some existing approaches in a wide range of problems, with respect to both variable selectivity and classification accuracy. Supplementary files for this article are available online.

Minimum class variance multiple kernel learning

The purpose of multiple kernel learning (MKL) is to learn an appropriate kernel from a set of predefined base kernels. Most of the MKL methods follow the basic idea of support vector machine (SVM) to learn the optimal weights of base kernels and build the used classifier. However, SVM is a local method and ignores the structure information of the data in that its solution is exclusively determined by the so-called support vectors. In the paper, we propose an improved SVM-based MKL method called minimum class variance multiple kernel learning (MCVMKL). The key characteristic of MCVMKL is that it exploits the ellipsoidal structure of the data during learning the optimal weights and building the classifier. Besides, its formulation is invariant to scalings of the weights of base kernels. We develop two optimization strategies to handle the optimization model of MCVMKL. Further, we derive a rough upper bound for the objective function of MCVMKL and propose a variant called trace-constrained multiple kernel learning (TCMKL) by using the trace of the within-class scatter matrix. TCMKL enlarges the margin between different classes and simultaneously shrinks the region covering the data as much as possible. Moreover, it can automatically tune the regularization parameter and so saves the training time due to avoiding using the time-consuming cross-validation technique to select an appropriate regularization parameter. Finally, the comprehensive experiments are conducted and the results demonstrate that the proposed methods are effective and can achieve better performance over the competing methods.

Face Recognition Using RLDA Method Based on Mutated Cuckoo Search Algorithm to Extract Optimal Features

Regularized-LDA (R-LDA) is one of the most successful holistic approaches that is introduced to overcome the “small sample size” (SSS) problem of the LDA method, which is often encountered in Face Recognition (FR) tasks. R-LDA is based on reducing the high variance of principal components of the within-class scatter matrix to optimize the regularized Fisher's criterion. In this article, the authors assume that some of these components do not have significant information and they can be discarded. To this end, the authors propose CS-RLDA that uses a Cuckoo search (CS) algorithm to select the optimal eigenvectors from a within-class matrix. However, the CS algorithm has a slow convergence speed. To deal with this problem, and to create more diversity and better trade-off between exploitation and exploration around the best solutions, the authors have modified the basic cuckoo algorithm by using a mutation operator. The experimental results performed on the ORL and UMIST databases indicate that the proposed method enhances the performance of FR.

Locally Joint Sparse Marginal Embedding for Feature Extraction

Classical linear discriminant analysis (LDA) has the limitation that it requires the within-class scatter matrix to be nonsingular so that it can perform eigen-decomposition to obtain optimal solutions. To break through this limitation, many methods based on LDA have been proposed. However, these methods are either sensitive to outliers or lack joint sparsity for effective feature extraction. To release these problems, this paper proposes a locally joint sparse marginal embedding (LJSME) method. LJSME reconstructs the scatter matrices and utilizes the locality graph to weigh each pair of data, such that it is robust to outliers and able to preserve the neighborhood relationship of the data. Moreover, LJSME can easily avoid the small sample-size problem by a maximum margin criterion and obtain joint sparsity for effective feature extraction by using joint sparse regularization. The comprehensive analysis between the proposed LJSME and the related methods is presented, which indicates the advantages of the proposed method. A series of experiments was conducted to evaluate the performance of LJSME when compared with the state-of-the-art methods. The MATLAB code of LJSME can be downloaded from https://github.com/TungmeeMo/LJSME.git .

Modified eigenvector-based feature extraction for hyperspectral image classification using limited samples

Classical supervised feature extraction methods, such as linear discriminant analysis (LDA) and nonparametric weighted feature extraction (NWFE), and search for projection directions through which the ratio of a between-class scatter matrix to a within-class scatter matrix can be maximized. The two feature extraction methods can obtain good classification results when training samples are sufficient; however, the effect is nonideal when samples are insufficient. In this study, the eigenvector spectra of LDA and NWFE are modified using spectral distribution information, which is locally unstable under the condition of a few samples. Experiments demonstrate that the proposed method outperforms several conventional feature extraction methods.

Spectral feature extraction of hyperspectral remote sensing images based on class pair-weighted criterion

Hyperspectral remote sensing images (HRSIs) face the problem of extracting efficient discriminant features when foreign objects with similar spectra and outlier classes exist. For solving the problem, first, we derive a class pair (CP) form of the classical Fisher criterion, namely CP Fisher criterion, where the between-class scatter matrix and within-class scatter matrix are both decomposed into the form of CPs. And then, a CP-weighted criterion is proposed to weight the between-class scatter matrix and within-class scatter matrix of each CP according to its separability so that the separabilities of CPs are balanced in the CP-weighted feature subspace. The Bayesian classifier and k-nearest neighbors classifier are used to evaluate the feature extraction performance. Experimental results on three real HRSIs show that the presented CP-weighted subspace method improves the classification accuracies of the CPs with small separabilities so that it is superior to the linear discriminant analysis subspace method and the weighted pairwise Fisher subspace method.

A robust geometric mean-based subspace discriminant analysis feature extraction approach for image set classification

Discriminant analysis technique is an important research topic in image set classification because it can extract discriminative features. However, most existing discriminant analysis methods almost fail to work for feature extraction of data because there is only a small amount of valid discriminant information. The main weakness of most existing discriminant analysis models is that the class mean vector is constructed by using class sample average. That is not sufficient to provide an accurate estimation of the class mean. In this paper, we proposed a robust geometric mean-based subspace discriminant analysis feature extraction method for image set classification. This method is a combination of geometric mean vector, subspace and covariance matrix to jointly represent an image set because they are contain different discriminative information. The above three representation ways lie on different spaces. To reduce the dissimilarity between the heterogeneous spaces, a robust geometric mean-based subspace discriminant analysis learning (GMSDA) framework is developed, which includes three steps operation. At first, uses a new geometric mean vector to construct geometric between-class scatter matrix (Sgb) and geometric within-class scatter matrix (Sgw) instead of traditional mean vector of many methods. Secondly, maximizes the geometric between-class scatter matrix to increase the difference between extracted features. Finally, maximizes the subspace between-class scatter (SgbS) and minimizes the subspace within-class scatter (SgwS) simultaneously in the subspace of original space. Experiments on five datasets illustrate that our proposed GMSDA method works the best in small sample size situation by using maximum likelihood classification (MLC).