Supervised Dimensionality Reduction Algorithm Research Articles

Chemical space deconstruction-based dynamic model ensemble architecture for molecular property prediction

Green solvents, catalysts, functional materials, drugs, and other chemical products now have a quick engine for design thanks to machine learning (ML) based prediction of molecular properties. However, the accuracy and stability of ML-based models can be impeded by poor data quality, which is rarely studied in chemical product discovery and design. Inspired by the dynamic ensemble selection (DES), an improved DES based on chemical space deconstruction is proposed in this work to accommodate the prediction task of molecular properties. We innovatively developed a chemical space representation and deconstruction model based on a self-organizing mapping (SOM) neural network, facilitating the rapid implementation of the improved DES on molecular samples. Consequently, a novel dynamic model ensemble architecture (SOM-DES) is proposed as a model enhancement technology to build a more accurate and stable ensemble model, aiming to improve the predictive performance on the chemical subspace within poor-quality data. To achieve the architecture, a supervised dimensionality reduction algorithm has been improved to enhance the deep mining of molecular feature information for DES optimization. Additionally, a novel resampling strategy based on the combination of the geometric synthetic minority oversampling technique (G-SMOTE) algorithm and chemical space deconstruction, as a data augmentation technology, has been proposed for mitigating the disadvantage of unbalanced data during DES training. The prediction task for enthalpy of formation of the ideal gas is applied as a case study to demonstrate the superiority of the proposed SOM-DES. The results indicate that the proposed SOM-DES (R2 = 0.9731, RMSE = 55.4639) outperforms the traditional static ensemble strategy (SES, R2 = 0.9552, RMSE = 71.5045) in terms of global chemical spatial precision. More importantly, for chemical subspaces that are difficult to predict due to low data quality, SOM-DES shows a significant reduction in prediction errors compared to SES.

Supervised Dimensionality Reduction of Proportional Data Using Exponential Family Distributions

Most well-known supervised dimensionality reduction algorithms suffer from the curse of dimensionality while handling high-dimensional sparse data due to ill-conditioned second-order statistics matrices. They also do not deal with multi-modal data properly since they construct neighborhood graphs that do not discriminate between multi-modal classes of data and single-modal ones. In this paper, a novel method that mitigates the above problems is proposed. In this method, assuming the data is from two classes, they are projected into the low-dimensional space in the first step which removes sparsity from the data and reduces the time complexity of any operation drastically afterwards. These projected data are modeled using a mixture of exponential family distributions for each class, allowing the modeling of multi-modal data. A measure for the similarity between the two projected classes is used as an objective function for constructing an optimization problem, which is then solved using a heuristic search algorithm to find the best separating projection. The conducted experiments show that the proposed method outperforms the rest of the compared algorithms and provides a robust effective solution to the problem of dimensionality reduction even in the presence of multi-modal and sparse data.

EEG emotion recognition based on enhanced SPD matrix and manifold dimensionality reduction

Recently, Riemannian geometry-based pattern recognition has been widely employed to brain computer interface (BCI) researches, providing new idea for emotion recognition based on electroencephalogram (EEG) signals. Although the symmetric positive definite (SPD) matrix manifold constructed from the traditional covariance matrix contains large amount of spatial information, these methods do not perform well to classify and recognize emotions, and the high dimensionality problem still unsolved. Therefore, this paper proposes a new strategy for EEG emotion recognition utilizing Riemannian geometry with the aim of achieving better classification performance. The emotional EEG signals of 32 healthy subjects were from an open-source dataset (DEAP). The wavelet packets were first applied to extract the time-frequency features of the EEG signals, and then the features were used to construct the enhanced SPD matrix. A supervised dimensionality reduction algorithm was then designed on the Riemannian manifold to reduce the high dimensionality of the SPD matrices, gather samples of the same labels together, and separate samples of different labels as much as possible. Finally, the samples were mapped to the tangent space, and the K-nearest neighbors (KNN), Random Forest (RF) and Support Vector Machine (SVM) method were employed for classification. The proposed method achieved an average accuracy of 91.86%, 91.84% on the valence and arousal recognition tasks. Furthermore, we also obtained the superior accuracy of 86.71% on the four-class recognition task, demonstrated the superiority over state-of-the-art emotion recognition methods.

Self-centralized jointly sparse maximum margin criterion for robust dimensionality reduction

Linear discriminant analysis (LDA) is among the most popular supervised dimensionality reduction algorithms, which has been largely followed in the fields of pattern recognition and data mining. However, LDA has three major drawbacks. One is the challenge brought by small-sample-size (SSS) problem; second makes it sensitive to outliers due to the use of squared L2-norms in the scatter loss evaluation; the third is the case that the feature loadings in projection matrix are relatively redundant and there is a risk of overfitting. In this paper, we put forward a novel functional expression for LDA, which combines maximum margin criterion (MMC) with a weighted strategy formulated by L1,2-norms to against outliers. Meanwhile, we simultaneously realize the adaptive calculation of weighted intra-class and global centroid to further reduce the influence of outliers, and employ the L2,1-norm to constrain row sparsity so that subspace learning and feature selection could be performed cooperatively. Besides, an effective alternating iterative algorithm is derived and its convergence is verified. From the complexity analysis, our proposed algorithm can deal with large-scale data processing. Our proposed model can address the sensitivity problem of outliers and extract the most representative features while preventing overfitting effectively. Experiments performed on several benchmark databases demonstrate that the proposed algorithm is more effective than some other state-of-the-art methods and has better generalization performance.

Gaussian discriminative component analysis for early detection of Alzheimer’s disease: A supervised dimensionality reduction algorithm

BackgroundUsing multiple modalities of biomarkers, several machine leaning-based approaches have been proposed to characterize patterns of structural, functional and metabolic differences discernible from multimodal neuroimaging data for Alzheimer’s disease (AD). Current investigations report several studies using binary classification often augmented with local feature selection methods, while fewer other studies address the challenging problem of multiclass classification. New methodTo assess the merits of each of these research directions, this study introduces a supervised Gaussian discriminative component analysis (GDCA) algorithm, which can effectively delineate subtle changes of early mild cognitive impairment (EMCI) group in relation to the cognitively normal control (CN) group. Using 251 CN, 297 EMCI, 196 late MCI (LMCI), and 162 AD subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) and considering both structural and functional (metabolic) information from magnetic resonance imaging (MRI) and positron emission tomography (PET) modalities as input, the proposed method conducts a dimensionality reduction algorithm taking into consideration the interclass information to define an optimal eigenspace that maximizes the discriminability of selected eigenvectors. ResultsThe proposed algorithm achieves an accuracy of 79.25 % for delineating EMCI from CN using 38.97 % of Gaussian discriminative components (i.e., dimensionality reduction). Moreover, for detecting the different stages of AD, a multiclass classification experiment attained an overall accuracy of 67.69 %, and more notably, discriminates MCI and AD groups from the CN group with an accuracy of 75.28 % using 48.90 % of the Gaussian discriminative components. Comparison with existing method(s)The classification results of the proposed GDCA method outperform the more recently published state-of-the-art methods in AD-related multiclass classification tasks, and seems to be the most stable and reliable in terms of relating the most relevant features to the optimal classification performance. ConclusionThe proposed GDCA model with its high prospects for multiclass classification has a high potential for deployment as a computer aided clinical diagnosis system for AD.

Semi-supervised partial least squares

Traditional supervised dimensionality reduction methods can establish a better model often under the premise of a large number of samples. However, in real-world applications where labeled data are scarce, traditional methods tend to perform poorly because of overfitting. In such cases, unlabeled samples could be useful in improving the performance. In this paper, we propose a semi-supervised dimensionality reduction method by using partial least squares (PLS) which we call semi-supervised partial least squares (S2PLS). To combine the labeled and unlabeled samples into a S2PLS model, we first apply the PLS algorithm to unsupervised dimensionality reduction. Then, the final S2PLS model is established by ensembling the supervised PLS model and the unsupervised PLS model which using the basic idea of principal model analysis (PMA) method. Compared with unsupervised or supervised dimensionality reduction algorithms, S2PLS not only can improve the prediction accuracy of the samples but also enhance the generalization ability of the model. Meanwhile, it can obtain better results even there are only a few or no labeled samples. Experimental results on five UCI data sets also confirmed the above properties of S2PLS algorithm.

Gaussian Process Graph-Based Discriminant Analysis for Hyperspectral Images Classification

Dimensionality Reduction (DR) models are highly useful for tackling Hyperspectral Images (HSIs) classification tasks. They mainly address two issues: the curse of dimensionality with respect to spectral features, and the limited number of labeled training samples. Among these DR techniques, the Graph-Embedding Discriminant Analysis (GEDA) framework has demonstrated its effectiveness for HSIs feature extraction. However, most of the existing GEDA-based DR methods largely rely on manually tuning the parameters so as to obtain the optimal model, which proves to be troublesome and inefficient. Motivated by the nonparametric Gaussian Process (GP) model, we propose a novel supervised DR algorithm, namely Gaussian Process Graph-based Discriminate Analysis (GPGDA). Our algorithm takes full advantage of the covariance matrix in GP to constructing the graph similarity matrix in GEDA framework. In this way, more superior performance can be provided with the model parameters tuned automatically. Experiments on three real HSIs datasets demonstrate that the proposed GPGDA outperforms some classic and state-of-the-art DR methods.

Harmonic Mean Linear Discriminant Analysis

In machine learning and data mining, dimensionality reduction is one of the main tasks. Linear Discriminant Analysis (LDA) is a widely used supervised dimensionality reduction algorithm and it has attracted a lot of research interests. Classical Linear Discriminant Analysis finds a subspace to minimize within-class distance and maximize between-class distance, where between-class distance is computed using arithmetic mean of all between-class distances. However, arithmetic mean between-class distance has some limitations. First, arithmetic mean gives equal weight to all between-class distances, and large between-class distance could dominate the result. Second, it does not consider pairwise between-class distance and thus some classes may overlap with each other in the subspace. In this paper, we propose two formulations of harmonic mean based Linear Discriminant Analysis: HLDA and HLDAp, to demonstrate the benefit of harmonic mean between-class distance and overcome the limitations of classical LDA. We compare our algorithm with 11 existing single-label algorithms on seven datasets and five existing multi-label algorithms on two datasets. On some single-label experiment data, the classification accuracy absolute percentage increase can reach 39 percent compared to state-of-art existing algorithms; on multi-label data, significant improvement on five evaluation metric has been achieved compared to existing algorithms.

Nonlinear supervised dimensionality reduction via smooth regular embeddings

The recovery of the intrinsic geometric structures of data collections is an important problem in data analysis. Supervised extensions of several manifold learning approaches have been proposed in the recent years. Meanwhile, existing methods primarily focus on the embedding of the training data, and the generalization of the embedding to initially unseen test data is rather ignored. In this work, we build on recent theoretical results on the generalization performance of supervised manifold learning algorithms. Motivated by these performance bounds, we propose a supervised manifold learning method that computes a nonlinear embedding while constructing a smooth and regular interpolation function that extends the embedding to the whole data space in order to achieve satisfactory generalization. The embedding and the interpolator are jointly learnt such that the Lipschitz regularity of the interpolator is imposed while ensuring the separation between different classes. Experimental results on several image data sets show that the proposed method outperforms traditional classifiers and the supervised dimensionality reduction algorithms in comparison in terms of classification accuracy in most settings.

Discriminant sparse and collaborative preserving embedding for bearing fault diagnosis

Abstract Background noise and small sample size would usually add to the difficulty of extracting most effective information for bearing fault diagnosis in rotating electrical machines. To address this issue, a novel supervised dimensionality reduction algorithm referred to as discriminant sparse and collaborative preserving embedding (DSCPE) is proposed in this paper for bearing defect classification, which utilizes collaborative representation (CR) for an intrinsic graph and sparse representation (SR) for a penalty graph. In the intrinsic graph, CR helps to involve more bases of the dictionary composed by the same labeled samples and generate less sparse solutions; meanwhile in the penalty graph, SR would avoid the mix-class interference when using the dictionary constituted by different labeled samples. DSCPE aims to seek the optimal projection directions that could minimize the intraclass compactness and maximize the interclass separability. After dimension reduction and data projection by DSCPE, the 1-Nearest Neighbor method is applied to classify the bearing defects. The experimental results demonstrate that DSCPE possesses more effective and robust classification performance than other compared algorithms when dealing with small-sample-sized problem and interfered vibration signals.

Spectral regression based marginal Fisher analysis dimensionality reduction algorithm

Abstract Traditional nonlinear dimensionality reduction methods, such as multiple kernel dimensionality reduction and nonlinear spectral regression (SR), are generally regarded as extended versions of linear discriminant analysis (LDA) in the supervised case. As is well known, LDA has the restrictive assumption that the data of each class is of a Gaussian distribution. Thus, the performance of these methods will be degraded if such an assumption is not hold. Although some methods based on marginal Fisher analysis are proposed to overcome the drawback of LDA, they have to solve the problem of dense metrics generalized eigenvalue decomposition, which is very time-consuming. To address these issues, in this paper, marginal Fisher analysis criterion based on extreme learning machine (ELM) is proposed to improve spectral regression and kernel marginal Fisher analysis. It is proved that the proposed marginal Fisher analysis is a special case of traditional kernel marginal Fisher analysis. Based on the proposed criterion, a novel supervised dimensionality reduction algorithm is presented by virtue of ELM and spectral regression. Experimental results on benchmark datasets validate that the proposed algorithm outperforms the state-of-the-art nonlinear dimensionality reduction methods in supervised scenarios.

Graph embedding and extensions: a general framework for dimensionality reduction.

Over the past few decades, a large family of algorithms - supervised or unsupervised; stemming from statistics or geometry theory - has been designed to provide different solutions to the problem of dimensionality reduction. Despite the different motivations of these algorithms, we present in this paper a general formulation known as graph embedding to unify them within a common framework. In graph embedding, each algorithm can be considered as the direct graph embedding or its linear/kernel/tensor extension of a specific intrinsic graph that describes certain desired statistical or geometric properties of a data set, with constraints from scale normalization or a penalty graph that characterizes a statistical or geometric property that should be avoided. Furthermore, the graph embedding framework can be used as a general platform for developing new dimensionality reduction algorithms. By utilizing this framework as a tool, we propose a new supervised dimensionality reduction algorithm called Marginal Fisher Analysis in which the intrinsic graph characterizes the intraclass compactness and connects each data point with its neighboring points of the same class, while the penalty graph connects the marginal points and characterizes the interclass separability. We show that MFA effectively overcomes the limitations of the traditional Linear Discriminant Analysis algorithm due to data distribution assumptions and available projection directions. Real face recognition experiments show the superiority of our proposed MFA in comparison to LDA, also for corresponding kernel and tensor extensions.

A supervised dimensionality reduction method-based sparse representation for face recognition

The sparse representation-based classification (SRC) method is a powerful tool to present high-dimensionality data and its superiority in many fields, especially in face recognition application has been proved. With sparsity appropriately harnessed, the SRC can solve face classification problems caused by varying expression, illumination as well as occlusion and disguise. However, face images as high-dimensionality data are usually noisy and the dimensionality is always larger than the number of training sample in real-world applications, which bring a disadvantage for the performance of SRC. Therefore, it is beneficial to perform dimensionality reduction (DR) before utilizing the SRC method. But most prevalent DR methods have no direct connection to SRC. In this paper, we proposed a supervised DR algorithm which suits SRC well and improves the discriminating ability in the low-dimensionality space. The proposed method utilizes the fisher discriminant criterion and low-dimensionality reconstructive restriction to extract the discriminating structure of data. The extensive experiments on public face databases verified the effectiveness of the supervised DR with the model of sparse representation.

Incremental maximum margin criterion based on eigenvalue decomposition updating algorithm

Dimensionality reduction has been proven to be a critical data processing step for face recognition. Maximum margin criterion (MMC) is one of the popular supervised dimensionality reduction algorithms. However, the original implementation of MMC is not suitable for incremental learning problem. In this paper, we first propose an eigenvalue decomposition updating algorithm (EVDU) for symmetric matrix. Then, based on our proposed EVDU technique, we propose an incremental MMC (EVDU-IMMC) method which can update the discriminant vectors of MMC when new samples are inserted into the training set. Experiments on ORL and PIE face databases show that the proposed EVDU-IMMC gives the same performance as the batch MMC with much lower computational complexity. The experimental results also show that our proposed EVDU-IMMC gives better performance than other IMMC method in terms of recognition accuracy and computational efficiency.

Sparse discriminant learning with ℓ1-graph for hyperspectral remote-sensing image classification

Graph-embedding (GE) algorithms have been widely used for dimensionality reduction (DR) of hyperspectral imagery (HSI), and k-nearest neighbour and ϵ-radius ball are usually used for graph construction in GE. However, the two approaches are sensitive to data noise and the optimum of k (or ϵ) is datum-dependent. In this paper, we propose a new supervised DR algorithm, called sparse discriminant learning (SDL), based on -graph for HSI classification. It constructs an inter- and an intra-manifold weight matrix that are computed from -graph, which is robust to data noise and the number of neighbours is adaptively selected to each sample. Then, the SDL algorithm seeks optimal projections with inter- and intra-manifold scatter, which can be formulated based on the modified sparse reconstruction weights. SDL not only reserves sparse reconstructive relations through -graph, but also enhances inter-manifold separability. Experiments on synthetic data and two real hyperspectral image data sets collected by AVIRIS and HDYICE sensors are performed to demonstrate the effectiveness of the SDL algorithm.

Local and Global Geometric Structure Preserving and Application to Hyperspectral Image Classification

Locality Preserving Projection (LPP) has shown great efficiency in feature extraction. LPP captures the locality by theK-nearest neighborhoods. However, recent progress has demonstrated the importance of global geometric structure in discriminant analysis. Thus, both the locality and global geometric structure are critical for dimension reduction. In this paper, a novel linear supervised dimensionality reduction algorithm, calledLocality and Global Geometric Structure Preserving(LGGSP) projection, is proposed for dimension reduction. LGGSP encodes not only the local structure information into the optimal objective functions, but also the global structure information. To be specific, two adjacent matrices, that is, similarity matrix and variance matrix, are constructed to detect the local intrinsic structure. Besides, a margin matrix is defined to capture the global structure of different classes. Finally, the three matrices are integrated into the framework of graph embedding for optimal solution. The proposed scheme is illustrated using both simulated data points and the well-known Indian Pines hyperspectral data set, and the experimental results are promising.

Multi-label classification of chronically ill patients with bag of words and supervised dimensionality reduction algorithms

ObjectiveThis research is motivated by the issue of classifying illnesses of chronically ill patients for decision support in clinical settings. Our main objective is to propose multi-label classification of multivariate time series contained in medical records of chronically ill patients, by means of quantization methods, such as bag of words (BoW), and multi-label classification algorithms. Our second objective is to compare supervised dimensionality reduction techniques to state-of-the-art multi-label classification algorithms. The hypothesis is that kernel methods and locality preserving projections make such algorithms good candidates to study multi-label medical time series. MethodsWe combine BoW and supervised dimensionality reduction algorithms to perform multi-label classification on health records of chronically ill patients. The considered algorithms are compared with state-of-the-art multi-label classifiers in two real world datasets. Portavita dataset contains 525 diabetes type 2 (DT2) patients, with co-morbidities of DT2 such as hypertension, dyslipidemia, and microvascular or macrovascular issues. MIMIC II dataset contains 2635 patients affected by thyroid disease, diabetes mellitus, lipoid metabolism disease, fluid electrolyte disease, hypertensive disease, thrombosis, hypotension, chronic obstructive pulmonary disease (COPD), liver disease and kidney disease. The algorithms are evaluated using multi-label evaluation metrics such as hamming loss, one error, coverage, ranking loss, and average precision. ResultsNon-linear dimensionality reduction approaches behave well on medical time series quantized using the BoW algorithm, with results comparable to state-of-the-art multi-label classification algorithms. Chaining the projected features has a positive impact on the performance of the algorithm with respect to pure binary relevance approaches. ConclusionsThe evaluation highlights the feasibility of representing medical health records using the BoW for multi-label classification tasks. The study also highlights that dimensionality reduction algorithms based on kernel methods, locality preserving projections or both are good candidates to deal with multi-label classification tasks in medical time series with many missing values and high label density.

Discriminant similarity and variance preserving projection for feature extraction

In this paper, a novel supervised dimensionality reduction algorithm called discriminant similarity and variance preserving projection (DSVPP) is presented for feature extraction and recognition. More specifically, we redefine the intrinsic graph and penalty graph to model the intra-class compactness and inter-class separability of data points, where the intrinsic graph characterizes the similarity information of the same-class points and the penalty graph characterizes the variance information of the not-same-class points. Using the two graphs, the within-class scatter and the between-class scatter are computed, and then a concise feature extraction criterion is raised via minimizing the difference between them. Experimental results on the Wine data set, ORL, FERET and AR face databases show the effectiveness of the proposed method.

Discriminant embedding by sparse representation and nonparametric discriminant analysis for face recognition

A novel supervised dimensionality reduction algorithm, named discriminant embedding by sparse representation and nonparametric discriminant analysis (DESN), was proposed for face recognition. Within the framework of DESN, the sparse local scatter and multi-class nonparametric between-class scatter were exploited for within-class compactness and between-class separability description, respectively. These descriptions, inspired by sparse representation theory and nonparametric technique, are more discriminative in dealing with complex-distributed data. Furthermore, DESN seeks for the optimal projection matrix by simultaneously maximizing the nonparametric between-class scatter and minimizing the sparse local scatter. The use of Fisher discriminant analysis further boosts the discriminating power of DESN. The proposed DESN was applied to data visualization and face recognition tasks, and was tested extensively on the Wine, ORL, Yale and Extended Yale B databases. Experimental results show that DESN is helpful to visualize the structure of high-dimensional data sets, and the average face recognition rate of DESN is about 9.4%, higher than that of other algorithms.

A novel supervised dimensionality reduction algorithm: Graph-based Fisher analysis

In this paper, a novel supervised dimensionality reduction (DR) algorithm called graph- based Fisher analysis (GbFA) is proposed. More specifically, we redefine the intrinsic and penalty graph and trade off the importance degrees of the same-class points to the intrinsic graph and the importance degrees of the not-same-class points to the penalty graph by a strictly monotone decreasing function; then the novel feature extraction criterion based on the intrinsic and penalty graph is applied. For the non-linearly separable problems, we study the kernel extensions of GbFA with respect to positive definite kernels and indefinite kernels, respectively. In addition, experiments are provided for analyzing and illustrating our results.