Semi-supervised Discriminant Analysis Research Articles

Sparse feature selection using hypergraph Laplacian-based semi-supervised discriminant analysis

Feature selection, as a dimension reduction technique in data mining and pattern recognition, aims to select the most discriminative features and improve the learning performance. With an abundance of unlabeled data readily available across various applications, semi-supervised feature selection has emerged as a promising approach. While most semi-supervised feature selection methods rely on simple graphs to preserve the geometrical structure of data, this approach often fails in capturing the high-order relationships present in many real-world applications. In contrast, hypergraphs offer the ability to encode more complex structures of data beyond what a simple graph can achieve. In this paper, we propose a feature selection method formulated in the trace ratio form, integrating hypergraph Laplacian-based semi-supervised discriminant analysis (SDA) and the mixed convex and non-convex ℓ2,p-norm (0<p≤1) regularization. The proposed trace ratio-based method, called HSDAFS, leverages the discriminative information from labeled data to maximize class separability while also utilizing the hypergraph Laplacian to capture the geometrical structure and high-order relationships within both labeled and unlabeled data. The ℓ2,p-norm regularization in the proposed HSDAFS provides improved sparsity over the ℓ2,1-norm. It ensures that the projection matrix is row-sparse, enabling the effective joint selection of discriminative features across all data. To solve the trace ratio-based HSDAFS method, we convert it into a trace difference method and propose an iterative algorithm. Experiments on several datasets demonstrate that HSDAFS is more effective in selecting the most discriminative features compared to other methods.

ISL-GKFDA: An incomplete supervision manifold learning framework based on propagation graph of data

To improve manifold classification results using unlabeled data with high dimensional redundancy, we propose an incomplete supervision manifold learning framework. In this framework, Euclidean distance and Euclidean neighborhood are replaced by manifold distance and manifold neighborhood respectively. The pseudo label of unlabeled data is generated by constructing label propagation network in manifold neighborhood. By adding the local manifold neighborhood structure into KFDA (Kernel Fisherś Discriminant Analysis) as regularization term, the local manifold structure is stressed. Subsequently, we propose the ISL-GKFDA (Incomplete Supervision Learning (ISL) with propagation Graph of data in Kernel Fisher Discriminant Analysis) algorithm. We validate ISL-GKFDA by comparing it with FDA (Fisher's Discrimination Analysis), KFDA (Kernel FDA), ISL-FDA (ISL version FDA), ISL-KFDA (ISL version KFDA), SSDA (semi-supervised discriminant analysis based on manifold distance), and KSSDA (Kernel SSDA). In typical manifold datasets, ISL-GKFDA can more accurately capture the boundaries of binary classification; and visually in handwritten digital dataset, ISL-GKFDA can separate three data types more clearly. Finally, we use the UCI database and compare multiple algorithms. When measured by accuracy, the proposed ISL-GKFDA reaches a maximum in at least three datasets of five when labeling ratio is 5%, 15% and 30%, respectively. When measured by F1, ISL-GKFDA reaches the maximum in all datasets.

Stockwell transform and semi-supervised feature selection from deep features for classification of BCI signals

Over the past few years, the processing of motor imagery (MI) electroencephalography (EEG) signals has been attracted for developing brain-computer interface (BCI) applications, since feature extraction and classification of these signals are extremely difficult due to the inherent complexity and tendency to artifact properties of them. The BCI systems can provide a direct interaction pathway/channel between the brain and a peripheral device, hence the MI EEG-based BCI systems seem crucial to control external devices for patients suffering from motor disabilities. The current study presents a semi-supervised model based on three-stage feature extraction and machine learning algorithms for MI EEG signal classification in order to improve the classification accuracy with smaller number of deep features for distinguishing right- and left-hand MI tasks. Stockwell transform is employed at the first phase of the proposed feature extraction method to generate two-dimensional time–frequency maps (TFMs) from one-dimensional EEG signals. Next, the convolutional neural network (CNN) is applied to find deep feature sets from TFMs. Then, the semi-supervised discriminant analysis (SDA) is utilized to minimize the number of descriptors. Finally, the performance of five classifiers, including support vector machine, discriminant analysis, k-nearest neighbor, decision tree, random forest, and the fusion of them are compared. The hyperparameters of SDA and mentioned classifiers are optimized by Bayesian optimization to maximize the accuracy. The presented model is validated using BCI competition II dataset III and BCI competition IV dataset 2b. The performance metrics of the proposed method indicate its efficiency for classifying MI EEG signals.

Fast algorithms for incremental and decremental semi-supervised discriminant analysis

Incremental and decremental problems are challenging tasks in semi-supervised learning. The incremental semi-supervised discriminant analysis (ISSDA) method proposed by Dhamecha et al. is an efficient method for incremental semi-supervised learning. However, one deficiency of the ISSDA method is that the total scatter matrix remains unchanged during incremental learning, which is impractical in practice. On the other hand, there may be a series of incorrectly artificial labeling in the public data set, and it is interesting to consider the decremental problem in semi-supervised learning. To the best of our knowledge, however, there are few decremental algorithms for semi-supervised discriminant analysis. The contributions of this work are as follows. First, a new incremental semi-supervised discriminant analysis method is proposed, in which we consider updating the total scatter matrix and the between-class scatter matrix simultaneously when new samples are added. Second, we show how to solve the large eigenproblem of the updated total scatter matrix efficiently. Third, we propose two decremental algorithms for semi-supervised discriminant analysis. Numerical experiments demonstrate the superiority of the proposed algorithms over many state-of-the-art algorithms for semi-supervised discriminant analysis.

Semi-supervised Heterogeneous Defect Prediction with Open-source Projects on GitHub

The heterogeneous defect prediction (HDP) technique can predict defects in a target company using heterogeneous metric data from external company, which has received substantial research attention. However, existing HDP methods assume that source data is labeled but labeling data is expensive. Semi-supervised defect prediction technique can perform defect prediction with few labeled data. In this paper, we investigate a new problem — semi-supervised HDP (SHDP). To solve this problem, we propose a new approach named cost-sensitive kernel semi-supervised correlation analysis (CKSCA) as a solution of SHDP problem. It introduces unified metric representation and canonical correlation analysis to make the data distributions of different company projects more similar. CKSCA also designs a cost-sensitive kernel semi-supervised discriminant analysis mechanism to utilize the limited labeled data and sufficient real-life unlabeled data from different companies. Besides we collect lots of open-source projects from GitHub website to construct a new large-scale unlabeled dataset called GITHUB dataset. It contains 26,407 modules and is greater than each public project dataset. It has been public online and can be extended continuously. Experiments on the GITHUB dataset and other public datasets indicate that unlabeled GITHUB data can help prediction model improve prediction performance, and CKSCA is effective and efficient for solving SHDP problem.

Semi-supervised orthogonal discriminant analysis with relative distance : integration with a MOO approach

In discriminant analysis, trace ratio is an important criterion for minimizing the between-class similarity and maximizing the within-class similarity, simultaneously. In brief, we address the trace ratio problem associated with many semi-supervised discriminant analysis algorithms as they use the normal Euclidean distances between training data samples. Based on this problem, we propose a new semi-supervised orthogonal discriminant analysis technique with relative distance constraints called SSODARD. Different from the existing semi-supervised dimensionality reduction algorithms, our algorithm is more consistent in propagating the label information from the labeled data to the unlabeled data because of the use of relative distance function instead of normal Euclidean distance function. For finding this appropriate relative distance function, we use pairwise constraints generated from labeled data and satisfy them using Bregman projection. Since the projection is not orthogonal, we require an appropriate subset of constraints. In order to select such a subset of constraints, we further develop a framework called MO-SSODARD, which uses evolutionary algorithm while optimizing various validity indices simultaneously. The experimental results on various datasets show that our proposed approaches are superior than the state-of-the-art discriminant algorithms with respect to various validity indices.

Fast semi-supervised discriminant analysis for binary classification of large data sets

High-dimensional data requires scalable algorithms. We propose and analyze four scalable and related algorithms for semi-supervised discriminant analysis (SDA). These methods are based on Krylov subspace methods and therefore exploit data sparsity and the shift-invariance of Krylov subspaces. In addition, centralization was derived for the semi-supervised setting. The proposed methods are evaluated on an industry-scale data set from a pharmaceutical company to predict compound activity on target proteins. The results show that our methods only require a few seconds, significantly improving computation time on the state of the art.

Cluster-Based Tensorial Semisupervised Discriminant Analysis for Feature Extraction of SAR Images

Several features have been developed to characterize the land cover in synthetic aperture radar (SAR) data with speckle noise. Feature extraction has become an essential task for SAR image processing. However, how to preserve the original intrinsic structural information and enhance the discriminant ability to reduce the impact of noise is still a challenge in this area. In this paper, using a clustering method to maintain the nonlocal information in images and tensors with the ability to preserve spatial neighborhood structure information, a new cluster-based tensorial semisupervised discriminant analysis (CTSDA) method is proposed for feature extraction of SAR images. In the CTSDA, the block clustering algorithm is employed to generate several high-order clustering tensors of multifeature SAR images, which preserves the intrinsic nonlocal spatial information and neighborhood structure. In the multiple manifold structures of the cluster tensors, the improved discriminant analysis enhances the feature discrimination by considering the local structure and labeled and unlabeled information through the Laplace matrix, and the fusion of tensor algebraic analysis and improved discriminant analysis produces multiple new projection directions of the cluster tensors. Finally, feature extraction is achieved by rearranging the projected cluster tensors. The experimental results on the simulated SAR data and four real SAR images demonstrate the superiority of the proposed method over several state-of-the-art approaches.

Using Semi-supervised Discriminant Analysis to Predict Subcellular Localization of Gram-negative Bacterial Proteins

In this paper, an effective dimension reduction approach called semi-supervised discriminant analysis (SDA) is employed to deal with the protein subcellular localization problem. Firstly, a novel protein sequence encoding method that consists of pseudo amino acid composition (PseAAC) and dipeptide composition (DC) is introduced to represent a protein. Secondly, the SDA algorithm is applied to extract the essential discriminant features from the combined feature data set consisting of PseAAC and DC. Finally, the K-nearest neighbor (K-NN) classifier is used to identify the subcellular localization of Gram-positive bacterial proteins. The proposed method can effective utilize both manifold information and the class information of the protein samples to guide the produce of protein subcellular localization. To evaluate the prediction performance of the proposed algorithm, a jackknife test based on nearest neighbor algorithm is employed on the gram-negative bacterial proteins data set. The results show that we can get a high total accuracy in a low-dimensional feature space, which indicates that the proposed approach is effective and practical.

An iterative paradigm of joint feature extraction and labeling for semi-supervised discriminant analysis

Semi-supervised discriminant analysis (SDA) has been attracting much attention as it utilizes both of labeled and unlabeled data simultaneously. Current SDA method estimates the labels of unlabeled data just in the original data space. However, in real application, the original features are often contaminated and those original features contain much redundant information. Furthermore, accurate estimation for labels of unlabeled samples plays a critical role for semi-supervised discriminant analysis. In this paper, a novel method called Iterative Semi-supervised discriminant analysis (Iterative-SDA) is proposed, which takes an iterative manner to estimate the labels of unlabeled samples. Each iteration of Iterative-SDA mainly contains two steps: (1) Extract low-dimensional features based on the estimated labels F of samples; (2) Estimate the labels based on the extracted low-dimensional features. It is an NP hard problem to optimize the binary label matrix F, so we explicitly impose nonnegative and orthogonal constrains to relax binary label matrix F during the process of iteration. Experiments are done on four benchmark databases including AR, ORL, CMU PIE and COIL20, the experimental results demonstrate that the proposed method is more effective than LDA and some other state-of-art semi-supervised methods.

A Novel Graph Constructor for Semisupervised Discriminant Analysis: Combined Low-Rank and k-Nearest Neighbor Graph

Semisupervised Discriminant Analysis (SDA) is a semisupervised dimensionality reduction algorithm, which can easily resolve the out-of-sample problem. Relative works usually focus on the geometric relationships of data points, which are not obvious, to enhance the performance of SDA. Different from these relative works, the regularized graph construction is researched here, which is important in the graph-based semisupervised learning methods. In this paper, we propose a novel graph for Semisupervised Discriminant Analysis, which is called combined low-rank and k-nearest neighbor (LRKNN) graph. In our LRKNN graph, we map the data to the LR feature space and then the kNN is adopted to satisfy the algorithmic requirements of SDA. Since the low-rank representation can capture the global structure and the k-nearest neighbor algorithm can maximally preserve the local geometrical structure of the data, the LRKNN graph can significantly improve the performance of SDA. Extensive experiments on several real-world databases show that the proposed LRKNN graph is an efficient graph constructor, which can largely outperform other commonly used baselines.

面向高光谱图像分类的空谱半监督局部判别分析

面向高光谱图像分类的空谱半监督局部判别分析

Fault Identification Method Based on Graph-implanted Probability-based Semi-supervised Discriminant Analysis

Fault Identification Method Based on Graph-implanted Probability-based Semi-supervised Discriminant Analysis

Recursively global and local discriminant analysis for semi-supervised and unsupervised dimension reduction with image analysis

Semi-supervised discriminant analysis (SDA) is a recently-developed semi-supervised dimension reduction method for improving the performance of Fisher linear discriminant analysis (FDA), which attempts to mine the local structures of both labeled and unlabeled data. In this paper, we develop new semi-supervised and unsupervised discriminant analysis techniques. Our semi-supervised method, referred as to as recursively global and local discriminant analysis (RGLDA), is modeled based on the characterizations of “locality” and “non-locality”, such that the manifold regularization in the formulation has a more direct connection to classification. The objective of RGLDA is a “concave-convex” programming problem based on the hinge loss. Its solution follows from solving multiple related SVM-type problems. In addition, we also propose a simple version (called URGLDA) for unsupervised dimension reduction. The experiments tried out on several image databases show the effectiveness of RGLDA and URGLDA.

Semi-supervised double sparse graphs based discriminant analysis for dimensionality reduction

Discriminant analysis (DA) is a well-known dimensionality reduction tool in pattern classification. With enough efficient labeled samples, the optimal projections could be found by maximizing the between-class scatter variance meanwhile minimizing the within-class scatter variance. However, the acquisition of label information is difficult in practice. So, semi-supervised discriminant analysis has attracted much attention in recent years, where both few labeled samples and many unlabeled samples are utilized during learning process. Sparse graph learned by sparse representation contains local structure information about data and is widely employed in dimensionality reduction. In this paper, semi-supervised double sparse graphs (sDSG) based dimensionality reduction is proposed, which considers both the positive and negative structure relationship of data points by using double sparse graphs. Aiming to explore the discriminant information among unlabeled samples, joint k nearest neighbor selection strategy is proposed to select pseudo-labeled samples which contain some precise discriminant information. In the following procedures, the data subset consisting of labeled samples and pseudo-labeled samples are used instead of the original data. Based on two different criterions, two sDSG based discriminant analysis methods are designed and denoted by sDSG-dDA (distance-based DA) and sDSG-rDA (reconstruction-based DA), which also use different strategies to reduce the effect of pseudo-labels’ inaccuracy. Finally, the experimental results both on UCI datasets and hyperspectral images validate the effectiveness and advantage of the proposed methods compared with some classical dimensionality reduction methods.

Low-Rank Kernel-Based Semisupervised Discriminant Analysis

Semisupervised Discriminant Analysis (SDA) aims at dimensionality reduction with both limited labeled data and copious unlabeled data, but it may fail to discover the intrinsic geometry structure when the data manifold is highly nonlinear. The kernel trick is widely used to map the original nonlinearly separable problem to an intrinsically larger dimensionality space where the classes are linearly separable. Inspired by low-rank representation (LLR), we proposed a novel kernel SDA method called low-rank kernel-based SDA (LRKSDA) algorithm where the LRR is used as the kernel representation. Since LRR can capture the global data structures and get the lowest rank representation in a parameter-free way, the low-rank kernel method is extremely effective and robust for kinds of data. Extensive experiments on public databases show that the proposed LRKSDA dimensionality reduction algorithm can achieve better performance than other related kernel SDA methods.

Semi-supervised discriminant analysis method for face recognition

In face recognition (FR), a lot of algorithms just utilize one single type of facial features namely global feature or local feature, and cannot obtain better performance under the complicated variations of the facial images. To extract robust facial features, this paper proposes a novel Semi-Supervised Discriminant Analysis (SSDA) criterion via nonlinearly combining the global feature and local feature. To further enhance the discriminant power of SSDA features, the geometric distribution weight information of the training data is also incorporated into the proposed criterion. We use SSDA criterion to design an iterative algorithm which can determine the combination parameters and the optimal projection matrix automatically. Moreover, the combination parameters are guaranteed to fall into the interval [0, 1]. The proposed SSDA method is evaluated on the ORL, FERET and CMU PIE face databases. The experimental results demonstrate that our method achieves superior performance.

Large-scale image recognition based on parallel kernel supervised and semi-supervised subspace learning

Kernel discriminant subspace learning technique is effective to exploit the structure of image dataset in the high-dimensional nonlinear space. However, for large-scale image recognition applications, this technique usually suffers from large computational burden. Although some kernel accelerating methods have been presented, how to greatly reduce computing time and simultaneously keep favorable recognition accuracy is still challenging. In this paper, we introduce the idea of parallel computing into kernel subspace learning and build a parallel kernel discriminant subspace learning framework. In this framework, we firstly design a random non-overlapping equal data division strategy to divide the whole training set into several subsets and assign each computational node a subset. Then, we separately learn kernel discriminant subspaces from these subsets without mutual communications and finally select the most appropriate subspace to classify test samples. Under the built framework, we propose two novel kernel subspace learning approaches, i.e., parallel kernel discriminant analysis (PKDA) and parallel kernel semi-supervised discriminant analysis (PKSDA). We show the superiority of the proposed approaches in terms of time complexity as compared with related methods, and provide the fundamental supports for our framework. For experiment, we establish a parallel computing environment and employ three public large-scale image databases as experiment data. Experimental results demonstrate the efficiency and effectiveness of the proposed approaches.

On incremental semi-supervised discriminant analysis

In various pattern classification problems, semi-supervised discriminant analysis has shown its effectiveness in utilizing unlabeled data to yield better performance than linear discriminant analysis. However, many of these semi-supervised classifiers operate in batch-mode and do not allow to incrementally update the existing model, which is one of the major limitations. This paper presents an incremental semi-supervised discriminant analysis algorithm, which utilizes the unlabeled data for enabling incremental learning. The major contributions of this research are (1) utilizing large unlabeled training set to estimate the total scatter matrix, (2) incremental learning approach that requires updating only the between-class scatter matrix and not the total scatter matrix, and (3) utilizing manifold regularization for robust estimation of total variability and sufficient spanning set representation for incremental learning. Using face recognition as the case study, evaluation is performed on the CMU-PIE, CMU-MultiPIE, and NIR-VIS-2.0 datasets. The experimental results show that the incremental model is consistent with the batch counterpart and reduces the training time significantly.

Learning from normalized local and global discriminative information for semi-supervised regression and dimensionality reduction

Semi-supervised dimensionality reduction is one of the important topics in pattern recognition and machine learning. During the past decade, Laplacian Regularized Least Square (LapRLS) and Semi-supervised Discriminant Analysis (SDA) are the two widely-used semi-supervised dimensionality reduction methods. In this paper, we show that SDA and LapRLS can be unified into a constrained manifold regularized least square framework. The manifold term, however, cannot fully utilize the underlying discriminative information. We thus introduce a new and effective semi-supervised dimensionality reduction method, called Learning from Local and Global Information (LLGDI), to solve the problem. The proposed LLGDI method adopts a set of local classification functions to preserve both local geometrical and discriminative information of dataset. It also adopts a global classification function to preserve the global discriminative information, and an uncorrelated constraint to calculate the projection matrix for simultaneously solving regression and dimensionality reduction problem. As a result, the LLGDI method is able to preserve local discriminative, manifold information as well as the global discriminative information. Theoretical analysis and extensive simulations presented in the paper show the effectiveness of the LLGDI algorithm. The results also demonstrate LLGDI can achieve superior performance compared with other existing methods.