Semi-supervised Feature Extraction Research Articles

Feature extraction framework based on contrastive learning with adaptive positive and negative samples

Feature extraction is an efficient approach for alleviating the issue of dimensionality in high-dimensional data. As a popular self-supervised learning method, contrastive learning has recently garnered considerable attention. In this study, we propose a unified feature extraction framework based on contrastive learning with adaptive positive and negative samples (CL-FEFA) that is suitable for unsupervised, supervised, and semi-supervised feature extraction. CL-FEFA constructs adaptively positive and negative samples from the result of feature extraction, which makes them more appropriate and accurate. Meanwhile, the discriminative features are extracted based on adaptive positive and negative samples, which will make the intra-class embedded samples more compact and the inter-class embedded samples more dispersed. In the process, using the potential structure information of subspace samples to dynamically construct positive and negative samples can make our framework more robust to noisy data. Furthermore, it is proven that CL-FEFA actually maximizes the mutual information of positive samples, which captures non-linear statistical dependencies between similar samples in potential structure space and thus can act as a measure of true dependence. This also provides theoretical support for its advantages in feature extraction. The final numerical experiments prove that the proposed framework has a strong advantage over traditional feature extraction methods and contrastive learning methods.

Classification of myocardial fibrosis in DE-MRI based on semi-supervised semantic segmentation and dual attention mechanism

ObjectiveIt is essential to utilize cardiac delayed-enhanced magnetic resonance imaging (DE-MRI) to diagnose cardiovascular disease. By segmenting myocardium DE-MRI images, it provides critical information for the evaluation and treatment of myocardial infarction. As a consequence, it is vital to investigate the segmentation and classification technique of myocardial DE-MRI. MethodsFirstly, an end-to-end minimally supervised and semi-supervised semantic DE-MRI myocardial fibrosis segmentation framework is proposed, which combines image classification and semantic segmentation branches based on the self-attention mechanism. Following that, a residual hole network fused with the dual attention mechanism was built, and a double attention metabolic pathway classification method for cardiac fibrosis in DE-MRI images was developed. ResultsBy adding pixel-level labels to an extra 40 training images, the segmentation model may enhance semantic segmentation performance by 2.6 percent (from 61.2 percent to 63.8 percent). When the number of pixel-level labels is increased to 80, semi-supervised feature extraction increases by 4.7 percent when compared to weakly guided semantic segmentation. Adding an attention mechanism to the critical network DRN (Deep Residual Network) can increase the classifier's performance by a small amount. Experiments revealed that the models worked effectively. ConclusionThis paper investigates the segmentation and classification of cardiac fibrosis in DE-MRI data using a semi-supervised semantic segmentation and dual attention mechanism, dealing with the issue that existing segmentation algorithms have difficulty segmenting myocardial fibrosis tissue. In the future, we can consider optimizing the design of the attention module to reduce the module computation.

Improved Locality Preserving Projection for Hyperspectral Image Classification in Probabilistic Framework

Unlabeled samples and transformation matrix are two main parts of unsupervised and semi-supervised feature extraction (FE) algorithms. In this manuscript, a semi-supervised FE method, locality preserving projection in the probabilistic framework (LPPPF), to find a sufficient number of reliable and unmixed unlabeled samples from all classes and constructing an optimal projection matrix is proposed. The LPPPF has two main steps. In the first step, a number of reliable unlabeled samples are selected based on the training samples, spectral features, and spatial information in the probabilistic framework. In this way, the spectral and spatial probability distribution function is calculated for each unlabeled sample. Therefore, the spectral features and spatial information are integrated together with a joint probability distribution function. Finally, a sufficient number of unlabeled samples with the highest joint probability distribution are selected. In the second step, the selected unlabeled samples are applied to construct the transformation matrix based on the spectral and spatial information of the unlabeled samples. The adjacency graph is improved by using new weights based on spectral and spatial information. This method is evaluated on three data sets: Indian Pines, Pavia University, and Kennedy Space Center (KSC) and compared with some recent and well-known supervised, semi-supervised, and unsupervised FE methods. Various experiments demonstrate the efficiency of the LPPPF in comparison with the other FE methods. LPPPF has also considerable performance with limited training samples.

Semisupervised collaborative representation graph embedding for hyperspectral imagery

Graph embedding (GE) frameworks are used for extracting the discriminative features of hyperspectral images (HSIs). However, it is difficult to select a proper neighborhood size for graph construction. To overcome this difficulty, a semisupervised feature extraction (FE) method, called semisupervised collaborative representation graph embedding (SCRGE), is proposed. The proposed algorithm utilizes collaborative representation (CR) to obtain the collaborative coefficients of labeled and unlabeled samples. Then, a semisupervised graph is constructed using the collaborative coefficients of the labeled samples within the same class and the collaborative coefficients of the unlabeled samples, and an interclass graph is constructed using the collaborative coefficients of the labeled samples in different classes. Finally, a projection matrix for FE is obtained by embedding these graphs into a low-dimensional space. SCRGE not only inherits the merits of CR to reveal the collaborative reconstructive properties of data but also enhances intraclass compactness and interclass separability to improve the discriminating power for classification. Experimental results on three real HSIs datasets demonstrate that SCRGE outperforms other state-of-the-art FE methods in terms of classification accuracy.

Collaborative representation-based semisupervised feature extraction of hyperspectral images using attraction points

Feature extraction (FE) is an important preprocessing step for classification of high-dimensional data. However, when the number of labeled training samples is small, less discriminant information induces some FE methods that use statistical moments, for example, linear discriminant analysis, even fail to work. We present a collaborative representation (CR)-based semisupervised FE method using attraction points (CRSUAP). By CR, CRSUAP defines a membership matrix that contains the correlation between multiple samples. Then, a nonmembership matrix is designed to enrich the discriminant information and further enhance the separability of classes. To avoid estimating statistical moments, an attraction point is selected from each class to calculate the projection matrix, avoiding matrix singularity problem. The experimental results on three real hyperspectral images demonstrate that CRSUAP has better performance than other related FE methods in a small labeled sample size situation.

DeepVM: A Deep Learning-based approach with automatic feature extraction for 2D input data Virtual Metrology

Industry 4.0 encapsulates methods, technologies, and procedures that transform data into informed decisions and added value in an industrial context. In this regard, technologies such as Virtual Metrology or Soft Sensing have gained much interest in the last two decades due to their ability to provide valuable knowledge for production purposes at limited added expense. However, these technologies have struggled to achieve wide-scale industrial adoption, largely due to the challenges associated with handling complex data structures and the feature extraction phase of model building. This phase is generally hand-engineered and based on specific domain knowledge, making it time consuming, difficult to automate, and prone to loss of information, thus ultimately limiting portability. Moreover, in the presence of complex data structures, such as 2-dimensional input data, there are no established procedures for feature extraction. In this paper, we present a Deep Learning approach for Virtual Metrology, called DeepVM, that exploits semi-supervised feature extraction based on Convolutional Autoencoders. The proposed approach is demonstrated using a real world Semiconductor Manufacturing dataset where the Virtual Metrology input data is 2-dimensional Optical Emission Spectrometry data. The feature extraction method is tested with different types of state-of-the-art autoencoder.

Recursive Maximum Margin Active Learning

Active learning is an important technique to alleviate the problem when there is abundant unlabeled data but scarce labeled data. It aims to choose the most valuable samples to label in order to build powerful predictive models with minimal supervision. However, under the setup of active learning, when the data are characterized by high-dimensional features, it will be difficult to get a reliable estimate on the model parameters, as the labeled data are limited. Most existing works tackle this problem by learning a low-dimensional representation of data before active learning, but it cannot guarantee to obtain promising results, as traditional feature extraction techniques and active learning algorithms are designed independently. In this paper, we propose an efficient hybrid active learning algorithm, called Recursive Maximum Margin Active Learning. We optimize the active learning and semi-supervised feature extraction under a unified framework to tackle the high-dimensional features' problems and select the most representative samples in the low-dimensional space. By introducing the semi-supervised maximum margin criterion into active learning, we can conduct sample selection and feature extraction recursively at each iteration of active learning to learn more accurate models. The extensive experimental results show that the proposed method outperforms several state-of-the-art active learning methods on publicly available datasets.

Semi-Supervised Tensorial Locally Linear Embedding for Feature Extraction Using PolSAR Data

Recently, the semi-supervised feature extraction, which uses both a few labeled samples and a large quantity of unlabeled samples to improve discrimination of extracted features, has attracted attention in PolSAR data processing. However, most of the feature extraction methods are based on matrix operations and need to re-arrange the multiple features into vectors. Such a vectorization way may break the spatial structure and correlation among the neighboring pixels, which leads to the degradation of classification performance. To address this issue and incorporate the superiority of semi-supervised learning, a novel semi-supervised tensorial locally linear embedding (STLLE) method is proposed for PolSAR data. Each pixel and its spatial neighbors in a local window are represented as a third-order tensor, whose first two orders represent the neighborhood pixels, and the third order denotes the multiple features. The feature extraction is converted to find an optimal projection that can transform the high-dimensional features to low-dimensional ones. This can be solved by designing an objective function, in which the tensorial discriminant item includes the between-class and within-class scatter matrices on each mode of the labeled tensors. A regularizer is based on tensorial locally linear embedding graph using labeled and unlabeled tensor samples. Moreover, the pairwise class constraints are also introduced. Finally, we can use all the extracted features with a classifier such as K-Nearest Neighbor. Experimental results on three real PolSAR datasets demonstrate that STLLE significantly improves classification accuracies compared with state-of-the-art methods, which clearly verifies that the extracted features preserve the data structures well and are more discriminative.

Rail Suspension System Fault Detection using Deep Semi-Supervised Feature Extraction with One-class data

In this paper we propose a novel semi-supervised fault detection methodology for a vehicle suspension system with one-class multi-sensor data. Supervised data-driven methods have been applied in fault detection successfully in recent studies. However, it is difficult and expensive to collect data under faulty condition for supervised learning while data collection under normal condition is much easier and cheaper. Fault detection under such situation is a one-class classification problem that requires classification models to identify the positive class when the negative class is either absent or poorly sampled. The efficiency of classifiers is constrained by defining the normal class boundary only with the knowledge of positive class as well as the problem of biased or incorrect feature extracted from the positive class. In proposed method, A deep semi-supervised learning method integrated with physical-based domain knowledge is applied for feature extraction. The suspension system for a full car model is modeled using a simulation tool, SIMPACK to generate the synthetic multi-sensor data. Our results show the effectiveness of the proposed in fault detection and diagnostics with one-class data.

A Novel Semi-Supervised Feature Extraction Method and Its Application in Automotive Assembly Fault Diagnosis Based on Vision Sensor Data.

The fault diagnosis of dimensional variation plays an essential role in the production of an automotive body. However, it is difficult to identify faults based on small labeled sample data using traditional supervised learning methods. The present study proposed a novel feature extraction method named, semi-supervised complete kernel Fisher discriminant (SS-CKFDA), and a new fault diagnosis flow for automotive assembly was introduced based on this method. SS-CKFDA is a combination of traditional complete kernel Fisher discriminant (CKFDA) and semi-supervised learning. It adjusts the Fisher criterion with the data global structure extracted from large unlabeled samples. When the number of labeled samples is small, the global structure that exists in the measured data can effectively improve the extraction effects of the projected vector. The experimental results on Tennessee Eastman Process (TEP) data demonstrated that the proposed method can improve diagnostic performance, when compared to other Fisher discriminant algorithms. Finally, the experimental results on the optical coordinate data proves that the method can be applied in the automotive assembly process, and achieve a better performance.

Spatially Constrained Semisupervised Local Angular Discriminant Analysis for Hyperspectral Images

Dimensionality reduction is an important preprocessing step for hyperspectral image analysis. Hyperspectral images contain the reflectances measured at a very large number of wavelengths. Due to the over-dimensionality and potential redundancy of information, extraction of features which are meaningful for the purpose of classification becomes a step of prime importance. This is especially true when we have computational constraints (both in terms of time and memory) or a small number of samples available for training the classifier. For most applications, the number of labeled training samples for hyperspectral images is usually small, though the number of unlabeled samples are abundantly available. In this paper, we propose a semisupervised feature extraction algorithm which minimizes the angular similarity between the spectrally similar spatial neighbors and maximizes the angular separation between the pixels belonging to different classes in the projected lower dimensional subspace by utilizing unlabeled samples. We propose a method to incorporate the spatial information from the neighborhood pixels to generate a spatial–spectral dimension reduction algorithm for a semisupervised dimension reduction approach and show that it outperforms the present state-of-the-art feature extraction approaches. In general our approach for incorporating the spatial information can be applied to other supervised and unsupervised feature extraction methods.

A Convolutional Autoencoder Approach for Feature Extraction in Virtual Metrology

Exploiting the huge amount of data collected by industries is definitely one of the main challenges of the so-called Big Data era. In this sense, Machine Learning has gained growing attention in the scientific community, as it allows to extract valuable information by means of statistical predictive models trained on historical process data. In Semiconductor Manufacturing, one of the most extensively employed data-driven applications is Virtual Metrology, where a costly or unmeasurable variable is estimated by means of cheap and easy to obtain measures that are already available in the system. Often, these measures are multi-dimensional, so traditional Machine Learning algorithms cannot handle them directly. Instead, they require feature extraction, that is a preliminary step where relevant information is extracted from raw data and converted into a design matrix. Features are often hand-engineered and based on specific domain knowledge. Moreover, they may be difficult to scale and prone to information loss, affecting the effectiveness and maintainability of machine learning procedures. In this paper, we present a Deep Learning method for semi-supervised feature extraction based on Convolutional Autoencoders that is able to overcome the aforementioned problems. The proposed method is tested on a real dataset for Etch rate estimation. Optical Emission Spectrometry data, that exhibit a complex bi-dimensional time and wavelength evolution, are used as input.

Semi-supervised fuzzy neighborhood preserving analysis for feature extraction in hyperspectral remote sensing images

Semi-supervised feature extraction methods are an important focus of interest in data mining and machine learning areas. These methods are improved methods based on learning from a combination of labeled and unlabeled data. In this study, a semi-supervised feature extraction method called as semi-supervised fuzzy neighborhood preserving analysis (SFNPA) is proposed to improve the classification accuracy of hyperspectral remote sensing images. The proposed method combines the principal component analysis (PCA) method, which is an unsupervised feature extraction method, and the supervised fuzzy neighborhood preserving analysis (FNPA) method and increases the classification accuracy by using a limited number of labeled data. Experimental results on four popular hyperspectral remote sensing datasets show that the proposed method significantly improves classification accuracy on hyperspectral remote sensing images compared to the well-known semi-supervised dimension reduction methods.

Kernel Feature Extraction for Hyperspectral Image Classification Using Chunklet Constraints

A novel semi-supervised kernel feature extraction algorithm to combine an efficient metric learning method, i.e. relevant component analysis (RCA), and kernel trick is presented for hyperspectral imagery land-cover classification. This method obtains projection of the input data by learning an optimal nonlinear transformation via a chunklet constraints-based FDA criterion, and called chunklet-based kernel relevant component analysis (CKRCA). The proposed method is appealing as it constructs the kernel very intuitively for the RCA method and does not require any labeled information. The effectiveness of the proposed CKRCA is successfully illustrated in hyperspectral remote sensing image classification. Experimental results demonstrate that the proposed method can greatly improve the classification accuracy compared with traditional linear and conventional kernel-based methods.

Semi-supervised transfer discriminant analysis based on cross-domain mean constraint

In this paper, a novel semi-supervised feature extraction algorithm, i.e., semi-supervised transfer discriminant analysis (STDA) with knowledge transfer capability is proposed, based on the traditional algorithm that cannot get adapted in the change of the learning environment. By using both the pseudo label information from target domain samples and the actual label information from source domain samples in the label iterative refinement process, not only the between-class scatter is maximized while that within-class scatter is minimized, but also the original space structure is maintained via Laplacian matrix, and the distribution difference is reduced by using maximum mean discrepancy as well. Moreover, semi-supervised transfer discriminant analysis based on cross-domain mean constraint (STDA-CMC) is proposed. In this algorithm, the cross-domain mean constraint term is incorporated into STDA, such that knowledge transfer between domains is facilitated by making source and target samples after being projected are located more closely in the low-dimensional feature subspace. The proposed algorithm is proved efficient and feasible from experiments on several datasets.

A joint-L2,1-norm-constraint-based semi-supervised feature extraction for RNA-Seq data analysis

It is of urgency to effectively identify differentially expressed genes from RNA-Seq data. In this paper, we proposed a novel method, joint-L2,1-norm-constraint-based semi-supervised feature extraction (L21SFE), to analyze RNA-Seq data. Our scheme was shown as follows. Firstly, we constructed a graph Laplacian matrix and refined it by using the labeled samples. Our graph construction method can make full use of a large number of unlabelled samples. Secondly, we found semi-supervised optimal maps by solving a generalized eigenvalue problem. Thirdly, we solved an optimal problem via the joint L2,1-norm constraint to obtain a projection matrix. It can diminish the impact of noises and outliers by using the L2,1-norm constraint and produce more precise results. Finally, we identified differentially expressed genes based on the projection matrix. The results on simulation and real RNA-Seq data sets demonstrated the feasibility and effectiveness of our method.

Dual structural consistency based multi-modal correlation propagation projections for data representation

Canonical correlation analysis (CCA) is a powerful tool for analyzing multi-dimensional paired data. However, when facing semi-supervised multi-modal data (Also called multi-view Hou et al. (Pattern Recog 43(3):720–730, 2010) or multi-represented Kailing et al. (Clustering multi-represented objects with noise. In: Proceedings of the eighth Pacific-Asia conference on knowledge discovery and data mining (PAKDD). Sydney, Australia, pp 394–403) data. For convenience, we will uniformly call them multi-modal data hereafter.) which widely exist in real-world applications, CCA usually performs poorly due to ignoring useful supervised information. Meanwhile, due to the limited labeled training samples in the semi-supervised scenario, supervised extensions of CCA suffer from overfitting. Several semi-supervised extensions of CCA have been proposed recently. Nevertheless, they either just utilize the global structural information captured from the unlabeled data, or propagate label information by discovering the affinities just between the labeled and unlabeled data points in advance. In this paper, we propose a robust multi-modal semi-supervised feature extraction and fusion framework, termed as dual structural consistency based multi-modal correlation propagation projections (SCMCPP). SCMCPP guarantees the consistency between representation structure and hypotaxis structure in each modality and ensures the consistency of hypotaxis structure between two different modalities. By iteratively propagating labels and learning affinities, discriminative information of both given labels and estimated labels is utilized to improve the affinity construction and infer the remaining unknown labels. Moreover, probabilistic within-class scatter matrices in each modality and probabilistic correlation matrix between two modalities are constructed to enhance the discriminative power of features. Extensive experiments on several benchmark face databases demonstrate the effectiveness of our approach.

Spatial-spectral locality preserving projection for hyperspectral image classification with limited training samples

ABSTRACTLocality preserving projection (LPP) is a popular unsupervised feature extraction (FE) method. In this paper, the spatial-spectral LPP (SSLPP) method is proposed, which uses both the spectral and spatial information of hyperspectral image (HSI) for FE. The proposed method consists of two parts. In the first part, unlabelled samples are selected in a spatially homogeneous neighbourhood from filtered HSI. In the second part, the transformation matrix is calculated by an LPP-based method and by using the spectral and spatial information of the selected unlabelled samples. Experimental results on Indian Pines (IP), Kennedy Space Center (KSC), and Pavia University (PU) datasets show that the performance of SSLPP is superior to spectral unsupervised, supervised, and semi-supervised FE methods in small and large sample size situations. Moreover, the proposed method outperforms other spatial-spectral semi-supervised FE methods for PU dataset, which has high spatial resolution. For IP and KSC datasets, spectral regularized local discriminant embedding (SSRLDE) has the best performance by using spectral and spatial information of labelled and unlabelled samples, and SSLPP is ranked just behind it. Experiments show that SSLPP is an efficient unsupervised FE method, which does not use training samples as preparation of them is so difficult, costly, and sometimes impractical. SSLPP results are much better than LPP. Also, it decreases the storage and calculation costs using less number of unlabelled samples.

Semisupervised Feature Extraction With Neighborhood Constraints for Polarimetric SAR Classification

The supervised feature extraction methods have a relative high performance since the discriminating information of classes is introduced from large quantities of labeled samples. However, it is labor intensive to obtain labeled samples for terrain classification. In this paper, in order to reduce the cost of labeled samples, a novel semisupervised algorithm with neighborhood constraints (SNC) is proposed for polarimetric synthetic aperture radar (PolSAR) feature extraction and terrain classification. A number of PolSAR features of each pixel and its neighbors are used to construct a spatial group, which can represent the central pixel and weaken the influence of speckle noise. Then, with the class information from a few of pixels and the neighborhood constraints, an objective function is designed for the estimation of a nonlinear low-dimensional space. Finally, the spatial groups in the original high-dimensional space are projected to this low-dimensional space, and a low-dimensional feature set is obtained. The redundancy among the features is reduced. Additionally, unlike the conventional semisupervised algorithms, because the local spatial relation of PolSAR image is utilized, the extracted features not only are discriminating but also preserve the structure of the PolSAR data, which can enhance the classification accuracy. Experiments using the extracted features for classification are performed on both the synthesized PolSAR and real PolSAR data which are from the AIRSAR, RADARSAT-2, and EMISAR. Quantitative results indicate that SNC improves the separability of features and is superior to state-of-the-art feature extraction algorithms with a few labeled pixels.

Inductive and flexible feature extraction for semi-supervised pattern categorization

This paper proposes a novel discriminant semi-supervised feature extraction method for generic classification and recognition tasks. This method, called inductive flexible semi-supervised feature extraction, is a graph-based embedding method that seeks a linear subspace close to a non-linear one. It is based on a criterion that simultaneously exploits the discrimination information provided by the labeled samples, maintains the graph-based smoothness associated with all samples, regularizes the complexity of the linear transform, and minimizes the discrepancy between the unknown linear regression and the unknown non-linear projection. We extend the proposed method to the case of non-linear feature extraction through the use of kernel trick. This latter allows to obtain a nonlinear regression function with an output subspace closer to the learned manifold than that of the linear one. Extensive experiments are conducted on ten benchmark databases in order to study the performance of the proposed methods. Obtained results demonstrate a significant improvement over state-of-the-art algorithms that are based on label propagation or semi-supervised graph-based embedding.