Semi-supervised Feature Selection Method Research Articles

SFS-AGGL: Semi-Supervised Feature Selection Integrating Adaptive Graph with Global and Local Information

As the feature dimension of data continues to expand, the task of selecting an optimal subset of features from a pool of limited labeled data and extensive unlabeled data becomes more and more challenging. In recent years, some semi-supervised feature selection methods (SSFS) have been proposed to select a subset of features, but they still have some drawbacks limiting their performance, for e.g., many SSFS methods underutilize the structural distribution information available within labeled and unlabeled data. To address this issue, we proposed a semi-supervised feature selection method based on an adaptive graph with global and local constraints (SFS-AGGL) in this paper. Specifically, we first designed an adaptive graph learning mechanism that can consider both the global and local information of samples to effectively learn and retain the geometric structural information of the original dataset. Secondly, we constructed a label propagation technique integrated with the adaptive graph learning in SFS-AGGL to fully utilize the structural distribution information of both labeled and unlabeled data. The proposed SFS-AGGL method is validated through classification and clustering tasks across various datasets. The experimental results demonstrate its superiority over existing benchmark methods, particularly in terms of clustering performance.

Multi-filter semi-supervised transformer model for fault diagnosis

Dissolved Gas Analysis (DGA) is the most commonly used method for power transformer fault diagnosis. However, very few reliable and labeled fault DGA samples are available in the transformer substation whilst DGA data without labels is easier to obtain, which makes it difficult to train fault detectors in high-dimensional input space or select features using wrapper methods. Therefore, in order to improve the fault diagnosis accuracy using limited labeled DGA samples but more unlabeled DGA data, this paper proposes a novel multi-filter semi-supervised feature selection method for selecting optimal DGA features and building effective fault diagnosis models. A confidence criterion is also proposed for selecting high confidence unlabeled data to expand the training data set. Five filter techniques based on different evaluation criteria are employed to rank input DGA features, and a feature combination method is then applied to aggregate feature ranks by multiple filters and form a lower-dimensional candidate feature subset. The proposed method has been tested by using the IEC T10 dataset and compared with traditional supervised diagnostic models. The results show that the proposed method works well in optimizing DGA features and improving fault diagnosis accuracy significantly. Besides, the robustness of the selection of optimal feature subset is validated by testing DGA samples from the local power utility.

Iterative constraint score based on hypothesis margin for semi-supervised feature selection

To remove redundant features and avoid the curse of dimensionality, the most important features should be selected for downstream tasks, including semi-supervised learning. Several semi-supervised constraint scores using pairwise constraints have been proposed to estimate the relevance of features. However, these methods evaluate the features individually and ignore the correlations between them. Thus, we propose a semi-supervised feature selection method called the iterative constraint score based on the hypothesis margin (HM-ICS), which uses forward sequential selection to select an optimal feature subset with a good ability to maintain the constraint structure of the data and distinguish samples that belong to different classes. HM-ICS iteratively modifies the classical constraint score method to measure the relevance between features and maintain the constraint structure of the data. By introducing the hypothesis margin, HM-ICS can ensure strong discriminative power of the optimal feature subset. Extensive experiments were conducted on nine UCI and five high-dimensional datasets, and the experimental results confirmed that HM-ICS can achieve better performance than state-of-the-art supervised and semi-supervised methods.

A unified view of feature selection based on Hilbert-Schmidt independence criterion

Feature selection is a challenging and increasing important task in the machine learning and data mining community. According to different learning scenarios such as the availability of supervision information, feature selection can be broadly categorized as supervised, unsupervised and semi-supervised methods. Traditionally, feature selection algorithms are scenario-specific, i.e., different types of feature selection algorithms were proposed and investigated separately for specific learning scenario. In this paper, we aim to develop a unified view of supervised, unsupervised and semi-supervised feature selection methods. Based on the Hilbert-Schmidt independence criterion (HSIC), we first show that supervised, unsupervised and semi-supervised feature selection methods share the same objective. In discussing the connection between the unified framework and HSIC least absolute shrinkage and selection operator (HSIC Lasso), we find that the proposed framework not only has a clear statistical interpretation that minimum redundant features with maximum dependence on the target output variable can be found in terms of the HSIC, but also enables the global optimal solution to be computed efficiently by solving a Lasso optimization problem. Under this unified view, a new unsupervised feature selection algorithm is also proposed and demonstrated with several benchmark examples.

Neurodynamics-driven holistic approaches to semi-supervised feature selection

Feature selection is a crucial part of machine learning and pattern recognition, which aims at selecting a subset of informative features from the original dataset. Because of label information, supervised feature selection performs better than unsupervised feature selection without label information. However, in the presence of a small number of labeled data and a large number of unlabeled data, it is challenging for supervised feature selection methods to select relevant features. In this paper, we propose three neurodynamics-driven holistic approaches to semi-supervised feature selection via semi-supervised feature redundancy minimization and semi-supervised feature relevancy maximization. We first define information-theoretic semi-supervised similarity coefficient matrix and semi-supervised feature relevancy vector based on multi-information, unsupervised symmetric uncertainty, and entropy to measure feature redundancy and relevancy. We then formulate a fractional programming problem and an iteratively weighted quadratic programming problem based on the semi-supervised similarity coefficient matrix and semi-supervised feature relevancy vector for semi-supervised feature selection. To solve the formulated problems, we delineate three neurodynamic optimization approaches based on two projection neural networks. We elaborate on the experimental results on six benchmark datasets to demonstrate the superior classification performance of the proposed neurodynamic approaches against six existing supervised and semi-supervised feature selection methods.

SemiACO: A semi-supervised feature selection based on ant colony optimization

Feature selection is one of the most efficient procedures for reducing the dimensionality of high-dimensional data by choosing a practical subset of features. Since labeled samples are not always available and labeling data may be time-consuming or costly, the importance of semi-supervised learning becomes apparent. Semi-supervised learning deals with data that includes both labeled and unlabelled instances. This article proposes a method based on Ant Colony Optimization (ACO) for the semi-supervised feature selection problem called SemiACO. The SemiACO algorithm finds features by considering the minimum redundancy between features and the maximum relevancy between the features and the class label. The SemiACO uses a nonlinear heuristic function instead of a linear one. The heuristic learning technique for the ACO heuristic function utilize a Temporal Difference (TD) reinforcement learning algorithm. We characterize the feature selection search space as a Markov Decision Process (MDP), where features indicate the states, and selecting the unvisited features by each ant represents a set of actions. We contrast the efficiency of SemiACO based on various experiments on 14 benchmark datasets, comparing eight semi-supervised feature selection methods.

Semisupervised Feature Selection via Generalized Uncorrelated Constraint and Manifold Embedding.

Ridge regression is frequently utilized by both supervised learning and semisupervised learning. However, the results cannot obtain the closed-form solution and perform manifold structure when ridge regression is directly applied to semisupervised learning. To address this issue, we propose a novel semisupervised feature selection method under generalized uncorrelated constraint, namely SFS. The generalized uncorrelated constraint equips the framework with the elegant closed-form solution and is introduced to the ridge regression with embedding the manifold structure. The manifold structure and closed-form solution can better save data's topology information compared to the deep network with gradient descent. Furthermore, the full rank constraint of the projection matrix also avoids the occurrence of excessive row sparsity. The scale factor of the constraint that can be adaptively obtained also provides the subspace constraint more flexibility. Experimental results on data sets validate the superiority of our method to the state-of-the-art semisupervised feature selection methods.

BRNS + SSFSM-DTI: A hybrid method for drug-target interaction prediction based on balanced reliable negative samples and semi-supervised feature selection

De novo drug discovery is a time-consuming and costly process. Drug repositioning, which means finding new applications for existing drugs, is one of the most effective approaches to reduce time and cost of detecting a new drug. Predicting drug-target interactions (DTIs) can facilitate the drug repositioning and consequently accelerate the process of drug discovery. The diversity of drug descriptors and protein features as well as the lack of access to experimentally-confirmed non-interacting drug-target pairs as negative samples are the major challenges in predicting new DTIs. In this study, we present a modified algorithm for extracting balanced reliable negative samples named BRNS. Moreover, we propose a semi-supervised feature selection method called SSFSM-DTI and compare the performance of our hybrid predictive model with the models constructed by random selection of negative samples and the other well-known feature selection methods on benchmark DTI datasets. The results show that the combination of BRNS and SSFSM-DTI is superior to the random selection of negative samples and the state-of-the-art feature selection methods in most cases. Hence, it can be used as a guideline in the drug repositioning process. The source codes and all datasets used in this study are available at https://github.com/LBDSoft/BRNS.

Semi-supervised feature selection with minimal redundancy based on local adaptive

With the speedy development of network technology, diverse data increase by hundreds of millions per hour, causing increasing pressure on the acquisition of data labels. Semi-supervised feature selection has been among the forefront of dimensionality reduction research due to the outstanding achievements of “small labels” and “high efficiency”. Especially, the graph-based methods use data with missing labels completely and effectively, prompting it to become a research hotspots in semi-supervised feature selection. However, the existing graph-based methods do not take into account the effects of outliers, noise, and redundancy of selected features simultaneously. To solve those problems, a novel semi-supervised feature selection method based on local adaptive and minimal redundancy is proposed. The local structure is flexibly assigned weights according to the data conditions, thereby reducing the impact of outliers and noise; moreover, a high similarity penalty mechanism is introduced in the feature mapping matrix to promote discrimination and low redundancy of the selected feature subset. In addition, an iterative method is designed and its convergence is proved theoretically and experimentally. Finally, the proposed algorithm is verified to be stable and effective through experiments from five aspects on sixteen public datasets.

Adaptive discriminant analysis for semi-supervised feature selection

As semi-supervised feature selection is becoming much more popular among researchers, many related methods have been proposed in recent years. However, many of these methods first compute a similarity matrix prior to feature selection, and the matrix is then fixed during the subsequent feature selection process. Clearly, the similarity matrix generated from the original dataset is susceptible to the noise features. In this paper, we propose a novel adaptive discriminant analysis for semi-supervised feature selection, namely, SADA. Instead of computing a similarity matrix first, SADA simultaneously learns an adaptive similarity matrix S and a projection matrix W with an iterative process. Moreover. we introduce the ℓ2,p norm to control the sparsity of S by adjusting p. Experimental results show that S will become sparser with the decrease of p. The experimental results for synthetic datasets and nine benchmark datasets demonstrate the superiority of SADA, in comparison with 6 semi-supervised feature selection methods.

Top-k Feature Selection Framework Using Robust 0-1 Integer Programming.

Feature selection (FS), which identifies the relevant features in a data set to facilitate subsequent data analysis, is a fundamental problem in machine learning and has been widely studied in recent years. Most FS methods rank the features in order of their scores based on a specific criterion and then select the k top-ranked features, where k is the number of desired features. However, these features are usually not the top- k features and may present a suboptimal choice. To address this issue, we propose a novel FS framework in this article to select the exact top- k features in the unsupervised, semisupervised, and supervised scenarios. The new framework utilizes the l0,2 -norm as the matrix sparsity constraint rather than its relaxations, such as the l1,2 -norm. Since the l0,2 -norm constrained problem is difficult to solve, we transform the discrete l0,2 -norm-based constraint into an equivalent 0-1 integer constraint and replace the 0-1 integer constraint with two continuous constraints. The obtained top- k FS framework with two continuous constraints is theoretically equivalent to the l0,2 -norm constrained problem and can be optimized by the alternating direction method of multipliers (ADMM). Unsupervised and semisupervised FS methods are developed based on the proposed framework, and extensive experiments on real-world data sets are conducted to demonstrate the effectiveness of the proposed FS framework.

Semi-supervised neighborhood discrimination index for feature selection

Neighborhood discriminant index (NDI) is an effective feature selection method for supervised learning. In reality, it is easy to obtain unlabeled data and is costly to tag them all. Thus, the given dataset commonly has only a small amount of tagged samples and a large amount of unlabeled ones, which cannot be handled by supervised learning methods. For this situation, we propose a semi-supervised feature selection method called semi-supervised neighborhood discriminant index (SSNDI) that combines NDI and the Laplacian score method to effectively deal with both labeled and unlabeled samples. The goal of SSNDI is to find an optimal feature subset that has a good ability to keep local geometrical structure and to distinguish samples belonging to different classes. In SSNDI, the classical Laplacian score method is modified to cooperate the iterative form of NDI. In each iteration, SSNDI picks up an important feature according to the new criterion that is a mixture of NDI and the modified Laplacian score. Extensive experiments are conducted on UCI and microarray gene datasets. Experimental results confirm that SSNDI can achieve a better performance than NDI and the other state-of-the-art semi-supervised methods.

Local preserving logistic I-Relief for semi-supervised feature selection

Logistic I-Relief (LIR), a supervised feature selection method, has been successfully extended for semi-supervised feature selection. However, exiting semi-supervised methods based on LIR are incapable of maintaining the local structure of data. In this paper, we propose a local preserving logistic I-Relief (LPLIR) algorithm for multi-class semi-supervised feature selection. The goal of LPLIR is to maximize the expected margin of the given labeled data and remain the local structural information of all given data, where the local structural information is represented by a local preserving regularization. Additionally, extensive experiments conducted on real-world datasets demonstrate that LPLIR is superior to the existing state-of-the-art semi-supervised feature selection methods.

Semi-Supervised Feature Selection with Adaptive Discriminant Analysis

In this paper, we propose a novel Adaptive Discriminant Analysis for semi-supervised feature selection, namely SADA. Instead of computing fixed similarities before performing feature selection, SADA simultaneously learns an adaptive similarity matrix S and a projection matrix W with an iterative method. In each iteration, S is computed from the projected distance with the learned W and W is computed with the learned S. Therefore, SADA can learn better projection matrix W by weakening the effect of noise features with the adaptive similarity matrix. Experimental results on 4 data sets show the superiority of SADA compared to 5 semisupervised feature selection methods.

Semi-Supervised Feature Selection via Insensitive Sparse Regression with Application to Video Semantic Recognition

Feature selection plays a significant role in dealing with high-dimensional data to avoid the curse of dimensionality. In many real applications, like video semantic recognition, handling few labeled and large unlabeled data samples from the same population is a recently addressed challenge in feature selection. To solve this problem, we propose a novel semi-supervised feature selection method via insensitive sparse regression (ISR). Specifically, we compute the soft label matrix by the special label propagation, which can predict the labels of the unlabeled data. To guarantee the robustness of ISR to the false labeled instances or outliers, we propose Insensitive Regression Model (IRM) by capped $l_2$ - $l_p$ -norm loss. The soft label is imposed as the weights of IRM to fully utilize the label information. Meanwhile, to perform feature selection, we incorporate $l_{2,q}$ -norm regularizer with IRM as the structural sparsity constraint when $0 . Moreover, we put forward an effective approach for solving the formulated non-convex optimization problem. We analyze the performance of convergence rigorously and discuss the parameter determination problem. Extensive experimental results on several public data sets verify the effectiveness of our proposed algorithm in comparison with the state-of-art feature selection methods. Finally, we apply our method to video semantic recognition successfully.

Semi-supervised sparse feature selection via graph Laplacian based scatter matrix for regression problems

Semi-supervised feature selection, which exploits both labeled and unlabeled data to select the relevant features, has an important role in many real world applications. Most semi-supervised feature selection methods have been exclusively designed for classification problems. In this paper, a semi-supervised framework based on graph Laplacian and mixed convex and non-convex l2,p-norm (0 < p ≤ 1) regularization is proposed for regression problems. In the proposed framework, a semi-supervised graph Laplacain based scatter matrix constructed for regression problems is used to encode the label information of labeled data and the local structure of both labeled and unlabeled data. To solve the mixed convex and non-convex regularized l2,p-norm framework, a unified iterative algorithm is proposed. The convergence of the proposed unified algorithm is theoretically and experimentally proved. To evaluate the performance of the proposed framework for regression problems, we perform extensive experiments on different computational drug design regression datasets. The results demonstrate the superiority of the proposed framework in comparison with other feature selection methods in selecting the relevant features.

Discriminative Semi-Supervised Feature Selection via Rescaled Least Squares Regression-Supplement

In this paper, we propose a Discriminative Semi-Supervised Feature Selection (DSSFS) method. In this method, a ε-dragging technique is introduced to the Rescaled Linear Square Regression in order to enlarge the distances between different classes. An iterative method is proposed to simultaneously learn the regression coefficients, ε-draggings matrix and predicting the unknown class labels. Experimental results show the superiority of DSSFS.

Feature selection in machine learning: A new perspective

High-dimensional data analysis is a challenge for researchers and engineers in the fields of machine learning and data mining. Feature selection provides an effective way to solve this problem by removing irrelevant and redundant data, which can reduce computation time, improve learning accuracy, and facilitate a better understanding for the learning model or data. In this study, we discuss several frequently-used evaluation measures for feature selection, and then survey supervised, unsupervised, and semi-supervised feature selection methods, which are widely applied in machine learning problems, such as classification and clustering. Lastly, future challenges about feature selection are discussed.

A combined Fisher and Laplacian score for feature selection in QSAR based drug design using compounds with known and unknown activities.

Quantitative structure-activity relationship (QSAR) is an effective computational technique for drug design that relates the chemical structures of compounds to their biological activities. Feature selection is an important step in QSAR based drug design to select the most relevant descriptors. One of the most popular feature selection methods for classification problems is Fisher score which aim is to minimize the within-class distance and maximize the between-class distance. In this study, the properties of Fisher criterion were extended for QSAR models to define the new distance metrics based on the continuous activity values of compounds with known activities. Then, a semi-supervised feature selection method was proposed based on the combination of Fisher and Laplacian criteria which exploits both compounds with known and unknown activities to select the relevant descriptors. To demonstrate the efficiency of the proposed semi-supervised feature selection method in selecting the relevant descriptors, we applied the method and other feature selection methods on three QSAR data sets such as serine/threonine-protein kinase PLK3 inhibitors, ROCK inhibitors and phenol compounds. The results demonstrated that the QSAR models built on the selected descriptors by the proposed semi-supervised method have better performance than other models. This indicates the efficiency of the proposed method in selecting the relevant descriptors using the compounds with known and unknown activities. The results of this study showed that the compounds with known and unknown activities can be helpful to improve the performance of the combined Fisher and Laplacian based feature selection methods.

An Adaptive Semisupervised Feature Analysis for Video Semantic Recognition.

Video semantic recognition usually suffers from the curse of dimensionality and the absence of enough high-quality labeled instances, thus semisupervised feature selection gains increasing attentions for its efficiency and comprehensibility. Most of the previous methods assume that videos with close distance (neighbors) have similar labels and characterize the intrinsic local structure through a predetermined graph of both labeled and unlabeled data. However, besides the parameter tuning problem underlying the construction of the graph, the affinity measurement in the original feature space usually suffers from the curse of dimensionality. Additionally, the predetermined graph separates itself from the procedure of feature selection, which might lead to downgraded performance for video semantic recognition. In this paper, we exploit a novel semisupervised feature selection method from a new perspective. The primary assumption underlying our model is that the instances with similar labels should have a larger probability of being neighbors. Instead of using a predetermined similarity graph, we incorporate the exploration of the local structure into the procedure of joint feature selection so as to learn the optimal graph simultaneously. Moreover, an adaptive loss function is exploited to measure the label fitness, which significantly enhances model's robustness to videos with a small or substantial loss. We propose an efficient alternating optimization algorithm to solve the proposed challenging problem, together with analyses on its convergence and computational complexity in theory. Finally, extensive experimental results on benchmark datasets illustrate the effectiveness and superiority of the proposed approach on video semantic recognition related tasks.