Unsupervised Feature Selection Method Research Articles

Dual-level feature assessment for unsupervised multi-view feature selection with latent space learning

Recently, numerous unsupervised multi-view feature selection methods have been presented. However, these methods assess the significance of data features in each view individually or jointly evaluate the significance of data features across multiple views by concatenating data features from all data views. So, they disregard either inter-view or intra-view feature connections of data. Moreover, they ignore the complementary information from multiple insufficient views, each of which only captures partial information of multi-view data and is insufficient to characterize data distribution. To address these problems, this paper proposes a model named Dual-level Feature Assessment for Unsupervised Multi-view Feature Selection with Latent Space Learning (DFA-LSL). It first investigates the underlying complementary information from multiple views in a latent space while simultaneously learning latent representations for data. Compared with the original data representations, the latent representations contain the entire information of multi-view data. Then, it explores connections and distinguishing ability of features at two levels, namely the inter-view and intra-view levels, by jointly assessing the significance of features in the latent space and individually assessing the significance of features in each view concurrently. Following that, an effective optimization algorithm is presented. Extensive experiments demonstrate that the proposed work surpasses several state-of-the-art models.

DELVE: feature selection for preserving biological trajectories in single-cell data

Single-cell technologies can measure the expression of thousands of molecular features in individual cells undergoing dynamic biological processes. While examining cells along a computationally-ordered pseudotime trajectory can reveal how changes in gene or protein expression impact cell fate, identifying such dynamic features is challenging due to the inherent noise in single-cell data. Here, we present DELVE, an unsupervised feature selection method for identifying a representative subset of molecular features which robustly recapitulate cellular trajectories. In contrast to previous work, DELVE uses a bottom-up approach to mitigate the effects of confounding sources of variation, and instead models cell states from dynamic gene or protein modules based on core regulatory complexes. Using simulations, single-cell RNA sequencing, and iterative immunofluorescence imaging data in the context of cell cycle and cellular differentiation, we demonstrate how DELVE selects features that better define cell-types and cell-type transitions. DELVE is available as an open-source python package: https://github.com/jranek/delve.

Boosted unsupervised feature selection for tumor gene expression profiles

AbstractIn an unsupervised scenario, it is challenging but essential to eliminate noise and redundant features for tumour gene expression profiles. However, the current unsupervised feature selection methods treat all samples equally, which tend to learn discriminative features from simple samples. Besides, it is difficult for these methods to guide the feature selection process by multiple feature subspaces comprehensively. In order to address these problems, a boosted unsupervised feature selection method (BoostUFS) is proposed for tumour gene expression profiles. Specifically, the authors design a boosting scheme to sequentially learn multiple compressed feature subspaces by focusing on ambiguous samples. The uncertainty of samples and the confidence of feature subspaces can be evaluated adaptively by minimising the overall loss of feature subspaces learning. Furthermore, we provide a consensus objective function with L2,1‐norm regularisation to combine these weighted feature subspaces and select discriminative features. Extensive experiments on several real‐world datasets of tumour gene expression profiles are carried out to demonstrate the effectiveness of the proposed method.

Selective Deep Autoencoder for Unsupervised Feature Selection

In light of the advances in big data, high-dimensional datasets are often encountered. Incorporating them into data-driven models can enhance performance; however, this comes at the cost of high computation and the risk of overfitting, particularly due to abundant redundant features. Identifying an informative subset of the features helps in reducing the dimensionality and enhancing model interpretability. In this paper, we propose a novel framework for unsupervised feature selection, called Selective Deep Auto-Encoder (SDAE). It aims to reduce the number of features used in unlabeled datasets without compromising the quality of information obtained. It achieves this by selecting sufficient features - from the original feature set - capable of representing the entire feature space and reconstructing them. Architecturally, it leverages the use of highly nonlinear latent representations in deep Autoencoders and intrinsically learns, in an unsupervised fashion, the relevant and globally representative subset of features through a customized Selective Layer. Extensive experimental results on three high-dimensional public datasets have shown promising feature selection performance by SDAE in comparison to other existing state-of-the-art unsupervised feature selection methods.

Unsupervised Feature Selection With Flexible Optimal Graph.

In the unsupervised feature selection method based on spectral analysis, constructing a similarity matrix is a very important part. In existing methods, the linear low-dimensional projection used in the process of constructing the similarity matrix is too hard, it is very challenging to construct a reliable similarity matrix. To this end, we propose a method to construct a flexible optimal graph. Based on this, we propose an unsupervised feature selection method named unsupervised feature selection with flexible optimal graph and l2,1 -norm regularization (FOG-R). Unlike other methods that use linear projection to approximate the low-dimensional manifold of the original data when constructing a similarity matrix, FOG-R can learn a flexible optimal graph, and by combining flexible optimal graph learning and feature selection into a unified framework to get an adaptive similarity matrix. In addition, an iterative algorithm with a strict convergence proof is proposed to solve FOG-R. l2,1 -norm regularization will introduce an additional regularization parameter, which will cause parameter-tuning trouble. Therefore, we propose another unsupervised feature selection method, that is, unsupervised feature selection with a flexible optimal graph and l2,0 -norm constraint (FOG-C), which can avoid tuning additional parameters and obtain a more sparse projection matrix. Most critically, we propose an effective iterative algorithm that can solve FOG-C globally with strict convergence proof. Comparative experiments conducted on 12 public datasets show that FOG-R and FOG-C perform better than the other nine state-of-the-art unsupervised feature selection algorithms.

Unsupervised feature selection via dual space-based low redundancy scores and extended OLSDA

Spectral clustering is a widely used method for unsupervised feature selection (UFS) to generate pseudo labels. Nonetheless, it is acknowledged that graph algorithms suffer from issues such as redundancy and the dissatisfaction of connectivity, which greatly affect the quality of the learning local manifold. Moreover, existing UFS methods usually ignore the linear dependency among features and the role of non-negative semantics of the predicted labels. Hence, a novel algorithm using dual space-based low redundancy scores and extended orthogonal least square discriminant analysis (OLSDA), abbreviated as DLSEO, is proposed in this paper. Specifically, extended OLSDA is employed to derive non-negative clustering labels and a manifold structure, which avoids explicitly constructing a Laplacian graph. The dual space-based low redundancy scores related to mutual information eliminate redundant data and features, preventing their interference in the feature selection process. Moreover, ℓ2,1−2-norm is introduced to simultaneously guarantee the feature weight matrix's sparsity and ensure the selection of salient features. The results of experiments conducted on twelve benchmark datasets compared with several relevant methods show the superiority of DLSEO.

Robust autoencoder feature selector for unsupervised feature selection

Unsupervised Feature Selection (UFS) methods are known to produce models with excellent ability to select high-quality features. This advantage, however, is challenged when analyzing noisy real-world data, where anomalies are prevalent. For example, there may be a feature (anomalous feature) that is corrupted across many samples or a sample (anomalous sample) that has more corruptions than its peers. Previous literature focused on addressing anomalous samples, and methods that are robust to both types of anomalies have been under-explored. This paper proposes a novel general framework for reconstruction-based UFS methods, which can be embedded into the feature learning process to simultaneously remove anomalous samples and features. Specifically, the framework learns double binary weight vectors to assign 0 weights to samples or features with the highest reconstruction errors and 1 weights to the others when computing reconstruction errors. By discarding the 0-weighted samples and features when updating the model parameters, the anomalies in the data are excluded. This allows the model to focus more on learning from the clean part of the noisy data. Our proposed framework is then integrated with AutoEncoder Feature Selector (AEFS [10]) to develop a new method, which jointly performs anomaly removal and feature selection. The experimental results demonstrate the effectiveness of the proposed framework. Particularly, processing both types of anomalies provides better robustness than processing only one type. Moreover, our proposed method outperforms several state-of-the-art methods on various real-world datasets.

Joint Anchor Graph Embedding and Discrete Feature Scoring for Unsupervised Feature Selection.

The success of existing unsupervised feature selection (UFS) methods heavily relies on the assumption that the intrinsic relationships among original high-dimensional (HD) data samples exist in the discriminative low-dimension (LD) subspace. However, previous UFS methods commonly construct pairwise graphs and employ l2,1 -norm regularization to severally preserve the local structure and calculate the score of features, which is computationally complex and easy to get stuck into local optimum, so that those approaches cannot be applied in dealing with large-scale datasets in practice. To overcome this challenge, we propose a novel UFS method, in which a novel anchor graph embedding paradigm is designed to extract the local neighborhood relationships among data samples by reducing the computational complexity of graph construction to be linear in the number of data. Moreover, to improve the optimality of selected features as well as the performance of downstream tasks, we propose a discrete feature scoring mechanism, which imposes orthogonal l2,0 -norm constraints on learned projections, in order to enhance the distinction of feature scores as well as reduce the probability of falling into local optimum. In addition, solving the proposed nonconvex and nonsmooth NP-hard problem is challenging, and we present an efficient optimization algorithm to address it and acquire a closed-form solution of the transformation matrix. Extensive experiments demonstrate the effectiveness and efficiency of the proposed UFS by comparison with several state-of-the-art approaches to clustering and image segmentation tasks.

Unsupervised Feature Selection via Controllable Adaptive Graph Learning and Discriminative Feature Learning.

Unsupervised feature selection is challenging in machine learning, pattern recognition, and data mining. The crucial difficulty is to learn a moderate subspace that preserves the intrinsic structure and to find uncorrelated or independent features simultaneously. The most common solution is first to project the original data into a lower dimensional space and then force them to preserve the similar intrinsic structure under linear uncorrelation constraint. However, there are three shortcomings. First, the final graph generated by the iterative learning process differs significantly from the initial graph in which the original intrinsic structure is embedded. Second, it requires prior knowledge about a moderate dimension of subspace. Third, it is inefficient when dealing with high-dimensional datasets. The first shortcoming, which is longstanding and undiscovered, makes the previous methods fail to achieve their expected results. The last two ones increase the difficulty of applying in different fields. Therefore, two unsupervised feature selection methods are proposed based on controllable adaptive graph learning and uncorrelated/independent feature learning (CAG-U and CAG-I) to address the abovementioned issues. In the proposed methods, the final graph that preserves intrinsic structure can be adaptively learned while the difference between the two graphs can be well controlled. Besides, relatively uncorrelated/independent features can be selected using a discrete projection matrix. The experimental results on 12 datasets in different fields show the superiority of CAG-U and CAG-I.

Double-Structured Sparsity Guided Flexible Embedding Learning for Unsupervised Feature Selection.

In this article, we propose a novel unsupervised feature selection model combined with clustering, named double-structured sparsity guided flexible embedding learning (DSFEL) for unsupervised feature selection. DSFEL includes a module for learning a block-diagonal structural sparse graph that represents the clustering structure and another module for learning a completely row-sparse projection matrix using the l2,0 -norm constraint to select distinctive features. Compared with the commonly used l2,1 -norm regularization term, the l2,0 -norm constraint can avoid the drawbacks of sparsity limitation and parameter tuning. The optimization of the l2,0 -norm constraint problem, which is a nonconvex and nonsmooth problem, is a formidable challenge, and previous optimization algorithms have only been able to provide approximate solutions. In order to address this issue, this article proposes an efficient optimization strategy that yields a closed-form solution. Eventually, through comprehensive experimentation on nine real-world datasets, it is demonstrated that the proposed method outperforms existing state-of-the-art unsupervised feature selection methods.

Unsupervised Discriminative Feature Selection via Contrastive Graph Learning.

Due to many unmarked data, there has been tremendous interest in developing unsupervised feature selection methods, among which graph-guided feature selection is one of the most representative techniques. However, the existing feature selection methods have the following limitations: (1) All of them only remove redundant features shared by all classes and neglect the class-specific properties; thus, the selected features cannot well characterize the discriminative structure of the data. (2) The existing methods only consider the relationship between the data and the corresponding neighbor points by Euclidean distance while neglecting the differences with other samples. Thus, existing methods cannot encode discriminative information well. (3) They adaptively learn the graph in the original or embedding space. Thus, the learned graph cannot characterize the data's cluster structure. To solve these limitations, we present a novel unsupervised discriminative feature selection via contrastive graph learning, which integrates feature selection and graph learning into a uniform framework. Specifically, our model adaptively learns the affinity matrix, which helps characterize the data's intrinsic and cluster structures in the original space and the contrastive learning. We minimize l1,2 -norm regularization on the projection matrix to preserve class-specific features and remove redundant features shared by all classes. Thus, the selected features encode discriminative information well and characterize the discriminative structure of the data. Generous experiments indicate that our proposed model has state-of-the-art performance.

Pseudo-Label Guided Structural Discriminative Subspace Learning for Unsupervised Feature Selection.

In this article, we propose a new unsupervised feature selection method named pseudo-label guided structural discriminative subspace learning (PSDSL). Unlike the previous methods that perform the two stages independently, it introduces the construction of probability graph into the feature selection learning process as a unified general framework, and therefore the probability graph can be learned adaptively. Moreover, we design a pseudo-label guided learning mechanism, and combine the graph-based method and the idea of maximizing the between-class scatter matrix with the trace ratio to construct an objective function that can improve the discrimination of the selected features. Besides, the main existing strategies of selecting features are to employ l2,1 -norm for feature selection, but this faces the challenges of sparsity limitations and parameter tuning. For addressing this issue, we employ the l2,0 -norm constraint on the learned subspace to ensure the row sparsity of the model and make the selected feature more stable. Effective optimization strategy is given to solve such NP-hard problem with the determination of parameters and complexity analysis in theory. Ultimately, extensive experiments conducted on nine real-world datasets and three biological ScRNA-seq genes datasets verify the effectiveness of the proposed method on the data clustering downstream task.

Efficient and Stable Unsupervised Feature Selection Based on Novel Structured Graph and Data Discrepancy Learning.

Unsupervised feature selection is an important tool in data mining, machine learning, and pattern recognition. Although data labels are often missing, the number of data classes can be known and exploited in many scenarios. Therefore, a structured graph, whose number of connected components is identical to the number of data classes, has been proposed and is frequently applied in unsupervised feature selection. However, methods based on the structured graph learning face two problems. First, their structured graphs are not always guaranteed to maintain the same number of connected components as the data classes with existing optimization algorithms. Second, they usually lack strategies for choosing moderate hyperparameters. To solve these problems, an efficient and stable unsupervised feature selection method based on a novel structured graph and data discrepancy learning (ESUFS) is proposed. Specifically, the novel structured graph, consisting of a pairwise data similarity matrix and an indicator matrix, can be efficiently learned by solving a discrete optimization problem. Data discrepancy learning focuses on features that maximize the difference among data and helps in selecting discriminative features. Extensive experiments conducted on various datasets show that ESUFS outperforms state-of-the-art methods not only in accuracy (ACC) but also in stability and speed.

Unsupervised feature selection with high-order similarity learning

Graph-based unsupervised feature selection methods have successfully processed high-dimensional data since they can effectively preserve data structure information. However, most existing methods are facing two challenges: (1) it is unreliable to exploit the intrinsic spatial structure from raw data because of the existence of redundant information and noise; (2) they only consider the first-order similar information but ignore the high-order similar information (i.e., samples with similar neighborhood network structure should be similar to each other). This study proposes an unsupervised feature selection with high-order similarity learning (HSL) to tackle the above problems. The projection matrix, first-order similar information, and high-order similar information are simultaneously learned in a unified framework, such that the intrinsic structure information can be exploited in the clean data that come from a latent low-dimensional embedding subspace. Specifically, we project raw data into a low-dimensional embedding subspace to effectively eliminate redundant information and noise. Moreover, we simultaneously consider the first-order similarity and the high-order similarity in the low-dimensional embedding space so that a better graph Laplacian matrix can be learned to fully preserve the intrinsic structure information of raw data and simultaneously utilize the label information. Furthermore, we design an effective optimization algorithm to tackle the proposed HSL. Comprehensive experiments on real-world datasets verify the superiority and effectiveness of HSL.

Unsupervised Feature Selection with Latent Relationship Penalty Term

With the exponential growth of high dimensional unlabeled data, unsupervised feature selection (UFS) has attracted considerable attention due to its excellent performance in machine learning. Existing UFS methods implicitly assigned the same attribute score to each sample, which disregarded the distinctiveness of features and weakened the clustering performance of UFS methods to some extent. To alleviate these issues, a novel UFS method is proposed, named unsupervised feature selection with latent relationship penalty term (LRPFS). Firstly, latent learning is innovatively designed by assigning explicitly an attribute score to each sample according to its unique importance in clustering results. With this strategy, the inevitable noise interference can be removed effectively while retaining the intrinsic structure of data samples. Secondly, an appropriate sparse model is incorporated into the penalty term to further optimize its roles as follows: (1) It imposes potential constraints on the feature matrix to guarantee the uniqueness of the solution. (2) The interconnection between data instances is established by a pairwise relationship situation. Extensive experiments on benchmark datasets demonstrate that the proposed method is superior to relevant state-of-the-art algorithms with an average improvement of 10.17% in terms of accuracy.

Filter unsupervised spectral feature selection method for mixed data based on a new feature correlation measure

In recent years, Unsupervised Feature Selection (UFS) methods have attracted considerable interest in different research areas due to their wide application in problems where unlabeled data appears. Nevertheless, few UFS methods can process datasets described by both numerical and non-numerical features (mixed data). This paper introduces a new filter UFS method based on a new feature correlation measure for mixed data to select a relevant and non-redundant feature subset. The proposed method addresses the feature selection problem in two stages through a strategy that combines Spectral Feature Selection to identify relevant features and a Pair-wise Redundancy Analysis to remove those features with a high correlation with others. According to our experiments, the proposed feature selection method based on the introduced feature correlation measure provides better quality results than previous filter UFS methods for mixed data reported in the literature.

Consistency–exclusivity guided unsupervised multi-view feature selection

Unsupervised multi-view feature selection (UMFS) is an effective dimension reduction for multi-view data. It aims to obtain the important feature subset from multi-view data, which can significantly minimize the impacts of noises, outliers and redundancy. Although previous UMFS methods achieve remarkable achievements, they fail to fully take into account the consistency or the exclusivity hidden in multi-view data. To address this, this article presents a consistency–exclusivity guided unsupervised multi-view feature selection (CE-UMFS) method. Specifically, we design a multi-view matrix factorization model to simultaneously explore the consistency and exclusivity in multi-view data. Meanwhile, we employ the Hilbert–Schmidt independence criterion (HSIC) to preserve the exclusivity of different views. Furthermore, we impose a nuclear norm on the consistent representation matrix to explore the consistency across views. At last, promising experimental results demonstrate the superiority of the proposed method compared with some state-of-the-art methods.

Unsupervised feature selection by learning exponential weights

Unsupervised feature selection has gained considerable attention for extracting valuable features from unlabeled datasets. Existing approaches typically rely on sparse mapping matrices to preserve local neighborhood structures. However, this strategy favors large-weight features, potentially overlooking smaller yet valuable ones and distorting data distribution and feature structure. Besides, some methods focus on local structure information, failing to explore global information. To address these limitations, we introduce an exponential weighting mechanism to induce a rational feature distribution and explore data structure in the feature subspace. Specifically, we propose a unified framework incorporating local structure learning and exponentially weighted sparse regression for optimal feature combinations, preserving global and local information. Experimental results demonstrate the superiority of our approach over existing unsupervised feature selection methods.

Low-Redundant Unsupervised Feature Selection based on Data Structure Learning and Feature Orthogonalization

An orthogonal representation of features can offer valuable insights into feature selection as it aims to find a representative subset of features in which all features can be accurately reconstructed by a set of features that are linearly independent, uncorrelated, and perpendicular to each other. In this paper, a novel feature selection method, called Low-Redundant Unsupervised Feature Selection based on Data Structure Learning and Feature Orthogonalization (LRDOR), is presented. In the first stage, the suggested LRDOR method makes use of the QR factorization over the whole set of features to find the orthogonal representation of the feature space. Then, LRDOR utilizes the directional distance based on the matrix factorization in order to determine the distance among the set of considered features and the orthogonal set obtained from the original features. Moreover, LRDOR simultaneously takes into account the local correlation of features and the data manifold as dual information into the feature selection process, which can lead to a low level of redundancy and maintain the geometric data structure when reducing the data dimension. In addition to providing a proficient iterative algorithm, the convergence analysis is also included to solve the objective function of LRDOR. The results of the experiments demonstrate that for clustering purposes, LRDOR works better than other related state-of-the-art unsupervised feature selection methods on ten real-world datasets.

Weak Monotonicity With Trend Analysis for Unsupervised Feature Evaluation.

Performance in an engineering system tends to degrade over time due to a variety of wearing or ageing processes. In supervisory controlled processes there are typically many signals being monitored that may help to characterize performance degradation. It is preferred to select the least amount of information to obtain high quality of predictive analysis from a large amount of collected data, in which labeling the data is not always feasible. To this end a novel unsupervised feature selection method, robust with respect to significant measurement disturbances, is proposed using the notion of ``weak monotonicity'' (WM). The robustness of this notion makes it very attractive to identify the common trend in the presence of measurement noises and population variation from the collected data. Based on WM, a novel suitability indicator is proposed to evaluate the performance of each feature. This new indicator is then used to select the key features that contribute to the WM of a family of processes when noises and variations among processes exist. In order to evaluate the performance of the proposed framework of the WM and suitability, a comparative study with other nine state-of-the-arts unsupervised feature evaluation and selection methods is carried out on well-known benchmark datasets. The results show a promising performance of the proposed framework on unsupervised feature evaluation in the presence of measurement noises and population variations.