Unsupervised Feature Selection Algorithm Research Articles

A New Fast Filter-based Unsupervised Feature Selection Algorithm Using Cumulative and Shannon Entropy

A New Fast Filter-based Unsupervised Feature Selection Algorithm Using Cumulative and Shannon Entropy

Improve the prediction in the digital Era: Causal feature selection with minimum redundancy

Effective decision-making in complex environments requires discerning the relevant from the irrelevant, a challenge that becomes pronounced with large multivariate time-series data. However, existing feature selection algorithms often suffer from complexity and a lack of interpretability, making it difficult for decision-makers to extract value, manage risks, and adhere to compliance regulations in a thoroughly explainable way. To address these challenges, we propose a novel causality-based feature selection technique that embeds an explainable unsupervised feature selection algorithm. We refer to our proposed method as Causal Feature Selection with Minimum Redundancy (CFSMR). Our method yields a minimum viable feature set without compromising model performance while ensuring interpretability. We conduct an experimental study to compare the proposed technique with conventional feature selection techniques. Our results demonstrate that our proposed method outperforms or performs on par with existing techniques, making it a promising approach for decision-makers seeking an effective and interpretable feature selection method.

Unsupervised Feature Selection With Flexible Optimal Graph.

In the unsupervised feature selection method based on spectral analysis, constructing a similarity matrix is a very important part. In existing methods, the linear low-dimensional projection used in the process of constructing the similarity matrix is too hard, it is very challenging to construct a reliable similarity matrix. To this end, we propose a method to construct a flexible optimal graph. Based on this, we propose an unsupervised feature selection method named unsupervised feature selection with flexible optimal graph and l2,1 -norm regularization (FOG-R). Unlike other methods that use linear projection to approximate the low-dimensional manifold of the original data when constructing a similarity matrix, FOG-R can learn a flexible optimal graph, and by combining flexible optimal graph learning and feature selection into a unified framework to get an adaptive similarity matrix. In addition, an iterative algorithm with a strict convergence proof is proposed to solve FOG-R. l2,1 -norm regularization will introduce an additional regularization parameter, which will cause parameter-tuning trouble. Therefore, we propose another unsupervised feature selection method, that is, unsupervised feature selection with a flexible optimal graph and l2,0 -norm constraint (FOG-C), which can avoid tuning additional parameters and obtain a more sparse projection matrix. Most critically, we propose an effective iterative algorithm that can solve FOG-C globally with strict convergence proof. Comparative experiments conducted on 12 public datasets show that FOG-R and FOG-C perform better than the other nine state-of-the-art unsupervised feature selection algorithms.

A novel scheme based on modified hierarchical time-shift multi-scale amplitude-aware permutation entropy for rolling bearing condition assessment and fault recognition

To avoid production interruptions and equipment damage caused by rolling bearing failure, this study presents a novel diagnosis scheme applicable to both condition monitoring and fault recognition. In detail, to fully exploit feature information, a modified Hierarchical Time-Shift Multi-scale Amplitude-Aware Permutation Entropy (MHTSMAAPE) is put forward to map the raw vibration signal into a high-dimensional vector space of features. This method is developed from the AAPE algorithm and integrates the time-shift procedure and modified hierarchical analysis, extracting more feature details while ensuring entropy stability. Afterwards, the averaged value of the feature vector is regarded as an indicator to assess the condition of rolling bearing, this indicator exceeds a specific value indicates the rolling bearing steps into faulty. Then, the Unsupervised Discriminative Feature Selection (UDFS) algorithm is first introduced to directly pick fault-sensitive features from the vector space, relieving the calculation burden of the fault recognition task. The fault identification algorithm is based on Harris Hawks Optimization-Joint Opposite Selection optimized for Support Vector Machine (HHO-JOS-SVM) to improve classification accuracy. Finally, the proposed scheme fused by MHTSMAAPE, UDFS, and HHO-JOS-SVM is validated in two different cases with four common evaluation metrics, respectively. The experimental results demonstrate its reliability and robustness in rolling bearing condition assessment and fault recognition.

PFA-Nipals: An Unsupervised Principal Feature Selection Based on Nonlinear Estimation by Iterative Partial Least Squares

Unsupervised feature selection (UFS) has received great interest in various areas of research that require dimensionality reduction, including machine learning, data mining, and statistical analysis. However, UFS algorithms are known to perform poorly on datasets with missing data, exhibiting a significant computational load and learning bias. In this work, we propose a novel and robust UFS method, designated PFA-Nipals, that works with missing data without the need for deletion or imputation. This is achieved by considering an iterative nonlinear estimation of principal components by partial least squares, while the relevant features are selected through minibatch K-means clustering. The proposed method is successfully applied to select the relevant features of a robust health dataset with missing data, outperforming other UFS methods in terms of computational load and learning bias. Furthermore, the proposed method is capable of finding a consistent set of relevant features without biasing the explained variability, even under increasing missing data. Finally, it is expected that the proposed method could be used in several areas, such as machine learning and big data with applications in different areas of the medical and engineering sciences.

CGUFS: A clustering-guided unsupervised feature selection algorithm for gene expression data

(Aim) Gene expression data is typically high dimensional with a limited number of samples and contain many features that are unrelated to the disease of interest. Existing unsupervised feature selection algorithms primarily focus on the significance of features in maintaining the data structure while not taking into account the redundancy among features. Determining the appropriate number of significant features is another challenge. (Method) In this paper, we propose a clustering-guided unsupervised feature selection (CGUFS) algorithm for gene expression data that addresses these problems. Our proposed algorithm introduces three improvements over existing algorithms. For the problem that existing clustering algorithms require artificially specifying the number of clusters, we propose an adaptive k-value strategy to assign appropriate pseudo-labels to each sample by iteratively updating a change function. For the problem that existing algorithms fail to consider the redundancy among features, we propose a feature grouping strategy to group highly redundant features. For the problem that the existing algorithms cannot filter the redundant features, we propose an adaptive filtering strategy to determine the feature combinations to be retained by calculating the potentially effective features and potentially redundant features of each feature group. (Result) Experimental results show that the average accuracy (ACC) and matthews correlation coefficient (MCC) indexes of the C4.5 classifier on the optimal features selected by the CGUFS algorithm reach 74.37% and 63.84%, respectively, significantly superior to the existing algorithms. (Conclusion) Similarly, the average ACC and MCC indexes of the Adaboost classifier on the optimal features selected by the CGUFS algorithm are significantly superior to the existing algorithms. In addition, statistical experiment results show significant differences between the CGUFS algorithm and the existing algorithms.

A Possibilistic Information Fusion-Based Unsupervised Feature Selection Method Using Information Quality Measures

The main goal of most information quality (IQ)-based measures is to combine data provided by multiple information sources to enhance the quality of information essential for decision makers to perform their tasks. However, there is few work to fuse multi-source information from the perspective of possibility distribution (PD) and use IQ as the evaluation criteria for feature selection. The PD is one of important concepts in possibility theory, which is a generally acknowledged method for describing a kind of uncertain knowledge. In this paper, we propose a novel representation model of possibility distributions based on fuzzy multisets, namely, a possibility distribution information system (PDIS). Then, several IQ measures are defined in the PDIS, including Gini entropy, compatibility, conflict, credibility and separability degrees. In view of this, a minimal-separability-minimal-uncertainty-based unsupervised feature selection algorithm (UmSMU) is designed. The proposed UmSMU can sufficiently fuse multiple possibilistic information. Meanwhile, the selected features maintain as much information as possible while minimizing the uncertainty of information. The experimental results show that the proposed algorithm performs well, especially when it comes to selecting fewer features and improving performance.

Unsupervised feature selection algorithm based on redundancy learning and sparse regression

In recent years, feature selection methods based on sparse regression have attracted much attention from researchers, and how to select more representative feature is the key point. In this paper, an unsupervised feature selection method based on redundancy learning and sparse regression(RSUFS) is proposed. Firstly, to make the model robust to outliers, this paper uses the l2,1-norm regression model as the loss function to learn the feature weight matrix. Secondly, in order to get exact k top features, l2,0-norm constraint is introduced. At the same time, the cosine similarity between features is taken into account to select more valuable features by reducing the redundancy between features. Finally, an efficient algorithm based on Augmented Lagrangian method is derived to solve the above optimization problem. Comparison experiments are made with some benchmark datasets and seven well-known unsupervised feature selection algorithms and the results show that the given algorithm is effective.

Sparse robust subspace learning via boolean weight

Unsupervised feature selection is a hot topic in the field of machine learning, which is more convenient and important because it does not require labeled data. Currently, unsupervised feature selection algorithms based on ℓ2,1-norm are very mature and have certain robustness to outliers, but they still have some problems that cannot be ignored. For example, the sparsity of ℓ2,1-norm is not ideal, they cannot achieve row sparsity but only element sparsity. Second, fine-tuning meaningless regularization parameters increases cost and makes it easy to fall into suboptimal solutions. To release the above problems, we propose an unsupervised feature selection algorithm via Sparse Robust Subspace Learning (SRSL), which combines reconstruction term and variance term so that the model simultaneously preserves reconstruction information and enhances separability. What is more, using the ℓ2,0-norm constraint on transformation matrix makes our model have a row-sparse property. In addition, we design the boolean weight so that the model not only eliminates outliers fundamentally to enhance robustness, but also achieves the effect of anomaly detection. To solve this NP-hard problem, we carefully design an optimization algorithm, which has strict convergence guarantees and obtains a closed-form solution. Experimental results on several real-world datasets demonstrate that our algorithm outperforms other comparison algorithms in both clustering and anomaly detection applications.

Compactness score: a fast filter method for unsupervised feature selection

The rapid development of big data era incurs the generation of huge amount of data day by day in various fields. Due to the large-scale and high-dimensional characteristics of these data, it is often difficult to achieve better decision-making in practical applications. Therefore, an efficient big data analytical method is urgently necessary. For feature engineering, feature selection seems to be an important research topic which is anticipated to select “excellent” features from candidate ones. The implementation of feature selection can not only achieve the purpose of dimensionality reduction, but also improve the computational efficiency and result performance of the model. In many classification tasks, researchers found that data seem to be usually close to each other if they are from the same class; thus, local compactness is of great importance for the evaluation of a feature. Based on this discovery, we propose a fast unsupervised feature selection algorithm, named Compactness Score (CSUFS), to select desired features. To prove the superiority of the proposed algorithm, several public data sets are considered with extensive experiments being performed. The experiments are presented by applying feature subsets selected through several different algorithms to the clustering task. The performance of clustering tasks is indicated by two well-known evaluation metrics, while the efficiency is reflected by the corresponding running time. As demonstrated, our proposed algorithm is more accurate and efficient compared with existing ones.

Adaptive graph regularization and self-expression for noise-aware feature selection

Many traditional unsupervised feature selection algorithms utilize manifold information to mine the local structure of the data. However, the noise existing in the raw data reduces the accuracy of the manifold information of data, which affects the learning effect of the entire algorithm. In order to solve the above problems and more fully find the internal structure inside the data, this paper proposes an adaptive graph regularization and self-expression for noise-aware feature selection (ASNFS). Firstly, the algorithm adopts non-negative matrix factorization to decompose the raw data matrix, and utilizes the low-dimensional matrix generated after decomposition to replace the raw high-dimensional data matrix. This allows the algorithm to reveal some internal structural information of the raw data while reducing the dimensionality of the data. ASNFS also introduces the orthogonal basis clustering with excellent clustering effect, and the interpretability of the algorithm is enhanced. Secondly, in addition to preserving the manifold information in the low-dimensional projection subspace, the algorithm also preserves the manifold information in the non-negative matrix factorization subspace. Meanwhile, the adaptive graph regularization term added to the objective function enables the algorithm to continuously update the similarity matrix. It can effectively remove the noise inside the raw data, and prevent the occurrence of over-fitting phenomenon of the experimental results caused by the fixed similarity matrix. Finally, the similarity matrix retains the local structure information of the data with each iteration, and the results of the feature selection are reused for the construction of the similarity matrix. ASNFS adopts an alternate iterative method to optimize the objective function, which is simple and effective. Then, the algorithm complexity and convergence are analyzed. ASNFS is compared with seven feature selection algorithms on nine datasets, and the experimental results reflect the effectiveness of ASNFS in feature selection.

A unified view of feature selection based on Hilbert-Schmidt independence criterion

Feature selection is a challenging and increasing important task in the machine learning and data mining community. According to different learning scenarios such as the availability of supervision information, feature selection can be broadly categorized as supervised, unsupervised and semi-supervised methods. Traditionally, feature selection algorithms are scenario-specific, i.e., different types of feature selection algorithms were proposed and investigated separately for specific learning scenario. In this paper, we aim to develop a unified view of supervised, unsupervised and semi-supervised feature selection methods. Based on the Hilbert-Schmidt independence criterion (HSIC), we first show that supervised, unsupervised and semi-supervised feature selection methods share the same objective. In discussing the connection between the unified framework and HSIC least absolute shrinkage and selection operator (HSIC Lasso), we find that the proposed framework not only has a clear statistical interpretation that minimum redundant features with maximum dependence on the target output variable can be found in terms of the HSIC, but also enables the global optimal solution to be computed efficiently by solving a Lasso optimization problem. Under this unified view, a new unsupervised feature selection algorithm is also proposed and demonstrated with several benchmark examples.

Soft-label guided non-negative matrix factorization for unsupervised feature selection

Unsupervised feature selection (UFS) aims to select the most representative features from the original data, which can efficiently reduce the influence of redundancy, outliers and noises. Over the past decades, various UFS algorithms have been proposed. However, these methods often do not consider the necessity of sparsity or ignore the fuzziness of the data. To tackle these shortcomings, in this paper, a novel soft-label guided non-negative matrix factorization (SLNMF) method is proposed. Specifically, both the convex NMF and ℓ2,1−norm regularization are introduced to ensure the sparsity of the feature selection matrix. Furthermore, the soft-label matrix based on local distance is used to supervise the feature selection, and a linear regression is developed to find the correlation between the low-dimensional representation and the soft-label space. Finally, extensive experiments on several benchmark datasets are conducted. The results show that the proposed method is advanced over several state-of-the-art UFS methods.

Unsupervised spectral feature selection algorithms for high dimensional data

Unsupervised spectral feature selection algorithms for high dimensional data

Unsupervised feature selection through combining graph learning and [formula omitted]-norm constraint

Graph-based unsupervised feature selection algorithms have been shown to be promising for handling unlabeled and high-dimensional data. Whereas, the vast majority of those algorithms usually involve two independent processes, i.e., similarity matrix construction and feature selection. This incurs a poor similarity matrix that is obtained from original data, which retains constant for the following feature selection process and heavily affects the corresponding performance. Aiming to integrate these two processes into a unified framework, this paper proposes a novel unsupervised feature selection algorithm, named Graph Learning Unsupervised Feature Selection (GLUFS) to ensure the two processes proceed simultaneously. In particular, a new similarity matrix is derived from the original one, while the new matrix can adaptively maintain the manifold structure of data. Due to the fact that good individual features do not necessarily guarantee efficient combinations, the GLUFS algorithm adopts the ℓ2,0-norm sparsity constraint to achieve group feature selection. Eventually, we perform experiments on six public datasets with sufficient analysis, while the obtained results illustrate the effectiveness and superiority of our GLUFS over the considered algorithms.

Unsupervised Feature Selection Using Iterative Shrinking and Expansion Algorithm

In this article, we propose an unsupervised feature selection algorithm based on the recently developed shrinking and expansion algorithm (SEA). The SEA is already an established, efficient algorithm to identify dense subgraphs within a weighted graph. To conduct SEA on a weighted graph, the dataset is first mapped onto an equivalent graph notation where each vertex corresponds to a feature, and the weight of each edge represents the normalized mutual information between the associated two features that form the edge. As a result, the feature selection problem is presented as the best way to obtain several dense sub-feature spaces by using the SEA on a weighted feature graph. The proposed feature selection algorithm consists of a two-stage structure, in which the first stage is utilized to find a number of dense feature subgraphs, whereas the second stage is used to identify a representative feature from each of these feature subgraphs. The main advantage of the proposed algorithm is that the users are not required to provide the number of features to be selected, which is a major flaw in most of the existing algorithms. The current findings from the researchers' method are better than the other state-of-the-art algorithms because the feature selection process identifies approximately one-fifth to one-third of the number of original features while providing a high accuracy score. The superiority of the proposed algorithm over the other conventional methods of feature selection is established for a number of real-life datasets. In addition, the performance of the proposed feature selection algorithm is also found to be better than other algorithms in one of the innovative application areas of computational intelligence such as ambient intelligence.

Unsupervised feature selection via discrete spectral clustering and feature weights

Most of the existing unsupervised feature selection methods learn the cluster structure through spectral clustering, and then use various regression models to introduce the data matrix into the indicator matrix to obtain feature selection matrix. In these methods, the clustering indicator matrix is usually continuous value, which is not the best choice for the matrix in terms of its supervising role in feature selection. Based on this, unsupervised feature selection via discrete spectral clustering and feature weights (FSDSC) is proposed in this paper. First, FSDSC integrates regression model and spectral clustering in a unified framework for feature selection, and introduces a feature weight matrix, which intuitively expresses the importance of each feature with its diagonal elements. Compared with the common feature selection matrix that requires constraints such as sparse regular items, the appearance of the feature weight matrix reduces the complexity of the model and simplifies the calculation process of feature evaluation. Secondly, for the value of the indicators matrix, the spectral clustering is improved to obtain a discrete clustering indicator matrix, which provides clearer guidance information for feature selection. Finally, in order to avoid trivial solutions, the transformation matrix is constrained by orthogonal constraint. The combination of the orthogonal regression model and spectral clustering enables the algorithm to perform feature selection and manifold information learning at the same time, thereby preserving the local geometric structure of data. Compared with other excellent unsupervised feature selection algorithms, the experimental results prove the effectiveness of the proposed algorithm.

Robust unsupervised feature selection via sparse and minimum-redundant subspace learning with dual regularization

Unsupervised feature selection, one of the important dimensionality reduction methods for high-dimensional data, has been paid more and more attention by researchers due to its effectiveness of processing the data without labels. However, most unsupervised feature selection methods do not take into account the fact that real data tends to be mixed with noise or outliers, or only consider the sensitivity of the model to them. In addition, the local manifold structure of data is usually preserved by a simple graph based on fixed k nearest neighbors. On the one hand, the different sample distributions of different data are considered without distinction. On the other hand, simple graph can not well capture the inherent relations between samples when facing multi-modal data. In view of the above problems, a robust unsupervised feature selection algorithm, sparse and minimum-redundant subspace learning with dual regularization (SMRSDR), is proposed in this paper. Specifically, a self-paced learning regularization based on soft weight allocation is introduced into SMRSDR to strictly control the samples’ entry into model learning. Normal samples are allowed to enter model learning in order of their importance, while noise or outliers are strictly blocked out of model learning. Besides, considering the difference in sample distribution and the multimodal nature of data, the traditional simple graph Laplacian regularization is replaced by the hypergraph Laplacian regularization based on adaptive nearest neighbors selection. Furthermore, the sparsity and minimum-redundancy constraints are imposed on the subspace learning framework so that the most representative and minimum-redundant features can be selected. Finally, the objective function of SMRSDR is solved by an alternating iterative optimization algorithm. And a series of experiments are performed to comprehensively prove the effectiveness and superiority of SMRSDR.

A multiple association-based unsupervised feature selection algorithm for mixed data sets

Companies have an increasing access to very large datasets within their domain. Analysing these datasets often requires the application of feature selection techniques in order to reduce the dimensionality of the data and prioritize features for downstream knowledge generation tasks. Effective feature selection is a key part of clustering, regression and classification. It presents a myriad of opportunities to improve the machine learning pipeline: eliminating redundant and irrelevant features, reducing model over-fitting, faster model training times and more explainable models. By contrast, and despite the widespread availability and use of categorical data in practice, feature selection for categorical and/or mixed data has received relatively little attention in comparison to numerical data. Furthermore, existing feature selection methods for mixed data are sensitive to number of objects by having nonlinear time complexities with respect to number of objects. In this work, we propose a generic multiple association measure for mixed datasets and a novel feature selection algorithm that uses multiple association across features. Our algorithm is based upon the belief that the most representative chosen set of features should be as diverse and minimally dependent on each other as possible. The proposed algorithm formulates the problem of feature selection as an optimization problem, searching for the set of features that have minimum association amongst them. We present a generic multiple association measure and two associated feature selection algorithms: Naive and Greedy Feature Selection Algorithms called NFSA and GFSA, respectively. Our proposed GFSA algorithm is evaluated on 15 benchmark datasets, and compared to four existing state of the art feature selection techniques. We demonstrate that our approach provides comparable downstream classification performance outperforming other leading techniques on several datasets. Both time complexity analysis and experimental results show that our proposed algorithm significantly reduces the processing time required for unsupervised feature selection algorithms especially for long datasets which have a huge number of objects, whilst also yielding comparable clustering and classification performance. On the other hand, we do not recommend our approach for wide datasets where the number of features is huge with respect to the number of objects e.g., image, text and genome datasets.

A Novel Unsupervised Approach to Heterogeneous Feature Selection Based on Fuzzy Mutual Information

Aiming at the problem of effectively selecting relevant features from heterogeneous data without decision, a novel feature selection approach is studied based on fuzzy mutual information in fuzzy rough set theory. First, the fuzzy relevance of each feature is defined by using fuzzy mutual information, and then, the fuzzy conditional relevance is further given. Next, the fuzzy redundancy is defined by using the difference between the fuzzy relevance and the fuzzy conditional relevance. Thereby, the evaluation index of the feature importance is obtained by using the idea of unsupervised minimum redundancy and maximum relevance. Finally, a fuzzy-mutual-information-based unsupervised feature selection algorithm is designed to select feature sequences. Extensive experiments are conducted on public datasets, and six unsupervised feature selection algorithms are compared. The selected features are evaluated by classification, clustering, and outlier detection methods. Experimental results show that the proposed algorithm can select fewer heterogeneous features to maintain or improve the performance of learning algorithms.