Unsupervised Feature Selection Approach Research Articles

Precise feature selection via non-convex regularized graph embedding and self-representation for unsupervised learning

In graph embedding learning with unsupervised feature selection fields, similarity matrices are usually obtained only from the initial noise-laden samples, and the probable association between various features is often ignored. To tackle the issues described above, this article proposes an effective unsupervised group feature selection approach via non-convex regularized graph embedding and self-representation (NLGMS). NLGMS integrates the similarity matrix and feature selection as a joint system on the basis of graph. It combines self-representation learning into the system to preserve the global structure and learns the nonconvex projection matrix to direct the procedure of feature selection. Specifically, first, NLGMS presents a unified framework for global structure learning, local structure learning, and feature selection process so the data’s intrinsic structure is excellently captured and learned adaptively. Second, nonconvex ℓ2,0-norm is enforced upon the projection matrix to perform the process of feature selection, ensuring that the optimal feature subset can be picked precisely. Nonconvex ℓ2,0-norm constraint considers the probable association between various features so that it can deliver optimal performance by considering the feature subset as an integral whole, which other convex regularized constraints often fail to achieve. Finally, the introduction to self-representation learning can facilitate the reconstruction of samples in low-rank subspaces, strengthening the model’s robustness. To tackle this challenging nonconvex model, an innovative alternative optimization technique is exploited. In addition, to validate the superiority of NLGMS, exhaustive experiments are performed on one synthetic dataset, six benchmark datasets, and one fish image dataset. The source code is available at: https://github.com/hrbai/NLGMS.

A Novel Unsupervised Feature Selection for High-Dimensional Data Based on FCM and k -Nearest Neighbor Rough Sets.

Large amounts of high-dimensional unlabeled data typically contain only a small portion of truly effective information. Consequently, the issue of unsupervised feature selection methods has gained significant attention in research. However, current unsupervised feature selection approaches face limitations when dealing with datasets that exhibit uneven density, and they also require substantial computational time. To address this problem, this research article proposes a feature extraction technique that combines the Fuzzy C-Means (FCM) and k -nearest neighbor rough sets. FCM is a clustering algorithm grounded in fuzzy theory, which takes into account the inherent data structure and the correlations between different features. Consequently, FCM is particularly well-suited for datasets with uneven density. Our proposed method consists of three steps. First, the FCM algorithm is used to cluster the unlabeled data. Second, a measure that evaluates the importance of features is defined and sorted based on the clustering results. Finally, redundant features are filtered using k -nearest neighbor rough sets while retaining important features, significantly reducing the running time. In addition, we designed the feature selection algorithm (KND-UFS) and conducted experiments on 12 public datasets. We compared KND-UFS with eight existing algorithms in terms of running time, classification accuracy, and the number of selected features. The experimental results provided strong evidence supporting the superior performance of the KND-UFS algorithm.

An unsupervised feature selection approach for actionable warning identification

An unsupervised feature selection approach for actionable warning identification

RMFRASL: Robust Matrix Factorization with Robust Adaptive Structure Learning for Feature Selection

In this paper, we present a novel unsupervised feature selection method termed robust matrix factorization with robust adaptive structure learning (RMFRASL), which can select discriminative features from a large amount of multimedia data to improve the performance of classification and clustering tasks. RMFRASL integrates three models (robust matrix factorization, adaptive structure learning, and structure regularization) into a unified framework. More specifically, a robust matrix factorization-based feature selection (RMFFS) model is proposed by introducing an indicator matrix to measure the importance of features, and the L21-norm is adopted as a metric to enhance the robustness of feature selection. Furthermore, a robust adaptive structure learning (RASL) model based on the self-representation capability of the samples is designed to discover the geometric structure relationships of original data. Lastly, a structure regularization (SR) term is designed on the learned graph structure, which constrains the selected features to preserve the structure information in the selected feature space. To solve the objective function of our proposed RMFRASL, an iterative optimization algorithm is proposed. By comparing our method with some state-of-the-art unsupervised feature selection approaches on several publicly available databases, the advantage of the proposed RMFRASL is demonstrated.

Big data analysis using a parallel ensemble clustering architecture and an unsupervised feature selection approach

Ensemble clustering is known as a challenging research direction in data mining. The results of several individual clustering methods are combined to produce higher quality final clusters. This study introduces a parallel hierarchical clustering approach based on the divide-and-conquer strategy, which is an attempt to realize faster and more efficient ensemble clustering. Here, we propose a cluster consensus selection approach that selects a subset of meriting primary clusters to participate in the final consensus. Considering the sample-cluster and cluster–cluster similarity on the selected primary clusters, we form the final clusters based on the clusters clustering technique as a consensus function. In addition, the proposed scheme is equipped with an unsupervised feature selection approach to remove features that do not contribute significantly to clustering. Extensive evaluations have been performed on datasets of different dimensions from the University of California Irvine (UCI) machine learning repository. The simulation results guarantee the efficiency of the proposed scheme and improves the average performance between 6% and 24% compared to the state-of-the-art clustering methods.

Parkinson’s Disease Classification and Clinical Score Regression via United Embedding and Sparse Learning From Longitudinal Data

Parkinson's disease (PD) is known as an irreversible neurodegenerative disease that mainly affects the patient's motor system. Early classification and regression of PD are essential to slow down this degenerative process from its onset. In this article, a novel adaptive unsupervised feature selection approach is proposed by exploiting manifold learning from longitudinal multimodal data. Classification and clinical score prediction are performed jointly to facilitate early PD diagnosis. Specifically, the proposed approach performs united embedding and sparse regression, which can determine the similarity matrices and discriminative features adaptively. Meanwhile, we constrain the similarity matrix among subjects and exploit the l2,p norm to conduct sparse adaptive control for obtaining the intrinsic information of the multimodal data structure. An effective iterative optimization algorithm is proposed to solve this problem. We perform abundant experiments on the Parkinson's Progression Markers Initiative (PPMI) data set to verify the validity of the proposed approach. The results show that our approach boosts the performance on the classification and clinical score regression of longitudinal data and surpasses the state-of-the-art approaches.

Deep unsupervised feature selection by discarding nuisance and correlated features

Modern datasets often contain large subsets of correlated features and nuisance features, which are not or loosely related to the main underlying structures of the data. Nuisance features can be identified using the Laplacian score criterion, which evaluates the importance of a given feature via its consistency with the Graph Laplacians’ leading eigenvectors. We demonstrate that in the presence of large numbers of nuisance features, the Laplacian must be computed on the subset of selected features rather than on the complete feature set. To do this, we propose a fully differentiable approach for unsupervised feature selection, utilizing the Laplacian score criterion to avoid the selection of nuisance features. We employ an autoencoder architecture to cope with correlated features, trained to reconstruct the data from the subset of selected features. Building on the recently proposed concrete layer that allows controlling for the number of selected features via architectural design, simplifying the optimization process. Experimenting on several real-world datasets, we demonstrate that our proposed approach outperforms similar approaches designed to avoid only correlated or nuisance features, but not both. Several state-of-the-art clustering results are reported. Our code is publically available at https://github.com/jsvir/lscae.

Low-rank dictionary learning for unsupervised feature selection

There are many high-dimensional data in real-world applications such as biology, computer vision, and social networks. Feature selection approaches are devised to confront high-dimensional data challenges with the aim of efficient learning technologies as well as reduction of models complexity. Due to the hardship of labeling on these datasets, there are a variety of approaches for the feature selection process in an unsupervised setting by considering some important characteristics of data. In this paper, we introduce a novel unsupervised feature selection approach by applying dictionary learning idea in a low-rank representation. Low-rank dictionary learning not only enables us to provide a new data representation but also maintains feature correlation. Then, spectral analysis is employed to preserve sample similarities. Finally, a unified objective function for unsupervised feature selection is proposed in a sparse way by an ℓ2,1-norm regularization. Furthermore, an efficient numerical algorithm is designed to solve the corresponding optimization problem. We demonstrate the performance of the proposed method based on a variety of standard datasets from different applied domains. Our experimental findings reveal that the proposed method outperforms the state-of-the-art algorithms.

Self-paced non-convex regularized analysis–synthesis dictionary learning for unsupervised feature selection

Due to the ability to prevent over-fitting, reduce computational complexity and storage cost, and enhance interpretability, unsupervised feature selection (UFS) has received widespread attention in a large number of application scenarios. However, a majority of the existing UFS methods cannot make full use of the diversity between data points, which usually leads to a suboptimal solution. In this paper, a novel UFS approach, referred to as self-paced analysis–synthesis dictionary learning (SPASDL) is proposed by integrating dictionary learning and self-paced learning into a joint framework. Specifically, we reconstruct and code data with the synthesis dictionary and analysis dictionary, respectively. This strategy can avoid using pseudo labels of data, which may mislead the feature selection process. Further, we introduce self-paced learning into dictionary learning to realize the reconstruction of data points from easy to hard in the training process. To identify more discriminative features, we impose the non-convex sparse constraint on the synthesis dictionary. For the purpose of optimizing the resulting model, we exploit an efficient iterative optimization algorithm based on alternative search strategy (ASS). Besides, the theoretical results on the optimization algorithm, including the convergence analysis and computational complexity, are investigated. Finally, experimental results on public available real-world datasets demonstrate the effectiveness and the superiority of the proposed method.

Graph regularized virtual label regression for unsupervised feature selection

Feature selection is an important technique to deal with high-dimensional unlabeled data. Over the past decades, various unsupervised feature selection methods have been presented. However, some of these methods fail to consider the discriminative ability due to lack of label information or ignoring the local structure. To address this issue, we propose a novel unsupervised feature selection approach, called graph regularized virtual label regression (GVLR). In GVLR, we learn a virtual label matrix to guide the selection of features, which can ensure the selected features be more discriminative. Besides, we utilize graph Laplacian to preserve the geometric structure of the feature space. To improve the robustness of our model, we further adopt an ℓ2,1-norm constraint on the feature selection matrix and loss function. Finally, we incorporate the label regression and graph Laplacian into the subspace learning based feature selection framework. Thus, our method can enhance the discriminative ability of selected features, and preserve the local structural information of features simultaneously. Extensive experimental results on several public datasets show that the proposed GVLR method can obtain superior performance in comparison with some state-of-the-art feature selection algorithms.

A Novel Unsupervised Feature Selection Approach Using Genetic Algorithm on Partitioned Data

A novel feature selection approach is presented in this paper. Sammon’s Stress Function transforms the high dimension data to a lower dimension data set. A data set is divided into small partitions. The features are assigned randomly to these partitions. Using GA with Sammon Error as fitness value, a small, desired number of features are selected from every partition. The combination of the reduced subsets of the features from these partitions is again divided into small partitions. After a certain number of iterating the process, a desired small number of features is obtained. For experimental validation, the proposed method has been tested on 11 standard datasets with three classifiers namely, Decision Tree, MLP and KNN. The classification accuracies obtained by the proposed method is highest on most of the considered datasets against the results reported in literature. Moreover, the proposed method selects comparatively less number of features in comparison to considered methods. The optimistic results obtained from the proposed method justify its strength.

Graph regularized locally linear embedding for unsupervised feature selection

As one of the important dimensionality reduction techniques, unsupervised feature selection (UFS) has enjoyed amounts of popularity over the last few decades, which can not only improve learning performance, but also enhance interpretability and reduce computational costs. The existing UFS methods often model the data in the original feature space, which cannot fully exploit the discriminative information. In this paper, to address this issue, we investigate how to strengthen the relationship between UFS and the feature subspace, so as to select relevant features more straightforwardly and effectively. Methodologically, a novel UFS approach, referred to as Graph Regularized Local Linear Embedding (GLLE), is proposed by integrating local linear embedding (LLE) and manifold regularization constrained in feature subspace into a unified framework. To be more specific, we explicitly define a feature selection matrix composed of 0 and 1, which can realize the process of UFS. For the purpose of modelling the feature selection matrix, we propose to preserve the local linear reconstruction relationship among neighboring data points in the feature subspace, which corresponds to LLE constrained in the feature subspace. To make the feature selection matrix more accurate, we propose to use manifold regularization as an assistant of LLE to find the relevant and representative features such that the selected features can make each sample under the feature subspace be accordance with the manifold assumption. A tailored iterative algorithm based on Alternative Direction Method of Multipliers (ADMM) is designed to solve the proposed optimization problem. Extensive experiments on twelve real-world benchmark datasets are conducted, and the more promising results are achieved compared with the state-of-the-arts approaches.

Expect the Unexpected: Unsupervised Feature Selection for Automated Sensor Anomaly Detection

With the growth of IoT applications, sensor data quality has become increasingly important to ensure the success of these data-driven applications. Sensor data riddled with errors are redundant and affects the accuracy of decision-making results. Moreover, several constraints are inherent in anomaly detection for IoT applications such as limited manpower, time, bandwidth, computational resources and the lack of labelled datasets. Most machine learning algorithms also require a large training set to fit a satisfactory model. In this paper, we propose a fully automated anomaly detection framework, which combines systematic time series feature engineering with unsupervised feature selection. The unsupervised feature selection approach automatically selects time series features based on their predictive power with respect to the statistics of near future measurements. The proposed framework also only needs a short calibration phase with respect to the deployment phase to detect anomalies despite not having seen any in training, which is critical for real-world applications. Its feature selection reduces the amount of data required in the deployment phase on an IoT application. The selected features are suitable for building a reliable anomaly detection model while achieving a similar or better anomaly detection performance than established methods, which are operating on the raw data. Results of the evaluation on two publicly available environmental monitoring datasets show that our proposed unsupervised feature selection approach is a crucial step for having a more accurate anomaly detection while providing complex application-specific time series features, which are safeguarding the sensor system against unseen sensor anomalies.

Unsupervised feature selection via multi-step markov probability relationship

Feature selection is a widely used dimension reduction technique to select feature subsets because of its interpretability. Many methods have been proposed and achieved good results, in which the relationships between adjacent data points are mainly concerned. But the possible associations between data pairs that are not adjacent are always neglected. Different from previous methods, we propose a novel and very simple approach for unsupervised feature selection, named MMFS (Multi-step Markov Probability Relationship for Feature Selection). The idea is using multi-step Markov transition probability to describe the relation between any data pair. Two ways from the positive and negative viewpoints are employed respectively to keep the data structure after feature selection. From the positive viewpoint, the maximum transition probability that can be reached in a certain number of steps is used to describe the relation between two points. Then, the features which can keep the compact data structure are selected. From the viewpoint of negative, the minimum transition probability that can be reached in a certain number of steps is used to describe the relation between two points. On the contrary, the features that least maintain the loose data structure are selected. The two ways can also be combined. Thus three algorithms are proposed. Our main contributions are a novel feature section approach which uses multi-step transition probability to characterize the data structure, and three algorithms proposed from the positive and negative aspects for keeping data structure and select the features to preserve such structure. The performance of our approach is compared with the state-of-the-art methods on eight real-world data sets, and the experimental results show that the proposed MMFS is effective in unsupervised feature selection.

Unsupervised Feature Selection With Extended OLSDA via Embedding Nonnegative Manifold Structure.

As to unsupervised learning, most discriminative information is encoded in the cluster labels. To obtain the pseudo labels, unsupervised feature selection methods usually utilize spectral clustering to generate them. Nonetheless, two related disadvantages exist accordingly: 1) the performance of feature selection highly depends on the constructed Laplacian matrix and 2) the pseudo labels are obtained with mixed signs, while the real ones should be nonnegative. To address this problem, a novel approach for unsupervised feature selection is proposed by extending orthogonal least square discriminant analysis (OLSDA) to the unsupervised case, such that nonnegative pseudo labels can be achieved. Additionally, an orthogonal constraint is imposed on the class indicator to hold the manifold structure. Furthermore, l2,1 regularization is imposed to ensure that the projection matrix is row sparse for efficient feature selection and proved to be equivalent to l2,0 regularization. Finally, extensive experiments on nine benchmark data sets are conducted to demonstrate the effectiveness of the proposed approach.

ACO Inspired Computer-aided Detection/Diagnosis (CADe/CADx) Model for Medical Data Classification

Background: Computer-Assisted Diagnosis (CAD) has become a common practice of use in the healthcare industry due to its improved accuracy and reliability. The CAD systems are expected to improve the quality of medical care by assisting healthcare professionals with a wide range of clinical decisions. A CAD system is a combination of Computer-Assisted Detection (CADe) and Computer-Assisted Diagnosis (CADx) system. Objective: The objective of this research article is to generate an optimized rule-set for medical diagnosis capable of providing improved accuracy. It is evident from the literature that keeping a balance between these performance parameters is a real challenge. Methods: In order to achieve the desired objective, the following two contributions have been proposed to improve diagnosis accuracy: 1) an unsupervised feature selection approach based on ACO Meta-heuristic is used to design the CADe system, and 2) an ACO assisted decision tree classifier technique is employed to make CADx system. Results: Three popular UCI (Wisconsin Breast Cancer, Pima Indian Diabetes and Liver Disorder) medical domain datasets have been used to evaluate the performance of the proposed model. The exploratory result analysis shows the efficiency of the proposed work as compared to existing work.

CNN-based salient features in HSI image semantic target prediction

ABSTRACTDeep networks have escalated the computational performance in the sensor-based high dimensional imaging such as hyperspectral images (HSI), due to their informative feature extraction competency. Therefore in this work, we have extracted the informative features from different CNN models for the benchmark HSI datasets. The deep features have concatenated with spectral features to increase the informative knowledge in the image datacube. The feature concatenation has massively increased the size of datacube. Therefore, we have applied an unsupervised maximum object identification-based salient feature selection to identify the most informative features of datacube and discard the less informative features to reduce the computational time without compromising the accuracy. It is an unsupervised feature selection approach that transforms the data into scale space and achieved robust and strong features. In the previous CNN-based methods, raw features have directly fed to the MLP (multilayer perception) layers for target prediction whereas we have provided our salient features into a multi-core SVM-based set-up and have achieved high accuracy with low computational time as compared to the previous state-of-art techniques.

Unsupervised feature selection with multi-subspace randomization and collaboration

Unsupervised feature selection has been an important technique in high-dimensional data analysis. Despite significant success, most of the existing unsupervised feature selection methods tend to estimate the underlying structure of data in the original feature space, but lack the ability to explore various subspaces in the high-dimensional space. In this paper, we argue that the use of a large number of random subspaces can significantly benefit the unsupervised feature selection accuracy. Particularly, we propose a new unsupervised feature selection approach based on multi-subspace randomization and collaboration. A balanced subspace randomization scheme is first presented to produce multiple basic feature partitions with similar-sized random subspaces. Then, multiple K-nearest neighbor graphs are constructed in these subspaces, based on which the Laplacian scores of the features in each subspace are obtained w.r.t. their locality preserving power. Thereafter, the obtained feature score vectors of different subspaces in different basic feature partitions are integrated into a full score vector of all features, which takes into account the structure information of various subspaces and can significantly enhance the performance of unsupervised feature selection. Experiments conducted on twenty high-dimensional datasets have demonstrated the high efficiency and robustness of our approach. The MATLAB source code is available at https://www.researchgate.net/publication/334520672.

Self-Tuned Discrimination-Aware Method for Unsupervised Feature Selection.

Unsupervised feature selection is fundamentally important for processing unlabeled high-dimensional data, and several methods have been proposed on this topic. Most existing embedded unsupervised methods just emphasize the data structure in the input space, which may contain large noise. Therefore, they are limited to perceive the discriminative information implied within the low-dimensional manifold. In addition, these methods always involve several parameters to be tuned, which is time-consuming. In this paper, we present a self-tuned discrimination-aware (STDA) approach for unsupervised feature selection. The main contributions of this paper are threefold: 1) it adopts the advantage of discriminant analysis technique to select the valuable features; 2) it learns the local data structure adaptively in the discriminative subspace to alleviate the effect of data noise; and 3) it performs feature selection and clustering simultaneously with an efficient optimization strategy, and saves the additional efforts to tune parameters. Experimental results on a toy data set and various real-world benchmarks justify the effectiveness of STDA on both feature selection and data clustering, and demonstrate its promising performance against the state of the arts.

Ordinal preserving matrix factorization for unsupervised feature selection

Feature selection aims to remove the irrelevant and redundant features to reduce the dimensionality of data and increase the efficiency of learning algorithms. Specifically, unsupervised feature selection without any label information has become a challenging and significant task in machine learning applications. In this paper, a novel algorithm called Ordinal Preserving Matrix Factorization (OPMF), which incorporates matrix factorization, ordinal locality structure preserving and inner-product regularization into a unified framework, is proposed for feature selection. The advantages of our algorithm are three-fold. First, the ordinal locality property of original data is preserved by introducing a triplet-based loss function to the selected features, which is of great importance for distance-based classification and clustering tasks. Second, an inner product regularization term is incorporated into the proposed framework, so that the selected features obtained by our OPMF can be sparse and low redundant. Third, a simple and efficient iteratively updating algorithm is derived to solve the objective function of the proposed algorithm. Extensive experimental results on six datasets demonstrate that the proposed OPMF can obtain competitive performance compared to the existing state-of-the-art unsupervised feature selection approaches.