Unsupervised Feature Selection Technique Research Articles

An Agglomerative Clustering Combined with an Unsupervised Feature Selection Approach for Structural Health Monitoring

Structural health monitoring (SHM) is critical for ensuring the safety and longevity of structures, yet existing methodologies often face challenges such as high data dimensionality, lack of interpretability, and reliance on extensive label datasets. Current research in SHM has primarily focused on supervised approaches, which require significant manual effort for data labeling and are less adaptable to new environments. Additionally, the large volume of data generated from dynamic structural monitoring campaigns often includes irrelevant or redundant features, further complicating the analysis and reducing computational efficiency. This study addresses these issues by introducing an unsupervised learning approach for SHM, employing an agglomerative clustering model alongside an unsupervised feature selection technique utilizing box-plot statistics. The proposed method is assessed through raw acceleration signals obtained from four dynamic structural monitoring campaigns, including 44 features with temporal, statistical, and spectral information. In addition, these features are also evaluated in terms of their relevance, and the most important ones are selected for a new execution of the computational procedure. The proposed feature selection not only reduces data dimensionality but also enhances model interpretability, improving the clustering performance in terms of homogeneity, completeness, V-measure, and adjusted Rand score. The results obtained for the four analyzed cases provide clear insights into the patterns of behavior and structural anomalies.

Dual-dual subspace learning with low-rank consideration for feature selection

The performance of machine learning algorithms can be affected by redundant features of high-dimensional data. Furthermore, these irrelevant features increase the time of computation for learning model. These problems can be addressed by leveraging different techniques including feature selection and dimensionality reduction. Unsupervised feature selection has drawn increased attention due to the difficulty of label collection for supervised feature selection. To this end, we developed an innovative approach based on nonnegative matrix factorization (NMF) to remove redundant information. In this technique, for the first time, the local information preserving regularization and global information preserving regularization are applied for both feature weight matrix and representation matrix which is why we called Dual-Dual regularized feature selection. Furthermore, Schatten p-norm is utilized to extract inherent low-rank properties of data. To demonstrate the effectiveness of the proposed method, experimental studies are conducted on six benchmark datasets. The computational results show that the proposed method in comparison with state-of-the-art unsupervised feature selection techniques is more efficient for feature selection.

Subspace learning for feature selection via rank revealing QR factorization: Fast feature selection

The identification of informative and distinguishing features from high-dimensional data has gained significant attention in the field of machine learning. Recently, there has been growing interest in employing matrix factorization-based techniques, such as non-negative matrix factorization, for feature selection. The primary objective of feature selection using matrix factorization is to extract a lower-dimensional subspace that captures the essence of the original space. This study introduces a novel unsupervised feature selection technique that leverages rank revealing QR (RRQR) factorization. Compared to singular value decomposition (SVD) and non-negative matrix factorization (NMF), RRQR is more computationally efficient. The uniqueness of this technique lies in the utilization of the permutation matrix of QR for feature selection. Additionally, we integrate QR factorization into the objective function of NMF to create a new unsupervised feature selection method. Furthermore, we propose a hybrid feature selection algorithm by combining RRQR and a Genetic algorithm. The algorithm eliminates redundant features using RRQR factorization and selects the most distinguishing subset of features using the Genetic algorithm. Experimental comparisons with state-of-the-art feature selection algorithms in supervised, unsupervised, and semi-supervised settings demonstrate the reliability and robustness of the proposed algorithm. The evaluation is conducted on eight microarray datasets using KNN, SVM, and C4.5 classifiers. The experimental results indicate that the proposed method achieves comparable performance to the state-of-the-art feature selection methods. Our empirical findings demonstrate that the proposed method exhibits a significantly lower computational cost compared to other techniques.

Dual regularized subspace learning using adaptive graph learning and rank constraint: Unsupervised feature selection on gene expression microarray datasets

High-dimensional problems have increasingly drawn attention in gene selection and analysis. To add insult to injury, usually the number of features is greater than number of samples in microarray gene dataset which leads to an ill-posed underdetermined equation system. Poor performance and high computational time for learning algorithms are consequences of redundant features in high-dimensional data. Feature selection is a noteworthy pre-processing method to ameliorate the curse of dimensionality with aim of maximum relevancy and minimum redundancy information preservation. Likewise, unsupervised feature selection has been important since collecting labels for data is expensive. In this paper, we develop a novel robust unsupervised feature selection to select discriminative subset of features for unlabeled data based on rank constrained and dual regularized nonnegative matrix factorization. The major focus of the proposed technique is to discard redundant features while keeping the informative features. Proposed feature selection technique consists of nonnegative matrix factorization to decompose the data into feature weight matrix and representation matrix, inner product norm as regularization for both feature weight matrix and representation matrix, adaptive structure learning to preserve local information and Schatten-p norm as rank constraint. To demonstrate the effectiveness of the proposed method, numerical studies are conducted on six benchmark microarray datasets. The results show that the proposed technique outperforms eight state-of-art unsupervised feature selection techniques in terms of clustering accuracy and normalized mutual information.

A Machine Learning Method for Prediction of Yogurt Quality and Consumers Preferencesusing Sensory Attributes and Image Processing Techniques

Prediction of quality and consumers’ preferences is essential task for food producers to improve their market share and reduce any gap in food safety standards. In this paper, we develop a machine learning method to predict yogurt preferences based on the sensory attributes and analysis of samples’ images using image processing texture and color feature extraction techniques. We compare three unsupervised ML feature selection techniques (Principal Component Analysis and Independent Component Analysis and t-distributed Stochastic Neighbour Embedding) with one supervised ML feature selection technique (Linear Discriminant Analysis) in terms of accuracy of classification. Results show the efficiency of the supervised ML feature selection technique over the traditional feature selection techniques.

Feature Identification Framework for Back Injury Risk in Repetitive Work With Application in Sheep Shearing.

Lower back injuries are a serious global problem. Most of these injuries occur over time with repeated sub-acute stresses. Neuromuscular control dysfunction could predict injury, however injuries are almost never observed alongside this data. No labels are available to identify important features that may be predictive of injury. While there are many individual differences in injury development, the population trend is that each individual's injury tolerance decreases over time with exposure, indicating a monotonic process. This paper proposes a framework for identifying key features of injury using an unsupervised technique that exploits knowledge of injury aetiology by analysing which features contribute to the popular trend using weak monotonicity from data segmented by task repetitions. The feature selection also evaluates feature redundancy. The efficacy of the framework is demonstrated through data from on-site sheep shearers over one day using 17 wearable inertial measurement units and 16 surface electromyography (sEMG) sensors. Consistent with literature, the results demonstrate sEMG features derived from the erector spinae and multifidus muscles are the most important indicators for lower back injury. To evaluate the performance of the proposed population-trend based unsupervised feature selection technique, the self-reported fatigue information is treated as some 'ground truth' information so that this proposed technique can compare with 5 existing unsupervised feature selection techniques. The proposed technique is shown to be the most consistent with the self-reported fatigue information, demonstrating the effectiveness of the proposed method.

Unsupervised text feature selection using NSGA II with Hill Climbing local search

This paper introduces a novel unsupervised text feature selection technique that combines the multi-objective evolutionary algorithm NSGA II with a local Hill Climbing based search. The objective functions in NSGA II are an adapted version of the Mean Absolute Difference criterion and the Holes projection pursuit index, both of which incorporate information regarding the cardinality of the selected feature subset. The local Hill Climbing procedure aims to further improve the solutions provided by NSGA II through elimination/replacement of redundant features. The redundancy level of a feature F in a chromosome is measured via the lexical and semantic similarities between F and the other features represented in the chromosome. Experiments conducted on three real-world textual subdatasets indicate that our proposed approach achieves very good results in comparison to four state-of-the-art evolutionary unsupervised feature selection techniques and to one classical feature extraction method with proven effectiveness (Principal Component Analysis).

Unsupervised Feature Selection via Orthogonal Basis Clustering and Local Structure Preserving.

Due to the "curse of dimensionality" issue, how to discard redundant features and select informative features in high-dimensional data has become a critical problem, hence there are many research studies dedicated to solving this problem. Unsupervised feature selection technique, which does not require any prior category information to conduct with, has gained a prominent place in preprocessing high-dimensional data among all feature selection techniques, and it has been applied to many neural networks and learning systems related applications, e.g., pattern classification. In this article, we propose an efficient method for unsupervised feature selection via orthogonal basis clustering and reliable local structure preserving, which is referred to as OCLSP briefly. Our OCLSP method consists of an orthogonal basis clustering together with an adaptive graph regularization, which realizes the functionality of simultaneously achieving excellent cluster separation and preserving the local information of data. Besides, we exploit an efficient alternative optimization algorithm to solve the challenging optimization problem of our proposed OCLSP method, and we perform a theoretical analysis of its computational complexity and convergence. Eventually, we conduct comprehensive experiments on nine real-world datasets to test the validity of our proposed OCLSP method, and the experimental results demonstrate that our proposed OCLSP method outperforms many state-of-the-art unsupervised feature selection methods in terms of clustering accuracy and normalized mutual information, which indicates that our proposed OCLSP method has a strong ability in identifying more important features.

Understanding and Predicting the Usage of Shared Electric Scooter Services on University Campuses

Electric vehicles (EVs) have been progressing rapidly in urban transport systems given their potential in reducing emissions and energy consumptions. The Shared Free-Floating Electric Scooter (SFFES) is an emerging EV publicized to address the first-/last-mile problem in travel. It also offers alternatives for short-distance journeys using cars or ride-hailing services. However, very few SFFES studies have been carried out in developing countries and for university populations. Currently, many universities are facing an increased number of short-distance private car travels on campus. The study is designed to explore the attitudes and perceptions of students and staff towards SFFES usage on campus and the corresponding influencing factors. Three machine learning models were used to predict SFFES usage. Eleven important factors for using SFFESs on campus were identified via the supervised and unsupervised feature selection techniques, with the top three factors being daily travel mode, road features (e.g., green spaces) and age. The random forest model showed the highest accuracy in predicting the usage frequency of SFFESs (93.5%) using the selected 11 variables. A simulation-based optimization analysis was further conducted to discover the characterization of SFFES users, barriers/benefits of using SFFESs and safety concerns.

Sufficiently Informative and Relevant Features: An Information-Theoretic and Fourier-Based Characterization

A fundamental challenge in learning is the presence of nonlinear redundancies and dependencies in the data. To address this, we propose a Fourier-based approach to characterize feature redundancies, in unsupervised learning, and feature-label dependencies, in the supervised variant of the problem. We first develop a novel Fourier expansion for functions (more generally stochastic mappings) of correlated binary random variables. This is a generalization of the standard Fourier expansion on the Boolean cube beyond product probability spaces. As an important application of this analysis, we investigate learning with feature subset selection. In the unsupervised variant of this problem, we characterize feature redundancies via the Shannon entropy and group the features into sufficiently informative and redundant. Then, we make a connection to the proposed Fourier expansion and derive an upper bound on the joint entropy. Based on that, we propose a measure to quantify feature redundancies and present an unsupervised learning algorithm. We test our method on various real-world and synthetic datasets and demonstrate improvements on conventional unsupervised feature selection techniques. Then, we investigate the supervised feature subset selection and reformulate it in the Fourier domain. Bridging the Bayesian error rate with the Fourier coefficients, we demonstrate that the Fourier expansion provides a powerful tool to characterize nonlinear feature-label dependencies. Further, we introduce a computationally efficient measure for selecting relevant features. Via a theoretical analysis, we show that our proposed measure finds provably <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">asymptotically optimal</i> feature subsets. Lastly, we present an algorithm based on this measure and via numerical experiments demonstrate its improvements on various supervised feature selection algorithms.

An Unsupervised Neural Network Feature Selection and 1D Convolution Neural Network Classification for Screening of Parkinsonism.

Parkinson’s disease (PD) is the second most common neurodegenerative disorder after Alzheimer’s disease. It has a slow progressing neurodegenerative disorder rate. PD patients have multiple motor and non-motor symptoms, including vocal impairment, which is one of the main symptoms. The identification of PD based on vocal disorders is at the forefront of research. In this paper, an experimental study is performed on an open source Kaggle PD speech dataset and novel comparative techniques were employed to identify PD. We proposed an unsupervised autoencoder feature selection technique, and passed the compressed features to supervised machine-learning (ML) algorithms. We also investigated the state-of-the-art deep learning 1D convolutional neural network (CNN-1D) for PD classification. In this study, the proposed algorithms are support vector machine, logistic regression, random forest, naïve Bayes, and CNN-1D. The classifier performance is evaluated in terms of accuracy score, precision, recall, and F1 score measure. The proposed 1D-CNN model shows the highest result of 0.927%, and logistic regression shows 0.922% on the benchmark dataset in terms of F1 measure. The major contribution of the proposed approach is that unsupervised neural network feature selection has not previously been investigated in Parkinson’s detection. Clinicians can use these techniques to analyze the symptoms presented by patients and, based on the results of the above algorithms, can diagnose the disease at an early stage, which will allow for improved future treatment and care.

Unsupervised fuzzy multivariate symmetric uncertainty feature selection based on constructing virtual cluster representative

The data readability, complexity reduction of learning algorithms, and enhancing predictability are the most important reasons for using feature selection methods, especially when there exist lots of features. In recent years, unsupervised feature selection techniques are well explored. Among the various methods of feature selection, algorithms based on information theory are effective in selecting useful attributes and eliminating useless ones. Mutual information and symmetric uncertainty based methods can effectively estimate feature relevancy, but the use of bivariate measures ignores possible dependencies among more than two features. To address this limitation, this research introduces a novel unsupervised feature selection method, called Fuzzy Multivariate Symmetric Uncertainty-Feature Selection (FMSU-FS). The proposed method also overcomes the need for discretization for continuous features that causes information loss. To evaluate the effectiveness of the proposed method, FMSU is compared with conventional methods based on information theory. The experimental results on benchmark datasets show improvement of our approach in measuring clustering performance.

Detection of Breast Cancer Through Clinical Data Using Supervised and Unsupervised Feature Selection Techniques

Breast cancer is one the most critical disease and suffered many people around the world. The efficient and correct detection of breast cancer is still needed to ensure this medical issue although the researchers around the world are proposed different diagnostic methods for detection of this disease, however these existing methods still needed further improvement to correct and efficient detection of this disease. In this study, we proposed a new breast cancer identification method by using machine learning algorithms and clinical data. In the proposed method supervised (Relief algorithm) and unsupervised (Autoencoder, PCA algorithms) techniques have been used for related features selection from data set and then these selected features have been used for training and testing of classifier support vector machine for accurate and on time detection of breast cancer. Additionally, in the proposed approach k fold cross validation method has been used for model validation and best hyperparameters selection. The model performance evaluation metrics have been used for model performance evaluation. The BC data sets have been used for testing of the proposed method. The analysis of experimental results has been demonstrated that the features selected by Relief algorithm are more related for accurate detection of Breast cancer instead of features selected by Auotencoder and PCA algorithms. The proposed method has been attained high results in terms of accuracy on selected feature selected by Relief algorithm and achieved 99.91% accuracy. We have been employed McNemar's statistical test for performance comparison of our different models. Further, the proposed method performance has been compared with baseline methods in the literature and the proposed method performance is high as compared to base line methods. Due to the high performance of the proposed method (Relief-Support vector machine) we highly recommended it for the diagnosis of breast cancer. In addition, the proposed method can be easily incorporated into the healthcare system for reliable diagnosis of Breast cancer.

Essential gene prediction using limited gene essentiality information-An integrative semi-supervised machine learning strategy.

Essential gene prediction helps to find minimal genes indispensable for the survival of any organism. Machine learning (ML) algorithms have been useful for the prediction of gene essentiality. However, currently available ML pipelines perform poorly for organisms with limited experimental data. The objective is the development of a new ML pipeline to help in the annotation of essential genes of less explored disease-causing organisms for which minimal experimental data is available. The proposed strategy combines unsupervised feature selection technique, dimension reduction using the Kamada-Kawai algorithm, and semi-supervised ML algorithm employing Laplacian Support Vector Machine (LapSVM) for prediction of essential and non-essential genes from genome-scale metabolic networks using very limited labeled dataset. A novel scoring technique, Semi-Supervised Model Selection Score, equivalent to area under the ROC curve (auROC), has been proposed for the selection of the best model when supervised performance metrics calculation is difficult due to lack of data. The unsupervised feature selection followed by dimension reduction helped to observe a distinct circular pattern in the clustering of essential and non-essential genes. LapSVM then created a curve that dissected this circle for the classification and prediction of essential genes with high accuracy (auROC > 0.85) even with 1% labeled data for model training. After successful validation of this ML pipeline on both Eukaryotes and Prokaryotes that show high accuracy even when the labeled dataset is very limited, this strategy is used for the prediction of essential genes of organisms with inadequate experimentally known data, such as Leishmania sp. Using a graph-based semi-supervised machine learning scheme, a novel integrative approach has been proposed for essential gene prediction that shows universality in application to both Prokaryotes and Eukaryotes with limited labeled data. The essential genes predicted using the pipeline provide an important lead for the prediction of gene essentiality and identification of novel therapeutic targets for antibiotic and vaccine development against disease-causing parasites.

Artificial Neural Networks in Motion Analysis-Applications of Unsupervised and Heuristic Feature Selection Techniques.

The use of machine learning to estimate joint angles from inertial sensors is a promising approach to in-field motion analysis. In this context, the simplification of the measurements by using a small number of sensors is of great interest. Neural networks have the opportunity to estimate joint angles from a sparse dataset, which enables the reduction of sensors necessary for the determination of all three-dimensional lower limb joint angles. Additionally, the dimensions of the problem can be simplified using principal component analysis. Training a long short-term memory neural network on the prediction of 3D lower limb joint angles based on inertial data showed that three sensors placed on the pelvis and both shanks are sufficient. The application of principal component analysis to the data of five sensors did not reveal improved results. The use of longer motion sequences compared to time-normalised gait cycles seems to be advantageous for the prediction accuracy, which bridges the gap to real-time applications of long short-term memory neural networks in the future.

Gene encoder: a feature selection technique through unsupervised deep learning-based clustering for large gene expression data

Cancer is a severe condition of uncontrolled cell division that results in a tumor formation that spreads to other tissues of the body. Therefore, the development of new medication and treatment methods for this is in demand. Classification of microarray data plays a vital role in handling such situations. The relevant gene selection is an important step for the classification of microarray data. This work presents gene encoder, an unsupervised two-stage feature selection technique for the cancer samples’ classification. The first stage aggregates three filter methods, namely principal component analysis, correlation, and spectral-based feature selection techniques. Next, the genetic algorithm is used, which evaluates the chromosome utilizing the autoencoder-based clustering. The resultant feature subset is used for the classification task. Three classifiers, namely support vector machine, k-nearest neighbors, and random forest, are used in this work to avoid the dependency on any one classifier. Six benchmark gene expression datasets are used for the performance evaluation, and a comparison is made with four state-of-the-art related algorithms. Three sets of experiments are carried out to evaluate the proposed method. These experiments are for the evaluation of the selected features based on sample-based clustering, adjusting optimal parameters, and for selecting better performing classifier. The comparison is based on accuracy, recall, false positive rate, precision, F-measure, and entropy. The obtained results suggest better performance of the current proposal.

Radiomic feature selection for lung cancer classifiers

Machine learning methods with quantitative imaging features integration have recently gained a lot of attention for lung nodule classification. However, there is a dearth of studies in the literature on effective features ranking methods for classification purpose. Moreover, optimal number of features required for the classification task also needs to be evaluated. In this study, we investigate the impact of supervised and unsupervised feature selection techniques on machine learning methods for nodule classification in Computed Tomography (CT) images. The research work explores the classification performance of Naive Bayes and Support Vector Machine(SVM) when trained with 2, 4, 8, 12, 16 and 20 highly ranked features from supervised and unsupervised ranking approaches. The best classification results were achieved using SVM trained with 8 radiomic features selected from supervised feature ranking methods and the accuracy was 100%. The study further revealed that very good nodule classification can be achieved by training any of the SVM or Naive Bayes with a fewer radiomic features. A periodic increment in the number of radiomic features from 2 to 20 did not improve the classification results whether the selection was made using supervised or unsupervised ranking approaches.

Radiomics for Distinguishing Myocardial Infarction from Myocarditis at Late Gadolinium Enhancement at MRI: Comparison with Subjective Visual Analysis.

To evaluate whether radiomics features of late gadolinium enhancement (LGE) regions at cardiac MRI enable distinction between myocardial infarction (MI) and myocarditis and to compare radiomics with subjective visual analyses by readers with different experience levels. In this retrospective, institutional review board-approved study, consecutive MRI examinations of 111 patients with MI and 62 patients with myocarditis showing LGE were included. By using open-source software, classification performances attained from two-dimensional (2D) and three-dimensional (3D) texture analysis, shape, and first-order descriptors were compared, applying five different machine learning algorithms. A nested, stratified 10-fold cross-validation was performed. Classification performances were compared through Wilcoxon signed-rank tests. Supervised and unsupervised feature selection techniques were tested; the effect of resampling MR images was analyzed. Subjective image analysis was performed on 2D and 3D image sets by two independent, blinded readers with different experience levels. When trained with recursive feature elimination (RFE), a support vector machine achieved the best results (accuracy: 88%) for 2D features, whereas linear discriminant analysis (LDA) showed the highest accuracy (85%) for 3D features (P <.05). When trained with principal component analysis (PCA), LDA attained the highest accuracy with both 2D (86%) and 3D (89%; P =.4) features. Results found for classifiers trained with spline resampling were less accurate than those achieved with one-dimensional (1D) nearest-neighbor interpolation (P <.05), whereas results for classifiers trained with 1D nearest-neighbor interpolation and without resampling were similar (P =.1). As compared with the radiomics approach, subjective visual analysis performance was lower for the less experienced and higher for the experienced reader for both 2D and 3D data. Radiomics features of LGE permit the distinction between MI and myocarditis with high accuracy by using either 2D features and RFE or 3D features and PCA.© RSNA, 2019Supplemental material is available for this article.

KSUFS: A Novel Unsupervised Feature Selection Method Based on Statistical Tests for Standard and Big Data Problems

The typical inaccuracy of data gathering and preparation procedures makes erroneous and unnecessary information to be a common issue in real-world applications. In this context, feature selection methods are used in order to reduce the harmful impact of such information in data analysis by removing irrelevant features from datasets. This research presents a novel feature selection method in the field of unsupervised learning, in which the complexity arises from the fact that the class labels cannot be used to select the most discriminative features as it is traditionally performed in supervised learning. The technique designed, which is called Kolmogorov-Smirnov test-based Unsupervised Feature Selection ( KSUFS ), is based on the computation of estimated feature distributions that are later compared to the original ones using non-parametric statistical tests to provide the most representative input variables. Two versions of the KSUFS are presented in this study: one of them is particularly designed to deal with standard data, in which the accuracy of the method prevalences over other of its aspects; the other version is designed to treat with big data problems, in which the computational complexity is improved due to the characteristics of this type of datasets. The KSUFS is successfully compared to other state-of-the-art unsupervised feature selection techniques in a thorough experimental study, which considers both standard and big data problems. The results obtained show that the method proposed is able to outperform the rest of reference unsupervised feature selection methods considered in the comparisons, selecting the first most influential features for standard datasets and particularly highlighting when big data problems are treated.

Unsupervised Feature Selection Based on Ultrametricity and Sparse Training Data: A Case Study for the Classification of High-Dimensional Hyperspectral Data

In this paper, we investigate the potential of unsupervised feature selection techniques for classification tasks, where only sparse training data are available. This is motivated by the fact that unsupervised feature selection techniques combine the advantages of standard dimensionality reduction techniques (which only rely on the given feature vectors and not on the corresponding labels) and supervised feature selection techniques (which retain a subset of the original set of features). Thus, feature selection becomes independent of the given classification task and, consequently, a subset of generally versatile features is retained. We present different techniques relying on the topology of the given sparse training data. Thereby, the topology is described with an ultrametricity index. For the latter, we take into account the Murtagh Ultrametricity Index (MUI) which is defined on the basis of triangles within the given data and the Topological Ultrametricity Index (TUI) which is defined on the basis of a specific graph structure. In a case study addressing the classification of high-dimensional hyperspectral data based on sparse training data, we demonstrate the performance of the proposed unsupervised feature selection techniques in comparison to standard dimensionality reduction and supervised feature selection techniques on four commonly used benchmark datasets. The achieved classification results reveal that involving supervised feature selection techniques leads to similar classification results as involving unsupervised feature selection techniques, while the latter perform feature selection independently from the given classification task and thus deliver generally versatile features.