Simultaneous Feature Selection Research Articles

Genetic Programming for Automatically Evolving Multiple Features to Classification

Abstract Performing classification on high-dimensional data poses a significant challenge due to the huge search space. Moreover, complex feature interactions introduce an additional obstacle. The problems can be addressed by using feature selection to select relevant features or feature construction to construct a small set of high-level features. However, performing feature selection or feature construction only might make the feature set suboptimal. To remedy this problem, this study investigates the use of genetic programming for simultaneous feature selection and feature construction in addressing different classification tasks. The proposed approach is tested on 16 datasets and compared with seven methods including both feature selection and feature constructions techniques. The results show that the obtained feature sets with the constructed and/or selected features can significantly increase the classification accuracy and reduce the dimensionality of the datasets. Further analysis reveals the complementarity of the obtained features leading to the promising classification performance of the proposed method.

Simultaneous feature selection and SVM optimization based on fuzzy signature and chaos GOA

Simultaneous feature selection and SVM optimization based on fuzzy signature and chaos GOA

Network dynamics and therapeutic aspects of mRNA and protein markers with the recurrence sites of pancreatic cancer

Pancreatic ductal adenocarcinoma (PDAC) is a deadly disease that typically manifests late patient presentation and poor outcomes. Furthermore, PDAC recurrence is a common challenge. Distinct patterns of PDAC recurrence have been associated with differential activation of immune pathway-related genes and specific inflammatory responses in their tumour microenvironment. However, the molecular associations between and within cellular components that underpin PDAC recurrence require further development, especially from a multi-omics integration perspective. In this study, we identified stable molecular associations across multiple PDAC recurrences and utilised integrative analytics to identify stable and novel associations via simultaneous feature selection. Spatial transcriptome and proteome datasets were used to perform univariate analysis, Spearman partial correlation analysis, and univariate analyses by Machine Learning methods, including regularised canonical correlation analysis and sparse partial least squares. Furthermore, networks were constructed for reported and new stable associations. Our findings revealed gene and protein associations across multiple PDAC recurrence groups, which can provide a better understanding of the multi-layer disease mechanisms that contribute to PDAC recurrence. These findings may help to provide novel association targets for clinical studies for constructing precision medicine and personalised surveillance tools for patients with PDAC recurrence.

Mathematical programming for simultaneous feature selection and outlier detection under l1 norm

The goal of simultaneous feature selection and outlier detection is to determine a sparse linear regression vector by fitting a dataset possibly affected by the presence of outliers.The problem is well-known in the literature. In its basic version it covers a wide range of tasks in data analysis. Simultaneously performing feature selection and outlier detection strongly improves the application potential of regression models in more general settings, where data governance is a concern. To trigger this potential, flexible training models are needed, with more parameters under control of decision makers.The use of mathematical programming, although pertinent, is scarce in this context and mostly focusing on the least-squares setting. Instead we consider the least absolute deviation criterion, proposing two mixed-integer linear programs, one adapted from existing studies, and the other obtained from a disjunctive programming argument. We show theoretically and computationally that the disjunctive-based formulation is better in terms of both continuous relaxation quality and integer optimality convergence.We experimentally benchmark against existing methodologies from the literature. We identify the characteristics of contamination patterns, in which mathematical programming is better than state-of-the-art algorithms in combining prediction quality, sparsity and robustness against outliers. Additionally, the mathematical programming approaches allow the decision maker to directly control parameters like the number of features or outliers to tolerate, those based on least absolute deviations performing best. On real world datasets, where privacy is a concern, our approach compares well to state-of-the-art methods in terms of accuracy, being at the same time more flexible.

Prediction of conversion from mild cognitive impairment to Alzheimer’s disease and simultaneous feature selection and grouping using Medicaid claim data

BackgroundDue to the heterogeneity among patients with Mild Cognitive Impairment (MCI), it is critical to predict their risk of converting to Alzheimer’s disease (AD) early using routinely collected real-world data such as the electronic health record data or administrative claim data.MethodsThe study used MarketScan Multi-State Medicaid data to construct a cohort of MCI patients. Logistic regression with tree-guided lasso regularization (TGL) was proposed to select important features and predict the risk of converting to AD. A subsampling-based technique was used to extract robust groups of predictive features. Predictive models including logistic regression, generalized random forest, and artificial neural network were trained using the extracted features.ResultsThe proposed TGL workflow selected feature groups that were robust, highly interpretable, and consistent with existing literature. The predictive models using TGL selected features demonstrated higher prediction accuracy than the models using all features or features selected using other methods.ConclusionsThe identified feature groups provide insights into the progression from MCI to AD and can potentially improve risk prediction in clinical practice and trial recruitment.

A hybrid multimodal machine learning model for Detecting Alzheimer's disease

Alzheimer's disease (AD) diagnosis utilizing single modality neuroimaging data has limitations. Multimodal fusion of complementary biomarkers may improve diagnostic performance. This study proposes a multimodal machine learning framework integrating magnetic resonance imaging (MRI), positron emission tomography (PET) and cerebrospinal fluid (CSF) assays for enhanced AD characterization. The model incorporates a hybrid algorithm combining enhanced Harris Hawks Optimization (HHO) algorithm referred to as ILHHO, with Kernel Extreme Learning Machine (KELM) classifier for simultaneous feature selection and classification. ILHHO enhances HHO's search efficiency by integrating iterative mapping (IM) to improve population diversity and local escaping operator (LEO) to balance exploration-exploitation. Comparative analysis with other improved HHO algorithms, classic meta-heuristic algorithms (MHAs), and state-of-the-art MHAs on IEEE CEC2014 benchmark functions indicates that ILHHO achieves superior optimization performance compared to other comparative algorithms. The synergistic ILHHO-KELM model is evaluated on 202 AD Neuroimaging Initiative (ADNI) subjects. Results demonstrate superior multimodal classification accuracy over single modalities, validating the importance of fusing heterogeneous biomarkers. MRI + PET + CSF achieves 99.2 % accuracy for AD vs. normal control (NC), outperforming conventional and proposed methods. Discriminative feature analysis provides further insights into differential AD-related neurodegeneration patterns detected by MRI and PET. The differential PET and MRI features demonstrate how the two modalities provide complementary biomarkers. The neuroanatomical relevance of selected features supports ILHHO-KELM's potential for extracting sensitive AD imaging signatures. Overall, the study showcases the advantages of capitalizing on complementary multimodal data through advanced feature learning techniques for improving AD diagnosis.

A Novel Semi-Supervised Learning Model for Smartphone-Based Health Telemonitoring

Telemonitoring is the use of electronic devices such as smartphones to remotely monitor patients. It provides great convenience and enables timely medical decisions. To facilitate the decision making for each patient, a model is needed to translate the data collected by the patient’s smartphone into a predicted score for his/her disease severity. To train a robust predictive model, semi-supervised learning (SSL) provides a viable approach by integrating both labeled and unlabeled samples to leverage all the available data from each patient. There are two challenging issues that need to be simultaneously addressed in using SSL for this problem: (1) feature selection from high-dimensional noisy telemonitoring data; and (2) instance selection from many, possibly redundant unlabeled samples. We propose a novel SSL model allowing for simultaneous feature and instance selection, namely the S2SSL model. We present a real-data application of telemonitoring for patients with Parkinson’s Disease using their smartphone-collected activity data such as tapping and speaking. A total of 382 features were extracted from the activity data of each patient. 74 labeled and 563 unlabeled instances from 37 patients were used to train S2SSL. The trained model achieved a high accuracy of 0.828 correlation between the true and predicted disease severity scores on a validation dataset. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Telemonitoring is an emerging health care platform enabled by smartphones and wearables. Because it allows for health data to be collected anytime and anywhere, patients can be frequently monitored and medical decisions can be made more timely and effectively. This paper addresses the data science challenges in leveraging the telemonitoring platform to benefit patient care. Specifically, we propose a new model, S2SSL, to tackle these challenges and provide better robustness, accuracy, and efficiency. This paper may be interesting to health care practitioners seeking advanced analytics capabilities to model and integrate the data collected through telemonitoring devices, with ultimate purposes of improving the decision in treating each patient and increasing patient access to specialized care.

Advancing Gene Expression Data Analysis: an Innovative Multi-objective Optimization Algorithm for Simultaneous Feature Selection and Clustering

Advancing Gene Expression Data Analysis: an Innovative Multi-objective Optimization Algorithm for Simultaneous Feature Selection and Clustering

DG-ALETSK: A High-Dimensional Fuzzy Approach With Simultaneous Feature Selection and Rule Extraction

Fuzzy or neuro-fuzzy systems have been successfully employed in many areas, but their limitation in solving high-dimensional problems remains a challenging task. On the other hand, fuzzy model based feature selection (FS) approaches have been well studied but, because of the above limitation, are rarely applied to high-dimensional data, which exactly needs feature reduction. In this paper, we design a novel adaptive Ln-Exp softmin operator as the approximator of the minimum T-norm, which is equipped in Takagi-Sugeno-Kang (TSK) fuzzy systems to make them capable of dealing with high-dimensional problems. The adaptive Ln-Exp softmin based TSK (ALETSK) model is developed. Then, we improve the gate function by introducing an enhanced scheme and propose a double groups of gates based ALETSK (DG-ALETSK) fuzzy approach to simultaneously conduct FS and rule extraction (RE), where the gate function is a function measuring the importance of the features or rules and it acts like a gate. More specifically, a group of feature gates are embedded in the antecedents of ALETSK for FS. Meanwhile, another group of rule gates are embedded in the consequents of ALETSK for RE. In one training phase, the gate parameters are trained along with the system parameters, so that the FS and RE are done together. Our proposed DG-ALETSK is effective and time-saving for FS and RE in the high-dimensional tasks, which is verified by the numerical experiments.

A data-driven evidential regression model for building hourly energy consumption prediction with feature selection and parameters learning

Building energy consumption prediction is critical for building energy management and energy policy formulation, and its inherent uncertainty can significantly affect the utilization of current energy market benefits for market participants. To capture the uncertainty of energy consumption and enhance the predictive capability of the model, in this study, a data-driven evidential regression (EVREG) model with integrated feature selection function is proposed based on Dempster–Shafer theory and mutual information, which can perform point prediction and interval prediction for building hourly energy consumption to describe its fluctuation and uncertainty. Different from the traditional EVREG model, this method enables simultaneous feature selection and model parameters learning instead of treating feature selection as a separate data pre-processing step. Specifically, an evaluation function is defined to describe the significance of a candidate feature, taking into account the predictive power of regression model and the redundancy between the candidate feature and already selected features. According to a search strategy, features with high significance are selected to minimize the objective function. A real dataset from a commercial building is used to evaluate the performance of the proposed method. The results demonstrate that the proposed method can select fewer features while achieving better prediction performance compared to traditional feature selection methods used as data preprocessing. The proposed method also achieves better or comparable performance compared to commonly applied point prediction and interval prediction methods.

A smoothing Group Lasso based interval type-2 fuzzy neural network for simultaneous feature selection and system identification

Inspired by the life philosophy, an ingenious gate (membership) function, which can mimic the open and close of the gate in the real world, is proposed to realize feature selection (FS) for interval type-2 fuzzy neural network. Based on the widths of gate functions, a smoothing integrated learning strategy is designed, and in the regularized FS error function the gate widths corresponding to one feature are grouped as one Group Lasso (GL) regularization term. During the training procedure of feature selection, all the group terms with respect to bad features will be punished, but for the good features the corresponding group terms converge to different nonzero values. When feature selection is finished, using the selected features, the interpretable clustering algorithm based initial interval type-2 rule generation method is detailedly introduced firstly, then a simple and efficient memory-based gradient method is presented for the tuning of rule structure parameters. Lastly, employing the simulation results of four regression problems and four classification problems, the effectiveness on simultaneous feature selection and system identification of our proposed model, i.e., the smoothing Group Lasso based interval type-2 fuzzy neural network (SGLIT2FNN), is proved, which illustrates that SGLIT2FNN not only simplifies the model structure by deleting the bad features, but also maintains the performance.

A Generalized Classification Framework with Simultaneous Feature Weighting and Selection Using Antlion Optimization Algorithm

ABSTRACT The use of machine-learning based algorithms on a large-scale nondestructive evaluation (NDE) data considerably advances the NDE techniques toward complete industrial automation. In this article, simultaneous feature selection and feature weighting are carried out on the magnetic Barkhausen emission (MBE) dataset to demonstrate the significance of optimization in NDE data. Antlion optimization is employed as a searching method to determine the optimum feature set that will maximize the classification performance. The proposed framework is validated for different magnetization frequencies separately and found to be frequency independent. The framework resulted in the selection of four significant features extracted from the MBE response thereby reducing the computational effort and improving the accuracy to 98.4% for AdaBoost classifier. The developed machine learning methodology is a potential strategy for processing industrial sensory data since material testing, property prediction, and categorization are frequent tasks in manufacturing and production engineering industries. Further, this research demonstrated the necessity of embedded intelligence in automation of NDE toward Industrial Revolution 4.0.

Double groups of gates based Takagi-Sugeno-Kang (DG-TSK) fuzzy system for simultaneous feature selection and rule extraction

Embedded feature selection methods based on fuzzy model have been well studied, in which the gate functions are usually used to distinguish the essential features. Fuzzy rules play a critical role in constructing fuzzy systems. However, few works simultaneously evaluate the quality of fuzzy rules during the fuzzy system based feature selection approaches. This article proposes double groups of gates based Takagi-Sugeno-Kang (DG-TSK) fuzzy model to simultaneously conduct feature selection and rule extraction in one training phase. Firstly, a novel gate function is introduced, which is embedded into the system. We call this function as M-gate since it is shaped like an M. Secondly, a new type of fuzzy rule base (FRB) is designed by transforming each data point into one fuzzy rule, which is named point-based FRB (P-FRB). Based on the M-gate and P-FRB, DG-TSK model is developed, in which M-gates are embedded in the antecedents for feature selection and consequents for rule extraction. Note that these two procedures are performed at the same time. We test DG-TSK on 18 classification datasets and compare it with other similar methods. The experiment results demonstrate that DG-TSK is effective.

Cost-sensitive max-margin feature selection for SVM using alternated sorting method genetic algorithm

This article introduces Alternated Sorting Method Genetic Algorithm (ASMGA), a simultaneous feature selection and model selection algorithm for Support Vector Machine (SVM) classifiers. It is a hybrid wrapper-filter algorithm that combines Genetic Algorithm (GA) with Max-Margin Feature Selection (MMFS). MMFS is a filter model that estimates feature importance based on feature relevance and redundancy. In this research, the idea of different error costs was used to introduce cost sensitivity to ASMGA. Thus, ASMGA selects relevant and independent features in a cost-sensitive manner. This research investigates the relationship between the cost sensitivity of SVM models produced by ASMGA and the feature selection process. ASMGA approximates a set of Pareto optimal feature subsets based on three objectives: cost-sensitive error rate, feature subset size, and MMFS-based estimates of feature relevance and redundancy. This research introduces a technique for handling multiple objectives. During the search, ASMGA alternates between two multi-objective sorting techniques: Weighted Sum (WS) of objectives and Non-dominated Sorting (NDS), according to a schedule of methods. This technique allows ASMGA to work as elitist GA for some iterations and as a Non-dominated Sorting Genetic Algorithm (NSGA-II) for the remaining iterations. The proposed algorithm was tested on 11 benchmark datasets and compared to canonical GA and NSGA-II. The algorithm and its variations performed on average 3.8% better than GA and NSGA-II for balanced datasets and 6.6% better for imbalanced datasets. The results and analysis in this article showcase the potential of ASMGA and help explain the interaction between cost sensitivity and feature selection.

Locally alignment based manifold learning for simultaneous feature selection and extraction in classification problems

Dimensionality reduction is an important step in increasing the performance of machine learning algorithms while decreasing the processing time. From feature reduction approaches, feature extraction is aimed to achieve better data representation while feature selection is meant to discard redundant features. The main contribution of this manuscript is to propose a hybrid feature selection and extraction approach which performs these tasks in a unified framework, simultaneously. Also, the proposed approach aims to maintain the manifold of data using locally alignment constrains. The main goal of these suggestions is to discriminate different classes while reducing the redundancy of the samples specially for the imbalanced data classification problems. The proposed approach optimizes a hybrid objective function that tries to both increase the between class discrimination and decrease the within class distribution while minimizing the information loss. Also, we have embedded a weighting factor into the objective function to achieve a basis for feature importance measurement and feature selection. To evaluate the proposed approach, a multiclass linear SVM classifier is applied on the reduced data in a k-fold cross-validation scheme, and the accuracy, as well as F-score, is used as the performance measure. Comparisons of the proposed method with some recent approaches on 12 datasets of UCI, show the superiority of the proposed method over previously proposed approaches. Also, Friedman and Nemenyi tests are applied which show significant improvement in the proposed approach.

Maximum A Posteriori Approximation of Hidden Markov Models for Proportional Sequential Data Modeling With Simultaneous Feature Selection.

One of the pillar generative machine learning approaches in time series data study and analysis is the hidden Markov model (HMM). Early research focused on the speech recognition application of the model with later expansion into numerous fields, including video classification, action recognition, and text translation. The recently developed generalized Dirichlet HMMs have proven efficient in proportional sequential data modeling. As such, we focus on investigating a maximum a posteriori (MAP) framework for the inference of its parameters. The proposed approach differs from the widely deployed Baum-Welch through the placement of priors that regularizes the estimation process. A feature selection paradigm is also integrated simultaneously in the algorithm. For validation, we apply our proposed approach in the classification of dynamic textures and the recognition of infrared actions.

Efficient Subject-Independent Detection of Anterior Cruciate Ligament Deficiency Based on Marine Predator Algorithm and Support Vector Machine.

Anterior cruciate ligament (ACL) deficiency not only reduces knee stability, but also increases the risk of more disease and impairs daily life, thus requiring efficient detection of ACL deficiency. To build an efficient subject-independent ACL deficiency detection model, this study proposes a new method called SVM-MPA that fuses marine predator algorithm (MPA) and support vector machine (SVM) for simultaneous feature selection, hyperparameter optimization and classification. 35ACL-deficient (ACLD) and 35 ACL-intact (ACLI) participants were recruited to collect 6-degree-of-freedom knee kinematic data. Then, 216-dimensional multi-domain features covering time domain, frequency domain, time-frequency domain and nonlinearity were extracted. The error rate of SVM classification based on 5-fold cross-validation was used to construct the fitness of MPA, and MPA served to select features and optimize two hyperparameters for SVM. The majority voting strategy-based post-processing was introduced to convert the gait cycle-level to knee-level ACL deficiency detection. Comparing with 7 well-known meta-heuristic algorithms and running all 20 times, the best average gait cycle-level ACL deficiency detection performance (sensitivity: 96.78±0.4.84%, specificity: 99.43±5.70%, and accuracy: 98.48±1.70%) was obtained using the proposed method. With post-processing, this study improved the best (final) detection performance (sensitivity: 97.78±4.97%, specificity: 100±0.00%, and accuracy: 99.13±1.94%). These results demonstrate the feasibility and effectiveness of the proposed method and shows that an efficient subject-independent ACL deficiency detection model can be constructed using the proposed method, which makes it possible to provide a non-invasive, objective and accurate preoperative auxiliary detection method for diagnosing ACL deficiency clinically.

A novel minorization–maximization framework for simultaneous feature selection and clustering of high-dimensional count data

A novel minorization–maximization framework for simultaneous feature selection and clustering of high-dimensional count data

Creating sparser prediction models of treatment outcome in depression: a proof-of-concept study using simultaneous feature selection and hyperparameter tuning

BackgroundPredicting treatment outcome in major depressive disorder (MDD) remains an essential challenge for precision psychiatry. Clinical prediction models (CPMs) based on supervised machine learning have been a promising approach for this endeavor. However, only few CPMs have focused on model sparsity even though sparser models might facilitate the translation into clinical practice and lower the expenses of their application.MethodsIn this study, we developed a predictive modeling pipeline that combines hyperparameter tuning and recursive feature elimination in a nested cross-validation framework. We applied this pipeline to a real-world clinical data set on MDD treatment response and to a second simulated data set using three different classification algorithms. Performance was evaluated by permutation testing and comparison to a reference pipeline without nested feature selection.ResultsAcross all models, the proposed pipeline led to sparser CPMs compared to the reference pipeline. Except for one comparison, the proposed pipeline resulted in equally or more accurate predictions. For MDD treatment response, balanced accuracy scores ranged between 61 and 71% when models were applied to hold-out validation data.ConclusionsThe resulting models might be particularly interesting for clinical applications as they could reduce expenses for clinical institutions and stress for patients.

A Simultaneous Feature Selection and Compositional Association Test for Detecting Sparse Associations in High-Dimensional Metagenomic Data.

Numerous metagenomic studies aim to discover associations between the microbial composition of an environment (e.g., gut, skin, oral) and a phenotype of interest. Multivariate analysis is often performed in these studies without critical a priori knowledge of which taxa are associated with the phenotype being studied. This approach typically reduces statistical power in settings where the true associations among only a few taxa are obscured by high dimensionality (i.e., sparse association signals). At the same time, low sample size and compositional sample space constraints may reduce beyond-study generalizability if not properly accounted for. To address these difficulties, we developed the Selection-Energy-Permutation (SelEnergyPerm) method, a nonparametric group association test with embedded feature selection that directly accounts for compositional constraints using parsimonious logratio signatures between taxonomic features, for characterizing and understanding alterations in microbial community structure. Simulation results show SelEnergyPerm selects small independent sets of logratios that capture strong associations in a range of scenarios. Additionally, our simulation results demonstrate SelEnergyPerm consistently detects/rejects associations in synthetic data with sparse, dense, or no association signals. We demonstrate the novel benefits of our method in four case studies utilizing publicly available 16S amplicon and whole-genome sequencing datasets. Our R implementation of Selection-Energy-Permutation, including an example demonstration and the code to generate all of the scenarios used here, is available at https://www.github.com/andrew84830813/selEnergyPermR.