Nonlinear Feature Selection Research Articles

Coal free-swelling index modeling by an ordinal-based soft computing approach

ABSTRACT Coking coal is the most important reductant agent in the steel-making industry and approximately does not have any substitution. Thus, it is globally still on the list of critical raw materials. The free swelling index (FSI) as a qualitative index is used to assess and rank the coking-ability of coal samples. Although the accurate classification of FSI values is one of the most crucial issues in the coke-making industry, it has become one of the most challenging data-driven problems due to the specific difficulties that lie in coal data characteristics. In this study, by analyzing a robust dataset with over 3600 records from U.S. Geological Survey Coal Quality, an insight was obtained into the FSI classification problem factors that have led to the poor performance of machine learning models. By settling the class imbalance at the data level and applying an efficient model training strategy based on the ordinal classification at the algorithmic level, a novel structure for FSI modeling has been proposed. This study also demonstrates direct relation between the model accuracy and the impact of nonlinear feature selection using Mutual Information instead of traditional linear methods.

SVM Based Intrusion Detection Method with Nonlinear Scaling and Feature Selection

Intrusion is one of major security issues of internet with the rapid growth in smart and Internet of Thing (IoT) devices, and it becomes important to detect attacks and set out alarm in IoT systems. In this paper, the support vector machine (SVM) and principal component analysis (PCA) based method is used to detect attacks in smart IoT systems. SVM with nonlinear scheme is used for intrusion classification and PCA is adopted for feature selection on the training and testing datasets. Experiments on the NSL-KDD dataset show that the test accuracy of the proposed method can reach 82.2% with 16 features selected from PCA for binary-classification which is almost the same as the result obtained with all the 41 features; and the test accuracy can achieve 78.3% with 29 features selected from PCA for multi-classification while 79.6% without feature selection. The Denial of Service (DoS) attack detection accuracy of the proposed method can achieve 8.8% improvement compared with existing artificial neural network based method.

Dynamic nonlinear process monitoring based on dynamic correlation variable selection and kernel principal component regression

In actual industrial processes, data are usually time series. Each process variable may have strong autocorrelation and cross-correlation with other variables with different delays. In addition, there are usually complex nonlinearities among variables. To further improve the monitoring performance for dynamic nonlinear processes, establishing a nonlinear filtering model for each variable is necessary. Therefore, a novel dynamic nonlinear process monitoring method based on dynamic nonlinear feature selection and kernel principal component regression (KPCR) is proposed in this study. First, dynamic nonlinear related variables are selected for each variable through mutual information by considering variables with different time delays. Second, process variables are divided into response and independent variable sets. Third, corresponding KPCR models are established to describe the dynamic relationships with the selected dynamic related variables as input variables and with response variables as output variables. To monitor the dynamic processes, kernel principal component analysis model is constructed on the basis of the residuals, where the residuals are obtained by comparing measured values of instruments with the predicted values of KPCR models. A support vector data description model is established to monitor the independent variables. Three cases are used to verify the performance of the novel approach. Results show that the proposed method is superior to and more effective than other advanced dynamic process monitoring methods.

Nonlinear gradient-based feature selection for precise prediction of diseases

Nonlinear gradient-based feature selection for precise prediction of diseases

Nonlinear gradient-based feature selection for precise prediction of diseases

Nonlinear gradient-based feature selection for precise prediction of diseases

Novel Feature Selection Method for Nonlinear Support Vector Regression

The development of sparse techniques presents a major challenge to complex nonlinear high‐dimensional data. In this paper, we propose a novel feature selection method for nonlinear support vector regression, called FS‐NSVR, which first attempts to solve the nonlinear feature selection problem in the regression technology field. FS‐NSVR preserves the representative features selected in the complex nonlinear system due to its use of a feature selection matrix in the original space. FS‐NSVR is a challenging mixed‐integer programming problem that is solved efficiently by using an alternate iterative greedy algorithm. Experimental results on three artificial datasets and five real‐world datasets confirm that FS‐NSVR effectively selects representative features and discards redundant features in a nonlinear system. FS‐NSVR outperforms L1‐norm support vector regression, L1‐norm least squares support vector regression, and Lp‐norm support vector regression on both feature selection ability and regression efficiency.

Joint nonlinear feature selection and continuous values regression network

The curse of dimensionality is a long-standing intractable issue in many machine learning and computer vision tasks. Feature selection, a data preprocessing technique, aims to select most discriminative feature subsets for improving performance of downstream machine learning tasks. However, most existing feature selection methods concentrate upon learning a linear relationship between data points and their labels, which causes that they are incapable of handling nonlinear complex data in real-world applications. In this paper, we first propose a novel end-to-end Nonlinear Feature Selective Networks (NFSN) that be able to select discriminative feature subsets while preserving their nonlinear structure by embedding a ℓ2,p-norm regularized hidden layer into designed continuous values regression networks. In addition, we propose an efficient optimization algorithm that joins back propagation algorithm and re-weighted optimization strategy to acquire derivative of all weights accurately. Experimental results on the nonlinear analog pulse signal and real-world datasets demonstrate the superiority of proposed method compared to some related methods on feature selection. Our source code available on: https://github.com/StevenWangNPU/NFSN.

Feature selection with kernelized multi-class support vector machine

Feature selection is an important procedure in machine learning because it can reduce the complexity of the final learning model and simplify the interpretation. In this paper, we propose a novel non-linear feature selection method that targets multi-class classification problems in the framework of support vector machines. The proposed method is achieved using a kernelized multi-class support vector machine with a fast version of recursive feature elimination. The proposed method selects features that work well for all classes, as the involved classifier simultaneously constructs multiple decision functions that separates each class from the others. We formulate the classifier as a large optimisation problem, and iteratively solve one decision function at a time, leading to a lower computational time complexity than when solving the large optimisation problem directly. The coefficients of the classifier are then used as a ranking criterion in the accelerated recursive feature elimination by adding batch elimination and a rechecking process. Experimental results on several datasets demonstrate the superior performance of the proposed feature selection method.

Prediction and Compensation of Contour Error of CNC Systems Based on LSTM Neural-Network

This article proposes a contour error estimation (CEE) and compensation method for computer numerical control (CNC) systems based on the long short-term memory neural network. This is achieved by performing modeling of each axis to predict the tracking error, calculating the actual trajectory, estimating the contour error, and modifying the reference trajectory. First, linear feature selection based on a simplified single-axis model and nonlinear feature selection based on a circular test are performed to achieve tracking error prediction. Then, a spline-approximation-based CEE method is proposed to estimate the contour error between the reference trajectory and the predicted trajectory. Finally, contour error compensation is performed on the reference trajectory before it is run on CNC systems. The proposed method is validated through experiments on a three-axis CNC system.

Abstract PS5-13: Holistic artificial intelligence-driven predictor in HER2-positive (HER2+) early breast cancer (BC) treated with neoadjuvant lapatinib and trastuzumab without chemotherapy: A correlative analysis from SOLTI-1114 PAMELA

Abstract Background: Positive partial response was observed in patients with primary HER2+ early BC with dual HER2 blockade that were not treated with chemotherapy. In this context and beyond, the low (partial) response rate of chemotherapy-free treatment strategies creates the necessity for patient stratification prior to treatment selection. Here, we tackle the challenging task of evaluating the ability of clinical, gene expression and histopathology data to predict response following dual HER2 blockade without chemotherapy. Our aim is through artificial intelligence to automatically decipher the complementarity of clinical, genomic and histopathology data through an evidence-driven approach towards a low dimensional holistic signature that determines outcomes and could be subsequently used as a clinical biomarker for treatment patient inclusion. Methods: PAMELA (Lancet Oncology 2017) was a prospective study in HER2+ BC designed to evaluate the ability of the PAM50 HER2-enriched intrinsic subtype to predict pCR following 18-weeks of neoadjuvant lapatinib and trastuzumab (and hormonal therapy if hormone receptor-positive [HR+]). Clinical-pathological variables (15) were included such as tumor cellularity, tumor-infiltrating lymphocytes (TILs), the expression of BC-related genes/signatures (567) along with histopathological data from pre-treatment samples. Imaging information was obtained from H/E slides, through an unsupervised deep learning approach using an attention network. The semantic segmentation was used to derive at the patch level image and shape characteristics resulted on a pathomics-derived feature vector of (300) variables. An integrative approach that harnessed clinical, genomics and pathomics data into a unified prediction framework was used. Patients were divided into a training set 80% and a testing set 20% with proportions of pCR and non-pCR corresponding to the ones observed. A 100-fold Cross-validation (CV) was performed on the training. Linear and non-linear robust feature selection were used to recover a low dimensional holistic signature along with an ensemble learning approach to select the top 5 machine learning/artificial intelligence methods for prognosis. Results: From the high dimensional feature (882) space, a low dimensional holistic signature of 8 predictive variables was automatically retrieved through. The signature consisted of 4 genomics variables (expression levels of ERBB2, ESR1, Luminal A signature and Risk of Relapse score), 2 clinical-pathological variables (histologic grade and ER-status) and 2 imaging variables (mean Short Run Low Gray Level Emphasis of the gray level run length matrix and the mean absolute deviation). To ensure the robustness and generalizability of the results, we present results averaged over 100 splits into training and test. On all cases the same holistic signature was uses and the same prediction methods/principles. The proposed AI-driven prognosis mechanism reached 75% balanced accuracy, 69% precision, 65% sensitivity, 86% specificity and 0.84 AUC demonstrating the relevance of the approach. It was observed a successful classification of 86% for the non-pCR and 65% for the pCR cases. Ablation studies were performed to determine the relevance of the different categories of variables. Genomics variables were the most informative since their removal led to the highest decrease of the metrics (11% in average). Conclusion: The proposed method has great potentials for an effective and clinically meaningful implementation of pre-selecting patients that will not achieve a pCR after neoadjuvant dual HER2 blockade. Besides, the generality of the method used here makes it transposable to any type of cancer or therapy. Citation Format: Enzo Batistella, Laia Paré, Mihir Sahasrabudhe, Tomás Pascual, Maria Vakalopoulou, Patricia Villagrasa, Eric Deutsch, Núria Chic, Guillermo Villacampa, Paolo Nuciforo, Javier Cortes, Antonio Llombart-Cussac, Nikos Paragios, Aleix Prat. Holistic artificial intelligence-driven predictor in HER2-positive (HER2+) early breast cancer (BC) treated with neoadjuvant lapatinib and trastuzumab without chemotherapy: A correlative analysis from SOLTI-1114 PAMELA [abstract]. In: Proceedings of the 2020 San Antonio Breast Cancer Virtual Symposium; 2020 Dec 8-11; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2021;81(4 Suppl):Abstract nr PS5-13.

A new machine learning ensemble model for class imbalance problem of screening enhanced oil recovery methods

Enhanced Oil Recovery (EOR) methods have received a lot of attention today due to the increase in global oil demand and the reduction of oil production capacity from natural extraction methods. EOR methods are the solution for extracting oil from heavy and extra-heavy reservoirs, but the problem for reservoir engineers is choosing the right EOR method for an oilfield. In fact, the number of EOR methods and the features associated with them are very high, so the financial risk of implementing the wrong EOR method leads to feeling the need for an expert screening tool. Until now, various individual classifiers have been used for this purpose, but due to the class imbalance problem of the EOR dataset, most of them do not have good results, therefore, to increase the accuracy of the model, EOR methods with low records in the dataset are ignored, while industry needs all EOR methods. In this study, an ensemble learning-based approach is used to overcome the problem of class imbalance at the algorithmic level instead of the data level. For this purpose, an effective model called B2S is proposed by gathering the advantages of each of the ensemble-based methods, including Bagging, Boosting, and Stacking. As a result, B2S by creating a balance between variance and bias achieved an average test accuracy of 96.94% on the dataset including 426 samples from 10 different EOR methods that had a highly skewed distribution. The average accuracy also increases to 98.12% with a standard deviation of 1.63%, after grouping some relatively similar EOR methods, which indicates the stability of the proposed method. The study also demonstrates direct relation between model performance and the impact of precise pre-processing, nonlinear feature selection using Mutual Information rather than conventional linear methods, and hyper-parameters tuning with newer methods such as Random Search.

A new artificial intelligence strategy for predicting the groundwater level over the Rafsanjan aquifer in Iran

This study presents a new strategy to predict the monthly groundwater level with short- and long-lead times over the Rafsanjan aquifer in Iran using an ensemble machine learning method called Gradient Boosting Regression (GBR). In this way, the satellite-based products, including the Tropical Rainfall Measuring Mission (TRMM) and the Gravity Recovery and Climate Experiment (GRACE) datasets, as well as the pumping rate, are used as the predictive variables in different lag times. To obtain the optimal input combinations, the Gamma Test (GT) is employed as a non-linear feature selection technique. The spatial analysis of the performance prediction is performed using several error metrics (e.g., R2, and NRMSE). Results indicate that the GBR provides a high prediction performance to predict the GWL, whereas the GRACE product is an accurate predictive variable. The correlation analysis between the predicted and observed GWL shows that the coefficient of determination values vary in the range of 0.66 to 0.94 over the different lead times. The spatial pattern of the prediction accuracy indicates that the regions with higher water depth and pumping rate offer higher performance. Moreover, the increasing trends in performance accuracy are observed from the north to south and the west to east of the Rafsanjan aquifer. In general, the proposed approach provides a reliable insight for water resources planner to make a decision based on the accurate modeling results.

Nonlinear feature selection on attributed networks

The acceleratinnsional nodal attributes in various data mining tasks highlights the significance of feature selection on the networked data. Due to the lack of class labels of nodes, many feature selection methods are proposed in semi-supervised or unsupervised manners in various scenarios instead of supervised ones. More often than not, features and (pseudo) labels are correlated in a nonlinear way that is more intricate than linearity. In these circumstances, the vast majority of existing linear algorithms could not work well since they select features according to how well the feature can linearly explain the variance of labels. Moreover, although some methods focus on nonlinear feature selection, with the neglect of the link relations between data, they are difficult to be applied to attributed networks directly. In this paper, we investigate how to achieve nonlinear feature selection on attributed networks with the help of both labeled and unlabeled data. Methodologically, we first propose a novel semi-supervised nonlinear framework FS-GCN based on graph convolutional networks (GCNs) to select high-quality features, which can elaborately catch the nonlinear dependency between nodal attributes and class labels. To verify the importance of nonlinearity precisely, we further explore the possibility of totally removing the label information so that a variant of FS-GCN is proposed in the unsupervised form, referred to as UFS-GCN. Besides, experimental results on several real-world datasets validate the superiority of FS-GCN as well as UFS-GCN in terms of the quality of selected features, suggesting their robustness in the condition of extremely low even zero label ratio.

Improved metabolomic data-based prediction of depressive symptoms using nonlinear machine learning with feature selection

To solve major limitations in algorithms for the metabolite-based prediction of psychiatric phenotypes, a novel prediction model for depressive symptoms based on nonlinear feature selection machine learning, the Hilbert–Schmidt independence criterion least absolute shrinkage and selection operator (HSIC Lasso) algorithm, was developed and applied to a metabolomic dataset with the largest sample size to date. In total, 897 population-based subjects were recruited from the communities affected by the Great East Japan Earthquake; 306 metabolite features (37 metabolites identified by nuclear magnetic resonance measurements and 269 characterized metabolites based on the intensities from mass spectrometry) were utilized to build prediction models for depressive symptoms as evaluated by the Center for Epidemiologic Studies-Depression Scale (CES-D). The nested fivefold cross-validation was used for developing and evaluating the prediction models. The HSIC Lasso-based prediction model showed better predictive power than the other prediction models, including Lasso, support vector machine, partial least squares, random forest, and neural network. l-leucine, 3-hydroxyisobutyrate, and gamma-linolenyl carnitine frequently contributed to the prediction. We have demonstrated that the HSIC Lasso-based prediction model integrating nonlinear feature selection showed improved predictive power for depressive symptoms based on metabolome data as well as on risk metabolites based on nonlinear statistics in the Japanese population. Further studies should use HSIC Lasso-based prediction models with different ethnicities to investigate the generality of each risk metabolite for predicting depressive symptoms.

Pixels of chemical structures correlate to chromatographic detector responses using genetic algorithm-adaptive neuro-fuzzy inference system as a novel nonlinear feature selection method

This paper introduces the genetic algorithm-adaptive neuro-fuzzy inference system (GA-ANFIS), as a novel nonlinear feature selection method. This hybrid technique combines genetic algorithms (GAs) as powerful optimization methods with ANFIS as a robust nonlinear statistical method. Multivariate image analysis whose descriptors achieved from bidimensional images coupled to principal component analysis (PCA) and the most significant principal components (PCs) were extracted. Eigen value ranking (EV), correlation ranking (CR), GA-partial least square (GA-PLS) as well the method proposed in this work were used to select the most relevant set of PCs as inputs for ANFIS to assess electron capture detector responses of 207 polychlorinated biphenyls. The results indicated that GA-ANFIS is superior over the others in both selecting the most relevant set of PCs and correlating the inputs (PCs) with the detector responses. The best model was statistically validated for its predictive power using cross-validation, applicability domain and Y-scrambling evaluation procedures. Moreover, the superiority of this model obtained from pixels of chemical structures over the nonlinear one obtained from original molecular descriptors in a previous work indicates that the image analysis is a powerful tool in quantitative structure-(chromatographic) property relationship (QSPR) studies.

Unsupervised Nonlinear Feature Selection from High-Dimensional Signed Networks

With the rapid development of social media services in recent years, relational data are explosively growing. The signed network, which consists of a mixture of positive and negative links, is an effective way to represent the friendly and hostile relations among nodes, which can represent users or items. Because the features associated with a node of a signed network are usually incomplete, noisy, unlabeled, and high-dimensional, feature selection is an important procedure to eliminate irrelevant features. However, existing network-based feature selection methods are linear methods, which means they can only select features that having the linear dependency on the output values. Moreover, in many social data, most nodes are unlabeled; therefore, selecting features in an unsupervised manner is generally preferred. To this end, in this paper, we propose a nonlinear unsupervised feature selection method for signed networks, called SignedLasso. This method can select a small number of important features with nonlinear associations between inputs and output from a high-dimensional data. More specifically, we formulate unsupervised feature selection as a nonlinear feature selection problem with the Hilbert-Schmidt Independence Criterion Lasso (HSIC Lasso), which can find a small number of features in a nonlinear manner. Then, we propose the use of a deep learning-based node embedding to represent node similarity without label information and incorporate the node embedding into the HSIC Lasso. Through experiments on two real world datasets, we show that the proposed algorithm is superior to existing linear unsupervised feature selection methods.

Machine learning model for Bitcoin exchange rate prediction using economic and technology determinants

In recent years, Bitcoin exchange rate prediction has attracted the interest of researchers and investors. Some studies have used traditional statistical and econometric methods to understand the economic and technology determinants of Bitcoin, few have considered the development of predictive models using these determinants. In this study, we developed a two-stage approach for exploring whether the information hidden in economic and technology determinants can accurately predict the Bitcoin exchange rate. In the first stage, two nonlinear feature selection methods comprising an artificial neural network and random forest are used to reduce the subset of potential predictors by measuring the importance of economic and technology factors. In the second stage, the potential predictors are integrated into long short-term memory (LSTM) to predict the Bitcoin exchange rate regardless of the previous exchange rate. Our results showed that by using the economic and technology determinants, LSTM could achieve better predictive performance than the autoregressive integrated moving average, support vector regression, adaptive network fuzzy inference system, and LSTM methods, which all use the previous exchange rate. Thus, information obtained from economic and technology determinants is more important for predicting the Bitcoin exchange rate than the previous exchange rate.

Nonlinear variable selection algorithms for surrogate modeling

AbstractHaving the ability to analyze, simulate, and optimize complex systems is becoming more important in all engineering disciplines. Decision‐making using complex systems usually leads to nonlinear optimization problems, which rely on computationally expensive simulations. Therefore, it is often challenging to detect the actual structure of the optimization problem and formulate these problems with closed‐form analytical expressions. Surrogate‐based optimization of complex systems is a promising approach that is based on the concept of adaptively fitting and optimizing approximations of the input–output data. Standard surrogate‐based optimization assumes the degrees of freedom are known a priori; however, in real applications the sparsity and the actual structure of the black‐box formulation may not be known. In this work, we propose to select the correct variables contributing to each objective function and constraints of the black‐box problem, by formulating the identification of the true sparsity of the formulation as a nonlinear feature selection problem. We compare three variable selection criteria based on Support Vector Regression and develop efficient algorithms to detect the sparsity of black‐box formulations when only a limited amount of deterministic or noisy data is available.

Sparse Nonlinear Feature Selection Algorithm via Local Structure Learning

In this paper, we propose a new unsupervised feature selection algorithm by considering the nonlinear and similarity relationships within the data. To achieve this, we apply the kernel method and local structure learning to consider the nonlinear relationship between features and the local similarity between features. Specifically, we use a kernel function to map each feature of the data into the kernel space. In the high-dimensional kernel space, different features correspond to different weights, and zero weights are unimportant features (e.g. redundant features). Furthermore, we consider the similarity between features through local structure learning, and propose an effective optimization method to solve it. The experimental results show that the proposed algorithm achieves better performance than the comparison algorithm.

Performance comparison of linear and non-linear feature selection methods for the analysis of large survey datasets.

Large survey databases for aging-related analysis are often examined to discover key factors that affect a dependent variable of interest. Typically, this analysis is performed with methods assuming linear dependencies between variables. Such assumptions however do not hold in many cases, wherein data are linked by way of non-linear dependencies. This in turn requires applications of analytic methods, which are more accurate in identifying potentially non-linear dependencies. Here, we objectively compared the feature selection performance of several frequently-used linear selection methods and three non-linear selection methods in the context of large survey data. These methods were assessed using both synthetic and real-world datasets, wherein relationships between the features and dependent variables were known in advance. In contrast to linear methods, we found that the non-linear methods offered better overall feature selection performance than linear methods in all usage conditions. Moreover, the performance of the non-linear methods was more stable, being unaffected by the inclusion or exclusion of variables from the datasets. These properties make non-linear feature selection methods a potentially preferable tool for both hypothesis-driven and exploratory analyses for aging-related datasets.