Two-stage Feature Selection Method Research Articles

A Two-Stage Feature Selection Method to Enhance Prediction of Daily PM2.5 Concentration Air Pollution

In recent decades, air pollution has negatively affected human health and the environment. One of the important features contributing to air pollution is called PM2.5. However, daily prediction of PM2.5 is still lacking, especially using feature selection infused into the model. Hence, the main objective of this research is to utilize the feature selection procedures by proposing two stages feature selection methods namely adjusted correlation sharing t-test (adjcorT) and radial basis function neural network (RBFNN) in identifying the important features. This consequently also helps enhance the prediction of daily PM2.5 concentrations. Secondary data were obtained from the Department of Environment Malaysia (DOE) from 2018 until 2022 that consists of 5 years of air pollutant daily data. The results found that adjcorT-RBFNN identified the NO2, PM2.5, PM10, CO, O3, wind speed and SO2 as important features. The finding revealed that the accuracy, sensitivity, specificity, precision, F1 score and AUROC value, for a day-ahead prediction in Shah Alam are 0.756, 0.801, 0.717, 0.717, 0.757, and 0.758 respectively. Additionally, the predicted model may serve as an instrument for an early warning system, providing local authorities with information on air quality for formulation of strategies of air quality improvement.

Financial Distress Prediction Using an Artificial Neural Network Integrating a Two-Stage Feature Selection Method

Financial distress prediction (FDP) is critical for companies, banks, and investors, and artificial neural networks (ANN) have been proven to be an efficient method for FDP. However, the “curse of dimensionality” in FDP not only increases the computational complexity, but also reduces the prediction accuracy. To solve this problem, this paper takes an ANN model as the basic classifier and presents a new two-stage feature selection method integrated with multiple filters and a wrapper method. The financial data of Chinese listed companies are applied for comparative analysis to verify the effectiveness of the constructed method. The results demonstrate that the proposed method achieves a smaller feature subset and better predictive effect than other methods, thus solving the “curse of dimensionality” more effectively and improving the accuracy. In addition, SHapley Additive exPlanations (SHAP) and Partial Dependence Plots (PDP) are employed to investigate the relative importance of selected features. Their results increase the credibility of the proposed model, giving users more confidence in using this “black box” model.

Performance prediction of sludge volume index of oxygenic photogranule based wastewater treatment system using machine learning algorithms

Oxygenic Photogranulation is a novel biotechnology that treats wastewater without external aeration and produces biomass in dense photogranules with high settling velocities. Oxygenic photogranules (OPG) based wastewater treatment (WWT) faces challenges during scaleup due to its dynamic and complex system variables, making troubleshooting costly. The Machine Learning (ML) approach can address this issue by creating a WWT process simulation. Moreover, traditional mechanistic models do not capture the interaction between input and output features due to their high dimensionality and the non-linear relationship, making them computationally expensive. In this study, the two-stage feature selection (FS) method is studied to enhance the prediction performance of SVI30, a critical operational parameter, to ensure optimal settleability and minimal loss of active photogranules. The optimal feature subsets generated by the two-stage selection method were evaluated using four regression models: decision tree, random forest, gradient boosting, and extreme gradient boosting. Results indicate that, among all regression models, the decision tree performs well having a prediction efficiency of 85 % with the subset of features obtained after Recursive Feature Elimination (RFE) of decision tree features in the second stage. This indicates the effectiveness of the two-stage FS method in identifying the most relevant features for predicting SVI30. The structured approach of FS and model evaluation highlights the potential of ML in addressing complex operational challenges in OPG WWT operations.

Prediction of the severity of marine accidents using improved machine learning

Although many studies have focused on the occurrence likelihood of marine accidents, few have focused on the analysis of the severity of the consequences, and even fewer on the prediction of the severity. To this end, a new research framework is proposed in this study to accurately predict the severity of marine accidents. First, a novel two-stage feature selection (FS) method was developed to select and rank Risk Influential Factors (RIFs) to improve the accuracy of the Machine Learning (ML) model and interpretability of the FS. Second, a comprehensive evaluation method is proposed to measure the performance of the FS methods based on stability, predictive performance improvement, and statistical tests. Third, six well-established ML models were used and compared to measure the performance of different predictors. The Light Gradient Boosting Machine (LightGBM) was found to have the best predictive performance for the severity prediction of marine accidents and was treated as the benchmark model. Finally, LightGBM was used to predict accident severity based on the RIFs selected by the proposed FS method, and the effect of risk control measures was counterfactually analysed from a quantitative perspective. This innovative study on the use of improved ML approaches can effectively analyse and predict the severity of marine accidents, providing a novel methodology for and triggering a new direction for using Artificial Intelligence (AI) technologies in safety assessment and accident prevention studies. The source code is publicly available at: https://github.com/FengYinLeo/PGI-SDMI.

Differential network knockoff filter with application to brain connectivity analysis.

The brain functional connectivity can typically be represented as a brain functional network, where nodes represent regions of interest (ROIs) and edges symbolize their connections. Studying group differences in brain functional connectivity can help identify brain regions and recover the brain functional network linked to neurodegenerative diseases. This process, known as differential network analysis focuses on the differences between estimated precision matrices for two groups. Current methods struggle with individual heterogeneity in measuring the brain connectivity, false discovery rate (FDR) control, and accounting for confounding factors, resulting in biased estimates and diminished power. To address these issues, we present a two-stage FDR-controlled feature selection method for differential network analysis using functional magnetic resonance imaging (fMRI) data. First, we create individual brain connectivity measures using a high-dimensional precision matrix estimation technique. Next, we devise a penalized logistic regression model that employs individual brain connectivity data and integrates a new knockoff filter for FDR control when detecting significant differential edges. Through extensive simulations, we showcase the superiority of our approach compared to other methods. Additionally, we apply our technique to fMRI data to identify differential edges between Alzheimer's disease and control groups. Our results are consistent with prior experimental studies, emphasizing the practical applicability of our method.

A two-stage feature selection method for hob state recognition

In mechanical field, tool state seriously affects the production quality and efficiency. Recently, machine learning is an important technique to realize tool state recognition. Extracting and selecting features from the raw input data of the machine learning model can effectively eliminate useless information and improve the model accuracy. Therefore, a two-stage feature selection method is proposed in this paper. We first qualitatively propose the non-discreteness and separability of signal features. For these two properties, quantitative calculation methods are then developed in two stages. The proposed two-stage feature selection method is applied to machine learning models for tool state recognition. In engineering application verification, the proposed method achieves an average recognition accuracy of 84.4% in hob wear recognition, and an average recognition accuracy of 99.8% and 94.4% for two data sets in hob fault recognition, which is significantly higher than other feature selection methods. The verification results indicate that the proposed method has good generalization and engineering application value.

A novel enterprise credit risk rating model with the perspective of multiple regulatory demands

With the number of market participants rising, the issue of contradiction between heavy supervision task and insufficient supervision resource have become increasingly urgent for regulatory authorities to consider. The credit risk rating of enterprises is therefore introduced to improve the efficiency of supervision. By analyzing the purposes of enterprise credit risk rating and the multiple demands for regulatory authorities, this paper offers a multi-classification ensemble model with hybrid decomposition strategy and two-stage multi-objective feature selection. First, an adaptive resampling technique is introduced into the hybrid decomposition strategy for multi-classification, so to appropriately allocate supervision resources in each credit risk class. Then, a two-stage multi-objective feature selection method is developed to combine the various demands of regulatory authorities for feature selection. In the first stage, results of multiple Filter feature selection methods are integrated as output to improve the performance of feature information; In the second stage, the feature information from the first stage is used as input to guide the Wrapper feature selection, which is based on the Improved Multi-objective Ant Colony Optimization. Finally, a real listed enterprise credit dataset of China is used to evaluate the performance of each module and the entire model. The empirical results show that the design of hybrid decomposition strategy and multi-objective feature selection can not only deal with the class-imbalanced problem during the decomposition process, but also take into account the balance of each demand well. More importantly, a desirable framework is set in this paper to simplify the process from imbalanced multi-classification to a balanced binary one.

CorrDQN-FS: A two-stage feature selection method for energy consumption prediction via deep reinforcement learning

Accurate energy consumption prediction is crucial for optimizing energy usage and achieving sustainable development goals, especially in the field of building engineering. However, constructing accurate prediction models faces challenges in feature selection, given the large pool of candidate features and their intricate relationships. To address this issue, we propose a novel two-stage feature selection method called CorrDQN-FS for energy consumption prediction. CorrDQN-FS leverages the Pearson correlation coefficient to rank features and eliminates features with obvious irrelevant coefficients. Subsequently, the method integrates deep reinforcement learning techniques to optimize the feature selection process, considering the complex nonlinear relationships among features. By combining correlation analysis and deep reinforcement learning, CorrDQN-FS aims to enhance the accuracy of energy consumption prediction models. To evaluate the effectiveness of CorrDQN-FS, we conduct experiments using real-world data. We compare the performance of CorrDQN-FS combined with different prediction methods, including DF-DDPG, DF-DQN, LSTM, RNN, MLR and DT, to access its ability to improve energy consumption prediction accuracy. Additionally, we compare CorrDQN-FS with other feature selection methods, including the Pearson correlation coefficient, Recursive Feature Elimination (RFE), Mutual Information, and Gradient Boosting Decision Trees (GBDT), to highlight its advantages in feature selection. The experimental results demonstrate that CorrDQN-FS outperforms other feature selection techniques, effectively enhancing the accuracy of energy consumption prediction. This advancement holds significant potential for optimizing energy management and furthering sustainable building engineering.

Detection of Electric Bicycle Indoor Charging for Electrical Safety: A NILM Approach

The increasing fire accidents caused by electric bicycle (EB) indoor charging have raised great concerns. To ensure residential electrical safety, EB is prohibited from charging indoors by regulations, but efficient monitoring of EB indoor charging behaviour in pratice still remains as a challenging problem. From the perspective of non-intrusive load monitoring (NILM), this paper proposes a holistic approach to detect the EB indoor charging behaviour. Our approach utilizes four key elements: (i) a multi-window based cumulative sum algorithm to capture the turning-on event from aggregate power; (ii) a twostage feature selection method to determine the optimal features to form an effective appliance signature for load identification; (iii) a one-class support vector machine classifier to identify whether the turned-on appliance is EB or other appliances in a semi-supervised way; (iv) a Raspberry-Pi based edge device to implement the overall NILM algorithm. The experiment results verify the effectiveness of our NILM based approach, which can obtain a high detection performance with a lowdimensional feature space, thus providing a cost-effective and high-performance solution for EB indoor charging detection.

Deep graph neural network with fish-inspired task allocation algorithm for heart disease diagnosis

Heart disease is a very hazardous disease and many people suffered from this disease globally. The major aim is “to diagnose the heart disease with higher accuracy through decreasing error rate including computational complexity”. The existing techniques did not proffer adequate accuracy and also increased the error rate. Therefore, a Deep Graph Neural Network with Fish-Inspired Task Allocation Algorithm is proposed in this manuscript for categorizing heart disease diagnosis (DGNN-FITA-HDD). Synthetic Minority Oversampling and standard scalar strategies are utilized for pre-processing process. The pre-processed output is given to feature selection process. Two-Stage Feature Selection method selects the most important features from pre-processing output. Extracted features are transferred to Deep Graph Neural Network (DGNN) for categorizing presence and absence of heart disease. DGNN does not expose any adoption of optimization strategies for calculating the optimum parameters to assure accurate prediction. Fish-Inspired Task Allocation approach is proposed for optimizing the weight parameters of DGNN. The proposed approach is executed at MATLAB. The performance of algorithm is analyzed with/without feature selection method. By this, the proposed DGNN-FITA-HDD method attains higher accuracy with feature selection of 13.41%, 18.53%, 10.38% and 9.31% and without feature selection attains 6.5%, 8.64%, 4.39%, and 10.28% compared with existing methods, like EDGA-AHHO-HDD, XGB-MAPO-HDD, AGAFL-HDD and RFBM-HDD respectively.

Fusion-based speech emotion classification using two-stage feature selection

Speech emotion recognition plays an important role in human–computer interaction, which uses speech signals to determine the emotional state. Previous studies have proposed various features and feature selection methods. However, few studies have investigated the two-stage feature selection method for speech emotion classification. In this study, we propose a novel speech emotion classification algorithm based on two-stage feature selection and two fusion strategies. Specifically, three types of features are extracted from speech signals: constant-Q spectrogram-based histogram of oriented gradients, openSMILE, and wavelet packet decomposition-based features. Then, two-stage feature selection using random forest and grey wolf optimization is applied to reduce feature dimension and model training time and improve the classification performance. In addition, both early and late fusion strategies are explored aiming to further improve the performance. Experimental results indicate that early fusion with two-stage feature selection can achieve the best performance. The highest classification accuracy for RAVDESS, SAVEE, EMOVO, and EmoDB is 86.97%, 88.79%, 89.24%, and 95.29%, respectively.

Diffusion-tensor imaging and dynamic susceptibility contrast MRIs improve radiomics-based machine learning model of MGMT promoter methylation status in glioblastomas

BackgroundDetermining the O6-methylguanine-DNA methyltransferase (MGMT) promoter methylation status, which is a predictor of response to standard radiotherapy treatment in patients with glioblastoma (GBM), remains an invasive and time-consuming process. Magnetic resonance imaging (MRI)-based radiomics studies have used machine learning models as a non-invasive and time-saving approach to predict MGMT methylation status. However, more studies need to use radiomics features extracted from Diffusion-Tensor Imaging (DTI) and Dynamic Susceptibility Contrast (DSC) scans that have shown benefits in GBM patients. Therefore, we aimed to use the radiomics features from conventional and advanced MRI scans, including DSC and DTI, to build an effective model for predicting MGMT methylation status. Materials and MethodsWe enrolled 182 GBM patients whose data were obtained from The Cancer Imaging Archive (TCIA). From the initial 4514 radiomics features (1,644 conventional MRI-based and 2,870 advanced MRI-based), we used a two-stage feature selection method to select the top conventional and advanced features to build Conventional and Advanced predictive models, respectively. Besides, our Full model was developed from combining all these features. We evaluated the performances of our models and compared them with that of a “state-of-the-art” model reconstructed by re-running a flowchart from a previous study, which demonstrated the highest performance in predicting MGMT methylation status, using our dataset for the rebuilding process. ResultsThe Conventional model yielded 0.71 sensitivity, 0.78 specificity, and 0.75 accuracy, while the Advanced model scored 0.67, 0.74, and 0.71, respectively. The prior state-of-the-art model achieved 0.65, 0.57, and 0.62. Our Full model outperformed them with 0.78, 0.84, and 0.80 and had the highest AUC value (0.81) compared to Conventional (0.78) and previous (0.68) models. ConclusionOur predictive model, which incorporates radiomics features from conventional MRI scans and advanced DSC and DTI imaging, offers an improved approach to predicting the MGMT methylation status in GBM patients.

Prediction of Proteins in Cerebrospinal Fluid and Application to Glioma Biomarker Identification.

Cerebrospinal fluid (CSF) proteins are very important because they can serve as biomarkers for central nervous system diseases. Although many CSF proteins have been identified with wet experiments, the identification of CSF proteins is still a challenge. In this paper, we propose a novel method to predict proteins in CSF based on protein features. A two-stage feature-selection method is employed to remove irrelevant features and redundant features. The deep neural network and bagging method are used to construct the model for the prediction of CSF proteins. The experiment results on the independent testing dataset demonstrate that our method performs better than other methods in the prediction of CSF proteins. Furthermore, our method is also applied to the identification of glioma biomarkers. A differentially expressed gene analysis is performed on the glioma data. After combining the analysis results with the prediction results of our model, the biomarkers of glioma are identified successfully.

A novel deep learning ensemble model based on two-stage feature selection and intelligent optimization for water quality prediction

Accurate prediction of effluent total nitrogen (E-TN) can assist in feed-forward control of wastewater treatment plants (WWTPs) to ensure effluent compliance with standards while reducing energy consumption. However, multivariate time series prediction of E-TN is a challenge due to the complex nonlinearity of WWTPs. This paper proposes a novel prediction framework that combines a two-stage feature selection model, the Golden Jackal Optimization (GJO) algorithm, and a hybrid deep learning model, CNN-LSTM-TCN (CLT), aiming to effectively capture the nonlinear relationships of multivariate time series in WWTPs. Specifically, convolutional neural network (CNN), long short-term memory (LSTM), and temporal convolutional network (TCN) combined to build a hybrid deep learning model CNN-LSTM-TCN (CLT). A two-stage feature selection method is utilized to determine the optimal feature subset to reduce the complexity and improve the accuracy of the prediction model, and then, the feature subset is input into the CLT. The hyperparameters of the CLT are optimized using GJO to further improve the prediction performance. Experiments indicate that the two-stage feature selection model learns the optimal feature subset to predict best, and the GJO-CLT achieves the best performance for different backtracking windows and prediction steps. These results demonstrate that the prediction system excels in the task of multivariate water quality time series prediction of WWTPs.

Two-Stage Feature Selection Method Created for 20 Neurons Artificial Neural Networks for Automatic Breast Cancer Detection

Breast cancer is a common deadly diseases in women. Initial recognition of breast cancer using mammogram images is a challenging task. Hence, this paper proposed a unique automatic diagnosis model for breast cancer. Initially, the mammogram images are preprocessed with a median filter and contrast limited adaptive histogram equalization (CLAHE). The pre-processed image is automatically segmented using the multilevel threshold method. Subsequently, statistical, texture, shape, and geometric features are extracted from the segmented image. So, the length of the feature vector is high, and it is important to identify optimum features. In this paper, the dimension of the feature vector is reduced by 2-stage feature selection methods. Initially, the feature vector is applied to the best first search method information gain (IG) with rank feature method, and then secondly, apply the Pearson correlation method (PCM). Artificial neural networks (ANNs) are used to increase the classification accuracy of a breast cancer diagnosis. In this model, the selection of appropriate neurons in a single hidden layer is used to avoid overfitting problems in an ANN model. Based on optimum feature selection, the appropriate number of neurons chosen in the hidden layer is 20, which was applied for the proposed IG+PCM+Boosted-ANN model. The proposed model is applied on 2 regular datasets mini-Mammographic Image Analysis Society (mini-MIAS) and Digital Database for Screening Mammography (DDSM). The proposed model was superior to other exiting models and the model in this study achieves the accuracy of 99 and 98.80 % for mini-MIAS and DDSM datasets, respectively.

A probabilistic forecasting approach for air quality spatio-temporal data based on kernel learning method

Reliable air quality index forecasting is an important guarantee for government to take early emergency measures to protect the ecological environment and public health. Nevertheless, providing accurate and reliable forecasting is still a difficult and challenging task, owing to the high nonlinearity and noise in the AQI (air quality index). This study develops a novel framework based on feature selection, the Error Correction Model, and the Residual estimation method with new structure fast kernels, to conveniently improve the prediction performance and efficiently quantify the uncertainty of the point prediction generated by any pre-trained model. This framework takes some representative ML (machine learning) tools as examples, with the significant input variables extracted by the two-stage feature selection method for accurate AQI point prediction, and models their residuals by the multi-output Gaussian process and the multi-output Student−t process, which rely on the new multi-output I/O kernels with chronological memory to embed the relevant information of the point prediction and the residuals as well as of different output for making predictions on residual distributions, thereby enhancing prediction performance and enriching uncertainty information. Besides, an acceleration method based on the Orthogonal Random Features method is also proposed to enable the framework to handle large-scale data at an acceptable time cost. The framework is validated by the AQI Spatio-temporal data of four observation points in Shijiazhuang, China, and the result reveals that the proposed framework provides valid prediction intervals and enhances the accuracy of the ML models’ point predictions. Specifically, the framework reduces the prediction error of almost all basic models, and provides shorter prediction intervals than other common methods. Especially, the multi-output GP model or multi-output TP model obtains the best improvement in point prediction and the shortest prediction intervals.

How to capture tourists’ search behavior in tourism forecasts? A two-stage feature selection approach

Search engine data have been widely used and shown to be useful in tourism demand forecasting. However, considering of the vast amounts of search keywords, how to better capture the tourists’ attention and explore the most predictive keyword combination remain unsolved. In this study, a two-stage feature selection-based methodology is proposed to address this question. Specifically, i.e., single feature selection method comparison for selecting a relative effective way to reduce the data dimension and ensure the quality of the initial subset, genetic algorithm in the second stage for obtaining feature subset better suitable for forecasting model with stronger predictive power. Experimental results indicate that the two-stage feature selection method outperforms all the considered benchmarks.

A Multi-Factor Driven Model for Locomotive Axle Temperature Prediction Based on Multi-Stage Feature Engineering and Deep Learning Framework

Recently, with the increasing scale of the volume of freight transport and the number of passengers, the study of railway vehicle fault diagnosis and condition management is becoming more significant than ever. The axle temperature plays a significant role in the locomotive operating condition assessment that sudden temperature changes may lead to potential accidents. To realize accurate real-time condition monitoring and fault diagnosis, a new multi-data-driven model based on reinforcement learning and deep learning is proposed in this paper. The whole modeling process contains three steps: In step 1, the feature crossing and reinforcement learning methods are applied to select the suitable features that could efficiently shorten the redundancy of the input. In step 2, the stack denoising autoencoder is employed to extract deep fluctuation information in the features after the reinforcement learning. In step 3, the bidirectional gated recurrent unit algorithm is utilized to accomplish the forecasting model and achieve the final results. These parts of the integrated modeling structure contributed to increased forecasting accuracy than single models. By analyzing the forecasting results of three different data series, it could be summarized that: (1) The proposed two-stage feature selection method and feature extraction method could greatly optimize the input for the predictor and form the optimal axle temperature forecasting model. (2) The proposed hybrid model can achieve satisfactory forecasting results which are better than the contrast algorithms proposed by other researchers.

A novel EEG-based major depressive disorder detection framework with two-stage feature selection

BackgroundMajor depressive disorder (MDD) is a common mental illness, characterized by persistent depression, sadness, despair, etc., troubling people’s daily life and work seriously.MethodsIn this work, we present a novel automatic MDD detection framework based on EEG signals. First of all, we derive highly MDD-correlated features, calculating the ratio of extracted features from EEG signals at frequency bands between beta and alpha. Then, a two-stage feature selection method named PAR is presented with the sequential combination of Pearson correlation coefficient (PCC) and recursive feature elimination (RFE), where the advantages lie in minimizing the feature searching space. Finally, we employ widely used machine learning methods of support vector machine (SVM), logistic regression (LR), and linear regression (LNR) for MDD detection with the merit of feature interpretability.ResultsExperiment results show that our proposed MDD detection framework achieves competitive results. The accuracy and F_{1} score are up to 0.9895 and 0.9846, respectively. Meanwhile, the regression determination coefficient R^2 for MDD severity assessment is up to 0.9479. Compared with existing MDD detection methods with the best accuracy of 0.9840 and F_1 score of 0.97, our proposed framework achieves the state-of-the-art MDD detection performance.ConclusionsDevelopment of this MDD detection framework can be potentially deployed into a medical system to aid physicians to screen out MDD patients.

Improving MGMT methylation status prediction of glioblastoma through optimizing radiomics features using genetic algorithm-based machine learning approach

O6-Methylguanine-DNA-methyltransferase (MGMT) promoter methylation was shown in many studies to be an important predictive biomarker for temozolomide (TMZ) resistance and poor progression-free survival in glioblastoma multiforme (GBM) patients. However, identifying the MGMT methylation status using molecular techniques remains challenging due to technical limitations, such as the inability to obtain tumor specimens, high prices for detection, and the high complexity of intralesional heterogeneity. To overcome these difficulties, we aimed to test the feasibility of using a novel radiomics-based machine learning (ML) model to preoperatively and noninvasively predict the MGMT methylation status. In this study, radiomics features extracted from multimodal images of GBM patients with annotated MGMT methylation status were downloaded from The Cancer Imaging Archive (TCIA) public database for retrospective analysis. The radiomics features extracted from multimodal images from magnetic resonance imaging (MRI) had undergone a two-stage feature selection method, including an eXtreme Gradient Boosting (XGBoost) feature selection model followed by a genetic algorithm (GA)-based wrapper model for extracting the most meaningful radiomics features for predictive purposes. The cross-validation results suggested that the GA-based wrapper model achieved the high performance with a sensitivity of 0.894, specificity of 0.966, and accuracy of 0.925 for predicting the MGMT methylation status in GBM. Application of the extracted GBM radiomics features on a low-grade glioma (LGG) dataset also achieved a sensitivity 0.780, specificity 0.620, and accuracy 0.750, indicating the potential of the selected radiomics features to be applied more widely on both low- and high-grade gliomas. The performance indicated that our model may potentially confer significant improvements in prognosis and treatment responses in GBM patients.