Forward Feature Selection Algorithm Research Articles

Identifying Key Factors for Securing a Champions League Position in French Ligue 1 Using Explainable Machine Learning Techniques

Introduction: Performance analysis is essential for coaches and a topic of extensive research. The advancement of technology and Artificial Intelligence (AI) techniques has revolutionized sports analytics. Aim: The primary aim of this article is to present a robust, explainable machine learning (ML) model that identifies the key factors that contribute to securing one of the top three positions in the standings of the French Ligue 1, ensuring participation in the UEFA Champions League for the following season. Materials and Methods: This retrospective observational study analyzed data from all 380 matches of the 2022–23 French Ligue 1 season. The data were obtained from the publicly-accessed website “whoscored” and included 34 performance indicators. This study employed Sequential Forward Feature Selection (SFFS) and various ML algorithms, including XGBoost, Support Vector Machine (SVM), and Logistic Regression (LR), to create a robust, explainable model. The SHAP (SHapley Additive Explanations) model was used to enhance model interpretability. Results: The K-means Cluster Analysis categorized teams into groups (TOP TEAMS, 3 teams/REST TEAMS, 17 teams), and the ML models provided significant insights into the factors influencing league standings. The LR classifier was the best-performing classifier, achieving an accuracy of 75.13%, a recall of 76.32%, an F1-score of 48.03%, and a precision of 35.17%. “SHORT PASSES” and “THROUGH BALLS” were features found to positively influence the model’s predictions, while “TACKLES ATTEMPTED” and “LONG BALLS” had a negative impact. Conclusions: Our model provided satisfactory predictive accuracy and clear interpretability of results, which gave useful information to stakeholders. Specifically, our model suggests adopting a strategy during the ball possession phase that relies on short passes (avoiding long ones) and aiming to enter the attacking third and the opponent’s penalty area with through balls.

Cognitive pairwise comparison forward feature selection with deep learning for astronomical object classification with sloan digital sky survey

This paper proposes a hybrid approach integrating the expert knowledge judgment approach using the Cognitive Pairwise Comparison (CPC) to the Deep Learning, a modern classification approach, for astronomic object classification. The astronomic data with ten thousand samples retrieved from Sloan Digital Sky Survey Sky Server Data Release 15 (SDSS SkyServer DR 15) are used for this study. The CPC is an approach to elicit and encode expert knowledge in the format of a Pairwise Opposite Matrix (POM) to evaluate expert preferences for the features. A forward feature selection algorithm taking the expert choices using CPC for the ordered features is used for the feature selection for the deep learning algorithm to build a heuristic training model based on the astronomic data. Whilst the accuracy of the case of improper feature selection is just 37.1%, the proposed hybrid approach can obtain a very high accuracy of 97.9% for the classification of the astronomic object using the eight scaled features (u, g, r, i, z redshift, ra, dec). To extend this research, the proposed CPC can be used as a human-centered tool to be applied to other areas of data sciences.

Uncertainty measures and feature selection based on composite entropy for generalized multigranulation fuzzy neighborhood rough set

With the continuous advancement of information technology, the information and data covered by various information systems become increasingly complex and diverse, it is essential to perform knowledge mining from multiple perspectives to extract valuable insights. Fuzzy neighborhood multigranulation rough set, as an excellent feature selection model, is capable of handling heterogeneous datasets more effectively, significantly improving learning efficiency. In this study, we investigate a feature selection method based on a generalized multigranulation fuzzy rough set (GMFNRS) in fuzzy decision systems. First, the concepts of fuzzy neighborhood rough sets and generalized multigranulation rough sets are introduced. Subsequently, the GMFNRS model is established to enable data mining and rule extraction from various perspectives. Secondly, from an informational perspective, the study investigates uncertainty measurement methods through fuzzy neighborhood joint entropy. Furthermore, a novel fuzzy neighborhood generalized composite entropy is proposed by integrating the GMFNRS model with uncertainty measures. Finally, a forward greedy feature selection algorithm is considered to extract essential information from complex datasets. Experimental results on 15 public datasets demonstrate that the proposed model effectively selects important features in fuzzy systems and exhibits excellent classification performance.

Improved Elman Deep Learning Model for Intrusion Detection System in Internet of Things

Many researchers have developed intrusion detection systems in the past using conventional techniques such as artificial neural networks, fuzzy clustering, evolutionary algorithms, association rule mining, and support vector machines. However, in terms of false negative rates and detection rates, these methods did not yield the best outcomes. To address these problems, we proposed a hybrid deep learning model (HDLM) based on intrusion detection and prevention in IoT devices. Initially, the data are collected from KDDCup-99 and NSL-KDD datasets. Then, the important features are extracted from the dataset using the Forward Feature Selection Algorithm (FFSA). Finally, the extracted features are given to the HDLM classifier. The proposed HDLM is a combination of Elman Recurrent Neural Network (ERNN) and Subtraction-Average-Based Optimizer (SABO). The performance of the suggested method is assessed using performance metrics including precision, recall, accuracy, sensitivity, specificity, and F_Measure. The experimental results show that the proposed method attained the maximum intrusion detection accuracy of 98.52%.

Forward feature selection: empirical analysis

Feature selection is an important preprocessing step in many data science and machine learning applications. Although there exist several sophisticated feature selection algorithms, their benefits are sometimes overshadowed by their complexity and slow execution. Therefore, in many cases, a more simple algorithm may be better suited. In this paper, we demonstrate that a rudimentary forward selection algorithm can achieve optimal performance with a low time complexity. Our study is based on an extensive empirical evaluation of the forward feature selection algorithm in the context of linear regression. Concretely, we compare the forward selection algorithm against the gold standard exhaustive search algorithm based on several datasets. The results show that the forward selection algorithm achieves high performance with relatively fast execution. Given the simplicity, accuracy, and speed of the forward feature selection algorithm, we recommend it as a primary feature selection method for most regression applications. Our results are particularly pertinent in the case of big data and real-time analysis.

A decision support system for osteoporosis risk prediction using machine learning and explainable artificial intelligence

Osteoporosis is a metabolic bone condition that occurs when bone mineral density and mass decrease. This makes the bones weak and brittle. The disorder is often undiagnosed and untreated due to its asymptomatic nature until the manifestation of a fracture. Machine Learning (ML) is extensively used in diverse healthcare domains to analyze precise outcomes, provide timely risk scores, and allocate resources. Hence, we have designed multiple heterogeneous machine-learning frameworks to predict the risk of Osteoporosis. An open-source dataset of 1493 patients containing bone density, blood, and physical tests is utilized. Thirteen distinct feature selection techniques were leveraged to extract the most salient parameters. The best-performing pipeline consisted of a Forward Feature Selection algorithm followed by a custom multi-level ensemble learning-based stack, which achieved an accuracy of 89 %. Deploying a layer of explainable artificial intelligence using tools such as SHAP (SHapley Additive Values), LIME (Local Interpretable Model Explainer), ELI5, Qlattice, and feature importance provided interpretability and rationale behind classifier prediction. With this study, we aim to provide the holistic risk prediction of Osteoporosis and concurrently present a system for automated screening to assist physicians in making diagnostic decisions.

Wind power plants hybridised with solar power: A generation forecast perspective

The association of different variable renewable technologies in hybrid power plants and the benefits of their aggregation for the operation of power systems is an area of recent research. Accurate forecasts are crucial for extracting those benefits and promote an optimal integration of such plants into power systems and electricity markets. This study focuses on the hybridisation of existing wind power plants with different shares of solar photovoltaic capacity and investigates how these power plants can reduce their combined forecast errors and thus, increasing profitability in electricity markets.The work uses a forecast methodology based on a sequential forward feature selection algorithm which employs two different objective functions and an artificial neural network approach previously presented but, in this case, it is applied to the specific case of hybrid power plants. The methodology uses as input data from a numerical weather prediction model and iteratively selects meteorological features to achieve the different objective functions implemented, namely i) minimisation of the root mean square error; or ii) maximisation of the market remuneration.The methodology developed was applied to three case studies in Portugal with different levels of wind and solar generation complementarity. The results show that the hybrid power plants can increase market value by up to 5% and total remuneration can increase by up to 30% when compared with the existing wind power plant, while it is possible to reduce the forecast errors by nearly 4%. The obtained results highlight the need to select the most relevant meteorological features to maximise the accuracy of the power forecast and the renewable power producers revenues in a market environment.

Improving prediction of drug-target interactions based on fusing multiple features with data balancing and feature selection techniques.

Drug discovery relies on predicting drug-target interaction (DTI), which is an important challenging task. The purpose of DTI is to identify the interaction between drug chemical compounds and protein targets. Traditional wet lab experiments are time-consuming and expensive, that's why in recent years, the use of computational methods based on machine learning has attracted the attention of many researchers. Actually, a dry lab environment focusing more on computational methods of interaction prediction can be helpful in limiting search space for wet lab experiments. In this paper, a novel multi-stage approach for DTI is proposed that called SRX-DTI. In the first stage, combination of various descriptors from protein sequences, and a FP2 fingerprint that is encoded from drug are extracted as feature vectors. A major challenge in this application is the imbalanced data due to the lack of known interactions, in this regard, in the second stage, the One-SVM-US technique is proposed to deal with this problem. Next, the FFS-RF algorithm, a forward feature selection algorithm, coupled with a random forest (RF) classifier is developed to maximize the predictive performance. This feature selection algorithm removes irrelevant features to obtain optimal features. Finally, balanced dataset with optimal features is given to the XGBoost classifier to identify DTIs. The experimental results demonstrate that our proposed approach SRX-DTI achieves higher performance than other existing methods in predicting DTIs. The datasets and source code are available at: https://github.com/Khojasteh-hb/SRX-DTI.

Innovative Forward Fusion Feature Selection Algorithm for Sentiment Analysis Using Supervised Classification

Sentiment analysis is considered one of the significant trends of the recent few years. Due to the high importance and increasing use of social media and electronic services, the need for reviewing and enhancing the provided services has become crucial. Revising the user services is based mainly on sentiment analysis methodologies for analyzing users’ polarities to different products and applications. Sentiment analysis for Arabic reviews is a major concern due to high morphological linguistics and complex polarity terms expressed in the reviews. In addition, the users can present their orientation towards a service or a product by using a hybrid or mix of polarity terms related to slang and standard terminologies. This paper provides a comprehensive review of recent sentiment analysis methods based on lexicon or machine learning (ML). The comparison provides a clear vision of the number of classes, the used dialect, the annotated algorithms, and their performance. The proposed methodology is based on cross-validation of Arabic data using a k-fold mechanism that splits the dataset into training and testing folds; subsequently, the data preprocessing is executed to clean sentiments from unwanted terms that can affect data analysis. A vectorization of the dataset is then applied using TF–IDF for counting word and polarity terms. Furthermore, a feature selection stage is processed using Pearson, Chi2, and Random Forest (RF) methods for mapping the compatibility between input and target features. This paper also proposed an algorithm called the forward fusion feature for sentiment analysis (FFF-SA) to provide a feature selection that applied different machine learning (ML) classification models for each chunk of k features and accumulative features on the Arabic dataset. The experimental results measured and scored all accuracies between the feature importance method and ML models. The best accuracy is recorded with the Naïve Bayes (NB) model with the RF method.

A tree growth based forward feature selection algorithm for intrusion detection system on convolutional neural network

With the rapid advancement of networking technologies, security system has become increasingly important to academics from several sectors. Intrusion detection (ID) provides a valuable protection by reducing the human resources required to keep an eye on intruders, improving the efficiency of detecting the various attacks in networks. Machine learning and deep learning are two key areas that have recently received a lot of attention, with a focus on improving the precision of detection classifiers. Using defense anvance research project agency (DARPA”98) datasets, a number of academics and research have developed intrusion detection systems. This paper discusses various approaches developed by different researchers, including scale-hybrid-IDS-AlertNet (SHIA), forward feature selection algorithm (FFSA), modified- mutual information feature selection (MMIFS), deep neural network (DNN), and the holes that remain to be filled, highlighting areas where these procedures can be improved, also are addressed and the proposed approach improved deep convolutional neural network (IDCNN) is compared with existing approach.

Feature Subset Selection With Multi-Scale Fuzzy Granulation

As a typical multigranularity data analysis model, multi-scale rough sets have attracted considerable attention in recent years. However, classical multi-scale rough sets and most of its extended models can only deal with discrete data, which limits its popularization and application. To overcome this problem, we investigate the fuzzy generalization of multi-scale rough sets as well as their application in feature selection for continuous data. To this end, a new type of decision systems, i.e., multi-scale fuzzy decision systems, is formalized to represent knowledge at different scales. Scaled fuzzy granules in terms of a family of scaled fuzzy relations are introduced, using which the granular structures of fuzzy lower and upper approximations are presented. A heuristic lattice-based optimal scale selection algorithm is then put forward from the viewpoint of maintaining the consistency of decision systems. Decision rules with strong generalization ability can be obtained by selecting appropriate scales. Finally, a forward feature selection algorithm was developed by means of the optimal scale to reduce redundant fuzzy relations. Extensive numerical experiments are further conducted to compare the proposed algorithm with some state-of-the-art algorithms. The experimental results show that our model can improve the generalization ability of fuzzy rough set, so as to be more feasible and effective.

Forecasting Coronary Heart Disease Risk With a 2-Step Hybrid Ensemble Learning Method and Forward Feature Selection Algorithm

Forecasting Coronary Heart Disease Risk With a 2-Step Hybrid Ensemble Learning Method and Forward Feature Selection Algorithm

Feature selection based on neighborhood rough sets and Gini index.

Neighborhood rough set is considered an essential approach for dealing with incomplete data and inexact knowledge representation, and it has been widely applied in feature selection. The Gini index is an indicator used to evaluate the impurity of a dataset and is also commonly employed to measure the importance of features in feature selection. This article proposes a novel feature selection methodology based on these two concepts. In this methodology, we present the neighborhood Gini index and the neighborhood class Gini index and then extensively discuss their properties and relationships with attributes. Subsequently, two forward greedy feature selection algorithms are developed using these two metrics as a foundation. Finally, to comprehensively evaluate the performance of the algorithm proposed in this article, comparative experiments were conducted on 16 UCI datasets from various domains, including industry, food, medicine, and pharmacology, against four classical neighborhood rough set-based feature selection algorithms. The experimental results indicate that the proposed algorithm improves the average classification accuracy on the 16 datasets by over 6%, with improvements exceeding 10% in five. Furthermore, statistical tests reveal no significant differences between the proposed algorithm and the four classical neighborhood rough set-based feature selection algorithms. However, the proposed algorithm demonstrates high stability, eliminating most redundant or irrelevant features effectively while enhancing classification accuracy. In summary, the algorithm proposed in this article outperforms classical neighborhood rough set-based feature selection algorithms.

Enhancing wind power forecast accuracy using the weather research and forecasting numerical model-based features and artificial neuronal networks

Forecasting with accuracy the quantity of energy produced by wind power plants is crucial to enabling its optimal integration into power systems and electricity markets. Despite the remarkable improvements in the wind forecasting systems in recent years, large errors can still be observed, especially for longer time horizons. This work focuses on identifying new numerical weather prediction (NWP)-based features aiming to improve the overall quality of wind power forecasts.The methodology also incorporates a sequential forward feature selection algorithm. This algorithm was designed to select iteratively the meteorological features which minimize the wind forecast errors.The methodology was applied separately to seven wind parks in Portugal with different climate characteristics. The proposed approach allowed a reduction between 13% and 37% in the root mean square errors of wind power forecasts, compared with a baseline scenario. While the meteorological features identified for each wind park showed similarities within regions with analogous wind power generation profiles, each wind park required specific meteorological parameters as input data to obtain the best performance. Thus, the results show to be crucial to select the most relevant features of a specific site to maximize the accuracy of a wind power forecast.

Load2Load: Day-ahead load forecasting at aggregated level

A reliable and accurate short-term load forecasting (STLF) helps utilities and energy providers deal with the challenges posed by supply and demand balance, higher penetration of renewable energies and the development of electricity markets with increasingly complex pricing strategies in future smart grids. Recent advances in deep learning have been successively utilized to STLF. However, there is no certain study that evaluates the performances of different STLF methods at an aggregated level on different datasets with different numbers of daily measurements.In this study, a deep learning STLF architecture called Load2Load is proposed for day-ahead forecasting. Different forecasting methods have been evaluated and compared on two datasets with different temporal resolutions and features. An additive ensemble method as well as a selective ensemble method that selects the outputs of different forecasters in an hourly manner are proposed. Moreover; a modified sequential forward feature selection algorithm is proposed, resulting in better performance with a much smaller number of features.Numerical results show that the proposed Load2Load architecture has a competing performance compared to other advanced forecasters. When used together with the proposed ensemble methods, Load2Load can significantly improve the forecasting performance. The proposed feature selection algorithm results in better performance for the majority of the cases while reducing the dimensionality. According to the results with two different datasets; the proposed methods are shown to be robust to temporal resolutions, feature types and sequence lengths.

Dependency maximization forward feature selection algorithms based on normalized cross-covariance operator and its approximated form for high-dimensional data

Supervised feature selection (FS) for classification aims at finding a more discriminative subset from original features, to facilitate classifier training, improve classification performance and enhance model interpretability. Primary normalized cross-covariance operator (NOCCO) is a nonlinear kernel-based dependency measure between features and labels, including two inverse matrices, and its approximated version (ANOCCO) is simplified via exploiting linear kernel for features, delta kernel for labels and Moore–Penrose inverse. In this paper, we apply NOCCO and ANOCCO to FS task. According to sequential forward selection, a forward NOCCO-based FS algorithm (i.e., FoNOCCO) is directly implemented via various accelerating strategies, but its computational complexity is extremely high. To this end, we propose its fast version via maximizing ANOCCO (i.e., FoANOCCO), where each candidate feature is evaluated efficiently according to column recursive algorithm for Moore–Penrose inverse. Theoretical analysis shows that evaluating a candidate feature needs a time complexity of O(N2) in FoANOCCO and O(N3) in FoNOCCO, where N is the number of instances. On eight small-size high-dimensional benchmark data sets, our detailed experiments illustrate that our two forward FS algorithms are superior to six state-of-the-art FS methods with four baseline classifiers, and FoANOCCO runs 70 times faster than FoNOCCO with a 1.79 % accuracy drop.

Computational prediction of Calu-3-based in vitro pulmonary permeability of chemicals

Pulmonary is a potential route for drug delivery and exposure to toxic chemicals. The human bronchial epithelial cell line Calu-3 is generally considered to be a useful in vitro model of pulmonary permeability by calculating the apparent permeability coefficient (Papp) values. Since in vitro experiments are time-consuming and labor-intensive, computational models for pulmonary permeability are desirable for accelerating drug design and toxic chemical assessment. This study presents the first attempt for developing quantitative structure-activity relationship (QSAR) models for addressing this goal. A total of 57 chemicals with Papp values based on Calu-3 experiments was first curated from literature for model development and testing. Subsequently, eleven descriptors were identified by a sequential forward feature selection algorithm to maximize the cross-validation performance of a voting regression model integrating linear regression and nonlinear random forest algorithms. With applicability domain adjustment, the developed model achieved high performance with correlation coefficient values of 0.935 and 0.824 for cross-validation and independent test, respectively. The preliminary results showed that computational models could be helpful for predicting Calu-3-based in vitro pulmonary permeability of chemicals. Future works include the collection of more data for further validating and improving the model.

A Data-Driven Short-Term PV Generation and Load Forecasting Approach for Microgrid Applications

The data-driven (DD) is a systematic approach to improve the data and model by deriving/adding features to address the problem identified during the iterative loop of forecasting model development. This article proposes a DD framework for forecasting short-term PV generation and load demand. A framework of three stages with a unique contribution in each stage, such as generalizing data preprocessing steps (stage-1), multivariate feature generation and selection (stage-2), and model hyperparameter tuning (stage-3) for further improvement in forecasting is proposed. It focuses on data as well as forecasting models. The whole process is analyzed using the time-series measured data collected from a real-life demonstration project in Ireland. Data preprocessing is generalized for both generation and demand forecasting under the same framework. The relevant features are selected with the help of the proposed random forest sequential forward feature selection algorithm. Hyperparameters are tuned through tree-structured Parzen estimator algorithm for further improvement. In addition, the performance of the classical autoregressive integrated moving average model is compared with the machine learning-based gate recurrent unit, long short term memory, recurrent neural network, and convolutional neural network models. Results show that the data-driven forecasting model framework systematically improves the model performance. The seasonal variation has also a high impact on the model performances.

Neighborhood rough set based multi‐label feature selection with label correlation

SummaryNeighborhood rough set (NRS) is considered as an effective tool for feature selection and has been widely used in processing high‐dimensional data. However, most of the existing methods are difficult to deal with multi‐label data and are lack of considering label correlation (LC), which is an important issue in multi‐label learning. Therefore, in this article, we introduce a new NRS model with considering LC. First, we explore LC by calculating the similarity relation between labels and divide the related labels into several label subsets. Then, a new neighborhood relation is proposed, which can solve the problem of neighborhood granularity selection by using the nearest neighbor information distribution of instances under the related labels. On this basis, the NRS model is reconstructed by embedding LC information, and the related properties of the model are discussed. Moreover, we design a new feature significance function to evaluate the quality of features, which can well capture the specific relationship between features and labels. Finally, a greedy forward feature selection algorithm is designed. Extensive experiments which are conducted on different types of datasets verify the effectiveness of the proposed algorithm.

Interpretable Machine Learning Models for Malicious Domains Detection Using Explainable Artificial Intelligence (XAI)

With the expansion of the internet, a major threat has emerged involving the spread of malicious domains intended by attackers to perform illegal activities aiming to target governments, violating privacy of organizations, and even manipulating everyday users. Therefore, detecting these harmful domains is necessary to combat the growing network attacks. Machine Learning (ML) models have shown significant outcomes towards the detection of malicious domains. However, the “black box” nature of the complex ML models obstructs their wide-ranging acceptance in some of the fields. The emergence of Explainable Artificial Intelligence (XAI) has successfully incorporated the interpretability and explicability in the complex models. Furthermore, the post hoc XAI model has enabled the interpretability without affecting the performance of the models. This study aimed to propose an Explainable Artificial Intelligence (XAI) model to detect malicious domains on a recent dataset containing 45,000 samples of malicious and non-malicious domains. In the current study, initially several interpretable ML models, such as Decision Tree (DT) and Naïve Bayes (NB), and black box ensemble models, such as Random Forest (RF), Extreme Gradient Boosting (XGB), AdaBoost (AB), and Cat Boost (CB) algorithms, were implemented and found that XGB outperformed the other classifiers. Furthermore, the post hoc XAI global surrogate model (Shapley additive explanations) and local surrogate LIME were used to generate the explanation of the XGB prediction. Two sets of experiments were performed; initially the model was executed using a preprocessed dataset and later with selected features using the Sequential Forward Feature selection algorithm. The results demonstrate that ML algorithms were able to distinguish benign and malicious domains with overall accuracy ranging from 0.8479 to 0.9856. The ensemble classifier XGB achieved the highest result, with an AUC and accuracy of 0.9991 and 0.9856, respectively, before the feature selection algorithm, while there was an AUC of 0.999 and accuracy of 0.9818 after the feature selection algorithm. The proposed model outperformed the benchmark study.