Feature Selection Procedure Research Articles

Graph Random Forest: A Graph Embedded Algorithm for Identifying Highly Connected Important Features.

Random Forest (RF) is a widely used machine learning method with good performance on classification and regression tasks. It works well under low sample size situations, which benefits applications in the field of biology. For example, gene expression data often involve much larger numbers of features (p) compared to the size of samples (n). Though the predictive accuracy using RF is often high, there are some problems when selecting important genes using RF. The important genes selected by RF are usually scattered on the gene network, which conflicts with the biological assumption of functional consistency between effective features. To improve feature selection by incorporating external topological information between genes, we propose the Graph Random Forest (GRF) for identifying highly connected important features by involving the known biological network when constructing the forest. The algorithm can identify effective features that form highly connected sub-graphs and achieve equivalent classification accuracy to RF. To evaluate the capability of our proposed method, we conducted simulation experiments and applied the method to two real datasets-non-small cell lung cancer RNA-seq data from The Cancer Genome Atlas, and human embryonic stem cell RNA-seq dataset (GSE93593). The resulting high classification accuracy, connectivity of selected sub-graphs, and interpretable feature selection results suggest the method is a helpful addition to graph-based classification models and feature selection procedures.

On the challenges of predicting treatment response in Hodgkin’s Lymphoma using transcriptomic data

BackgroundDespite the advancements in multiagent chemotherapy in the past years, up to 10% of Hodgkin’s Lymphoma (HL) cases are refractory to treatment and, after remission, patients experience an elevated risk of death from all causes. These complications are dependent on the treatment and therefore an increase in the prognostic accuracy of HL can help improve these outcomes and control treatment-related toxicity. Due to the low incidence of this cancer, there is a lack of works comprehensively assessing the predictability of treatment response, especially by resorting to machine learning (ML) advances and high-throughput technologies.MethodsWe present a methodology for predicting treatment response after two courses of Adriamycin, Bleomycin, Vinblastine and Dacarbazine (ABVD) chemotherapy, through the analysis of gene expression profiles using state-of-the-art ML algorithms. We work with expression levels of tumor samples of Classical Hodgkin’s Lymphoma patients, obtained through the NanoString’s nCounter platform. The presented approach combines dimensionality reduction procedures and hyperparameter optimization of various elected classifiers to retrieve reference predictability levels of refractory response to ABVD treatment using the regulatory profile of diagnostic tumor samples. In addition, we propose a data transformation procedure to map the original data space into a more discriminative one using biclustering, where features correspond to discriminative putative regulatory modules.ResultsThrough an ensemble of feature selection procedures, we identify a set of 14 genes highly representative of the result of an fuorodeoxyglucose Positron Emission Tomography (FDG-PET) after two courses of ABVD chemotherapy. The proposed methodology further presents an increased performance against reference levels, with the proposed space transformation yielding improvements in the majority of the tested predictive models (e.g. Decision Trees show an improvement of 20pp in both precision and recall).ConclusionsTaken together, the results reveal improvements for predicting treatment response in HL disease by resorting to sophisticated statistical and ML principles. This work further consolidates the current hypothesis on the structural difficulty of this prognostic task, showing that there is still a considerable gap to be bridged for these technologies to reach the necessary maturity for clinical practice.

Feature selection translates drug response predictors from cell lines to patients.

Targeted therapies and chemotherapies are prevalent in cancer treatment. Identification of predictive markers to stratify cancer patients who will respond to these therapies remains challenging because patient drug response data are limited. As large amounts of drug response data have been generated by cell lines, methods to efficiently translate cell-line-trained predictors to human tumors will be useful in clinical practice. Here, we propose versatile feature selection procedures that can be combined with any classifier. For demonstration, we combined the feature selection procedures with a (linear) logit model and a (non-linear) K-nearest neighbor and trained these on cell lines to result in LogitDA and KNNDA, respectively. We show that LogitDA/KNNDA significantly outperforms existing methods, e.g., a logistic model and a deep learning method trained by thousands of genes, in prediction AUC (0.70-1.00 for seven of the ten drugs tested) and is interpretable. This may be due to the fact that sample sizes are often limited in the area of drug response prediction. We further derive a novel adjustment on the prediction cutoff for LogitDA to yield a prediction accuracy of 0.70-0.93 for seven drugs, including erlotinib and cetuximab, whose pathways relevant to anti-cancer therapies are also uncovered. These results indicate that our methods can efficiently translate cell-line-trained predictors into tumors.

Quantifying time perception during virtual reality gameplay using a multimodal biosensor-instrumented headset: a feasibility study.

We have all experienced the sense of time slowing down when we are bored or speeding up when we are focused, engaged, or excited about a task. In virtual reality (VR), perception of time can be a key aspect related to flow, immersion, engagement, and ultimately, to overall quality of experience. While several studies have explored changes in time perception using questionnaires, limited studies have attempted to characterize them objectively. In this paper, we propose the use of a multimodal biosensor-embedded VR headset capable of measuring electroencephalography (EEG), electrooculography (EOG), electrocardiography (ECG), and head movement data while the user is immersed in a virtual environment. Eight gamers were recruited to play a commercial action game comprised of puzzle-solving tasks and first-person shooting and combat. After gameplay, ratings were given across multiple dimensions, including (1) the perception of time flowing differently than usual and (2) the gamers losing sense of time. Several features were extracted from the biosignals, ranked based on a two-step feature selection procedure, and then mapped to a predicted time perception rating using a Gaussian process regressor. Top features were found to come from the four signal modalities and the two regressors, one for each time perception scale, were shown to achieve results significantly better than chance. An in-depth analysis of the top features is presented with the hope that the insights can be used to inform the design of more engaging and immersive VR experiences.

An improved one-class algorithm combined with NIR spectroscopy for detecting adulterated chemicals in weight-loss pills

Illegal chemicals may be added into weight-loss pills labeled as 100% natural. Taking into account public health and legal regulations, how to quickly and cheaply detect adulterants in these herbal pills is of great significance. Current efforts are focused on developing a near-infrared (NIR) spectroscopy-based non-destructive, rapid and cost-effective strategy for detecting phenolphthalein, the most commonly chemical adulterant in weight-loss pills. A total of 160 pure and adulterated samples were prepared. A feature selection procedure was used to compress variables. In the frame of ensemble, an improved class-modeling algorithm based on one-class partial least square (OCPLS) (named as “EOCPLS”) was proposed to construct the predictive model. Its performance is better than the original OCPLS. The sensitivity, specificity, and accuracy of the ECOPLS model on the test set were all equal to 1 based only on the score distance (SD) value. The results show that NIR spectroscopy together with the proposed procedure is effective and feasible for identifying the phenolphthalein in weight-loss pills.

Kratkoročna prognoza vidljivosti određena metodom slučajne šume

Accurate visibility forecasting is essential for safe aircraft operations. This study examines how various configurations of the Random Forest model can enhance visibility predictions. Preprocessing techniques are employed, including correlation analysis to identify fundamental relationships in weather observations. Time-series data is transformed into a regular Data Frame to facilitate analysis. This study proposes a classification framework for organizing visibility data and phenomena, which is then used to develop a visibility forecast using the Random Forest method. The study also presents procedures for hyperparameter tuning, feature selection, data balancing, and accuracy evaluation for this dataset. The main outcomes are the Random Forest model parameters for a three-hour visibility forecast, along with an analysis of errors in low visibility forecasts. Additionally, models for one-hour forecasts and visibility forecasting under precipitation are also examined. The resulting models demonstrate a deterministic forecast accuracy of approximately 78%, with a false alarm rate of around 6%, providing a comprehensive overview of the capabilities of the Random Forest model for visibility forecasting. As anticipated, the model demonstrated limitations in accurately simulating fast radiative cooling or abrupt decreases in visibility caused by precipitation. Specifically, in relation to precipitation, the model achieved an accuracy of 79%, yet exhibited a false alarm rate of 19%. Additionally, this method sets a foundation for enhancing prediction accuracy through the inclusion of supplementary forecast data, while its implementation on real-world datasets expands the reach of machine learning techniques to the members of the meteorological community.

Classification and feature selection methods based on fitting logistic regression to PU data

In our work, we examine the classification methods where the positive and unlabeled data are considered and where the conditional distribution of the true class label given the feature vector is governed by the model of logistic regression. Our first objective is to compute and compare the selected metrics allowing for the quality assessment of these methods. In this context, we investigate four methods of the posterior probability estimation, where the risk of logistic loss function is optimized: the naive approach, the weighted likelihood approach, as well as the quite recently proposed methods – the joint approach, and the LassoJoint method. The corresponding evaluations are basically performed for 13 machine learning models on some chosen – both low- and high-dimensional – datasets. Some of the mentioned machine learning model schemes have been directly borrowed from literature and some have been obtained through some modifications in the existing procedures. Our second goal is to establish the most stable and efficient approach for the posterior probability estimation. Moreover, we use the AdaSampling scheme for comparison of the considered classification methods. We also conducted comparisons of feature selection procedures – the Mutual Information-Based feature selection method and the LassoJoint approach. The current article is an enhancement of the conference paper Furmańczyk et al. (2022).

“Location, Location, Location”: Fluctuations in Real Estate Market Values after COVID-19 and the War in Ukraine Based on Econometric and Spatial Analysis, Random Forest, and Multivariate Regression

In this research, the authors aim to detect the marginal appreciation of construction and neighbourhood characteristics of property prices at three different time points: before the COVID-19 pandemic, two years after the first COVID-19 alert but before the War in Ukraine, and one year after the outbreak of the War. The marginal appreciations of the building’s features are analysed for a pilot case study in Northern Italy using a Random Forest feature importance analysis and a Multivariate Regression. Several techniques are integrated into this study, such as computer programming in Python language, multi-parametric value assessment techniques, feature selection procedures, and spatial analysis. The results may represent an interesting ongoing monitoring of how these anomalous events affect the buyer’s willingness to pay for specific characteristics of the buildings, with particular attention to the location features of the neighbourhood and accessibility.

An intelligent feature selection method using binary teaching-learning based optimization algorithm and ANN

The most challenging issue in dealing with big datasets is the large number of their dimensions. Feature selection is a technique for reducing the dimensionality of datasets by removing irrelevant and useless features from the dataset, finally selecting the most crucial features in the dataset, will ultimately increase the algorithm's efficiency. This paper presents a novel feature selection procedure that includes a binary-modified teaching learning-based optimization algorithm (BMTLBO). One of the most effective and popular methods is the TLBO algorithm. Although this algorithm has a good convergence rate, there is a chance to get stuck in a local optimum. Our goal is to balance between exploration and exploitation capabilities. The proposed method consists of two parts: First, the BMTLBO algorithm, which includes the improved binary version of the basic algorithm, is used for the feature selection problem. The idea of a pool is used in this algorithm that boosts the population's diversity while enhancing the algorithm's precision and convergence rate. Second, we are used from improved TLBO algorithm with the self-learning phase (SLTLBO) for neural network training to show the application of the classification problem to evaluate the performance of the procedures of the method. We are tested the proposed method on 14 datasets in the term of classification accuracy and the number of features. Additionally, a different chemical data set named Biodegradable is used to assess the proposed method. The evaluation results demonstrate that, in terms of accuracy, convergence rate, and success in achieving promising solutions, the proposed algorithm outperforms all other optimization algorithms compared. The results are very promising and close to optimal.

A two-step feature selection procedure for relevant markers of Squamous Cell Lung Carcinoma using different survival models

There are potentially infinite gene expression markers for Lung Squamous Cell Carcinoma. This results in a high-dimensional data with a large number of features. The selection of relevant markers for analysis is thus, of utmost importance. In our study, we have aimed to select a subset of prominent and significant features from 31918 features of gene expressions. Analysis is then performed on the selected features using the Cox Proportional Hazards Model to know how each marker affects the survival estimates of a patient. We have employed a two-step selection process to select a subset of markers. The first step is done by L1 regularized Cox PH. Then the selected markers are screened a second time by running a univariate Cox PH model and checking for the p-value of each bio-marker via Wald inference (p<0.05). Once the final selection is made, we estimate the Hazard Ratio and Confidence intervals using Maximum Likelihood Estimates (MLE) and the Bayesian Approach with the Cox Proportional Hazards Model (CPH) and the Accelerated Failure Time Model (AFT) as an alternative. A forest plot has also been generated to show the graphical representation of the meta-analysis done in the study. With the proposed selection procedure we have managed to find a suitable subset out of a large number of variables available. The features selected have been analyzed and their validity has been confirmed by using survival models.

Accurate prediction of pan-cancer types using machine learning with minimal number of DNA methylation sites.

DNA methylation analysis has been applied to determine the primary site of cancer; however, robust and accurate prediction of cancer types with a minimum number of sites is still a significant scientific challenge. To build an accurate and robust cancer type prediction tool with a minimum number of DNA methylation sites, we internally benchmarked different DNA methylation site selection and ranking procedures, as well as different classification models. We used The Cancer Genome Atlas dataset (26 cancer types with 8296 samples) to train and test models and used an independent dataset (17 cancer types with 2738 samples) for model validation. A deep neural network model using a combined feature selection procedure (named MethyDeep) can predict 26 cancer types using 30 methylation sites with superior performance compared with the known methods for both primary and metastatic cancers in independent validation datasets. In conclusion, MethyDeep is an accurate and robust cancer type predictor with the minimum number of DNA methylation sites; it could help the cost-effective clarification of cancer of unknown primary patients and the liquid biopsy-based early screening of cancers.

Automated feature selection procedure for particle jet classification

The high dimensionality of the data produced in high-energy physics experiments makes the use of machine learning algorithms, such as neural networks, necessary to improve the performance of reconstruction and classification of the analyzed events. Interpretability, i.e. the capability to explain the dynamics that lead the network to a certain outcome, emerged as a major need with architectures growing in complexity. In the analysis of pp collisions at the LHC, explainability firstly concern the assessment of the relative importance of high-level observables used to classify events. In this context, we have developed a method to select the most important features associated with a particle jet of which we want to establish the origin. Features are importance-sorted with a decision tree algorithm. A k-fold cross-validation is applied to raise the confidence in the extracted ranking. We tested the method with the case of highly boosted di-jet resonances decaying to two b-quarks, to be selected against an overwhelming QCD background with a Deep Neural network. We show that noisy and irrelevant features are rejected while relevant features occupy the top-ranking positions.

Analysis of Rank Aggregation Techniques for Rank Based on the Feature Selection Technique

In order to improve classification accuracy and lower future computation and data collecting costs, feature selection is the process of choosing the most crucial features from a group of attributes and removing the less crucial or redundant ones. To narrow down the features that need to be analyzed, a variety of feature selection procedures have been detailed in published publications. Chi-Square (CS), IG, Relief, GR, Symmetrical Uncertainty (SU), and MI are six alternative feature selection methods used in this study. The provided dataset is aggregated using four rank aggregation strategies: "rank aggregation," "Borda Count (BC) methodology," "score and rank combination," and "unified feature scoring" based on the outcomes of the six feature selection method (UFS). These four procedures by themselves were unable to generate a clear selection rank for the characteristic. To produce different ranks of traits, this ensemble of aggregating ranks is carried out. For this, the bagging method of majority voting was applied.

Leaf area index estimation of pergola-trained vineyards in arid regions using classical and deep learning methods based on UAV-based RGB images

Timely and accurate mapping of leaf area index (LAI) in vineyards plays an important role for management choices in precision agricultural practices. However, only a little work has been done to extract the LAI of pergola-trained vineyards using higher spatial resolution remote sensing data. The main objective of this study was to evaluate the ability of unmanned aerial vehicle (UAV) imageries to estimate the LAI of pergola-trained vineyards using shallow and deep machine learning (ML) methods. Field trials were conducted in different growth seasons in 2021 by collecting 465 LAI samples. Firstly, this study trained five classical shallow ML models and an ensemble learning model by using different spectral and textural indices calculated from UAV imageries, and the most correlated or useful features for LAI estimations in different growth stages were differentiated. Then, due to the classical ML approaches need the arduous computation of multiple indices and feature selection procedures, another ResNet-based convolutional neural network (CNN) model was constructed which can be directly fed by cropped images. Furthermore, this study introduced a new image data augmentation method which is applicable to regression problems. Results indicated that the textural indices performed better than spectral indices, while the combination of them can improve estimation results, and the ensemble learning method showed the best among classical ML models. By choosing the optimal input image size, the CNN model we constructed estimated the LAI most effectively without extracting and selecting the features manually. The proposed image data augmentation method can generate new training images with new labels by mosaicking the original ones, and the CNN model showed improved performance after using this method compared to those using only the original images, or augmented by rotation and flipping methods. This data augmentation method can be applied to other regression models to extract growth parameters of crops using remote sensing data, and we conclude that the UAV imagery and deep learning methods are promising in LAI estimations of pergola-trained vineyards.

Comparison of Machine Learning-based Approaches to Predict the Conversion to Alzheimer’s Disease from Mild Cognitive Impairment

In Mild Cognitive Impairment (MCI), identifying a high risk of conversion to Alzheimer’s Disease Dementia (AD) is a primary goal for patient management. Machine Learning (ML) algorithms are widely employed to pursue data-driven diagnostic and prognostic goals. An agreement on the stability of these algorithms –when applied to different biomarkers and other conditions– is far from being reached. In this study, we compared the different prognostic performances of three supervised ML algorithms fed with multimodal biomarkers of MCI subjects obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. Random Forest, Gradient Boosting, and eXtreme Gradient Boosting algorithms predict MCI conversion to AD. They can also be simultaneously employed –with the voting procedure– to improve predictivity. AD prediction accuracy is influenced by the nature of the data (i.e., neuropsychological test scores, cerebrospinal fluid AD-related proteins and APOE ε4, cerebral structural MRI (sMRI) data). In our study, independent of the applied ML algorithms, sMRI data showed the lowest accuracy (0.79) compared to other classes. Multimodal data were helpful in the algorithms’ performances by combining clinical and biological measures. Accordingly, using the three ML algorithms, the highest accuracy (0.90) was reached by employing neuropsychological and AD-related biomarkers. Finally, the feature selection procedure indicated that the most critical variables in the respective classes were the ADAS-Cog-13 scale, the medial temporal lobe and hippocampus atrophy, and the ratio between phosphorylated Tau and Aβ42 proteins. In conclusion, our data support the notion that using multiple ML algorithms and multimodal biomarkers helps make more accurate and solid predictions.

Feature selection in ultrahigh-dimensional additive models with heterogeneous frequency component functions

In this paper, we consider feature selection in ultrahigh-dimensional additive models with heterogeneous frequency component functions. Firstly, we introduce a new concept, weighted sum of squared conditional correlations (WSSCC), which measures the correlation between a random variable and its function. Afterwards, we propose a sure independence screening procedure based on WSSCC (WSSCCSIS), whose sure screening property is established. Furthermore, a sequential feature selection procedure based on WSSCC (WSSCCFR) is proposed. Numerical studies including comprehensive simulation and a real data analysis are carried out to demonstrate the advantage of our method over other existing approaches.

Deep learning based phishing website identification system using CNN-LSTM classifier

The term phishing refers to an attack that pretends to be the website of a large corporation, typically one dealing with money, such as a bank or other financial institution or an online retailer. Its primary objective is to acquire personally identifiable information from users, such as their social security numbers, credit card information, and passwords. Due to the rise of phishing attacks, various techniques have been developed in order to combat these threats. One of these is deep learning algorithms, which are capable of learning and analyzing massive datasets. Due to their capabilities, these algorithms are very useful in identifying and preventing phishing attacks. Due to the complexity of the phishing websites, many development systems have been created to detect them. Unfortunately, the output that was desired cannot be achieved by these systems, and they have a number of other flaws as well. The purpose of this paper is to propose a hybrid deep learning-based phishing detection system that is easy to put into practice. The quality of the input dataset is improved through the process of preprocessing the dataset. After that, the procedures of clustering and feature selection are carried out in order to improve the accuracy and decrease the amount of time required for the processing. The resulting features are then fed into the CNN_LSTM, which is a classification system that classifies websites that are phishing and legitimate. Proposed Hybrid deep learning models are proposed to combine the features of natural language processing (NLP) and character embedding. They can then reveal high-level connections between characters. In terms of the metric that is being used for the evaluation, the performance of the models that have been proposed is better than that of the other models.

Forecasting Daily Stock Movement Using a Hybrid Normalization Based Intersection Feature Selection and ANN

The potential financial benefits of stock market forecasting have drawn a lot of interest. Due to the various interconnected aspects, predicting these markets is a difficult endeavour that necessitates a thorough as well as efficient feature selection procedure to discover the highest useful aspects. Stock price changes are also influenced by previous trading days' movements, which is a time series problem. In stock forecasting, feature selection techniques are commonly used, although most known systems use a single feature selection methodology that probably can neglect some key notions about the regression function that is at the root of the problem relating the variables for input and output. This study employs an artificial neural network (ANN) based generative model to forecast pricing changes in the future by combining features preferred by different feature picking strategies to build an ideal optimal feature group. We begin by calculating an expanded set of 83 technical indicators using day-to-day stock data of six stock indices, and then we normalize them using the Hybrid-Normalization (HN) technique. The important features are selected using various types of feature selection techniques and then considering the common features for the stock movement prediction. For stock trend predictions, we used a variety of classifiers such as Support Vector Machine, K Nearest Neighbour and Artificial Neural Network and. The system was given a performance review after simulations were done on 6 stock indices from various portions of the international market. The outcomes show that joining highlighted features got by various feature choice calculations and taking care of them into a profound generative model beats best in class techniques.

Classification of Alzheimer's disease and frontotemporal dementia using routine clinical and cognitive measures across multicentric underrepresented samples: A cross sectional observational study.

Global brain health initiatives call for improving methods for the diagnosis of Alzheimer's disease (AD) and frontotemporal dementia (FTD) in underrepresented populations. However, diagnostic procedures in upper-middle-income countries (UMICs) and lower-middle income countries (LMICs), such as Latin American countries (LAC), face multiple challenges. These include the heterogeneity in diagnostic methods, lack of clinical harmonisation, and limited access to biomarkers. This cross-sectional observational study aimed to identify the best combination of predictors to discriminate between AD and FTD using demographic, clinical and cognitive data among 1794 participants [904 diagnosed with AD, 282 diagnosed with FTD, and 606 healthy controls (HCs)] collected in 11 clinical centres across five LAC (ReDLat cohort). A fully automated computational approach included classical statistical methods, support vector machine procedures, and machine learning techniques (random forest and sequential feature selection procedures). Results demonstrated an accurate classification of patients with AD and FTD and HCs. A machine learning model produced the best values to differentiate AD from FTD patients with an accuracy = 0.91. The top features included social cognition, neuropsychiatric symptoms, executive functioning performance, and cognitive screening; with secondary contributions from age, educational attainment, and sex. Results demonstrate that data-driven techniques applied in archival clinical datasets could enhance diagnostic procedures in regions with limited resources. These results also suggest specific fine-grained cognitive and behavioural measures may aid in the diagnosis of AD and FTD in LAC. Moreover, our results highlight an opportunity for harmonisation of clinical tools for dementia diagnosis in the region. This work was supported by the Multi-Partner Consortium to Expand Dementia Research in Latin America (ReDLat), funded by NIA/NIH (R01AG057234), Alzheimer's Association (SG-20-725707-ReDLat), Rainwater Foundation, Takeda (CW2680521), Global Brain Health Institute; as well as CONICET; FONCYT-PICT (2017-1818, 2017-1820); PIIECC, Facultad de Humanidades, Usach; Sistema General de Regalías de Colombia (BPIN2018000100059), Universidad del Valle (CI 5316); ANID/FONDECYT Regular (1210195, 1210176, 1210176); ANID/FONDAP (15150012); ANID/PIA/ANILLOS ACT210096; and Alzheimer's Association GBHI ALZ UK-22-865742.

An Effective Analysis of Detection of Erythemato Squamous using Machine Learning Algorithms

In the realm of biotechnology, machine learning methods are crucial for illness diagnosis. Expert systems that aid in disease prediction can be created with the knowledge gained via the use of machine learning techniques. This paper discusses various data mining methods for predicting skin diseases. Skin conditions known as erythemato-squamous diseases (ESDs) are widespread. Psoriasis, seboreic dermatitis, lichen planus, pityriasis rosea, chronic dermatitis, and pityriasis rubra pilaris are among the six different classifications. With very few exceptions, they all exhibit erythema and scaling as clinical characteristics. The skin illness dataset is also subjected to a feature selection procedure in order to get an accuracy of 99.68% when the Gradient Boosting ensemble method is used on RNC. On the dataset for skin diseases, we found the highest accuracy in the literature. In this paper , we will be discussing about the various machine learning algorithms which shows different classification accuracy with our system.