Deep Feature Selection Research Articles

Knowledge distillation: A novel approach for deep feature selection

High dimensional data in hyperspectral remote sensing leads to computational, analytical, and storage complexities. Dimensionality reduction serves as an efficient tool to remove redundant, irrelevant, and highly correlated features. Recently, deep learning approaches have received remarkable progress in hyperspectral data analysis. In this paper, a new end-to-end deep learning framework based on a teacher-student network inspired by knowledge distillation is proposed for deep feature selection. Initially, a complicated teacher deep neural network is employed on complex high dimensional data to learn its corresponding best low dimensional representation. Then, the knowledge from the network is transferred to a simple student network that performs feature selection. Hence, it eventually leads to deep neural network compression which is of prime concern in hyperspectral remote sensing. Limited studies have been carried out to explore the benefits of knowledge distillation on hyperspectral data. The proposed method could be employed to choose deep features for both supervised and unsupervised tasks. Experimental results reveal the performance of the proposed scheme using limited features. In comparison to 1D and simple autoencoder models, the 2D model based on convolutional autoencoder delivers greater classification accuracies, with a classification accuracy value of 96.15% for the Indian Pines dataset and 97.82% for the Pavia University dataset. A similar trend is reported with unsupervised learning as well. Furthermore, the proposed model has a low degree of sensitivity to parameter selection.

Joint Deep Learning for Covariate Decomposition and Treatment Effect Estimation

This study proposes a joint deep learning method, namely, confounding factor joint decomposition under counterfactual regression (CFJD-CFR), to identify a minimum adjustment covariate set and estimate the average treatment effect (ATE). CFJD-CFR includes two levels: feature learning and prediction model. Feature learning constructs the objective function based on clear variable attribute recognition criteria to identify and decompose a covariate set of different attributes into four disjoint sets of factors: instrumental, confounding, predicting, and noise variables. The output of feature learning is then introduced into the prediction model as its input to build a joint deep learning model and reduce the systematic errors caused by step-by-step learning. The proposed method combines deep representation learning, feature selection, and prediction under the structural equation modeling framework. Simulation studies show that CFJD-CFR can effectively identify the minimum adjustment covariate set and consistently outperform other methods for ATE estimation. The proposed method is applied to the Chinese General Social Survey study. New insights into the difference in social welfare policy preference between urban and rural residents in China are obtained.

Bias reduction in representation of histopathology images using deep feature selection

Appearing traces of bias in deep networks is a serious reliability issue which can play a significant role in ethics and generalization related concerns. Recent studies report that the deep features extracted from the histopathology images of The Cancer Genome Atlas (TCGA), the largest publicly available archive, are surprisingly able to accurately classify the whole slide images (WSIs) based on their acquisition site while these features are extracted to primarily discriminate cancer types. This is clear evidence that the utilized Deep Neural Networks (DNNs) unexpectedly detect the specific patterns of the source site, i.e, the hospital of origin, rather than histomorphologic patterns, a biased behavior resulting in degraded trust and generalization. This observation motivated us to propose a method to alleviate the destructive impact of hospital bias through a novel feature selection process. To this effect, we have proposed an evolutionary strategy to select a small set of optimal features to not only accurately represent the histological patterns of tissue samples but also to eliminate the features contributing to internal bias toward the institution. The defined objective function for an optimal subset selection of features is to minimize the accuracy of the model to classify the source institutions which is basically defined as a bias indicator. By the conducted experiments, the selected features extracted by the state-of-the-art network trained on TCGA images (i.e., the KimiaNet), considerably decreased the institutional bias, while improving the quality of features to discriminate the cancer types. In addition, the selected features could significantly improve the results of external validation compared to the entire set of features which has been negatively affected by bias. The proposed scheme is a model-independent approach which can be employed when it is possible to define a bias indicator as a participating objective in a feature selection process; even with unknown bias sources.

GestroNet: A Framework of Saliency Estimation and Optimal Deep Learning Features Based Gastrointestinal Diseases Detection and Classification.

In the last few years, artificial intelligence has shown a lot of promise in the medical domain for the diagnosis and classification of human infections. Several computerized techniques based on artificial intelligence (AI) have been introduced in the literature for gastrointestinal (GIT) diseases such as ulcer, bleeding, polyp, and a few others. Manual diagnosis of these infections is time consuming, expensive, and always requires an expert. As a result, computerized methods that can assist doctors as a second opinion in clinics are widely required. The key challenges of a computerized technique are accurate infected region segmentation because each infected region has a change of shape and location. Moreover, the inaccurate segmentation affects the accurate feature extraction that later impacts the classification accuracy. In this paper, we proposed an automated framework for GIT disease segmentation and classification based on deep saliency maps and Bayesian optimal deep learning feature selection. The proposed framework is made up of a few key steps, from preprocessing to classification. Original images are improved in the preprocessing step by employing a proposed contrast enhancement technique. In the following step, we proposed a deep saliency map for segmenting infected regions. The segmented regions are then used to train a pre-trained fine-tuned model called MobileNet-V2 using transfer learning. The fine-tuned model's hyperparameters were initialized using Bayesian optimization (BO). The average pooling layer is then used to extract features. However, several redundant features are discovered during the analysis phase and must be removed. As a result, we proposed a hybrid whale optimization algorithm for selecting the best features. Finally, the selected features are classified using an extreme learning machine classifier. The experiment was carried out on three datasets: Kvasir 1, Kvasir 2, and CUI Wah. The proposed framework achieved accuracy of 98.20, 98.02, and 99.61% on these three datasets, respectively. When compared to other methods, the proposed framework shows an improvement in accuracy.

Deep feature selection using local search embedded social ski-driver optimization algorithm for breast cancer detection in mammograms

Breast cancer has become a common malignancy in women. However, early detection and identification of this disease can save many lives. As computer-aided detection helps radiologists in detecting abnormalities efficiently, researchers across the world are striving to develop reliable models to deal with. One of the common approaches to identifying breast cancer is through breast mammograms. However, the identification of malignant breasts from mass lesions is a challenging research problem. In the current work, we propose a method for the classification of breast mass using mammograms which consists of two main stages. At first, we extract deep features from the input mammograms using the well-known VGG16 model while incorporating an attention mechanism into this model. Next, we apply a meta-heuristic called Social Ski-Driver (SSD) algorithm embedded with Adaptive Beta Hill Climbing based local search to obtain an optimal features subset. The optimal features subset is fed to the K-nearest neighbors (KNN) classifier for the classification. The proposed model is demonstrated to be very useful for identifying and differentiating malignant and healthy breasts successfully. For experimentation, we evaluate our model on the digital database for screening mammography (DDSM) database and achieve 96.07% accuracy using only 25% of features extracted by the attention-aided VGG16 model. The Python code of our research work is publicly available at: https://github.com/Ppayel/BreastLocalSearchSSD.

A hybrid deep feature selection framework for emotion recognition from human speeches

A hybrid deep feature selection framework for emotion recognition from human speeches

MRNA Biomarkers for Invasive Breast Cancer based on a Deep Feature Selection Approach

Early detection of breast cancer and effective identification of its correct stage remain major challenges for healthcare professionals. Testing the tumour for Oestrogen Receptor and Progesterone Receptor is a standard part of the initial evaluation of breast cancer diagnosis and treatment planning. Several expression profiling studies have illustrated that the expression of these hormone receptors is linked with diverse genetic variations, which means that several mutated genes can a affect the development and progression of breast cancer and contribute to its heterogeneity. Unfortunately, due to the high dimensionality and low sample size nature of microarray data, traditional statistical feature selection techniques fail to identify genes that could act as risk factors for breast cancer. Inspired by this, we developed a deep learning-based feature extraction module with a weight interpretation method to select a subset of robust biomarkers across three different mRNA expression data sets from The Cancer Genome Atlas program (TCGA). For a discovered feature (a gene) to be accepted for further investigation, it must have been independently selected by the weight interpretation method from each of the deep feature extraction modules (each having been trained on a different data set). The small panel of discovered biomarkers was then subsequently evaluated using a range of classifiers to ascertain their predictive ability with respect to the above hormone receptor status. We observed strong evidence that the upregulation in the expression levels of highly positively weighted genes within the deep feature selection modules and the down regulation in the expression levels of the highly negatively weighted genes both indicated the strong likelihood of a patient experiencing ER+/PR+ invasive breast cancer. In addition, we discovered a number of potentially novel biomarkers worthy of further consideration.

Detection of COVID-19 and its pulmonary stage using Bayesian hyperparameter optimization and deep feature selection methods.

Since the first case of COVID‐19 was reported in December 2019, many studies have been carried out on artificial intelligence for the rapid diagnosis of the disease to support health services. Therefore, in this study, we present a powerful approach to detect COVID‐19 and COVID‐19 findings from computed tomography images using pre‐trained models using two different datasets. COVID‐19, influenza A (H1N1) pneumonia, bacterial pneumonia and healthy lung image classes were used in the first dataset. Consolidation, crazy‐paving pattern, ground‐glass opacity, ground‐glass opacity and consolidation, ground‐glass opacity and nodule classes were used in the second dataset. The study consists of four steps. In the first two steps, distinctive features were extracted from the final layers of the pre‐trained ShuffleNet, GoogLeNet and MobileNetV2 models trained with the datasets. In the next steps, the most relevant features were selected from the models using the Sine–Cosine optimization algorithm. Then, the hyperparameters of the Support Vector Machines were optimized with the Bayesian optimization algorithm and used to reclassify the feature subset that achieved the highest accuracy in the third step. The overall accuracy obtained for the first and second datasets is 99.46% and 99.82%, respectively. Finally, the performance of the results visualized with Occlusion Sensitivity Maps was compared with Gradient‐weighted class activation mapping. The approach proposed in this paper outperformed other methods in detecting COVID‐19 from multiclass viral pneumonia. Moreover, detecting the stages of COVID‐19 in the lungs was an innovative and successful approach.

Deep features selection through genetic algorithm for cervical pre-cancerous cell classification

Deep features selection through genetic algorithm for cervical pre-cancerous cell classification

ID-RDRL: a deep reinforcement learning-based feature selection intrusion detection model

Network assaults pose significant security concerns to network services; hence, new technical solutions must be used to enhance the efficacy of intrusion detection systems. Existing approaches pay insufficient attention to data preparation and inadequately identify unknown network threats. This paper presents a network intrusion detection model (ID-RDRL) based on RFE feature extraction and deep reinforcement learning. ID-RDRL filters the optimum subset of features using the RFE feature selection technique, feeds them into a neural network to extract feature information and then trains a classifier using DRL to recognize network intrusions. We utilized CSE-CIC-IDS2018 as a dataset and conducted tests to evaluate the model’s performance, which is comprised of a comprehensive collection of actual network traffic. The experimental results demonstrate that the proposed ID-RDRL model can select the optimal subset of features, remove approximately 80% of redundant features, and learn the selected features through DRL to enhance the IDS performance for network attack identification. In a complicated network environment, it has promising application potential in IDS.

Fractured Elbow Classification Using Hand-Crafted and Deep Feature Fusion and Selection Based on Whale Optimization Approach

The fracture of the elbow is common in human beings. The complex structure of the elbow, including its irregular shape, border, etc., makes it difficult to correctly recognize elbow fractures. To address such challenges, a method is proposed in this work that consists of two phases. In Phase I, pre-processing is performed, in which images are converted into RGB. In Phase II, pre-trained convolutional models Darknet-53 and Xception are used for deep feature extraction. The handcrafted features, such as the histogram of oriented gradient (HOG) and local binary pattern (LBP), are also extracted from the input images. A principal component analysis (PCA) is used for best feature selection and is serially merged into a single-feature vector having the length of N×2125. Furthermore, informative features N×1049 are selected out of N×2125 features using the whale optimization approach (WOA) and supplied to SVM, KNN, and wide neural network (WNN) classifiers. The proposed method’s performance is evaluated on 16,984 elbow X-ray radiographs that are taken from the publicly available musculoskeletal radiology (MURA) dataset. The proposed technique provides 97.1% accuracy and a kappa score of 0.943% for the classification of elbow fractures. The obtained results are compared to the most recently published approaches on the same benchmark datasets.

Enhancing classification capacity of CNN models with deep feature selection and fusion: A case study on maize seed classification

Developing computer-assisted agricultural analysis systems using machine learning methods is a focused area in recent years. As in every field, it is aimed to improve the production process and product quality in agricultural applications with computer-assisted systems. The maize plant is a very important species in terms of providing sufficient food to the world’s population. The separation process of the haploid and diploid maize seeds is a critical issue in terms of maize breeding time and production efficiency. In this study, a haploid–diploid maize seed classification method is proposed using deep features obtained from convolutional neural networks (CNN). In the first stage, deep features were obtained from fully connected layers of different CNN models. Then, the best 100 features were selected by using the MRMR (Max-Relevance and Min-Redundancy) feature selection method for 1000 features obtained in each CNN model. These selected features have been fused according to different combinations of the CNN models. These fused features have been used in the training and testing stages of a conventional classifier method in the last stage. Eventually, according to the experimental results, it was determined that the general accuracy performance has been around 96.74%. It has been observed that the proposed approach exhibits high performance in the classification process of maize seeds.

An adaptive and altruistic PSO-based deep feature selection method for Pneumonia detection from Chest X-rays

Pneumonia is one of the major reasons for child mortality especially in income-deprived regions of the world. Although it can be detected and treated with very less sophisticated instruments and medication, Pneumonia detection still remains a major concern in developing countries. Computer-aided based diagnosis (CAD) systems can be used in such countries due to their lower operating costs than professional medical experts. In this paper, we propose a CAD system for Pneumonia detection from Chest X-rays, using the concepts of deep learning and a meta-heuristic algorithm. We first extract deep features from the pre-trained ResNet50, fine-tuned on a target Pneumonia dataset. Then, we propose a feature selection technique based on particle swarm optimization (PSO), which is modified using a memory-based adaptation parameter, and enriched by incorporating an altruistic behavior into the agents. We name our feature selection method as adaptive and altruistic PSO (AAPSO). The proposed method successfully eliminates non-informative features obtained from the ResNet50 model, thereby improving the Pneumonia detection ability of the overall framework. Extensive experimentation and thorough analysis on a publicly available Pneumonia dataset establish the superiority of the proposed method over several other frameworks used for Pneumonia detection. Apart from Pneumonia detection, AAPSO is further evaluated on some standard UCI datasets, gene expression datasets for cancer prediction and a COVID-19 prediction dataset. The overall results are satisfactory, thereby confirming the usefulness of AAPSO in dealing with varied real-life problems. The supporting source codes of this work can be found at https://github.com/rishavpramanik/AAPSO.

Improved mayfly optimization deep stacked sparse auto encoder feature selection scorched gradient descent driven dropout XLM learning framework for software defect prediction

SummarySoftware testing is the process of improving software quality by classifying and removing defects in the software development. Previously, several methods were used for software defect prediction, but any one method did not provide sufficient accuracy. To overcome this issue, an improved may fly optimization with deep stacked sparse auto encoder feature selection scorched gradient descent driven dropout extreme learning machine framework (SDP‐IMFOFS‐GDDDXLMC) is proposed in this article for software defect prediction. Here, an IMFO method is considered for feature selection techniques. In feature selection technique, features are selected, such as PC1, PC4, and MC1 for selecting the probable minimal attribute. Then, GDDDXLMC approach is used to classify buggy and clean software detection. The proposed approach is implemented in MATLAB platform. The performance metrics, such as accuracy, precision, F‐measure, sensitivity, specificity, Mathew correlation coefficient (MCC) is examined to examine the performance of the proposed method. The simulation performance of the proposed SDP‐IMFOFS‐GDDDXLMC method attains higher accuracy 99.75%, 97.85%, 95.13%, 14.89%, 16.34%, 17.89%, and 98.79, higher sensitivity 96.34%, 91.23%, 89.12%, 12.67%, 17.56%, 18.90%, and 87.25% higher specificity 14.89%, 16.89%, 20.67%, 93.67%, 92.37%, 98.47%, and 94.78% compared with the existing methods, like SDP‐MLP‐PSO, SDP‐BPNN‐RBFNN, SDP‐CNN‐RNN‐LSTM, SDP‐KBN‐LIME, SDP‐SLDeep‐LSTM, SDP‐K‐PCA‐ELM, and SDP‐CNN‐PHI forest, respectively.

AI-driven deep and handcrafted features selection approach for Covid-19 and chest related diseases identification.

To identify various pneumonia types, a gap of 15% value is being created every five years. To fill this gap, accurate detection of chest disease is required in the healthcare department to avoid any serious issues in the future. Testing the affected lungs to detect a Coronavirus 2019 (COVID-19) using the same imaging modalities may detect some other chest diseases. This wrong diagnosis strongly needs a multidisciplinary approach to the right diagnosis of chest-related diseases. Only a few works till now are targeting pathological x-ray images. Many studies target only a single chest disease that is not enough to automate chest disease detection. Only a few studies regarding the observation of the COVID-19, but more cases are those where it can be misclassified as detecting techniques not providing any generic solution for all types of chest diseases. However, the existing studies can only detect if the person has COVID-19 or not. The proposed work significantly contributes to detecting COVID-19 and other chest diseases by providing useful analysis of chest-related diseases. One of our testing approaches achieves 90.22% accuracy for 15 types of chest disease with 100% correct classification of COVID-19. Though it analyzes the perfect detection as the accuracy level is high enough, but it would be an excellent decision to consider the proposed study until doctors can visually inspect the input images used by models that lead to its detection.

Evolutionary deep feature selection for compact representation of gigapixel images in digital pathology

Despite the recent progress in Deep Neural Networks (DNNs) to characterize histopathology images, compactly representing a gigapixel whole-slide image (WSI) via salient features to enable computational pathology is still an urgent need and a significant challenge. In this paper, we propose a novel WSI characterization approach to represent, search and classify biopsy specimens using a compact feature vector (CFV) extracted from a multitude of deep feature vectors. Since the non-optimal design and training of deep networks may result in many irrelevant and redundant features and also cause computational bottlenecks, we proposed a low-cost stochastic method to optimize the output of pre-trained deep networks using evolutionary algorithms to generate a very small set of features to accurately represent each tissue/biopsy. The performance of the proposed method has been assessed using WSIs from the publicly available TCGA image data. In addition to acquiring a very compact representation (i.e., 11,000 times smaller than the initial set of features), the optimized features achieved 93% classification accuracy resulting in 11% improvement compared to the published benchmarks. The experimental results reveal that the proposed method can reliably select salient features of the biopsy sample. Furthermore, the proposed approach holds the potential to immensely facilitate the adoption of digital pathology by enabling a new generation of WSI representation for efficient storage and more user-friendly visualization.

Brain tumor detection from 3D MRI using Hyper‐Layer Convolutional Neural Networks and Hyper‐Heuristic Extreme Learning Machine

SummaryAutomated techniques for brain tumor classification using deep learning approaches have gained significant research interest in recent years. Yet, the difficulties in extracting and classifying the tumor regions from the 3D Magnetic Resonance Imaging (MRI) do not have a definite solution. The major challenge in utilizing machine and deep learning algorithms for brain cancer classification from 3D images is the time complexity in analyzing the multiple frames of a brain MRI. This paper introduces Hyper‐Layer Convolutional Neural Networks (HL‐CNN) and Hyper‐Heuristic Extreme Learning Machine (HH‐ELM). The proposed method consists of three main phases are pre‐processing, deep feature mining and selection, and classification. The input MRI images are pre‐processed through denoising and image enhancement methods in the first phase. In the second phase, the HL‐CNN is introduced for feature extraction. The hyper‐layer technique is a masking technique that also inherent the features of the specified layers instead of only considering the features at the last layer. The best features are selected using a simple correlation‐based selection approach through HL‐CNN validation to minimize the irrelevant features in the system. In the last phase, the HH‐ELM is introduced to classify the tumor images to identify the different types of tumors. HH‐ELM is an enhanced version of ELM through optimal tuning of ELM parameters using a hyper‐heuristic optimization algorithm. Evaluations are performed over the BRATS 2020 database of MRI images and the proposed method of HL‐CNN and HH‐ELM achieved dice scores of 0.9020, 0.9393, and 0.9589 for ED, WT, and TC tumor classes with 95.89% accuracy, 98.46% precision, 96% recall, and 97.21% f‐measure which are 2%–13% higher and processing time of 139.88 s which is 66%–78% lesser than the existing methods.

Detection of oral cancer using deep learning approach

Globally, oral cancer is becoming more and more of an issue, and in some nations, like Taiwan, India, and Sri Lanka, it is at the very top of the list. Tobacco, alcohol, and betel nut use are responsible for more than 95% of all mouth cancer cases (BQ). In Western nations, smoking and alcohol consumption are the two biggest risk factors, but in Asian nations, smoking and BQ usage are the two most risk factors. It is alarming how frequently people with advanced oral cancer arrive. The best method for minimising personal illness burden, lowering morbidity and mortality, and enhancing quality of life. The detection, evaluation, and treatment of oral cancer remain challenges for the dental profession. In the proposed approach, deep learning algorithm has been used to simulate the development of cancer diagnosis and therapy, and they are successful in predicting future outcomes of a cancer. For the best outcomes in the detection and diagnosis of oral cancer, an effective deep learning and feature selection approach utilising Alex net model has been applied.Overall, 500 images with different resolution were used in our system.

A Comparative Study on the Potential of Unsupervised Deep Learning-based Feature Selection in Radiomics.

In Radiomics, deep learning-based systems for medical image analysis play an increasing role. However, due to the better explainability, feature-based systems are still preferred, especially by physicians. Often, high-dimensional data and low sample size pose different challenges (e.g. increased risk of overfitting) to machine learning systems. By removing irrelevant and redundant features from the data, feature selection is an effective way of pre-processing. The research in this study is focused on unsupervised deep learning-based methods for feature selection. Five recently proposed algorithms are compared regarding their applicability and efficiency on seven data sets in three different sample applications. It was found that deep learning-based feature selection leads to improved classification results compared to conventional methods, especially for small feature subsets. Clinical Relevance - The exploration of distinctive features and the ability to rank their importance without the need for outcome information is a potential field of application for unsupervised feature selection methods. Especially in multiparametric radiology, the number of features is increasing. The identification of new potential biomarkers is important both for treatment and prevention.

Hybrid intrusion detection system for wireless IoT networks using deep learning algorithm

The intense growth of wireless communication and the digital revolution have increased the utilization of Internet of Things (IoT) applications. The number of internet users increase everyday which in turn increases the network traffic and data volume. The energy-limited sensor node resources in the IoT environment are vulnerable to attacks due to the networking procedures, open broadcast communication, etc. Intruders easily gain access to the network and perform different types of attacks which degrades the overall performance and quality of services. Intrusion detection systems are developed to detect different kinds of attacks that cannot be detected by the firewalls. Based on the features, the intrusion detection system classifies the normal and abnormal characteristics of the system. Various intrusion detection systems based on machine learning models have evolved so far. However, the feature selection process is much important to enhance the classification performance. Thus, in this research work, a deep learning-based feature selection procedure is presented to select the optimal features. The decision tree algorithm is utilized as a classifier to classify the deep features and detect attacks in the IoT network. Benchmark NSL-KDD dataset has been used for experimentation and the parameters like precision, recall, f1-score, and accuracy are evaluated for the proposed model and the conventional models to validate the superior performance. With maximum accuracy of 99.49%, the proposed hybrid model performs better than the conventional intrusion detection systems.