Ensemble Selection Research Articles

An automated classification framework for glaucoma detection in fundus images using ensemble of dynamic selection methods

Glaucoma is an optic neuropathy, which leads to vision loss and is irreversible due to damage in the optic nerve head mainly caused by increased intra-ocular pressure. Retinal fundus photography facilitates ophthalmologist in detection of glaucoma but is subjective to human intervention and is time-consuming. Computational methods such as image processing and machine learning classifiers can aid in computer-based glaucoma detection which helps in mass screening of glaucoma. In this context, the proposed method develops an automated glaucoma detection system, in the following steps: (i) pre-processing by segmenting the blood vessels using directional filter; (ii) segmenting the region of interest by using statistical features; (iii) extracting the clinical and texture-based features; and (iv) developing ensemble of classifier models using dynamic selection techniques. The proposed method is evaluated on two publically available datasets and 300 fundus images collected from a hospital. The best results are obtained using ensemble of random forest using META-DES dynamic ensemble selection technique, and the average specificity, sensitivity and accuracy for glaucoma detection on hospital dataset are 100%, respectively. For RIM-ONE dataset, the average specificity, sensitivity and accuracy for glaucoma detection are 100%, 93.85% and 97.86%, respectively. For Drishti dataset, the average specificity, sensitivity and accuracy for glaucoma detection are 90%, 100% and 97%, respectively. The quantitative results and comparative study indicate the ability of the developed method, and thus, it can be deployed in mass screening and also as a second opinion in decision making by the ophthalmologist for glaucoma detection.

Vibration signal-based defect detection method for railway signal relay using parameter-optimized VMD and ensemble feature selection

Railway signal relay (RSR) plays a critical role in the railway signaling system and its failure may bring serious consequences. At present, the manual-based RSR defect detection method has many shortcomings. A vibration-based defect detection method for RSR is proposed to solve this problem. First, since most of the variational mode decomposition (VMD) optimization strategies do not apply to the vibration signal of RSR, a parameter-optimized VMD based on a hybrid criterion of sample entropy (SampEn) and Pearson correlation coefficient (PCC) is proposed. Second, an ensemble feature selection method combining Filter method and Embedded method is proposed to select defect-sensitive features. The ensemble is considered as a multi-criteria decision-making (MCDM) problem and solved using the VIKOR method, which can integrate the advantages of each feature selection method to select the optimal feature subset. Finally, the effectiveness of the proposed method is verified by real vibration signals of RSR.

An adaptive selective ensemble algorithm for fault classification

With the aim of improving the classification performance of a single extreme learning machine (ELM) for fault diagnosis, an adaptive selective ensemble classification algorithm is proposed based on the idea of ensemble learning. The proposed algorithm achieves better classification performance than a single model and the selected ELM subnetworks are selected adaptively and the ensemble weights of selected ELM subnetworks are calculated adaptively, which avoids the complex process of manually selecting subnetworks and calculating ensemble weights. First, a regularized error weight adjustment ELM is constructed by introducing regularization parameters and error weight adjustment matrix to a standard ELM, where the error weight adjustment matrix is obtained by the method of adaptive Gaussian kernel density estimation. Then, discrepancy subnetworks are constructed using six different activation functionsand the ensemble weights of subnetworks are obtained adaptively according to the normalized ratio of mean and variance of subnetwork F-scores to achieve the ensemble of subnetworks. Finally, the adaptive selective ensemble classification algorithm is validated using the UCI dataset and experimental acoustic emission signals of gearbox faults. The results show that the adaptive selective ensemble method can improve the stability and accuracy of classification algorithms and the achieved classification accuracy for experimental acoustic emission was 0.9773.

A fuzzy clustering ensemble selection based on active full-link similarity

A fuzzy clustering ensemble selection based on active full-link similarity

Automated machine learning with dynamic ensemble selection

Automated machine learning with dynamic ensemble selection

Sensor fault analysis of aero-engine using ensemble SCNN and Bayesian interval estimation

Sensors of an aircraft engine have the problems of multiple kinds of faults and difficulty to diagnose. Moreover, the diagnosis performance cannot meet practical demands. Therefore, this paper proposes a novel approach for sensor fault diagnosis integrating the advantages of selective ensemble learning, siamese convolutional neural network (SCNN), and Bayesian interval estimation (BIE). The Bayesian interval estimation with hypothesis testing is developed with real-time data update to reduce the confidence interval. The attention mechanism is introduced to enhance fault diagnosis performance. Moreover, ensemble selection learning with Akaike information criteria is proposed to obtain optimal SCNNs. The theorems of convergence and optimal confidence data in ensemble learning are proven. Furthermore, this paper develops a novel improved learning algorithm based on AdaMod (Adaptive and momental bound method) and SGD (stochastic gradient descent), and achieves a new fault diagnosis algorithm to improve the diagnosis performance. The sensor data of aircraft engines from China Eastern Airlines is adopted for validation and experimental results show that proposed approach reaches an excellent tradeoff between sensor diagnosis performance, confidence interval, and runtime. The ablation studies are performed to analyze the role of leaner’s components. The proposed method improves training accuracy by 6.2%, fault diagnosis performance by 5.1%, fault prediction mean result by 4.6%, and fault prediction variance is reduced by 38.1% than the state-of-the-art methods.

Bi-level ensemble method for unsupervised feature selection

Unsupervised feature selection is an important machine learning task and thus attracts increasingly more attention. However, due to the absence of labels, unsupervised feature selection often suffers from stability and robustness problems. To tackle these problems, some works try to ensemble multiple feature selection results to obtain a consensus result. Most of the existing methods do the ensemble on the feature level, i.e., they directly ensemble feature selection results by feature ranking or voting aggregation, without paying any attention to the following downstream tasks. In this paper, we take clustering as the downstream task and wish to ensemble the base results to select features which are appropriate for clustering. To this end, we propose a novel bi-level feature selection ensemble method, which ensembles on two levels: the feature level and the clustering level. Together with feature level ensemble, we also learn a consensus clustering result from base feature selection results with self-paced learning. Then, we apply the consensus clustering result to guide the feature selection in turn. Extensive experiments are conducted to demonstrate that the proposed method outperforms other state-of-the-art feature selection and feature selection ensemble methods in the clustering task. The codes of this paper are released in https://doctor-nobody.github.io/codes/BLFSE.zip.

TF-Predictor: Transformer-Based Prerouting Path Delay Prediction Framework

Timing mismatch between different stages of physical design poses great challenges for circuit optimization to achieve the desired performance, power and area (PPA) tradeoff. The inaccurate timing estimation prior to routing may lead to over-design with unwanted power and area consumption or iterating back to cell placement at the cost of design turn-around time. Existing learning models could not predict post-routing circuit timing with satisfying accuracy and efficiency due to the limitations of the ignorance of delay correlation along the timing path and the empirical feature selection solutions. In this work, an accurate and efficient pre-routing path delay prediction framework is proposed by utilizing transformer network and residual model with an ensemble feature selection mechanism. Owing to the combined filter and wrapper methods, an ensemble feature selection mechanism is implemented to determine the optimal feature subset based on the timing and physical information at the placement stage for path delay prediction, which is extracted as feature sequences for each cell along the timing path to be trained by transformer network. With the residual model, the predicted timing mismatch between the placement and routing stages by transformer network is further calibrated to estimate the post-routing path delay. The proposed framework has been validated with ISCAS’85 and OpenCores benchmark circuits for the prediction of post-routing path delay, where the perdition error in terms of rRMSE is limited within 1.3% and 3.0% and the correlation coefficient R is higher than 0.999 and 0.995 for seen and unseen circuits respectively, indicating an error reduction by 2.3 10.6 times compared by prior learning-based models. In addition, the framework achieves average three orders of magnitude speedup compared with the commercial tools and is accelerated by a factor of 14 128 as against the competitive learning models, which is promising to be applied to guide design optimization prior to time-consuming routing stage.

An intelligent flower classification framework: optimal hybrid flower pattern extractor with adaptive dynamic ensemble transfer learning-based convolutional neural network

ABSTRACT The rapid development in computer technology plays an essential role in the research works for performing fast and accurate identification of flower species through the processing of flower images with the support of mobile devices that seeks more attention in the research areas. It is highly significant for maintaining the sensitivity of ecological balance, and therefore, the image-processing approaches have provided improved outcomes in recent days. To rectify these existing problems and improve the accuracy attains the flower classification, the CNN variants of ensemble techniques can be used for the dynamic ensemble selection of the CNN networks. This paper develops an enhanced ensemble deep learning-based flower classification model to get efficient classification results with optimal models. The pre-processing is done through the Contrast Limited Adaptive Histogram Equalization (CLAHE) and filtering techniques. The pre-processed images are considered for the optimal pattern extraction phase, where optimal hybrid patterns are extracted from the Local Binary Pattern (LBP) and Local Vector Pattern (LVP). Here, the optimal hybrid pattern extraction phase contains the optimization in it using the enhanced heuristic algorithm named Improved Rat Swarm Optimizer (IRSO). The flower classification is performed with an Adaptive Dynamic Ensemble Transfer learning-based Convolutional Neural Network (ADET-CNN). Here, the optimal models are selected with the support of the same RSO algorithm. The experimental results show a better efficacy of the developed flower classification framework.

Ensemble feature selection for single-label text classification: a comprehensive analytical study

Ensemble feature selection for single-label text classification: a comprehensive analytical study

Prediction of posttraumatic functional recovery in middle-aged and older patients through dynamic ensemble selection modeling.

Age-specific risk factors may delay posttraumatic functional recovery; complex interactions exist between these factors. In this study, we investigated the prediction ability of machine learning models for posttraumatic (6 months) functional recovery in middle-aged and older patients on the basis of their preexisting health conditions. Data obtained from injured patients aged ≥45 years were divided into training-validation (n = 368) and test (n = 159) data sets. The input features were the sociodemographic characteristics and baseline health conditions of the patients. The output feature was functional status 6 months after injury; this was assessed using the Barthel Index (BI). On the basis of their BI scores, the patients were categorized into functionally independent (BI >60) and functionally dependent (BI ≤60) groups. The permutation feature importance method was used for feature selection. Six algorithms were validated through cross-validation with hyperparameter optimization. The algorithms exhibiting satisfactory performance were subjected to bagging to construct stacking, voting, and dynamic ensemble selection models. The best model was evaluated on the test data set. Partial dependence (PD) and individual conditional expectation (ICE) plots were created. In total, nineteen of twenty-seven features were selected. Logistic regression, linear discrimination analysis, and Gaussian Naive Bayes algorithms exhibited satisfactory performances and were, therefore, used to construct ensemble models. The k-Nearest Oracle Elimination model outperformed the other models when evaluated on the training-validation data set (sensitivity: 0.732, 95% CI: 0.702-0.761; specificity: 0.813, 95% CI: 0.805-0.822); it exhibited compatible performance on the test data set (sensitivity: 0.779, 95% CI: 0.559-0.950; specificity: 0.859, 95% CI: 0.799-0.912). The PD and ICE plots showed consistent patterns with practical tendencies. Preexisting health conditions can predict long-term functional outcomes in injured middle-aged and older patients, thus predicting prognosis and facilitating clinical decision-making.

A classification algorithm based on dynamic ensemble selection to predict mutational patterns of the envelope protein in HIV-infected patients

BackgroundTherapeutics against the envelope (Env) proteins of human immunodeficiency virus type 1 (HIV-1) effectively reduce viral loads in patients. However, due to mutations, new therapy-resistant Env variants frequently emerge. The sites of mutations on Env that appear in each patient are considered random and unpredictable. Here we developed an algorithm to estimate for each patient the mutational state of each position based on the mutational state of adjacent positions on the three-dimensional structure of the protein.MethodsWe developed a dynamic ensemble selection algorithm designated k-best classifiers. It identifies the best classifiers within the neighborhood of a new observation and applies them to predict the variability state of each observation. To evaluate the algorithm, we applied amino acid sequences of Envs from 300 HIV-1-infected individuals (at least six sequences per patient). For each patient, amino acid variability values at all Env positions were mapped onto the three-dimensional structure of the protein. Then, the variability state of each position was estimated by the variability at adjacent positions of the protein.ResultsThe proposed algorithm showed higher performance than the base learner and a panel of classification algorithms. The mutational state of positions in the high-mannose patch and CD4-binding site of Env, which are targeted by multiple therapeutics, was predicted well. Importantly, the algorithm outperformed other classification techniques for predicting the variability state at multi-position footprints of therapeutics on Env.ConclusionsThe proposed algorithm applies a dynamic classifier-scoring approach that increases its performance relative to other classification methods. Better understanding of the spatiotemporal patterns of variability across Env may lead to new treatment strategies that are tailored to the unique mutational patterns of each patient. More generally, we propose the algorithm as a new high-performance dynamic ensemble selection technique.

Improving the accuracy of diagnosing and predicting coronary heart disease using ensemble method and feature selection techniques

Improving the accuracy of diagnosing and predicting coronary heart disease using ensemble method and feature selection techniques

Selective ensemble methods for deep learning segmentation of major vessels in invasive coronary angiography.

Invasive coronary angiography (ICA) is a primary imaging modality that visualizes the lumen area of coronary arteries for diagnosis and interventional guidance. In the current practice of quantitative coronary analysis (QCA), semi-automatic segmentation tools require labor-intensive and time-consuming manual correction, limiting their application in the catheterization room. This study aims to propose rank-based selective ensemble methods that improve the segmentation performance and reduce morphological errors that limit fully automated quantification of coronary artery using deep-learning segmentation of ICA. Two selective ensemble methods proposed in this work integrated the weighted ensemble approach with per-image quality estimation. The segmentation outcomes from five base models with different loss functions were ranked either by mask morphology or estimated dice similarity coefficient (DSC). The final output was determined by imposing different weights according to the ranks. The ranking criteria based on mask morphology were formulated from empirical insight to avoid frequent types of segmentation errors (MSEN), while the estimation of DSCs was performed by comparing the pseudo-ground truth generated from a meta-learner (ESEN). Five-fold cross-validation was performed with the internal dataset of 7426 coronary angiograms from 2924 patients, and prediction model was externally validated with 556 images of 226 patients. The selective ensemble methods improved the segmentation performance with DSCs up to 93.07% and provided a better delineation of coronary lesion with local DSCs of up to 93.93%, outperforming all individual models. Proposed methods also minimized the chances of mask disconnection in the most narrowed regions to 2.10%. The robustness of the proposed methods was also evident in the external validation. Inference time for major vessel segmentation was approximately one-sixth of a second. Proposed methods successfully reduced morphological errors in the predicted masks and were able to enhance the robustness of the automatic segmentation. The results suggest better applicability of real-time QCA-based diagnostic methods in routine clinical settings.

MTR-SDL: a soft computing based multi-tier rank model for shoulder X-ray classification.

Deep neural networks (DNN) effectiveness are contingent upon access to quality-labelled training datasets since label mistakes (label noise) in training datasets may significantly impair the accuracy of models trained on clean test data. The primary impediments to developing and using DNN models in the healthcare sector include the lack of sufficient label data. Labeling data by a domain expert are a costly and time-consuming task. To overcome this limitation, the proposed Multi-Tier Rank-based Semi-supervised deep learning (MTR-SDL) for Shoulder X-Ray Classification uses the small labelled dataset to generate a labelled dataset from unable dataset to obtain performance equivalent to approaches trained on the enormous dataset. The motivation behind the suggested model MTR-SDL approach is analogous to how physicians deal with unknown or suspicious patients in everyday life. Practitioners handle these questionable circumstances with the support of professional colleagues. Before initiating treatment, some patients consult with a range of skilled doctors. Patients are treated according to the most suitable professional diagnosis (vote count). In this article, we have proposed a new ensemble learning technique called "Rank based Ensemble Selection with machine learning models" (MTR-SDL) approach. In this technique, multiple machine learning models are trained on a labeled dataset, and their accuracy is ranked. A dynamic ensemble voting approach is then used to tag samples for each base model in the ensemble. The combination of these tags is used to generate a final tag for an unlabeled dataset. Our suggested MTR-SDL model has attained the best accuracy and specificity, sensitivity, precision, Matthew's correlation coefficient, false discovery rate, false positive rate, f1 score, negative predictive value, and false negative rate negative 92.776%, 97.376%, 86.932%, 96.192%, 85.644%, 3.808%, 2.624%, 91.072%, 90.85%, and 13.068% for unseen dataset, respectively. This approach has the potential to improve the performance of ensemble models by leveraging the strengths of multiple base models and selecting the most informative samples for each model. This study results in an improved Semi-supervised deep learning model that is more effective and precise.

Novel protein biomarkers for pneumonia and acute exacerbations in COPD: a pilot study.

Community-acquired pneumonia (CAP) and acute exacerbations of chronic obstructive pulmonary disease (AECOPD) result in high morbidity, mortality, and socio-economic burden. The usage of easily accessible biomarkers informing on disease entity, severity, prognosis, and pathophysiological endotypes is limited in clinical practice. Here, we have analyzed selected plasma markers for their value in differential diagnosis and severity grading in a clinical cohort. A pilot cohort of hospitalized patients suffering from CAP (n = 27), AECOPD (n = 10), and healthy subjects (n = 22) were characterized clinically. Clinical scores (PSI, CURB, CRB65, GOLD I-IV, and GOLD ABCD) were obtained, and interleukin-6 (IL-6), interleukin-8 (IL-8), interleukin-2-receptor (IL-2R), lipopolysaccharide-binding protein (LBP), resistin, thrombospondin-1 (TSP-1), lactotransferrin (LTF), neutrophil gelatinase-associated lipocalin (NGAL), neutrophil-elastase-2 (ELA2), hepatocyte growth factor (HGF), soluble Fas (sFas), as well as TNF-related apoptosis-inducing ligand (TRAIL) were measured in plasma. In CAP patients and healthy volunteers, we found significantly different levels of ELA2, HGF, IL-2R, IL-6, IL-8, LBP, resistin, LTF, and TRAIL. The panel of LBP, sFas, and TRAIL could discriminate between uncomplicated and severe CAP. AECOPD patients showed significantly different levels of LTF and TRAIL compared to healthy subjects. Ensemble feature selection revealed that CAP and AECOPD can be discriminated by IL-6, resistin, together with IL-2R. These factors even allow the differentiation between COPD patients suffering from an exacerbation or pneumonia. Taken together, we identified immune mediators in patient plasma that provide information on differential diagnosis and disease severity and can therefore serve as biomarkers. Further studies are required for validation in bigger cohorts.

Soft sensor modeling for small data scenarios based on data enhancement and selective ensemble

Data-driven soft sensors have been widely used to facilitate real-time estimations of difficult-to-measure variables. However, sufficient high-quality data are often difficult to obtain due to high cost or low sampling rate. Thus, a soft sensor method based on data enhancement and selective ensemble (DESE) is proposed. First, a generative model is proposed for generating virtual labeled samples by combining supervised variational autoencoder (SVAE) and Wasserstein GAN with gradient penalty (WGAN-gp), which is referred to as SV-WGANgp. Then, SV-WGANgp is trained on various resampled training subsets to generate different sets of virtual samples. Next, diverse enhanced base models are constructed with the extended training sets. Subsequently, a multi-objective optimization (MOO) is utilized to achieve ensemble pruning. Finally, the final prediction is obtained by fusing the selected models. The application results on an industrial chlortetracycline fermentation process and a simulated penicillin fermentation process verify the effectiveness and superiority of the proposed methods.

A Multi-Strategy Sparrow Search Algorithm with Selective Ensemble

Aiming at the deficiencies of the sparrow search algorithm (SSA), such as being easily disturbed by the local optimal and deficient optimization accuracy, a multi-strategy sparrow search algorithm with selective ensemble (MSESSA) is proposed. Firstly, three novel strategies in the strategy pool are proposed: variable logarithmic spiral saltation learning enhances global search capability, neighborhood-guided learning accelerates local search convergence, and adaptive Gaussian random walk coordinates exploration and exploitation. Secondly, the idea of selective ensemble is adopted to select an appropriate strategy in the current stage with the aid of the priority roulette selection method. In addition, the modified boundary processing mechanism adjusts the transgressive sparrows’ locations. The random relocation method is for discoverers and alerters to conduct global search in a large range, and the relocation method based on the optimal and suboptimal of the population is for scroungers to conduct better local search. Finally, MSESSA is tested on CEC 2017 suites. The function test, Wilcoxon test, and ablation experiment results show that MSESSA achieves better comprehensive performance than 13 other advanced algorithms. In four engineering optimization problems, the stability, effectiveness, and superiority of MSESSA are systematically verified, which has significant advantages and can reduce the design cost.

Clustered ensemble feature selection with M-GRU classification for efficient intrusion detection system of industrial systems

Industrial Control Systems (ICS) are susceptible to threats or attacks, and even minor changes or manipulation could cause major damage to industrial operations. Industrial control system cybersecurity is vital owing to the severe negative effects it could have on the economy, the environment, people, and politics. Therefore, it’s also crucial to design intrusion detection systems for industrial control systems. In this paper, an efficient intrusion detection system with clustered ensemble feature selection and a Multi-Level Modified Gated Recurrent Unit (M-GRU) classification model is proposed. This intrusion detection system with a general framework for clustered ensemble feature ranking approach is proposed to effectively find the best feature subset in network packet traffic data. The features designated are fed into a multi class classification algorithm Multi-Level Modified Gated Recurrent Unit (M-GRU) to efficiently detect the cyberattacks. Evaluation criteria including precision, accuracy, recall and F1 score are assessed and compared to other cutting-edge algorithms to assess the performance of the proposed model. The proposed model attained an average accuracy of 98.21 %. Results show that the suggested model increased the attack detection accuracy by an average of 5.935% and 0.116% when compared to the Gated Recurrent Unit, Long Short Term Memory, random forest and naïve bayes models.

Ensemble feature selection using Bonferroni, OWA and Induced OWA aggregation operators

Feature selection becomes inevitable owing to a rapid increase in digital technology which permits the generation of high dimensional data in a large quantity within a short time. Feature selection techniques not only improves classification accuracy but also decreases time complexity as well as computation cost and storage. Ensemble feature selection has lately emerged as a potential approach to data mining. Identifying numerous optimal features is one of the key advantages of ensemble feature selection. The objective of this study is to introduce an ensemble technique based on the rank aggregation procedure. Four operators are used for the purpose of aggregation, namely the induced ordered weighted averaging (IOWA) operator, the normalized Bonferroni weighted mean operator, the Bonferroni-ordered weighted averaging operator and the Bonferroni-induced ordered weighted averaging operator. These operators enable the evaluation of continuous aggregations, multiple assessments between each input and distance measurements in the same formulation. An objective weighting approach denoted as the entropy weight method, is utilized to measure the degree of disorder. A total of 10 benchmark data sets are employed to evaluate the superiority of the proposed methodologies. The effectiveness of the proposed method is evaluated and compared using the accuracy, F-measure, precision and recall performance metrics, and better results than those from other existing techniques are obtained.