Conventional Ensemble Methods Research Articles

On the use of machine learning based ensemble approaches to improve evapotranspiration estimates from croplands across a wide environmental gradient

Accurately mapping of regional-scale evapotranspiration (ET) from the croplands using remote sensing is currently challenged by limited spatial information on crop and field management to properly characterize the biophysical constraints on ET. A multi-model ensemble can potentially address this challenge, however, conventional ensemble models using the simple average (MEAN) or Bayesian Model Average (BMA) assign a fixed weight to each model and may not fully utilize the strengths of individual models. To this end, we developed four ensemble ET Models (EEMs) that use different machine learning (ML) classifiers, namely K-nearest neighbors, random forest, support vector machine, and multi-layer perception neural network (MLP), to assign varying weights to assemble six physically-driven remote sensing-based ET models. These ML-based EEMs were compared against the six individual ET models and two conventional ensemble methods (MEAN and BMA) using latent heat fluxes (λE) observations from 47 cropland eddy covariance flux sites covering diverse environments across the globe. Results suggested that while MEAN and BMA can reduce some uncertainties in the individual models, ML-based EEMs can better integrate the capabilities of multiple biophysical constraints on ET used across the individual models. The four ML-based EEMs yielded daily λE for training, validation, and testing datasets with the coefficient of determination (R2) and root mean squared error (RMSE) within 0.75 – 0.83 and 18 – 21 W m−2, respectively, among which the MLP algorithm was found to be the most efficient with respect to accuracies and costs. These performance metrics were much better than those from the conventional ensemble models (R2 = 0.69 – 0.71, RMSE = 23 – 25 W m−2) and six individual ET models (R2 = 0.53 – 0.69, RMSE = 26 – 35 W m−2). Results suggested that ML-based EEMs perform much better than the conventional approaches and hence can be viable tools for mapping cropland ET across a wide environmental gradient.

A New Approach to 3D Modeling of Inhomogeneous Populations of Viral Regulatory RNA.

Tertiary structure (3D) is the physical context of RNA regulatory activity. Retroviruses are RNA viruses that replicate through the proviral DNA intermediate transcribed by hosts. Proviral transcripts form inhomogeneous populations due to variable structural ensembles of overlapping regulatory RNA motifs in the 5′-untranslated region (UTR), which drive RNAs to be spliced or translated, and/or dimerized and packaged into virions. Genetic studies and structural techniques have provided fundamental input constraints to begin predicting HIV 3D conformations in silico. Using SimRNA and sets of experimentally-determined input constraints of HIVNL4-3 trans-activation responsive sequence (TAR) and pairings of unique-5′ (U5) with dimerization (DIS) or AUG motifs, we calculated a series of 3D models that differ in proximity of 5′-Cap and the junction of TAR and PolyA helices; configuration of primer binding site (PBS)-segment; and two host cofactors binding sites. Input constraints on U5-AUG pairings were most compatible with intramolecular folding of 5′-UTR motifs in energetic minima. Introducing theoretical constraints predicted metastable PolyA region drives orientation of 5′-Cap with TAR, U5 and PBS-segment helices. SimRNA and the workflow developed herein provides viable options to predict 3D conformations of inhomogeneous populations of large RNAs that have been intractable to conventional ensemble methods.

Transductive Ensemble Learning for Neural Machine Translation

Ensemble learning, which aggregates multiple diverse models for inference, is a common practice to improve the accuracy of machine learning tasks. However, it has been observed that the conventional ensemble methods only bring marginal improvement for neural machine translation (NMT) when individual models are strong or there are a large number of individual models. In this paper, we study how to effectively aggregate multiple NMT models under the transductive setting where the source sentences of the test set are known. We propose a simple yet effective approach named transductive ensemble learning (TEL), in which we use all individual models to translate the source test set into the target language space and then finetune a strong model on the translated synthetic corpus. We conduct extensive experiments on different settings (with/without monolingual data) and different language pairs (English↔{German, Finnish}). The results show that our approach boosts strong individual models with significant improvement and benefits a lot from more individual models. Specifically, we achieve the state-of-the-art performances on the WMT2016-2018 English↔German translations.

Improving Pronunciation Erroneous Tendency Detection with Multi-Model Soft Targets

Detecting pronunciation erroneous tendency (PET) can provide detailed instructive feedback for second language learners in computer aided pronunciation training (CAPT). In this paper, we proposed to apply soft targets from various models to improve the detection performance of PET. First, we examined the effectiveness of soft targets in three single systems by replacing hard targets with soft targets directly for mispronunciation detection. Furthermore, we proposed two kinds of methods using multi-model soft targets in this paper: 1) explicit combination, which used multi-model soft targets as the final targets by weighted linear combination; 2) implicit combination, which employed the multi-task framework to combine soft targets. Experimental results showed that the detection performance of PET could be improved by using both single soft targets and multi-model soft targets. Moreover, using multi-model soft targets within multi-task framework achieved the best results in pronunciation error detection task, and it was more efficient than conventional ensemble methods which required multiple decoding runs or forward passes.

Combining multiple forecasts for multivariate time series via state-dependent weighting.

We present a model-free forecast algorithm that dynamically combines multiple forecasts using multivariate time series data. The underlying principle is based on the fact that forecast performance depends on the position in the state space. This property is exploited to weight multiple forecasts via a local loss function. Specifically, additional weights are assigned to appropriate forecasts depending on their positions in a state space reconstructed via delay coordinates. The function evaluates the forecast error discounted by the distance in the space, whereas most existing methods discount the error in relation to time. In addition, forecasts are selected with the function for each time step based on the existing multiview embedding approach; by contrast, the original multiview embedding selects the reconstructions in advance before starting the forecast. The proposed prediction method has the smallest mean squared error among conventional ensemble methods for the Rössler and the Lorenz'96I models. The results of comparison of the proposed method with conventional machine-learning methods using a flood forecast example indicate that the proposed method yields superior accuracy. We also demonstrate that the proposed method might even correctly forecast the maximum water level of rivers without any prior knowledge.

Multiobjective sparse ensemble learning by means of evolutionary algorithms

Ensemble learning can improve the performance of individual classifiers by combining their decisions. The sparseness of ensemble learning has attracted much attention in recent years. In this paper, a novel multiobjective sparse ensemble learning (MOSEL) model is proposed. Firstly, to describe the ensemble classifiers more precisely the detection error trade-off (DET) curve is taken into consideration. The sparsity ratio (sr) is treated as the third objective to be minimized, in addition to false positive rate (fpr) and false negative rate (fnr) minimization. The MOSEL turns out to be augmented DET (ADET) convex hull maximization problem. Secondly, several evolutionary multiobjective algorithms are exploited to find sparse ensemble classifiers with strong performance. The relationship between the sparsity and the performance of ensemble classifiers on the ADET space is explained. Thirdly, an adaptive MOSEL classifiers selection method is designed to select the most suitable ensemble classifiers for a given dataset. The proposed MOSEL method is applied to well-known MNIST datasets and a real-world remote sensing image change detection problem, and several datasets are used to test the performance of the method on this problem. Experimental results based on both MNIST datasets and remote sensing image change detection show that MOSEL performs significantly better than conventional ensemble learning methods.

Adaptive fault feature extraction from wayside acoustic signals from train bearings

Wayside acoustic detection of train bearing faults plays a significant role in maintaining safety in the railway transport system. However, the bearing fault information is normally masked by strong background noises and harmonic interferences generated by other components (e.g. axles and gears). In order to extract the bearing fault feature information effectively, a novel method called improved singular value decomposition (ISVD) with resonance-based signal sparse decomposition (RSSD), namely the ISVD-RSSD method, is proposed in this paper. A Savitzky-Golay (S-G) smoothing filter is used to filter singular vectors (SVs) in the ISVD method as an extension of the singular value decomposition (SVD) theorem. Hilbert spectrum entropy and a stepwise optimisation strategy are used to optimize the S-G filter's parameters. The RSSD method is able to nonlinearly decompose the wayside acoustic signal of a faulty train bearing into high and low resonance components, the latter of which contains bearing fault information. However, the high level of noise usually results in poor decomposition results from the RSSD method. Hence, the collected wayside acoustic signal must first be de-noised using the ISVD component of the ISVD-RSSD method. Next, the de-noised signal is decomposed by using the RSSD method. The obtained low resonance component is then demodulated with a Hilbert transform such that the bearing fault can be detected by observing Hilbert envelope spectra. The effectiveness of the ISVD-RSSD method is verified through both laboratory field-based experiments as described in the paper. The results indicate that the proposed method is superior to conventional spectrum analysis and ensemble empirical mode decomposition methods.

Single-molecule manipulation and detection.

Compared to conventional ensemble methods, studying macromolecules at single-molecule level can reveal extraordinary clear and even surprising views for a biological reaction. In the past 20 years, single-molecule techniques have been undergoing a very rapid development, and these cutting edge technologies have revolutionized the biological research by facilitating single-molecule manipulation and detection. Here we give a brief review about these advanced techniques, including optical tweezers, magnetic tweezers, atomic force microscopy (AFM), hydrodynamic flow-stretching assay, and single-molecule fluorescence resonance energy transfer (smFRET). We are trying to describe their basic principles and provide a few examples of applications for each technique. This review aims to give a rather introductory survey of single-molecule techniques for audiences with biological or biophysical background.

Ensemble Methods for Classification of Physical Activities from Wrist Accelerometry.

To investigate whether the use of ensemble learning algorithms improve physical activity recognition accuracy compared to the single classifier algorithms, and to compare the classification accuracy achieved by three conventional ensemble machine learning methods (bagging, boosting, random forest) and a custom ensemble model comprising four algorithms commonly used for activity recognition (binary decision tree, k nearest neighbor, support vector machine, and neural network). The study used three independent data sets that included wrist-worn accelerometer data. For each data set, a four-step classification framework consisting of data preprocessing, feature extraction, normalization and feature selection, and classifier training and testing was implemented. For the custom ensemble, decisions from the single classifiers were aggregated using three decision fusion methods: weighted majority vote, naïve Bayes combination, and behavior knowledge space combination. Classifiers were cross-validated using leave-one subject out cross-validation and compared on the basis of average F1 scores. In all three data sets, ensemble learning methods consistently outperformed the individual classifiers. Among the conventional ensemble methods, random forest models provided consistently high activity recognition; however, the custom ensemble model using weighted majority voting demonstrated the highest classification accuracy in two of the three data sets. Combining multiple individual classifiers using conventional or custom ensemble learning methods can improve activity recognition accuracy from wrist-worn accelerometer data.

A hybrid ensemble pruning approach based on consensus clustering and multi-objective evolutionary algorithm for sentiment classification

Sentiment analysis is a critical task of extracting subjective information from online text documents. Ensemble learning can be employed to obtain more robust classification schemes. However, most approaches in the field incorporated feature engineering to build efficient sentiment classifiers.The purpose of our research is to establish an effective sentiment classification scheme by pursuing the paradigm of ensemble pruning. Ensemble pruning is a crucial method to build classifier ensembles with high predictive accuracy and efficiency. Previous studies employed exponential search, randomized search, sequential search, ranking based pruning and clustering based pruning. However, there are tradeoffs in selecting the ensemble pruning methods. In this regard, hybrid ensemble pruning schemes can be more promising.In this study, we propose a hybrid ensemble pruning scheme based on clustering and randomized search for text sentiment classification. Furthermore, a consensus clustering scheme is presented to deal with the instability of clustering results. The classifiers of the ensemble are initially clustered into groups according to their predictive characteristics. Then, two classifiers from each cluster are selected as candidate classifiers based on their pairwise diversity. The search space of candidate classifiers is explored by the elitist Pareto-based multi-objective evolutionary algorithm.For the evaluation task, the proposed scheme is tested on twelve balanced and unbalanced benchmark text classification tasks. In addition, the proposed approach is experimentally compared with three ensemble methods (AdaBoost, Bagging and Random Subspace) and three ensemble pruning algorithms (ensemble selection from libraries of models, Bagging ensemble selection and LibD3C algorithm). Results demonstrate that the consensus clustering and the elitist pareto-based multi-objective evolutionary algorithm can be effectively used in ensemble pruning. The experimental analysis with conventional ensemble methods and pruning algorithms indicates the validity and effectiveness of the proposed scheme.

Hybrid supervised clustering based ensemble scheme for text classification

PurposeThe immense quantity of available unstructured text documents serve as one of the largest source of information. Text classification can be an essential task for many purposes in information retrieval, such as document organization, text filtering and sentiment analysis. Ensemble learning has been extensively studied to construct efficient text classification schemes with higher predictive performance and generalization ability. The purpose of this paper is to provide diversity among the classification algorithms of ensemble, which is a key issue in the ensemble design.Design/methodology/approachAn ensemble scheme based on hybrid supervised clustering is presented for text classification. In the presented scheme, supervised hybrid clustering, which is based on cuckoo search algorithm and k-means, is introduced to partition the data samples of each class into clusters so that training subsets with higher diversities can be provided. Each classifier is trained on the diversified training subsets and the predictions of individual classifiers are combined by the majority voting rule. The predictive performance of the proposed classifier ensemble is compared to conventional classification algorithms (such as Naïve Bayes, logistic regression, support vector machines and C4.5 algorithm) and ensemble learning methods (such as AdaBoost, bagging and random subspace) using 11 text benchmarks.FindingsThe experimental results indicate that the presented classifier ensemble outperforms the conventional classification algorithms and ensemble learning methods for text classification.Originality/valueThe presented ensemble scheme is the first to use supervised clustering to obtain diverse ensemble for text classification

MediBoost: a Patient Stratification Tool for Interpretable Decision Making in the Era of Precision Medicine.

Machine learning algorithms that are both interpretable and accurate are essential in applications such as medicine where errors can have a dire consequence. Unfortunately, there is currently a tradeoff between accuracy and interpretability among state-of-the-art methods. Decision trees are interpretable and are therefore used extensively throughout medicine for stratifying patients. Current decision tree algorithms, however, are consistently outperformed in accuracy by other, less-interpretable machine learning models, such as ensemble methods. We present MediBoost, a novel framework for constructing decision trees that retain interpretability while having accuracy similar to ensemble methods, and compare MediBoost’s performance to that of conventional decision trees and ensemble methods on 13 medical classification problems. MediBoost significantly outperformed current decision tree algorithms in 11 out of 13 problems, giving accuracy comparable to ensemble methods. The resulting trees are of the same type as decision trees used throughout clinical practice but have the advantage of improved accuracy. Our algorithm thus gives the best of both worlds: it grows a single, highly interpretable tree that has the high accuracy of ensemble methods.

A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification

Typically performed by supervised machine learning algorithms, sentiment analysis is highly useful for extracting subjective information from text documents online. Most approaches that use ensemble learning paradigms toward sentiment analysis involve feature engineering in order to enhance the predictive performance. In response, we sought to develop a paradigm of a multiobjective, optimization-based weighted voting scheme to assign appropriate weight values to classifiers and each output class based on the predictive performance of classification algorithms, all to enhance the predictive performance of sentiment classification. The proposed ensemble method is based on static classifier selection involving majority voting error and forward search, as well as a multiobjective differential evolution algorithm. Based on the static classifier selection scheme, our proposed ensemble method incorporates Bayesian logistic regression, naïve Bayes, linear discriminant analysis, logistic regression, and support vector machines as base learners, whose performance in terms of precision and recall values determines weight adjustment. Our experimental analysis of classification tasks, including sentiment analysis, software defect prediction, credit risk modeling, spam filtering, and semantic mapping, suggests that the proposed classification scheme can predict better than conventional ensemble learning methods such as AdaBoost, bagging, random subspace, and majority voting. Of all datasets examined, the laptop dataset showed the best classification accuracy (98.86%).

InSAR processing for volcano monitoring and other near‐real time applications

AbstractRadar interferometry (InSAR, interferometric synthetic aperture radar) is routinely used to measure surface deformation prior to, during, and after volcanic events, although not in a monitoring capacity. The improved data availability of some current satellite missions presents us with the opportunity to do just that. We present here a fast and flexible algorithm to estimate coherence and select points on an interferogram‐by‐interferogram basis, which overcomes limitations of the conventional boxcar ensemble method in areas of marginal coherence. Time series methods, which offer an alternative way to select coherent points, are typically slow, and do not allow for insertion of new data without reprocessing the entire data set. Our new algorithm calculates the coherence for each point based on an ensemble of points with similar amplitude behavior throughout the data set. The points that behave similarly are selected prior to new images being acquired, on the assumption that the behavior of these nearby points does not change rapidly through time. The resulting coherence estimate is superior in resolution and noise level to the boxcar method. In contrast to most other time series methods, we select a different set of coherent points for each interferogram, avoiding the selection compromise inherent to other time series methods. The relative simplicity of this strategy compared to other time series techniques means we can process new images in about 1 h for a typical setup.

Incremental Semi-Supervised Clustering Ensemble for High Dimensional Data Clustering

Traditional cluster ensemble approaches have three limitations: ( $1$ ) They do not make use of prior knowledge of the datasets given by experts. ( $2$ ) Most of the conventional cluster ensemble methods cannot obtain satisfactory results when handling high dimensional data. ( $3$ ) All the ensemble members are considered, even the ones without positive contributions. In order to address the limitations of conventional cluster ensemble approaches, we first propose an incremental semi-supervised clustering ensemble framework (ISSCE) which makes use of the advantage of the random subspace technique, the constraint propagation approach, the proposed incremental ensemble member selection process, and the normalized cut algorithm to perform high dimensional data clustering. The random subspace technique is effective for handling high dimensional data, while the constraint propagation approach is useful for incorporating prior knowledge. The incremental ensemble member selection process is newly designed to judiciously remove redundant ensemble members based on a newly proposed local cost function and a global cost function, and the normalized cut algorithm is adopted to serve as the consensus function for providing more stable, robust, and accurate results. Then, a measure is proposed to quantify the similarity between two sets of attributes, and is used for computing the local cost function in ISSCE. Next, we analyze the time complexity of ISSCE theoretically. Finally, a set of nonparametric tests are adopted to compare multiple semi-supervised clustering ensemble approaches over different datasets. The experiments on 18 real-world datasets, which include six UCI datasets and 12 cancer gene expression profiles, confirm that ISSCE works well on datasets with very high dimensionality, and outperforms the state-of-the-art semi-supervised clustering ensemble approaches.

Ensemble Classification and Regression-Recent Developments, Applications and Future Directions [Review Article

Ensemble methods use multiple models to get better performance. Ensemble methods have been used in multiple research fields such as computational intelligence, statistics and machine learning. This paper reviews traditional as well as state-of-the-art ensemble methods and thus can serve as an extensive summary for practitioners and beginners. The ensemble methods are categorized into conventional ensemble methods such as bagging, boosting and random forest, decomposition methods, negative correlation learning methods, multi-objective optimization based ensemble methods, fuzzy ensemble methods, multiple kernel learning ensemble methods and deep learning based ensemble methods. Variations, improvements and typical applications are discussed. Finally this paper gives some recommendations for future research directions.

A discriminative random sampling strategy with individual-author feature selection for writeprint recognition of Chinese texts

The auto authorship recognition has become a novel technique to investigate cybercrimes. But the challenge of the research is that a huge number of features exist in the moderate-sized corpus, which causes the curse of over-training. Besides, it is hard to distinguish between potential authors only by a single feature set. In this paper, we proposed a random sampling style ensemble method with individual-author feature selection to exploit the high-dimensional feature space. The proposed method randomly picks writing-style features on each individual-author feature set (IAFS) partitioned from the whole feature set. The IAFSs are heuristically selected with training set of each author. Then, multiple base classifiers (BCs) are formed on the sampled feature sets. Finally, all BCs are fused to get a final decision. Experimental results on the real-life Chinese forum data verify the robustness of the proposed method compared with conventional ensemble methods. We also analyze the diversity of algorithm to reveal that the ensemble strategy is more effective and can construct more diverse BCs than random subspace methods.

Wisdom of Crowds cluster ensemble

The Wisdom of Crowds is a phenomenon described in social science that suggests four criteria applicable to groups of people. It is claimed that, if these criteria are satisfied, then the aggregate decisions made by a group will often be better than those of its individual members. Inspired by this concept, we present a novel feedback framework for the cluster ensemble problem, which we call Wisdom of Crowds Cluster Ensemble (WOCCE). Although many conventional cluster ensemble methods focusing on diversity have recently been proposed, WOCCE analyzes the conditions necessary for a crowd to exhibit this collective wisdom. These include decentralization criteria for generating primary results, independence criteria for the base algorithms, and diversity criteria for the ensemble members. We suggest appropriate procedures for evaluating these measures, and propose a new measure to assess the diversity. We evaluate the performance of WOCCE against some other traditional base algorithms as well as state-of-the-art ensemble methods. The results demonstrate the efficiency of WOCCE's aggregate decision-making compared to other algorithms.

DE2: Dynamic ensemble of ensembles for learning nonstationary data

Learning nonstationary data with concept drift has received much attention in machine learning and been an active topic in ensemble learning. Specifically, batch growing ensemble methods present one important direction for dealing with concept drift involved in nonstationary data. However, current batch growing ensemble methods combine all the available component classifiers only, each trained independently from a batch of non-stationary data. They simply discard interim ensembles and hence may lose useful information obtained from the fine-tuned interim ensembles. Distinctively, we introduce a comprehensive hierarchical approach called Dynamic Ensemble of Ensembles (DE2). The novel method combines classifiers as an ensemble of all the interim ensembles dynamically from consecutive batches of nonstationary data. DE2 includes two key stages: component classifiers and interim ensembles are dynamically trained; and the final ensemble is then learned by exponentially-weighted averaging with available experts, i.e., interim ensembles. Moreover. we engage Sparsity Learning to choose component classifiers selectively and intelligently. We also incorporate the techniques of Dynamic Weighted Majority, and Learn++.NSE for better integrating different classifiers dynamically. We perform experiments with two benchmark test sets in real nonstationary environments, and compare our DE2 method to other conventional competitive ensemble methods. Experimental results confirm that our approach consistently leads to better performance and has promising generalization ability for learning in nonstationary environments.

A novel classifier ensemble method with sparsity and diversity

We consider the classifier ensemble problem in this paper. Due to its superior performance to individual classifiers, class ensemble has been intensively studied in the literature. Generally speaking, there are two prevalent research directions on this, i.e., to diversely generate classifier components, and to sparsely combine multiple classifiers. While most current approaches are emphasized on either sparsity or diversity only, we investigate the classifier ensemble by learning both sparsity and diversity simultaneously. We manage to formulate the classifier ensemble problem with the sparsity or/and diversity learning in a general framework. In particular, the classifier ensemble with sparsity and diversity can be represented as a mathematical optimization problem. We then propose a heuristic algorithm, capable of obtaining ensemble classifiers with consideration of both sparsity and diversity. We exploit the genetic algorithm, and optimize sparsity and diversity for classifier selection and combination heuristically and iteratively. As one major contribution, we introduce the concept of the diversity contribution ability so as to select proper classifier components and evolve classifier weights eventually. Finally, we compare our proposed novel method with other conventional classifier ensemble methods such as Bagging, least squares combination, sparsity learning, and AdaBoost, extensively on UCI benchmark data sets and the Pascal Large Scale Learning Challenge 2008 webspam data. The experimental results confirm that our approach leads to better performance in many aspects.