Diverse Base Learners Research Articles

Estimation of Carbon Stock in Chinese Fir Stands Based on Stacking Ensemble Learning and LSTM Models

Abstract Forest carbon sinks, a critical component of the global carbon cycle, constitute nearly half of the total terrestrial carbon pool. This study employed correlation analysis and factor effect analysis to quantify the influences of various factors on the volume of Chinese fir stands. A novel modeling framework was developed using the stacking ensemble learning and LSTM (LongShort-TermMemory) models. This framework incorporated diverse base learners, including XGBoost, Adaboost, KNN, and DT, which are intelligently ensemble via GBDT as a meta learner. The model was further optimized by comparing activation functions and optimizers, with ReLU selected as the activation function and Adam as the optimizer. Model accuracy was evaluated using RMSE, MAE, and R2 metrics, significantly enhancing its learning ability and generalization performance. The findings are as follows: (1) Stand stocking showed strong positive correlations with depression, age group, average diameter at breast height (ADBH), average tree height (ATH), slope, and elevation. Conversely, it exhibited significant negative correlations with origin, stand density, and slope position. In investigating Chinese fir growth on slopes, no significant growth differences were observed between downslopes and midslopes; however, both differed significantly from upper slopes. (2) The stacking ensemble learning method constructed here surpassed all existing single models in terms of estimation and assessment indices, demonstrating superior comprehensive performance. (3) Among the LSTM models, Adam-LSTM performed the best (R2=0.844), followed by Sigmoid-LSTM (R2=0.656), while the RMSprop-LSTM model performed the worst (R2=0.618). Combined with artificial intelligence methods, our optimized carbon stock estimation model can help to improve the ability of forest land management and provide a theoretical basis for the scientific management of forest areas.

Ensemble Learning Based Gene Regulatory Network Inference

In the machine learning field, the technique known as ensemble learning aims at combining different base learners in order to increase the quality and the robustness of the predictions. Indeed, this approach has widely been applied to tackle, with success, real world problems from different domains, including computational biology. Nevertheless, despite their potential, ensembles combining results from different base learners have been understudied in the context of gene regulatory network inference. In this paper we applied genetic algorithms and frequent itemset mining, to design small but effective ensembles of gene regulatory network inference methods. These ensembles were evaluated and compared to well-established single and ensemble methods, on both real and synthetic datasets. Results showed that small ensembles, consisting of few but diverse base learners, enhance the exploration of the solution space, and compensate base learners biases, outperforming state-of-the-art methods. Results advocate for the use of such methods as gene regulatory network inference tools.

Genetic Algorithm-Based Ensemble Hybrid Sparse ELM for Grasp Stability Recognition With Multimodal Tactile Signals

Grasp stability recognition based on tactile perception has attracted increasing attention in the robotics community. In this article, we extract tactile features from multimodal tactile signals and propose a novel ensemble approach named genetic algorithm-based ensemble hybrid sparse extreme learning machine (GA-EHSELM) for the grasp stability recognition task. In contrast to traditional ensemble extreme learning machines (ELMs), the proposed approach takes the sparsity of the random weights assigned to the connection between the input layer and the hidden layer for each base learner into consideration, in order to avoid the problem of overfitting. In addition, The diversity of base learners is increased by constructing two types of sparse ELM (SELM). The random weights of the base SELMs are sampled from two different distributions with a certain probability. Furthermore, we utilize the genetic algorithm (GA) for the optimization of tactile features and the base sparse ELMs. To be specific, GA is employed for feature selection as well as the optimization of the sparsity of the random weights associated with different types of base learners and the number of each type of base learner. The effectiveness of the proposed approach has been demonstrated on a public tactile grasp stability dataset.

Feature Fusion based Ensemble Method for remaining useful life prediction of machinery

The signal analysis features and deep representation features have been widely utilized to predict the Remaining Useful Life (RUL) of machinery. However, existing studies rarely fuse these features for RUL prediction to explore their complementarity. Therefore, this paper proposes a Feature Fusion based Ensemble Method (FFEM) that makes full use of the characteristics of signal analysis features and deep representation features. First, features are extracted by signal analysis and deep learning methods, respectively. The time-domain features, frequency-domain features, and time–frequency​ domain features are extracted by different signal analysis methods, while the deep representation features are from the bidirectional long short-term memory networks. Then, an improved random subspace method is proposed, which fuses different types of features based on group-based sparse learning to use the complementarity among features. Furthermore, accurate and diverse base learners are generated and the aggregation strategy, mean rule, is adopted for predicting RUL. To validate the proposed FFEM, experiments on the run-to-failure datasets of bearings are conducted, and the experimental results verify that the proposed method greatly improves the RUL prediction performance and surpasses other existing ensemble learning methods.

A Multiobjective Evolutionary Nonlinear Ensemble Learning With Evolutionary Feature Selection for Silicon Prediction in Blast Furnace.

In the blast furnace ironmaking process, accurate prediction of silicon content in molten iron is of great significance for maintaining stable furnace conditions, improving hot metal quality, and reducing energy consumption. However, most of the current research works employ linear correlation coefficient methods to select input features in modeling, which may not fully take the nonlinear and coupling relationships between features into account. Therefore, this article considers the input feature selection issue of silicon content prediction model from a new perspective and proposes a multiobjective evolutionary nonlinear ensemble learning model with evolutionary feature selection mechanism (MOENE-EFS), in which extreme learning machine is adopted as the base learner. MOENE-EFS takes the input feature scheme of each base learner as well as their network structure and parameters as decision variables and proposes a modified nondominated sorting differential evolution algorithm to optimize two conflicting objectives, i.e., accuracy and diversity of base learners, simultaneously. Through the optimization, a set of Pareto optimal base learners with high accuracy and strong diversity can be obtained. Moreover, different from the linear ensemble methods commonly used in classical evolutionary ensemble learning, this article proposes a nonlinear ensemble method to combine the obtained base learners based on differential evolution. Experimental results indicate that the two proposed strategies, i.e., evolutionary feature selection and nonlinear ensemble, are very effective in improving the accuracy and stability of the prediction model. MOENE-EFS also outperforms the other prediction models in both benchmark data and practical industrial data. Furthermore, analysis on the input features of all Pareto optimal base learners shows that the evolutionary feature selection is capable of selecting essential features and is consistent with human experience, which indicates it is a promising method to deal with the input feature selection issue in silicon content prediction.

DOWELL: Diversity-Induced Optimally Weighted Ensemble Learner for Predictive Maintenance of Industrial Internet of Things Devices

The Industrial Internet of Things (I-IoT) enables a smarter maintenance approach for various industrial applications, such as manufacturing, logistics, etc. This approach is based on continuously observing system data to predict device failures and increase device efficiency. This smart maintenance, also known as predictive maintenance (PDM), finds an optimal maintenance schedule to reduce operational and capital costs. Accurate remaining useful life (RUL) prediction is critical for an effective PDM system. Data-driven RUL estimation methods are quite popular owing to their easier implementation. We observe that the performance of data-driven methods varies drastically based on the data set and underlying system parameters, thus making it difficult to have a single algorithm and a parameter set that work best for all settings. We propose an ensemble learning framework, where accurate and diverse base learners are selected out of 20 different state-of-the-art deep learning models. For accuracy, we discover the optimal weights of base learners by constructing an optimization problem. For diversity, we measure the similarity among base learner predictions and iteratively select the most diversified set of models while keeping the accuracy at a certain level. We show that our approach can have 39.2% faster retraining compared to an accuracy-based ensemble with only 3.4% loss in accuracy.

A Self-Training Hierarchical Prototype-based Ensemble Framework for Remote Sensing Scene Classification

Remote sensing scene classification plays a critical role in a wide range of real-world applications. Technically, however, scene classification is an extremely challenging task due to the huge complexity in remotely sensed scenes, and the difficulty in acquiring labelled data for model training such as supervised deep learning. To tackle these issues, a novel semi-supervised ensemble framework is proposed here using the self-training hierarchical prototype-based classifier as the base learner for chunk-by-chunk prediction. The framework has the ability to build a powerful ensemble model from both labelled and unlabelled images with minimum supervision. Different feature descriptors are employed in the proposed ensemble framework to offer multiple independent views of images. Thus, the diversity of base learners is guaranteed for ensemble classification. To further increase the overall accuracy, a novel cross-checking strategy was introduced to enable the base learners to exchange pseudo-labelling information during the self-training process, and maximize the correctness of pseudo-labels assigned to unlabelled images. Extensive numerical experiments on popular benchmark remote sensing scenes demonstrated the effectiveness of the proposed ensemble framework, especially where the number of labelled images available is limited. For example, the classification accuracy achieved on the OPTIMAL-31, PatternNet and RSI-CB256 datasets was up to 99.91%, 98. 67% and 99.07% with only 40% of the image sets used as labelled training images, surpassing or at least on par with mainstream benchmark approaches trained with double the number of labelled images.

Detecting rotating machinery faults under different working conditions with cross-domain negative correlated ensemble algorithm

When detecting faults of rotating machinery, variation in working conditions leads to the distribution mismatch between training and test data. Thus, the diagnosis accuracy of existing deep learning models deteriorates greatly. To address that, we propose a novel ensemble algorithm. The ensemble algorithm employs deep convolutional extreme learning machine (DCELM) as base learners to learn representative features from time-frequency images. Then, a selection scheme is designed to select diverse base learners which are negatively correlated. Meanwhile, by calculating the correlation term using both the source and the target domain data, the attained ensemble diversity is also valid in target domains. The proposed ensemble algorithm is applied to detect faults of rotating machinery components in three cases. Results show that it attains the best performance on all cross-condition tasks. On average, its accuracy is 1.9% higher than other state-of-the-art ensemble methods and achieves 3.6% enhancement compared with its base learner.

Probabilistic wind power forecasting using selective ensemble of finite mixture Gaussian process regression models

Ensemble learning models have been widely used for wind power forecasting to facilitate efficient dispatching of power systems. However, traditional ensemble methods cannot always function well due to insufficient accuracy and diversity of base learners, ignorance of ensemble pruning, as well as the lack of adaptation capability. Therefore, a novel probabilistic wind power forecasting method is proposed based on selective ensemble of finite mixture Gaussian process regression models (SEFMGPR). First, a set of diverse local Gaussian process regression (GPR) models are constructed through multimodal perturbation mechanism, i.e., perturbing the training data and input attributes simultaneously. Then, a set of finite mixture GPR models (FMGPR) is built by integrating local GPR models through finite mixture mechanism (FMM). Next, the highly influential FMGPR models are selected using genetic algorithm (GA) based ensemble pruning. When a new test sample comes, the component predictions from the selected FMGPR models are adaptively combined by using FMM again and the probabilistic prediction results of the SEFMGPR model are obtained. Besides, an incremental adaptation mechanism is used to alleviate performance degradation of SEFMGPR. The application results from a real wind farm dataset show that SEFMGPR outperforms the traditional global and ensemble wind power prediction methods, and can maintain high prediction accuracy by effectively handling time-varying changes of wind power data.

Global convergence of Negative Correlation Extreme Learning Machine

Ensemble approaches introduced in the Extreme Learning Machine literature mainly come from methods that rely on data sampling procedures, under the assumption that the training data are heterogeneously enough to set up diverse base learners. To overcome this assumption, it was proposed an ELM ensemble method based on the Negative Correlation Learning framework, called Negative Correlation Extreme Learning Machine (NCELM). This model works in two stages: (i) different ELMs are generated as base learners with random weights in the hidden layer, and (ii) a NCL penalty term with the information of the ensemble prediction is introduced in each ELM minimization problem, updating the base learners, (iii) second step is iterated until the ensemble converges. Although this NCL ensemble method was validated by an experimental study with multiple benchmark datasets, no information was given on the conditions about this convergence. This paper mathematically presents sufficient conditions to guarantee the global convergence of NCELM. The update of the ensemble in each iteration is defined as a contraction mapping function, and through Banach theorem, global convergence of the ensemble is proved.

ROTATION FORESTS AND RANDOM FOREST CLASSIFIERS FOR MONITORING OF VEGETATION IN PAYS DE BREST (FRANCE)

Abstract. Remote sensing is a potentially very useful source of information for spatial monitoring of natural or cultivated vegetation. The latest advances, in particular the arrival of new image acquisition programs, are changing the temporal approach to monitoring vegetation. The latest European satellites launched, delivering an image every 5 days for each point on the globe, allow the end of a growing season to be monitored. The main objective of this work is to identify and map the vegetation in the Pays de Brest area by using a multi sensors stacking of Sentinel-1 and Sentinel-2 satellites data via Random Forest, Rotation forests (RoF) and Canonical Correlation Forests (CCFs). RoF and CCF create diverse base learners using data transformation and subset features. Twenty four radar images and optical dataa representing different dates in 2017 were processed in time series stacks. The results of RoF and CCF were compared with the ones of RF.

Ensemble learning based on fitness Euclidean-distance ratio differential evolution for classification

Ensemble learning is a system that combines a set of base learners to improve the performance in machine learning, where accuracy and diversity of base learners are two important factors. However, these two factors are usually contradictory. To address this problem, in this paper, we propose a novel ensemble learning algorithm based on fitness Euclidean-distance ratio differential evolution, to train the neural network ensemble. FEFERR_ELA employs a multimodal evolutionary algorithm that is capable of producing diverse solutions to search for optimal solutions corresponding to parameters of base learners, where each optimal solution leads to one trained model. A dynamic ensemble selection scheme is applied to select appropriate individuals for the ensemble. The proposed algorithm is evaluated on several benchmark problems and compared with some related ensemble learning models. The experimental results demonstrate that the proposed algorithm outperforms the related works and can produce the neural network ensembles with better generalization.

Negative correlation learning in the extreme learning machine framework

Extreme learning machine (ELM) has shown to be a suitable algorithm for classification problems. Several ensemble meta-algorithms have been developed in order to generalize the results of ELM models. Ensemble approaches introduced in the ELM literature mainly come from boosting and bagging frameworks. The generalization of these methods relies on data sampling procedures, under the assumption that training data are heterogeneously enough to set up diverse base learners. The proposed ELM ensemble model overcomes this strong assumption by using the negative correlation learning (NCL) framework. An alternative diversity metric based on the orthogonality of the outputs is proposed. The error function formulation allows us to develop an analytical solution to the parameters of the ELM base learners, which significantly reduce the computational burden of the standard NCL ensemble method. The proposed ensemble method has been validated by an experimental study with a variety of benchmark datasets, comparing it with the existing ensemble methods in ELM. Finally, the proposed method statistically outperforms the comparison ensemble methods in accuracy, also reporting a competitive computational burden (specially if compared to the baseline NCL-inspired method).

Transfer Regression with Data-Augmented Ensemble Learning Framework

In biological measuring instruments design, it is time-consuming and expensive to acquire sufficient samples for calibration process. Transfer learning has provided an efficient approach for this problem by leveraging the labeled samples from calibration channels, considered as source domain, to annotate the target domain which has few labels under actual working conditions. In this paper, we design a general transfer regression framework improved by data-augmented ensemble learning called DA-TRTs. First, we modify two-stage TrAdaBoost.R2 to TrAdaBoost. RT (TRT) which concentrates more on hard examples during transfer boosting. Second, model stacking framework is introduced to improve the performance of regression, calling TRT as a base learner. Then, we introduce a simple data augmentation routine, based on label smoothing assumption. Data augmentation improves the strength in diversity of base learners and implicitly controls model complexity. Finally, to illustrate the performance of DA-TRTs, we conduct experiments on synthetic datasets.

Two-stage consumer credit risk modelling using heterogeneous ensemble learning

Modelling consumer credit risk is a crucial task for banks and non-bank financial institutions to support decision-making on granting loans. To model the overall credit risk of a consumer loan in terms of expected loss (EL), three key credit risk parameters must be estimated: probability of default (PD), loss given default (LGD) and exposure at default (EAD). Research to date has tended to model these parameters separately. Moreover, a neglected area in the field of LGD/EAD modelling is the application of ensemble learning, which by benefitting from diverse base learners reduces the over-fitting problem and enables modelling diverse risk profiles of defaulted loans. To overcome these problems, this paper proposes a two-stage credit risk model that integrates (1) class-imbalanced ensemble learning for predicting PD (credit scoring), and (2) an EAD prediction using a regression ensemble. Furthermore, multi-objective evolutionary feature selection is used to minimize both the misclassification cost (root mean squared error) of the PD and EAD models and the number of attributes necessary for modelling. For this task, we propose a misclassification cost metric suitable for consumer loans with fixed exposure because it combines opportunity cost and LGD. We show that the proposed credit risk model is not only more effective than single-stage credit risk models but also outperforms state-of-the-art methods used to model credit risk in terms of prediction and economic performance.

Improved Space Forest: A Meta Ensemble Method.

The performance of the ensemble algorithms is related with the individual accuracy of the base learners and their results diversity. Individual accuracy of a base learner is directly related to the similarity between the original training set and the base learner's training set. When a modified training set by randomly selecting features/classes/samples is given to the base learners, the diversity is created but the individual accuracy is decreased. From this point of view, different ensemble algorithms can be seen as a selection between having more accurate but less diverse base learners and having more diverse but less accurate base learners. We propose a meta ensemble method named as improved space forest which adds generated and (hopefully) more accurate features to the original features. The new features are obtained from randomly selected original features. When the new features are more distinctive than the original ones, they are selected by the learners. So, the ensemble may have more accurate base learners. However, a different improved space is generated for each learner to create diversity. The proposed method can be used with different ensemble methods. We compared original and improved space versions of bagging, random forest, and rotation forest algorithms. Improved space versions have generally better or comparable results than the original ones. We also present a theoretical framework to analyze the individual accuracies and diversities of the base learners.

Detecting outliers in complex nonlinear systems controlled by predictive control strategy

Detecting outliers in complicated nonlinear systems that are controlled by model predictive control is a significant work for engineering applications. Based on the features of data in practical systems, we propose a one-class classification ensemble method incorporating the notion of Feature Subspace with Bagging. Clustering and PCA (Principal Component Analysis) are integrated to obtain a more informative feature space, where Feature subspaces and bootstrap replications are implemented orderly to generate more accuracy and diverse base learners. A detector is constructed based on the above methodology, and a model updating strategy is also provided. By means of comparison with competitive methods, the effectiveness of the proposed detector has been verified.

Efficient Stereo Matching Leveraging Deep Local and Context Information

Stereo matching is a challenging problem with respect to weak texture, discontinuities, illumination difference and occlusions. Therefore, a deep learning framework is presented in this paper, which focuses on the first and last stage of typical stereo methods: the matching cost computation and the disparity refinement. For matching cost computation, two patch-based network architectures are exploited to allow the trade-off between speed and accuracy, both of which leverage multi-size and multi-layer pooling unit with no strides to learn cross-scale feature representations. For disparity refinement, unlike traditional handcrafted refinement algorithms, we incorporate the initial optimal and sub-optimal disparity maps before outlier detection. Furthermore, diverse base learners are encouraged to focus on specific replacement tasks, corresponding to the smooth regions and details. Experiments on different datasets demonstrate the effectiveness of our approach, which is able to obtain sub-pixel accuracy and restore occlusions to a great extent. Specifically, our accurate framework attains near-peak accuracy both in non-occluded and occluded region and our fast framework achieves competitive performance against the fast algorithms on Middlebury benchmark.

Spectral–Spatial Classification for Hyperspectral Data Using Rotation Forests With Local Feature Extraction and Markov Random Fields

In this paper, we propose a new spectral-spatial classification strategy to enhance the classification performances obtained on hyperspectral images by integrating rotation forests and Markov random fields (MRFs). First, rotation forests are performed to obtain the class probabilities based on spectral information. Rotation forests create diverse base learners using feature extraction and subset features. The feature set is randomly divided into several disjoint subsets; then, feature extraction is performed separately on each subset, and a new set of linear extracted features is obtained. The base learner is trained with this set. An ensemble of classifiers is constructed by repeating these steps several times. The weak classifier of hyperspectral data, classification and regression tree (CART), is selected as the base classifier because it is unstable, fast, and sensitive to rotations of the axes. In this case, small changes in the training data of CART lead to a large change in the results, generating high diversity within the ensemble. Four feature extraction methods, including principal component analysis (PCA), neighborhood preserving embedding (NPE), linear local tangent space alignment (LLTSA), and linearity preserving projection (LPP), are used in rotation forests. Second, spatial contextual information, which is modeled by MRF prior, is used to refine the classification results obtained from the rotation forests by solving a maximum a posteriori problem using the α-expansion graph cuts optimization method. Experimental results, conducted on three hyperspectral data with different resolutions and different contexts, reveal that rotation forest ensembles are competitive with other strong supervised classification methods, such as support vector machines. Rotation forests with local feature extraction methods, including NPE, LLTSA, and LPP, can lead to higher classification accuracies than that achieved by PCA. With the help of MRF, the proposed algorithms can improve the classification accuracies significantly, confirming the importance of spatial contextual information in hyperspectral spectral-spatial classification.

Corrective classification: Learning from data imperfections with aggressive and diverse classifier ensembling

Learning from imperfect (noisy) information sources is a challenging and reality issue for many data mining applications. Common practices include data quality enhancement by applying data preprocessing techniques or employing robust learning algorithms to avoid developing overly complicated structures that overfit the noise. The essential goal is to reduce noise impact and eventually enhance the learners built from noise-corrupted data. In this paper, we propose a novel corrective classification (C2) design, which incorporates data cleansing, error correction, Bootstrap sampling and classifier ensembling for effective learning from noisy data sources. C2 differs from existing classifier ensembling or robust learning algorithms in two aspects. On one hand, a set of diverse base learners of C2 constituting the ensemble are constructed via a Bootstrap sampling process; on the other hand, C2 further improves each base learner by unifying error detection, correction and data cleansing to reduce noise impact. Being corrective, the classifier ensemble is built from data preprocessed/corrected by the data cleansing and correcting modules. Experimental comparisons demonstrate that C2 is not only more accurate than the learner built from original noisy sources, but also more reliable than Bagging [4] or aggressive classifier ensemble (ACE) [56], which are two degenerated components/variants of C2. The comparisons also indicate that C2 is more stable than Boosting and DECORATE, which are two state-of-the-art ensembling methods. For real-world imperfect information sources ( i.e. noisy training and/or test data), C2 is able to deliver more accurate and reliable prediction models than its other peers can offer.