Bagging Predictors Research Articles

Remaining Useful Life Estimation based on Noise Injection and a Kalman Filter Ensemble of modified Bagging Predictors

Remaining Useful Life Estimation based on Noise Injection and a Kalman Filter Ensemble of modified Bagging Predictors

An Investigation of Sensitivity on Bagging Predictors: An Empirical Approach

As growing numbers of real world applications involve imbalanced class distribution or unequal costs for mis- classification errors in different classes, learning from imbalanced class distribution is considered to be one of the most challenging issues in data mining research. This study empirically investigates the sensitivity of bagging predictors with respect to 12 algorithms and 9 levels of class distribution on 14 imbalanced data-sets by using statistical and graphical methods to address the important issue of understanding the effect of vary- ing levels of class distribution on bagging predictors. The experimental results demonstrate that bagging NB and MLP are insensitive to various levels of imbalanced class distribution.

A new forecasting system for high-speed railway passenger demand based on residual component disposing

Accurate passenger demand forecasting of high-speed railway is of great significance for railway line planning and daily operation management. Capturing the random factors hidden in complex data is the key to achieve accurate forecasting. In view of this, this paper presents a new system with deterministic and probabilistic forecasting capacities based on the residual component disposing. First, the passenger demand is decomposed into trend, seasonality components and remainder using seasonal and trend decomposition using loess. Most likely some weak but indispensable information may still be contained in the remainder component, and random incidents occurred in the past may also occur in the future. Thus, moving block bootstrap is employed to bootstrap the remainder and generate one thousand similar samples by virtue of extrapolating the disaggregated subseries, thereby simulating the stochastic feature of future series. The forecasts of similar sample are acquired through reaggregating the results of extrapolations. Eventually, the bagging predictors is applied to attain the deterministic and probabilistic prediction. Two real-world case study manifests that the proposed hybrid system provides a more accurate assessment of passenger demand than all the benchmark models in various aspects, and the theoretical framework established in this paper is of certain enlightening significance for dealing with complex data.

Prediction of Mechanical Properties of Seamless Steel Tubes Using Artificial Neural Networks

A bottleneck of laboratory analysis in process industries including steelmaking plants is the low sampling rate. Inference models using only variables measured online have then been used to made such information available in advance. This study develops predictive models for key mechanical properties of seamless steel tubes, by strength, ultimate tensile strength and hardness. A plant in Brazil was used as the case study. The sample sizes of some steel tube families given namely, yield a particular property are discrepant and sometimes very small. To overcome this sample imbalance and lack of representativeness, committees of predictive neural network models based on bagging predictors, a type of ensemble method, were adopted. As a result, all steel families for all properties have been satisfactorily described showing the correlations between targets and model estimates close to 99%. These results were compared to multiple linear regression, support vector machine and a simpler neural network. Such information available in advance favors corrective actions before complete tube production mitigating rework costs in general.

Multi-level particle swarm optimisation and its parallel version for parameter optimisation of ensemble models: a case of sentiment polarity prediction

Ensemble learning is increasingly used in sentiment analysis. Determining the parameter settings of ensemble models, however, is not easy. Besides its own parameters, an ensemble model has base-predictors that have their individual parameters. Some ensemble models use a specific base-predictor and could be optimised using standard metaheuristics such as the Particle Swarm Optimisation (PSO) approach. Optimising ensemble models with multiple base-predictor candidates is more complicated and challenging, as there are multiple options to choose from. We therefore propose Multi-Level PSO (ML-PSO) and Parallel ML-PSO (PML-PSO) to optimise the parameters of ensemble models, especially those with multiple base-predictors, for sentiment analysis. The idea is to utilise multiple PSOs as particles of the main PSO. The main PSO optimises ensemble-model parameters and determines the best base-predictor, whereas PSOs within it optimise the corresponding base-predictor’s parameters. Experimental results using Bagging Predictors as the underlying ensemble model show that ML-PSO can improve prediction accuracy, while PML-PSO is able to speed up the processing time and further improve the accuracy.

Application of fractional Fourier transform for classification of power quality disturbances

Proper mitigation of power quality disturbances (PQDs) requires a fast, accurate and highly noise immune classification technique. This study, therefore, presents fractional Fourier transform (FRFT) based feature extraction as a new technique for classification of PQDs. FRFT is a generalised version of Fourier transform (FT) with an additional order control and can give time, frequency and intermediate time-frequency representations for a signal. The order control offers multi-domain feature extraction, such that most robust feature matrix is utilised for classification under any condition. An expression is derived for the optimal classification order corresponding to maximum overall accuracy. Based on IEEE-1159 standards, 15 PQDs are simulated and a database of pure and noisy signals is prepared. Features extracted from FRFT processed signals are tested with decision trees (DTs) and bagging predictors (BPs). The proposed technique shows better performance in most of the cases, when compared with Stockwell transform based classification under similar conditions. The classification accuracies of FRFT-DT and FRFT-BP are impressive even with significant reduction in training and features. Further, a validation using real PQDs obtained from an experimental setup is shown. The corresponding results closely resemble the simulation outcomes.

Financial technical indicator based on chaotic bagging predictors for adaptive stock selection in Japanese and American markets

In order to examine the predictability and profitability of financial markets, we introduce three ideas to improve the traditional technical analysis to detect investment timings more quickly. Firstly, a nonlinear prediction model is considered as an effective way to enhance this detection power by learning complex behavioral patterns hidden in financial markets. Secondly, the bagging algorithm can be applied to quantify the confidence in predictions and compose new technical indicators. Thirdly, we also introduce how to select more profitable stocks to improve investment performance by the two-step selection: the first step selects more predictable stocks during the learning period, and then the second step adaptively and dynamically selects the most confident stock showing the most significant technical signal in each investment. Finally, some investment simulations based on real financial data show that these ideas are successful in overcoming complex financial markets.

Dynamical Portfolio Theory by Nonlinear Bagging Predictors

Dynamical Portfolio Theory by Nonlinear Bagging Predictors

Mean-Variance Portfolio Model Modified by Nonlinear Bagging Predictors

In Markowitz' s mean-variance portfolio model, the probability distribution of a future return is composed of recent historical prices, and the future return and future risk are estimated as the mean and standard deviation of the distribution, respectively. Namely, the future return is predicted by a simple moving average, and the risk is simply the historical fluctuation. In this study, to improve the prediction accuracy of the future return, we apply a nonlinear prediction method following local spatial dynamics, and to estimate the future risk, we produce a probability distribution by aggregating predicted values by the bagging algorithm. Then, each risk is reduced by making a portfolio, that is, we apply the portfolio effect. Namely, our method attempts to simultaneously improve the prediction accuracy and reduce the risk of its prediction error. To confirm the validity of our method, we performed investment simulations. As a result, we could realize higher profit and lower risk in investment than by the conventional method.

Risk reduction for nonlinear prediction and its application to the surrogate data test

We propose a method for estimating nonlinear prediction risk using a bagging algorithm that involves ensemble learning. First we estimate the probability distribution of a future state as the ensemble set obtained using bagging predictors, and consider its standard deviation as the prediction risk. We can then improve the prediction reliability by avoiding dangerous predictions if the estimated prediction risk is high. As an application of this risk reduction method, we improve the power of surrogate data tests for system identification. Low prediction accuracy and poor system identification are caused by short and noisy data, so we perform simulations using short data derived from noisy chaotic models and real systems to confirm the validity of our method.

The effect of varying levels of class distribution on bagging for different algorithms: An empirical study

Many real world applications involve highly imbalanced class distribution. Research into learning from imbalanced class distribution is considered to be one of ten challenging problems in data mining research, and it has increasingly captured the attention of both academia and industry. In this work, we study the effects of different levels of imbalanced class distribution on bagging predictors by using under-sampling techniques. Despite the popularity of bagging in many real-world applications, some questions have not been clearly answered in the existing research, such as the effect of varying the levels of class distribution on different bagging predictors, e.g., whether bagging is superior to single learners when the levels of class distribution change. Most classification learning algorithms are designed to maximize the overall accuracy rate and assume that training instances are uniformly distributed; however, the overall accuracy does not represent correct prediction on the minority class, which is the class of interest to users. The overall accuracy metric is therefore ineffective for evaluating the performance of classifiers in extremely imbalanced data. This study investigates the effect of varying levels of class distribution on different bagging predictors based on the Area Under the Receiver Operating Characteristic (ROC) Curve (AUC) as a performance metric, using an under-sampling technique on 14 data-sets with imbalanced class distributions. Our experimental results indicate that Decision Table (DTable) and RepTree are the learning algorithms with the best bagging AUC performance. The AUC performances of bagging predictors are statistically superior to single learners, with the exception of Support Vector Machines (SVM) and Decision Stump (DStump).

An Empirical Study of Bagging Predictors for Different Learning Algorithms

Bagging is a simple yet effective design which combines multiple single learners to form an ensemble for prediction. Despite its popular usage in many real-world applications, existing research is mainly concerned with studying unstable learners as the key to ensure the performance gain of a bagging predictor, with many key factors remaining unclear. For example, it is not clear when a bagging predictor can outperform a single learner and what is the expected performance gain when different learning algorithms were used to form a bagging predictor. In this paper, we carry out comprehensive empirical studies to evaluate bagging predictors by using 12 different learning algorithms and 48 benchmark data-sets. Our analysis uses robustness and stability decompositions to characterize different learning algorithms, through which we rank all learning algorithms and comparatively study their bagging predictors to draw conclusions. Our studies assert that both stability and robustness are key requirements to ensure the high performance for building a bagging predictor. In addition, our studies demonstrated that bagging is statistically superior to most single base learners, except for KNN and Naïve Bayes (NB). Multi-layer perception (MLP), Naïve Bayes Trees (NBTree), and PART are the learning algorithms with the best bagging performance.

Making an accurate classifier ensemble by voting on classifications from imputed learning sets

Ensemble methods often produce effective classifiers by learning a set of base classifiers from a diverse collection of the training sets. In this paper, we present a system, voting on classifications from imputed learning sets (VCI), that produces those diverse training sets by randomly removing a small percentage of attribute values from the original training set, and then using an imputation technique to replace those values. VCI then runs a learning algorithm on each of these imputed training sets to produce a set of base classifiers. Later, the final prediction on a novel instance is the plurality classification produced by these classifiers. We investigate various imputation techniques here, including the state-of-the-art Bayesian multiple imputation (BMI) and expectation maximisation (EM). Our empirical results show that VCI predictors, especially those using BMI and EM as imputers, significantly improve the classification accuracy over conventional classifiers, especially on datasets that are originally incomplete; moreover VCI significantly outperforms bagging predictors and imputation-helped machine learners.

Bagging for linear classifiers

Classifiers built on small training sets are usually biased or unstable. Different techniques exist to construct more stable classifiers. It is not clear which ones are good, and whether they really stabilize the classifier or just improve the performance. In this paper bagging (bootstrapping and aggregating) [L. Breiman, Bagging predictors, Machine Learning J. 24(2), 123–140 (1996)] is studied for a number of linear classifiers. A measure for the instability of classifiers is introduced. The influence of regularization and bagging on this instability and the generalization error of linear classifiers is investigated. In a simulation study it is shown that in general bagging is not a stabilizing technique. It is also demonstrated that one can consider the instability of the classifier to predict how useful bagging will be. Finally, it is shown experimentally that bagging might improve the performance of the classifier only for very unstable situations.

Improved option pricing using artificial neural networks and bootstrap methods.

A hybrid neural network is used to predict the difference between the conventional option-pricing model and observed intraday option prices for stock index option futures. Confidence intervals derived with bootstrap methods are used in a trading strategy that only allows trades outside the estimated range of spurious model fits to be executed. Whilst hybrid neural network option pricing models can improve predictions they have bias. The hybrid option-pricing bias can be reduced with bootstrap methods. A modified bootstrap predictor is indexed by a parameter that allows the predictor to range from a pure bootstrap predictor, to a hybrid predictor, and finally the bagging predictor. The modified bootstrap predictor outperforms the hybrid and bagging predictors. Greatly improved performance was observed on the boundary of the training set and where only sparse training data exists. Finally, bootstrap bias estimates were studied.

Bagging predictors

Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy.

Bagging predictors

Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor. The aggregation averages over the versions when predicting a numerical outcome and does a plurality vote when predicting a class. The multiple versions are formed by making bootstrap replicates of the learning set and using these as new learning sets. Tests on real and simulated data sets using classification and regression trees and subset selection in linear regression show that bagging can give substantial gains in accuracy. The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy.

Weighted random subspace method for high dimensional data classification.

High dimensional data, especially those emerging from genomics and proteomics studies, pose significant challenges to traditional classification algorithms because the performance of these algorithms may substantially deteriorate due to high dimensionality and existence of many noisy features in these data. To address these problems, pre-classification feature selection and aggregating algorithms have been proposed. However, most feature selection procedures either fail to consider potential interactions among the features or tend to over fit the data. The aggregating algorithms, e.g. the bagging predictor, the boosting algorithm, the random subspace method, and the Random Forests algorithm, are promising in handling high dimensional data. However, there is a lack of attention to optimal weight assignments to individual classifiers and this has prevented these algorithms from achieving better classification accuracy. In this article, we formulate the weight assignment problem and propose a heuristic optimization solution.We have applied the proposed weight assignment procedures to the random subspace method to develop a weighted random subspace method. Several public gene expression and mass spectrometry data sets at the Kent Ridge biomedical data repository have been analyzed by this novel method. We have found that significant improvement over the common equal weight assignment scheme may be achieved by our method.