Stacked Generalization Method Research Articles

An ensemble framework-based approach for modeling stability of expansive soil slopes: fusion of machine learning algorithms and protection structure disease data.

Slope failures lead to catastrophic consequences in numerous countries, so accurate slope stability evaluation is critical in geological disaster prevention and control. In this study, the type and characteristics of slope protection structure disease were determined through the field investigation of an expansive soil area, and this information is incorporated into the numerical simulations and works to develop prediction models of slope stability. Four base machine learning (ML) methods are used to capture the relationship between protection structure diseases and factor of safety (FOS). Further, with the help of stacked generalization (SG), four ML models are combined, and the final SG model is used to predict the FOS. The results show that ML methods can effectively utilize this information and achieve excellent prediction results. The proposed SG model exhibits superior accuracy and robustness in predicting FOS compared to other ML methods. With FOS as the regression variable, the main feature contributions are slope height (37.05%) > slip distance of retaining wall (25.43%) > expansive force (18.03%) > slope gradient (12.00%); the coupling relationship among features is also captured by the proposed model. It is concluded that the SG method is particularly suitable for slope stability modeling under small sample conditions. Besides, the SG-based model effectively captures the impact of protection structure diseases on slope stability, enhances the interpretability of the ML model, and provides a reference for the maintenance and repair of the protection structure.

Hybrid method for accurate starch estimation in adulterated turmeric using Vis-NIR spectroscopy

Turmeric is widely used as a health supplement and foodstuff in South East Asian countries because of its medicinal benefits. Like several other plants and peppers, turmeric is prone to exploitation because of its economic value, rising consumer need, and essential food element that adds colour and flavour. Due to this, quick and comprehensive testing processes are needed to detect adulterants in turmeric. In this study, pure turmeric powders were mixed with starch in proportions ranging from 0 to 50% with a 1% variation to obtain different combinations. Reflectance spectra of pure turmeric and starch mixed samples were recorded using a JASCO-V770 spectrometer from 400 to 2050 nm. The recorded spectra were pre-processed using a Multiplicative Scatter Correction (MSC) and Standard Normal Variate (SNV). The Savitzky-Golay (SG) filter was initially applied to these original (X), MSC, and SNV-corrected spectra. Secondly, the Extra Tree Regressor (ETR) feature selection method was employed to select the best features. Finally, principal component analysis (PCA) was used to reduce the dimension of the selected features. The stacked generalization method was applied to improve the performance of this work. Both regressors and classifier stacking techniques have been tested with different classification and regression methods. The K-Nearest Neighbours (KNN), Decision Tree (DT), and Random Forest (RF) models were used as base learners, and Logistic Regression (LRC) was used as a meta-model for classification and Linear Regression (LR) for regression analysis. The proposed method achieved the best regression performance with r 2 of 0.999, Root Mean Square Error (RMSE) of 0.206, Ratio of Performance to Deviation (RPD) of 73.73, and Range Error Ratio (RER) of 480.58, whereas 100% F1 score and Matthew’s Correlation Coefficient (MCC) classification performance.

Mitigating anomalous electricity consumption in smart cities using an AI‐based stacked‐generalization technique

AbstractEnergy management and efficient asset utilization play an important role in the economic development of a country. The electricity produced at the power station faces two types of losses from the generation point to the end user. These losses are technical losses (TL) and non‐technical losses (NTL). TLs occurs due to the use of inefficient equipment. While NTLs occur due to the anomalous consumption of electricity by the customers, which happens in many ways; energy theft being one of them. Energy theft majorly happens to cut down on the electricity bills. These losses in the smart grid (SG) are the main issue in maintaining grid stability and cause revenue loss to the utility. The automatic metering infrastructure (AMI) system has reduced grid instability but it has opened up new ways for NTLs in the form of different cyber‐physical theft attacks (CPTA). Machine learning (ML) techniques can be used to detect and minimize CPTA. However, they have certain limitations and cannot capture the energy consumption patterns (ECPs) of all the users, which decreases the performance of ML techniques in detecting malicious users. In this paper, we propose a novel ML‐based stacked generalization method for the cyber‐physical theft issue in the smart grid. The original data obtained from the grid is preprocessed to improve model training and processing. This includes NaN‐imputation, normalization, outliers' capping, support vector machine‐synthetic minority oversampling technique (SVM‐SMOTE) balancing, and principal component analysis (PCA) based data reduction techniques. The pre‐processed dataset is provided to the ML models light gradient boosting (LGB), extra trees (ET), extreme gradient boosting (XGBoost), and random forest (RF), to accurately capture all consumers' overall ECP. The predictions from these base models are fed to a meta‐classifier multi‐layer perceptron (MLP). The MLP combines the learning capability of all the base models and gives an improved final prediction. The proposed structure is implemented and verified on the publicly available real‐time large dataset of the State Grid Corporation of China (SGCC). The proposed model outperformed the individual base classifiers and the existing research in terms of CPTA detection with false positive rate (FPR), false negative rate (FNR), F1‐score, and accuracy values of 0.72%, 2.05%, 97.6%, and 97.69%, respectively.

Predicting the potential distribution of wheatear birds using stacked generalization-based ensembles

Habitat suitability models, usually referred to as species distribution models (SDMs), are widely applied in ecology for many purposes, including species conservation, habitat discovery, and gain evolutionary insights by estimating the distribution of species. Machine learning algorithms as well as statistical models have been recently used to predict the distribution of species. However, they seemed to have some limitations due to the data and the models used. Therefore, this study proposes a novel approach for assessing habitat suitability based on ensemble learning techniques. Three heterogeneous ensembles were built using the stacked generalization method to model the distribution of four wheatear species (Oenanthe deserti, Oenanthe leucopyga, Oenanthe leucura, and Oenanthe oenanthe) located in Morocco. Initially, a set of base-learners were constructed by virtue of training for each specie's dataset six machine learning algorithms (Multi-Layer Perceptron (MLP), Support Vector Classifier (SVC), K-nearest neighbors (KNN), Decision Trees (DT), Gradient Boosting Classifier (GB), and Random Forest (RF)). Then, the predictions of these base learners were fed as training data to train three meta-learners (Logistic Regression (LR), SVC, and MLP). To evaluate and assess the performance of the proposed approaches, we used: (1) six performance criteria (accuracy, recall, precision, F1-score, AUC, and TSS), (2) Borda Count (BC) ranking method based on multiple criteria to rank the best-performing models, and (3) Scott Knott (SK) test to statistically compare the performance of the presented models. The results based on the six-evaluation metrics showed that stacked ensembles outperformed their singles in all species datasets, and the stacked model with SVC as a meta-learner outperformed the other two ensembles. The results showed the potential of using ensemble learning techniques to model species distribution and recommend the use of the stacked generalization technique as a combination strategy since it gave better results compared to single models in four wheatear species datasets. Moreover, to assess the impact of future climate changes on the distribution of the four wheatear species, the best-performing distribution model was selected and projected into the current and future climatic conditions. The distributions of the Moroccan wheatear birds were found to be slightly affected by future climate changes.

Black Carbon Emission Prediction of Diesel Engine Using Stacked Generalization

With the continuous growth of international maritime trade, black carbon (BC) emissions from ships have caused great harm to the natural environment and human health. Controlling the BC emissions from ships is of positive significance for Earth’s environmental governance. In order to accelerate the development process of ship BC emission control technologies, this paper proposes a BC emission prediction model based on stacked generalization (SG). The meta learner of the prediction model is Ridge Regression (RR), and the base learner combines four models: Extreme Gradient Boosting (XGB), Light Gradient Boosting Machine (LGB), Random Forest (RF), and Support Vector Regression (SVR). We used mutual information (MI) to measure the correlation between combustion characteristic parameters (CCPs) and BC emission concentration, and selected them as the features of the prediction model. The results show that the CCPs have a strong correlation with the BC emission concentration of the diesel engine under different working conditions, which can be used to describe the influence of the changes to the combustion process in the cylinder on the BC generation. The introduction of the stacked generalization method reconciles the inherent bias of various models. Compared with traditional models, the fusion model has achieved higher prediction accuracy on the same datasets. The research results of this paper can provide a reference for the research and development of ship black carbon emission control technologies and the formulation of relevant regulations.

Discrete Wavelet Transform for Generative Adversarial Network to Identify Drivers Using Gyroscope and Accelerometer Sensors

Driver identification is a central research area in intelligent transportation systems, with applications in commercial freight transport and usage-based insurance. One way to perform the identification is to use smartphones as the main sensor devices. After extracting features from smartphone-embedded sensors, various machine learning methods can be used to identify the driver. However, the accuracy often degrades as the number of drivers increases. This paper uses a Generative Adversarial Network (GAN) for data augmentation to obtain a driver identification algorithm that maintains excellent performance also when the number of drivers increases. Since GAN diversifies the drivers’ data, it makes it possible to apply the identification algorithm on a larger number of drivers without overfitting. Although GANs are commonly used in image processing for image augmentation, their use for driving signal augmentation is novel. However, GAN’s training on raw driving signals diverges. This challenge is solved by getting the Discrete Wavelet Transform (DWT) on driving signals before feeding to GAN. Our experiments prove the usefulness of GAN model for generating driving signals emanating from DWT on smartphones’ accelerometer and gyroscope signals. After collecting the augmented data, their histograms along the overlapped windows are fed to machine learning methods covered by a Stacked Generalization Method (SGM). The presented hybrid GAN-SGM approach identifies drivers with 97% accuracy, 98% precision, 97% recall, and 97% F1-measure that outperforms standard machine learning methods that utilize features extracted by the statistical, spectral, and temporal approaches.

A Novel Approach of Ensemble Methods Using the Stacked Generalization for High-dimensional Datasets

Stacked generalization-based heterogeneous ensemble methods combine the prediction of multiple classifiers to improve overall classification performance. Several stacking methods are available in the literature, but the criteria to select the number and type of classifiers are missing. This work analyzes the performance of stacked generalization-based ensemble machine learning methods for high-dimensional datasets. Also, the impact of the classifier selection for the first level ( L 0 ) of the stacked generalization method has been studied. Based on that, the criteria for selecting classifiers at the first level of the stacked generalization method are proposed. So, six stacked generalization approaches are presented and have been analyzed for the thirty high-dimensional datasets. The experiments and results indicate that the performance of stacking strategies based on proposed selection criteria performs better. Also, a comparative study about the choice of homogeneous ensemble classifiers in stacked generalization concerning the use of basic classifiers and fusion of basic with homogeneous ensemble methods has been made. It has been observed that the use of only homogeneous ensemble classifiers is not beneficial in the stacked generalization methods. Also, the performance of stacked generalization based on basic classifiers or a combination of homogeneous ensemble methods with basic classifiers is better than homogeneous ensemble methods. The proposed stacking approach based on a combination of the basic and ensemble classifiers has improved the accuracy 0.72% to 8.46%. The impact of removing redundant and non-relevant features on the proposed stacking approaches has been evaluated.

Mapping wetland using the object-based stacked generalization method based on multi-temporal optical and SAR data

Wetland ecosystems have experienced dramatic challenges in the past few decades due to natural and human factors. Wetland maps are essential for the conservation and management of terrestrial ecosystems. This study is to obtain an accurate wetland map using an object-based stacked generalization (Stacking) method on the basis of multi-temporal Sentinel-1 and Sentinel-2 data. Firstly, the Robust Adaptive Spatial Temporal Fusion Model (RASTFM) is used to get time series Sentinel-2 NDVI, from which the vegetation phenology variables are derived by the threshold method. Subsequently, both vertical transmit-vertical receive (VV) and vertical transmit-horizontal receive (VH) polarization backscatters (σ0 VV, σ0 VH) are obtained using the time series Sentinel-1 images. Speckle noise inherent in SAR data, resulting in over-segmentation or under-segmentation, can affect image segmentation and degrade the accuracies of wetland classification. Therefore, we segment Sentinel-2 multispectral images to delineate meaningful objects in this study. Then, in order to reduce data redundancy and computation time, we analyze the optimal feature combination using the Sentinel-2 multispectral images, Sentinel-2 NDVI time series, phenological variables and other vegetation index derived from Sentinel-2 multispectral images, as well as time series Sentinel-1 backscatters at the object level. Finally, the stacked generalization algorithm is utilized to extract the wetland information based on the optimal feature combination in the Dongting Lake wetland. The overall accuracy and Kappa coefficient of the object-based stacked generalization method are 92.46% and 0.92, which are 3.88% and 0.04 higher than that using the pixel-based method. Moreover, the object-based stacked generalization algorithm is superior to single classifiers in classifying vegetation of high heterogeneity areas.

Diagnosis and Prediction of Large-for-Gestational-Age Fetus Using the Stacked Generalization Method

An accurate and efficient Large-for-Gestational-Age (LGA) classification system is developed to classify a fetus as LGA or non-LGA, which has the potential to assist paediatricians and experts in establishing a state-of-the-art LGA prognosis process. The performance of the proposed scheme is validated by using LGA dataset collected from the National Pre-Pregnancy and Examination Program of China (2010–2013). A master feature vector is created to establish primarily data pre-processing, which includes a features’ discretization process and the entertainment of missing values and data imbalance issues. A principal feature vector is formed using GridSearch-based Recursive Feature Elimination with Cross-Validation (RFECV) + Information Gain (IG) feature selection scheme followed by stacking to select, rank, and extract significant features from the LGA dataset. Based on the proposed scheme, different features subset are identified and provided to four different machine learning (ML) classifiers. The proposed GridSearch-based RFECV+IG feature selection scheme with stacking using SVM (linear kernel) best suits the said classification process followed by SVM (RBF kernel) and LR classifiers. The Decision Tree (DT) classifier is not suggested because of its low performance. The highest prediction precision, recall, accuracy, Area Under the Curve (AUC), specificity, and F1 scores of 0.92, 0.87, 0.92, 0.95, 0.95, and 0.89 are achieved with SVM (linear kernel) classifier using top ten principal features subset, which is, in fact higher than the baselines methods. Moreover, almost every classification scheme best performed with ten principal feature subsets. Therefore, the proposed scheme has the potential to establish an efficient LGA prognosis process using gestational parameters, which can assist paediatricians and experts to improve the health of a newborn using computer aided-diagnostic system.

Predicting the impact of single nucleotide variants on splicing via sequence-based deep neural networks and genomic features.

Single nucleotide mutations in exonic regions can significantly affect gene function through a disruption of splicing, and various computational methods have been developed to predict the splicing-related effects of a single nucleotide mutation. We implemented a new method using ensemble learning that combines two types of predictive models: (a) base sequence-based deep neural networks (DNNs) and (b) machine learning models based on genomic attributes. This method was applied to the Massively Parallel Splicing Assay challenge of the Fifth Critical Assessment of Genome Interpretation, in which challenge participants predicted various experimentally-defined exonic splicing mutations, and achieved a promising result. We successfully revealed that combining different predictive models based upon the stacked generalization method led to significant improvement in prediction performance. In addition, whereas most of the genomic features adopted in constructing machine learning models were previously reported, feature values generated with DSSP, a DNN-based splice site prediction tool, were novel and helpful for the prediction. Learning the sequence patterns associated with normal splicing and the change in splicing site probabilities caused by a mutation was presumed to be helpful in predicting splicing disruption.

Classifying Imbalanced Data Sets by a Novel RE-Sample and Cost-Sensitive Stacked Generalization Method

Learning with imbalanced data sets is considered as one of the key topics in machine learning community. Stacking ensemble is an efficient algorithm for normal balance data sets. However, stacking ensemble was seldom applied in imbalance data. In this paper, we proposed a novel RE-sample and Cost-Sensitive Stacked Generalization (RECSG) method based on 2-layer learning models. The first step is Level 0 model generalization including data preprocessing and base model training. The second step is Level 1 model generalization involving cost-sensitive classifier and logistic regression algorithm. In the learning phase, preprocessing techniques can be embedded in imbalance data learning methods. In the cost-sensitive algorithm, cost matrix is combined with both data characters and algorithms. In the RECSG method, ensemble algorithm is combined with imbalance data techniques. According to the experiment results obtained with 17 public imbalanced data sets, as indicated by various evaluation metrics (AUC, GeoMean, and AGeoMean), the proposed method showed the better classification performances than other ensemble and single algorithms. The proposed method is especially more efficient when the performance of base classifier is low. All these demonstrated that the proposed method could be applied in the class imbalance problem.

Classifier Subset Selection for the Stacked Generalization Method Applied to Emotion Recognition in Speech.

In this paper, a new supervised classification paradigm, called classifier subset selection for stacked generalization (CSS stacking), is presented to deal with speech emotion recognition. The new approach consists of an improvement of a bi-level multi-classifier system known as stacking generalization by means of an integration of an estimation of distribution algorithm (EDA) in the first layer to select the optimal subset from the standard base classifiers. The good performance of the proposed new paradigm was demonstrated over different configurations and datasets. First, several CSS stacking classifiers were constructed on the RekEmozio dataset, using some specific standard base classifiers and a total of 123 spectral, quality and prosodic features computed using in-house feature extraction algorithms. These initial CSS stacking classifiers were compared to other multi-classifier systems and the employed standard classifiers built on the same set of speech features. Then, new CSS stacking classifiers were built on RekEmozio using a different set of both acoustic parameters (extended version of the Geneva Minimalistic Acoustic Parameter Set (eGeMAPS)) and standard classifiers and employing the best meta-classifier of the initial experiments. The performance of these two CSS stacking classifiers was evaluated and compared. Finally, the new paradigm was tested on the well-known Berlin Emotional Speech database. We compared the performance of single, standard stacking and CSS stacking systems using the same parametrization of the second phase. All of the classifications were performed at the categorical level, including the six primary emotions plus the neutral one.

Efficient performance of neural networks for nonlinearity error modeling of three-longitudinal-mode interferometer in nano-metrology system

Abstract Nano-metrology has a crucial role in order to produce nano-materials and devices with a high degree of accuracy and reliability. Laser heterodyne interferometers are non-contact, high-resolution measurement systems which are commonly used in the displacement measurement systems. In this paper, an approach based on neural networks (NNs) for nonlinearity modeling in a three-longitudinal-mode laser heterodyne interferometer (TLMI) for length and displacement measurements is presented considering the experimental deviation parameters. We model nonlinearity errors that arise from elliptically and non-orthogonally polarized laser beams, rotational error in the alignment of the laser head with respect to the polarizing beam splitter axis, rotational error in the alignment of the mixing polarizer, and unequal transmission coefficients of the polarizing beam splitter. Here, we use the neural network algorithms including radial basis function (RBF) and multi-layer perceptron (MLP) networks and stacked generalization method. The simulation results show that multi-layer feed forward perceptron network and stacked generalization method is successfully applicable to real noisy interferometer signals. The one-hidden layer network with 5 neurons gives a good quality of fit for the training and test sets for the measurement system with RBF and MLP networks and three MLP networks with one-hidden layer for stacked generalization method. The numbers of neurons and hidden layers are selected for the best mean square error (MSE) and minimum time consuming.

Improving ECG classification accuracy using an ensemble of neural network modules.

This paper illustrates the use of a combined neural network model based on Stacked Generalization method for classification of electrocardiogram (ECG) beats. In conventional Stacked Generalization method, the combiner learns to map the base classifiers' outputs to the target data. We claim adding the input pattern to the base classifiers' outputs helps the combiner to obtain knowledge about the input space and as the result, performs better on the same task. Experimental results support our claim that the additional knowledge according to the input space, improves the performance of the proposed method which is called Modified Stacked Generalization. In particular, for classification of 14966 ECG beats that were not previously seen during training phase, the Modified Stacked Generalization method reduced the error rate for 12.41% in comparison with the best of ten popular classifier fusion methods including Max, Min, Average, Product, Majority Voting, Borda Count, Decision Templates, Weighted Averaging based on Particle Swarm Optimization and Stacked Generalization.

Refractive index determination in fuel cells using high-resolution laser heterodyne interferometer

In this paper, we present an interferometry method for refractive index determination in membranes of fuel cells. This technique is based on the use of an improved laser heterodyne interferometer. The photocurrents of the avalanche photodiodes, resulting from reflected beams of the optical head, are led to the signal conditioner and digital signal processing sections. The optical path difference between the target and reference paths is fixed, and as a result, the phase shift is calculated in terms of the refractive index shift. In addition, nonlinearity of this system is analyzed and modeled with different neural networks and adaptive filter algorithms. For neural networks, the radial basis function (RBF), the multi-layer perceptron (MLP), and the stacked generalization method are simulated. In adaptive filter algorithms, the least mean square (LMS), the normalized least mean square (NLMS), the recursive least squares (RLS), and the affine projection algorithm (APA) are applied. The simulation results indicate that the RLS method is faster and contains minimum mean square error (MSE) compared to the other approaches. Also, comparison between two main approaches shows that the nonlinearity of refractive index determination can be effectively modeled with adaptive filter algorithms.

Modeling and Compensation of Periodic Nonlinearity in Two-mode Interferometer Using Neural Networks

The periodic nonlinearity in nano-metrology systems based on heterodyne interferometers is the most important limitation to the accuracy of displacement measurement. It is mainly produced due to the polarization-mixing and frequency-mixing. In this paper, a new approach based on an ensemble of neural networks for modeling and compensation of nonlinearity in a high-resolution laser heterodyne interferometer is presented. We model the periodic nonlinearity arising from elliptical polarization and non-orthogonality of the laser polarized light based on the neural network approaches, including the multi-layer perceptrons and radial basis function as single neural networks and stacked generalization method as ensemble of neural networks. It is also shown that by using the stacked generalization method, the primary periodic nonlinearity of 1.3 nm is significantly compensated by a factor of 168.

Data mining of time series using stacked generalizers

Data mining is the search for valuable information in large volumes of data. Finding patterns in time series databases is important to a variety of applications, including stock market trading and budget forecasting. This paper reports on an extension of neural network methods for planning and budgeting in the State of Utah. In particular, historical time series are analyzed using stacked generalization, a methodology devised to aid in developing models that generalize well to future time periods. Stacked generalization is compared to ARIMA and to stand-alone neural networks. The results are consistent and suggest promise for the stacked generalization method in other time series domains.