Stacked Generalization Ensemble Research Articles

Rent Price Prediction with Advanced Machine Learning Methods: A Comparison of California and Texas

The forecast of rent prices in dynamic housing markets is of fundamental importance to renters, landlords, investors, and politicians alike. Machine learning models offer flexibility, excel at modeling complex relationships, and provide outstanding forecast precision. This study compares advanced machine learning models, extreme gradient boosting regressor (XGBoost), light gradient boosting machine (LightGBM), random forest, ridge regression, and two ensemble approaches, to predict California and Texas rent prices. The two ensemble approaches include a hybrid regression of averaging base models and a 2-level stacked generalization model. The results revealed that a stacked generalization ensemble with base models random forest, XGBoost, LightGBM, and meta-model ridge regression achieved the best performance for the California dataset by generating the lowest MSE and highest R2 of 46116.3 and 0.8858, respectively. In contrast, random forest outperformed both ensemble models with the lowest MSE and MAPE of 18401.93 and 9.7003%, respectively, and the highest R2 of 0.7992. These methodologies can assess future rental property worth, serve as indicators for market dynamics, and aid in establishing real estate policies, thereby providing practical guidance to individuals, businesses, and policymakers.

IoT Data Collection and Short-Term Solar Power Forecasting Using Stacked Generalization Ensemble Model

IoT Data Collection and Short-Term Solar Power Forecasting Using Stacked Generalization Ensemble Model

Discussion of “Prediction of Maximum Scour Depth near Spur Dikes in Uniform Bed Sediment Using Stacked Generalization Ensemble Tree-Based Frameworks” by Manish Pandey, Mehdi Jamei, Masoud Karbasi, Iman Ahmadianfar, and Xuefeng Chu

Discussion of “Prediction of Maximum Scour Depth near Spur Dikes in Uniform Bed Sediment Using Stacked Generalization Ensemble Tree-Based Frameworks” by Manish Pandey, Mehdi Jamei, Masoud Karbasi, Iman Ahmadianfar, and Xuefeng Chu

Discussion of “Prediction of Maximum Scour Depth near Spur Dikes in Uniform Bed Sediment Using Stacked Generalization Ensemble Tree-Based Frameworks” by Manish Pandey, Mehdi Jamei, Masoud Karbasi, Iman Ahmadianfar, and Xuefeng Chu

Discussion of “Prediction of Maximum Scour Depth near Spur Dikes in Uniform Bed Sediment Using Stacked Generalization Ensemble Tree-Based Frameworks” by Manish Pandey, Mehdi Jamei, Masoud Karbasi, Iman Ahmadianfar, and Xuefeng Chu

A Deep Convolutional Neural Network Stacked Ensemble for Malware Threat Classification in Internet of Things

Malicious attacks to software applications are on the rise as more people use Internet of things (IoT) devices and high-speed internet. When a software system crash happens caused by malicious action, a malware imaging method can examine the application. In this study, we present a novel malware classification method that captures suspected operations in a variety of discrete size image features, allowing us to identify such IoT device malware families. To decrease deep neural network training time, essential local and global image features are selected using a combined local and global feature descriptor (LBP-GLCM). The classification performance of the proposed deep learning model is improved by combining the predictions of weak learners (CNNs) and using them as knowledge input to a multi-layer perceptron meta learner. This is a neural network ensemble with stacked generalization that is used to improve network generalization ability. The public dataset used for performance evaluation contains 5472 samples from 11 different malware families. In order to compare the proposed methodology to current malware detection systems, we developed a baseline experiment. The proposed approach improved malware classification results to 98.5% accuracy and 98.4% accuracy when using [Formula: see text] and [Formula: see text] image sizes, respectively. Overall, the results showed that the stacked generalization ensemble with multi-step extracting features is a more effective method for classification performance and response time.

A novel machine learning approach to shorten depression risk assessment for convenient uses

BackgroundDepression is a mental disorder affecting many people worldwide which has been exacerbated by the current pandemic. There is an urgent need for a reliable yet short scale for individuals to self-assess the risk of depression conveniently and rapidly on a regular basis. MethodsWe obtained a dataset of responses to the Depression, Anxiety, and Stress questionnaire (DASS-42) from a large sample of individuals worldwide (N = 31,715). With this dataset, important items from the questionnaire were extracted by applying feature selection techniques. Then, using the most important features, various machine learning algorithms were trained, tested, and validated in predicting depression status. ResultsThis study revealed that only seven items are needed to predict depression status with at least 90 % accuracy of the original full scale. This can be achieved through the Stacked Generalization Ensemble learning method of multiple models. The trained machine learning models from the best algorithm were then implemented as an online Depression Rapid Assessment tool, which allows the user to evaluate their current depression status conveniently and quickly (about 1 min). LimitationsThe sample size of the present study is still relatively small and has biases toward certain demographics (e.g., mostly young, Asian, and female). Further, memory issues with Stacked Generalization Ensemble prevent it from being trained in the same way as the other algorithm. ConclusionIt is possible to produce very short assessments that approximate the accuracy of the full scale for convenient and rapid self-assessment of depression risks.

Predicting quorum sensing peptides using stacked generalization ensemble with gradient boosting based feature selection.

Bacteria exist in natural environments for most of their life as complex, heterogeneous, and multicellular aggregates. Under these circumstances, critical cell functions are controlled by several signaling molecules known as quorum sensing (QS) molecules. In Gram-positive bacteria, peptides are deployed as QS molecules. The development of antibodies against such QS molecules has been identified as a promising therapeutic intervention for bacterial control. Hence, the identification of QS peptides has received considerable attention. Availability of a fast and reliable predictive model to effectively identify QS peptides can help the existing high throughput experiments. In this study, a stacked generalization ensemble model with Gradient Boosting Machine (GBM)-based feature selection, namely EnsembleQS was developed to predict QS peptides with high accuracy. On selected GBM features (791D), the EnsembleQS outperformed finely tuned baseline classifiers and demonstrated robust performance, indicating the superiority of the model. The accuracy of EnsembleQS is 4% higher than those resulting from ensemble model on hybrid dataset. When evaluating an independent data set of 40 QS peptides, the EnsembleQS model showed an accuracy of 93.4% with Matthew's Correlation Coefficient (MCC) and area under the ROC curve (AUC) values of 0.91 and 0.951, respectively. These results suggest that EnsembleQS will be a useful computational framework for predicting QS peptides and will efficiently support proteomics research. The source code and all datasets used in this study are publicly available at https://github.com/proteinexplorers/EnsembleQS .

Stacking machine learning models versus a locally weighted linear model to generate high-resolution monthly precipitation over a topographically complex area

This study applied a stacked generalization ensemble approach to generate high-resolution precipitation estimates and compared its performance with an optimized local weighted linear regression (LWLR) algorithm, a well-known local precipitation-elevation interpolation method incorporating physiographical factors. The stacked generalization ensemble consists of multilayer perceptron neural network (MLP), support vector machine (SVM), and random forest (RF) combined through a meta-learning algorithm with/without rescanning input covariates. Sixteen input covariates, including 2 topographic features, 5 cloud properties, 5 environmental variables, 3 precipitation products (PPs), and inverse distance weighted (IDW) estimates as the precipitation background field were fed into the machine learning models. Hold out approach was adopted for performance evaluation in which 50% of the 174 gauges were used for training, and the rest was used for validation. The results indicated that the overall monthly MAE, RMSE, and rBias of the proposed stacking model for the validation dataset were 3.3%, 6.8%, and 50%, respectively, less than that of the RF model, which is the best individual model. Also, the stacking model outperformed LWLR by decreasing monthly MAE, RMSE, and rBias 10.7%, 19.1%, and 28.6%, respectively. Further analysis implied that (1) the stacked model is more robust than LWLR and less dependent on the density of gauges, thus suitable for areas with scarce gauge coverage; (2) comparing the spatial distribution of mean monthly precipitation maps, generated by stacking and LWLR models, stacking algorithm can successfully screen out the bulls' eyes of background IDW precipitation field, and both patterns are consistent with the topography of the area; and (3) the stacking scheme is found to have a better extrapolation ability than LWLR over high elevations.

Web attacks detection using stacked generalization ensemble for LSTMs and word embedding

Web-based applications are prone to many web security attacks because they are openly-accessible and convenient. Most techniques used to prevent web attacks have some limitations; they cannot detect zero-day attacks and cannot analyze complex attacks, and should be maintained and updated regularly by security experts. Recently, there have been more research work on using deep learning for detecting web intrusions. Moreover, since most high risk web attacks are injected into HTTP web requests, detecting most web attacks needs classifying HTTP web requests into normal and anomalous. In this paper, we propose an approach based on Word2vec embedding and a stacked generalization ensemble model for LSTMs to detect malicious HTTP web requests. We evaluate our classification model performance using the HTTP CSIC 2010 dataset. We show that the combination of different word-level embeddings in a stacked generalization ensemble model for LSTMs has good performance both in terms of classification metrics and training time.

An Intelligent Forecasting Model for Disease Prediction Using Stack Ensembling Approach

This research work proposes a new stack-based generalization ensemble model to forecast the number of incidences of conjunctivitis disease. In addition to forecasting the occurrences of conjunctivitis incidences, the proposed model also improves performance by using the ensemble model. Weekly rate of acute Conjunctivitis per 1000 for Hong Kong is collected for the duration of the first week of January 2010 to the last week of December 2019. Pre-processing techniques such as imputation of missing values and logarithmic transformation are applied to pre-process the data sets. A stacked generalization ensemble model based on Auto-ARIMA (Autoregressive Integrated Moving Average), NNAR (Neural Network Autoregression), ETS (Exponential Smoothing), HW (Holt Winter) is proposed and applied on the dataset. Predictive analysis is conducted on the collected dataset of conjunctivitis disease, and further compared for different performance measures. The result shows that the RMSE (Root Mean Square Error), MAE (Mean Absolute Error), MAPE (Mean Absolute Percentage Error), ACF1 (Auto Correlation Function) of the proposed ensemble is decreased significantly. Considering the RMSE, for instance, error values are reduced by 39.23%, 9.13%, 20.42%, and 17.13% in comparison to Auto-ARIMA, NAR, ETS, and HW model respectively. This research concludes that the accuracy of the forecasting of diseases can be significantly increased by applying the proposed stack generalization ensemble model as it minimizes the prediction error and hence provides better prediction trends as compared to Auto-ARIMA, NAR, ETS, and HW model applied discretely

Analysis of the Ultrasonic Signal in Polymeric Contaminated Insulators Through Ensemble Learning Methods

Outdoor insulators may experience stress due to severe environmental conditions, such as pollution and contamination. Through the identification of partial discharges by ultrasonic noise, it is possible to assess the possibility of a power grid failure occurring. In this paper, ensemble models are used to analyze an ultrasonic signal from an ultrasonic microphone Pettersson M500. As the insulators are susceptible to developing irreversible failures, it will be evaluated whether the ultrasonic signal will remain over time, so that it is possible to assess whether the discharges being captured can result in a failure in contaminated polymeric insulators, evaluated in a high voltage laboratory under controlled conditions. The ensemble models were used in this paper because they typically require less computational effort than techniques based on deep learning and have acceptable performance for the problem at hand. The bagging, boosting, random subspace, bagging plus random subspace, and stacked generalization ensemble models are evaluated, and the best result of each model is used to compare the differences between the models. The bagging ensemble learning model proved to be faster and have lower error than other ensemble models, long short-term memory (LSTM), and nonlinear autoregressive (NAR).

An intelligent decision support system for crop yield prediction using hybrid machine learning algorithms

Background: In recent times, digitization is gaining importance in different domains of knowledge such as agriculture, medicine, recommendation platforms, the Internet of Things (IoT), and weather forecasting. In agriculture, crop yield estimation is essential for improving productivity and decision-making processes such as financial market forecasting, and addressing food security issues. The main objective of the article is to predict and improve the accuracy of crop yield forecasting using hybrid machine learning (ML) algorithms. Methods: This article proposes hybrid ML algorithms that use specialized ensembling methods such as stacked generalization, gradient boosting, random forest, and least absolute shrinkage and selection operator (LASSO) regression. Stacked generalization is a new model which learns how to best combine the predictions from two or more models trained on the dataset. To demonstrate the applications of the proposed algorithm, aerial-intel datasets from the github data science repository are used. Results: Based on the experimental results done on the agricultural data, the following observations have been made. The performance of the individual algorithm and hybrid ML algorithms are compared using cross-validation to identify the most promising performers for the agricultural dataset. The accuracy of random forest regressor, gradient boosted tree regression, and stacked generalization ensemble methods are 87.71%, 86.98%, and 88.89% respectively. Conclusions: The proposed stacked generalization ML algorithm statistically outperforms with an accuracy of 88.89% and hence demonstrates that the proposed approach is an effective algorithm for predicting crop yield. The system also gives fast and accurate responses to the farmers.

An intelligent decision support system for crop yield prediction using hybrid machine learning algorithms.

Background: In recent times, digitization is gaining importance in different domains of knowledge such as agriculture, medicine, recommendation platforms, the Internet of Things (IoT), and weather forecasting. In agriculture, crop yield estimation is essential for improving productivity and decision-making processes such as financial market forecasting, and addressing food security issues. The main objective of the article is to predict and improve the accuracy of crop yield forecasting using hybrid machine learning (ML) algorithms. Methods: This article proposes hybrid ML algorithms that use specialized ensembling methods such as stacked generalization, gradient boosting, random forest, and least absolute shrinkage and selection operator (LASSO) regression. Stacked generalization is a new model which learns how to best combine the predictions from two or more models trained on the dataset. To demonstrate the applications of the proposed algorithm, aerial-intel datasets from the github data science repository are used. Results: Based on the experimental results done on the agricultural data, the following observations have been made. The performance of the individual algorithm and hybrid ML algorithms are compared using cross-validation to identify the most promising performers for the agricultural dataset. The accuracy of random forest regressor, gradient boosted tree regression, and stacked generalization ensemble methods are 87.71%, 86.98%, and 88.89% respectively. Conclusions: The proposed stacked generalization ML algorithm statistically outperforms with an accuracy of 88.89% and hence demonstrates that the proposed approach is an effective algorithm for predicting crop yield. The system also gives fast and accurate responses to the farmers.

Prediction of Maximum Scour Depth near Spur Dikes in Uniform Bed Sediment Using Stacked Generalization Ensemble Tree-Based Frameworks

The scouring process near spur dikes could jeopardize the stability of riverbanks. Thus, accurate estimation of the maximum scour depth near spur dikes is crucial in river engineering. However, due to the complexity of the scour phenomenon around these structures, it has been a challenge to accurately estimate the maximum scour depth. Few efforts have been made to develop a machine learning (ML) approach for such a purpose. In this study, two novel multilayer stacked generalization frameworks are developed to model the scour depth near a spur dike in a uniform sediment condition. Stacked tree-based frameworks consist of three standalone ML approaches, including multivariate adaptive regression spline, multigene genetic programming, and kernel extreme learning machine as the first layer and the boosting regression tree (BRT) and bagging regression tree (BGT) as meta learners. A total of 186 data points were collected from previous experimental studies, and 32 flume experiments were further conducted under clear water conditions at the Indian Institute of Technology (IIT) Roorkee Laboratory in this study. The performances of the models were assessed using various statistical metrics [e.g., correlation coefficient (R), root-mean-square error (RMSE), and mean absolute percentage error (MAPE)], graphical criteria, and some existing empirical equations. The modeling results demonstrated that the stacked BRT and BGT frameworks were superior to all the standalone ML approaches (R=0.9786, RMSE=0.1654, and MAPE=11.68 for BRT; and R=0.9742, RMSE=0.1831, and MAPE =11.29 for BGT). This finding was also confirmed by a comparison between the empirical correlations and the artificial intelligence models. In addition, the sensitivity analysis proved that the mean sediment size ratio (L/d50) was the most influential variable in estimating the scour depth near spur dikes.

Convolutional neural network-based ensemble methods to recognize Bangla handwritten character.

In this era of advancements in deep learning, an autonomous system that recognizes handwritten characters and texts can be eventually integrated with the software to provide better user experience. Like other languages, Bangla handwritten text extraction also has various applications such as post-office automation, signboard recognition, and many more. A large-scale and efficient isolated Bangla handwritten character classifier can be the first building block to create such a system. This study aims to classify the handwritten Bangla characters. The proposed methods of this study are divided into three phases. In the first phase, seven convolutional neural networks i.e., CNN-based architectures are created. After that, the best performing CNN model is identified, and it is used as a feature extractor. Classifiers are then obtained by using shallow machine learning algorithms. In the last phase, five ensemble methods have been used to achieve better performance in the classification task. To systematically assess the outcomes of this study, a comparative analysis of the performances has also been carried out. Among all the methods, the stacked generalization ensemble method has achieved better performance than the other implemented methods. It has obtained accuracy, precision, and recall of 98.68%, 98.69%, and 98.68%, respectively on the Ekush dataset. Moreover, the use of CNN architectures and ensemble methods in large-scale Bangla handwritten character recognition has also been justified by obtaining consistent results on the BanglaLekha-Isolated dataset. Such efficient systems can move the handwritten recognition to the next level so that the handwriting can easily be automated.

Open-Source Neural Architecture Search with Ensemble and Pre-trained Networks

The training and optimization of neural networks, using pre-trained, super learner and ensemble approaches is explored. Neural networks, and in particular Convolutional Neural Networks (CNNs), are often optimized using default parameters. Neural Architecture Search (NAS) enables multiple architectures to be evaluated prior to selection of the optimal architecture. Our contribution is to develop, and make available to the community, a system that integrates open source tools for the neural architecture search (OpenNAS) of image classification models. OpenNAS takes any dataset of grayscale, or RGB images, and generates the optimal CNN architecture. Particle Swarm Optimization (PSO), Ant Colony Optimization (ACO) and pre-trained models serve as base learners for ensembles. Meta learner algorithms are subsequently applied to these base learners and the ensemble performance on image classification problems is evaluated. Our results show that a stacked generalization ensemble of heterogeneous models is the most effective approach to image classification within OpenNAS.

A Stacked Generalization of 3D Orthogonal Deep Learning Convolutional Neural Networks for Improved Detection of White Matter Hyperintensities in 3D FLAIR Images.

Accurate and reliable detection of white matter hyperintensities and their volume quantification can provide valuable clinical information to assess neurologic disease progression. In this work, a stacked generalization ensemble of orthogonal 3D convolutional neural networks, StackGen-Net, is explored for improving automated detection of white matter hyperintensities in 3D T2-FLAIR images. Individual convolutional neural networks in StackGen-Net were trained on 2.5D patches from orthogonal reformatting of 3D-FLAIR (n = 21) to yield white matter hyperintensity posteriors. A meta convolutional neural network was trained to learn the functional mapping from orthogonal white matter hyperintensity posteriors to the final white matter hyperintensity prediction. The impact of training data and architecture choices on white matter hyperintensity segmentation performance was systematically evaluated on a test cohort (n = 9). The segmentation performance of StackGen-Net was compared with state-of-the-art convolutional neural network techniques on an independent test cohort from the Alzheimer's Disease Neuroimaging Initiative-3 (n = 20). StackGen-Net outperformed individual convolutional neural networks in the ensemble and their combination using averaging or majority voting. In a comparison with state-of-the-art white matter hyperintensity segmentation techniques, StackGen-Net achieved a significantly higher Dice score (0.76 [SD, 0.08], F1-lesion (0.74 [SD, 0.13]), and area under precision-recall curve (0.84 [SD, 0.09]), and the lowest absolute volume difference (13.3% [SD, 9.1%]). StackGen-Net performance in Dice scores (median = 0.74) did not significantly differ (P = .22) from interobserver (median = 0.73) variability between 2 experienced neuroradiologists. We found no significant difference (P = .15) in white matter hyperintensity lesion volumes from StackGen-Net predictions and ground truth annotations. A stacked generalization of convolutional neural networks, utilizing multiplanar lesion information using 2.5D spatial context, greatly improved the segmentation performance of StackGen-Net compared with traditional ensemble techniques and some state-of-the-art deep learning models for 3D-FLAIR.

Improved sequence-based prediction of interaction sites in α-helical transmembrane proteins by deep learning.

Interactions between transmembrane (TM) proteins are fundamental for a wide spectrum of cellular functions, but precise molecular details of these interactions remain largely unknown due to the scarcity of experimentally determined three-dimensional complex structures. Computational techniques are therefore required for a large-scale annotation of interaction sites in TM proteins. Here, we present a novel deep-learning approach, DeepTMInter, for sequence-based prediction of interaction sites in α-helical TM proteins based on their topological, physiochemical, and evolutionary properties. Using a combination of ultra-deep residual neural networks with a stacked generalization ensemble technique DeepTMInter significantly outperforms existing methods, achieving the AUC/AUCPR values of 0.689/0.598. Across the main functional families of human transmembrane proteins, the percentage of amino acid sites predicted to be involved in interactions typically ranges between 10% and 25%, and up to 30% in ion channels. DeepTMInter is available as a standalone package at https://github.com/2003100127/deeptminter. The training and benchmarking datasets are available at https://data.mendeley.com/datasets/2t8kgwzp35.

LieToMe: An Ensemble Approach for Deception Detection from Facial Cues.

Deception detection is a relevant ability in high stakes situations such as police interrogatories or court trials, where the outcome is highly influenced by the interviewed person behavior. With the use of specific devices, e.g. polygraph or magnetic resonance, the subject is aware of being monitored and can change his behavior, thus compromising the interrogation result. For this reason, video analysis-based methods for automatic deception detection are receiving ever increasing interest. In this paper, a deception detection approach based on RGB videos, leveraging both facial features and stacked generalization ensemble, is proposed. First, a face, which is well-known to present several meaningful cues for deception detection, is identified, aligned, and masked to build video signatures. These signatures are constructed starting from five different descriptors, which allow the system to capture both static and dynamic facial characteristics. Then, video signatures are given as input to four base-level algorithms, which are subsequently fused applying the stacked generalization technique, resulting in a more robust meta-level classifier used to predict deception. By exploiting relevant cues via specific features, the proposed system achieves improved performances on a public dataset of famous court trials, with respect to other state-of-the-art methods based on facial features, highlighting the effectiveness of the proposed method.

Milling Tool Wear State Recognition by Vibration Signal Using a Stacked Generalization Ensemble Model

Milling tool wear state recognition plays an important role in controlling the quality of milled parts and reducing machine tool downtime. However, the characteristics of milling process limit the accuracy and stability of tool condition monitoring (TCM) employing vibration signals. To improve this problem, this paper explores the use of vibration signals as sensing approach for recognizing tool wear states during milling operation by using the stacked generalization (SG) ensemble model. In this study, vibration signals collected during the milling process are analyzed through the time domain, frequency domain, and time-frequency domain to extract signal features. The support vector machine recursive feature elimination (SVM-RFE) algorithm is used to select the main features which are most relevant to tool wear states. The SG ensemble model based on SVM, decision tree (DT), naive Bayes (NB), and SG ensemble strategy is constructed to recognize tool wear states. The proposed method is experimental verified, and the results show that the recognition accuracy of the established SG ensemble model is 98.74% and the overall G-mean and AUC evaluation value of the model is 0.98 and 0.98, respectively. In addition, compared with other ensemble models and single models, the SG ensemble model based on vibration signals has better recognition accuracy and stability than other models.