Ensemble Prediction Model Research Articles

Ensemble deep learning models for protein secondary structure prediction using bidirectional temporal convolution and bidirectional long short-term memory.

Protein secondary structure prediction (PSSP) is a challenging task in computational biology. However, existing models with deep architectures are not sufficient and comprehensive for deep long-range feature extraction of long sequences. This paper proposes a novel deep learning model to improve Protein secondary structure prediction. In the model, our proposed bidirectional temporal convolutional network (BTCN) can extract the bidirectional deep local dependencies in protein sequences segmented by the sliding window technique, the bidirectional long short-term memory (BLSTM) network can extract the global interactions between residues, and our proposed multi-scale bidirectional temporal convolutional network (MSBTCN) can further capture the bidirectional multi-scale long-range features of residues while preserving the hidden layer information more comprehensively. In particular, we also propose that fusing the features of 3-state and 8-state Protein secondary structure prediction can further improve the prediction accuracy. Moreover, we also propose and compare multiple novel deep models by combining bidirectional long short-term memory with temporal convolutional network (TCN), reverse temporal convolutional network (RTCN), multi-scale temporal convolutional network (multi-scale bidirectional temporal convolutional network), bidirectional temporal convolutional network and multi-scale bidirectional temporal convolutional network, respectively. Furthermore, we demonstrate that the reverse prediction of secondary structure outperforms the forward prediction, suggesting that amino acids at later positions have a greater impact on secondary structure recognition. Experimental results on benchmark datasets including CASP10, CASP11, CASP12, CASP13, CASP14, and CB513 show that our methods achieve better prediction performance compared to five state-of-the-art methods.

Evolutionary and ensemble machine learning predictive models for evaluation of water quality

Study regionBisham Qilla and Doyian stations, Indus River Basin of Pakistan Study focusWater pollution is an international concern that impedes human health, ecological sustainability, and agricultural output. This study focuses on the distinguishing characteristics of an evolutionary and ensemble machine learning (ML) based modeling to provide an in-depth insight of escalating water quality problems. The 360 temporal readings of electric conductivity (EC) and total dissolved solids (TDS) with several input variables are used to establish multi-expression programing (MEP) model and random forest (RF) regression model for the assessment of water quality at Indus River. New hydrological insight for the regionThe developed models were evaluated using several statistical metrics. The findings reveal that the determination coefficient (R2) in the testing phase (subject to unseen data) for the all the developed models is more than 0.95, indicating the accurateness of the developed models. Furthermore, the error measurements are much lesser with root mean square logarithmic error (RMSLE) nearly equals to zero for each developed model. The mean absolute percent error (MAPE) of MEP models and RF models falls below 10% and 5%, respectively, in all three phases (training, validation and testing). According to the sensitivity study of generated MEP models about the relevance of inputs on the predicted EC and TDS, shows that bi-carbonates and chlorine content have significant influence with a sensitiveness score more than 0.90, whereas the impact of sodium content is less pronounced. All the models (RF and MEP) have lower uncertainty based on the prediction interval coverage probability (PICP) calculated using the quartile regression (QR) approach. The PICP% of each model is greater than 85% in all three stages. Thus, the findings of the study indicate that developing intelligent models for water quality parameter is cost effective and feasible for monitoring and analyzing the Indus River water quality.

Oxidative Dehydrogenation of Ethane with CO2 over the Fe-Co/Al2O3 Catalyst: Experimental Data Assisted AI Models for Prediction of Ethylene Yield

In this work, Fe-Co-based mixed metal oxides supported on Al2O3 are proposed for ethylene production through oxidative dehydrogenation of ethane with CO2 (ODH-CO2). Thermodynamic feasibility analysis followed by a systematic experimental study is performed on catalyst synthesis and its composition optimization along with process condition optimization in a fixed bed reactor. The study revealed that 5% Fe loaded on 10% Co/Al2O3, 700 °C, and 1:1 are the optimal composition, temperature, and molar ratio of CO2 to ethane, respectively, achieving 29% ethane conversion and resulting in 16% ethylene yield. Further, the experimental data was used to develop different linear, nonlinear, and ensemble AI models for ethylene yield prediction through a systematic grid search and k-fold cross-validation procedure. Among all the models, the kernel ridge regression model is found to be the most accurate, exhibiting the highest R2 value of 0.966 and lowest root mean-squared error (RMSE) of 0.032 on test data, successfully capturing the underlying nonlinear dynamics of ODH-CO2.

An Ensemble Learning Approach to Improving Prediction of Case Duration for Spine Surgery: Algorithm Development and Validation.

Estimating surgical case duration accurately is an important operating room efficiency metric. Current predictive techniques in spine surgery include less sophisticated approaches such as classical multivariable statistical models. Machine learning approaches have been used to predict outcomes such as length of stay and time returning to normal work, but have not been focused on case duration. The primary objective of this 4-year, single-academic-center, retrospective study was to use an ensemble learning approach that may improve the accuracy of scheduled case duration for spine surgery. The primary outcome measure was case duration. We compared machine learning models using surgical and patient features to our institutional method, which used historic averages and surgeon adjustments as needed. We implemented multivariable linear regression, random forest, bagging, and XGBoost (Extreme Gradient Boosting) and calculated the average R2, root-mean-square error (RMSE), explained variance, and mean absolute error (MAE) using k-fold cross-validation. We then used the SHAP (Shapley Additive Explanations) explainer model to determine feature importance. A total of 3189 patients who underwent spine surgery were included. The institution's current method of predicting case times has a very poor coefficient of determination with actual times (R2=0.213). On k-fold cross-validation, the linear regression model had an explained variance score of 0.345, an R2 of 0.34, an RMSE of 162.84 minutes, and an MAE of 127.22 minutes. Among all models, the XGBoost regressor performed the best with an explained variance score of 0.778, an R2 of 0.770, an RMSE of 92.95 minutes, and an MAE of 44.31 minutes. Based on SHAP analysis of the XGBoost regression, body mass index, spinal fusions, surgical procedure, and number of spine levels involved were the features with the most impact on the model. Using ensemble learning-based predictive models, specifically XGBoost regression, can improve the accuracy of the estimation of spine surgery times.

A time-varying ensemble model for ship motion prediction based on feature selection and clustering methods

Effective ship motion prediction helps to avoid ship navigation hazards in time. However, there is insufficient attention to potentially valuable information in the multi-factor ship-related data space and the time-varying characteristics of ship motion. In this paper, a time-varying ensemble model based on feature selection and clustering methods is proposed to improve the performance of real-time ship motion prediction. The ensemble model consists of three modules that progressively improve the validity of the model. In Module I, the non-dominated sorting genetic algorithm-II (NSGA-II) algorithm is adopted for feature selection of the original multi-factor data to filter out the feature factors that are useful for ship motion prediction. In Module II, the self-organizing map (SOM) algorithm is applied to cluster the multi-factor data after feature selection to reorganize samples with similar attributes into a limited number of clusters. Then, an ensemble learning model is constructed for each cluster using multiple Elman neural networks and Adaboost techniques. In Module III, a time-varying prediction framework is proposed for real-time prediction of ship motion by combining the ensemble model corresponding to each cluster. An empirical study was conducted using three multifactorial datasets collected from a ship in the South China Sea in December 2020. The results show that the proposed model can be a competitive technique for solving complex multi-factor ship motion predictions and has the potential to be applied to real-time ship motion warning systems.

LGBM-ACp: an ensemble model for anticancer peptide prediction and in silico screening with potential drug targets.

Conventional cancer therapies are highly expensive and have serious complications. An alternative approach now emphasizes on the development of small, biologically active peptides without acute toxicity. Experimental screening to find curative anticancer peptides (ACP) often gives rise to multiple obstacles and is time dependent. Consequently, developing an effective computational technique to identify promising ACP candidates prior to preclinical research is in high demand. This study proposed a machine-learning framework that used the light gradient-boosting machine as a classifier and two compositional and two binary profile features as input. The ensemble model displayed an accuracy, MCC, and AUROC of 97.52%, 0.91, and 0.98, respectively, which outclassed most of the existing sequence-based computational tools. A distinct dataset of non-mutagenic, non-toxic, and non-inhibitory Cytochrome P-450 peptides was used to validate the hybrid model. The most relevant ACP in the alternative dataset was compared with two standard ACPs, beta defensin 2, and cecropin-A. Molecular docking of the predicted peptide revealed that it has a strong binding affinity with twenty-five anticancer drug targets, most notably phosphoenolpyruvate carboxykinase (- 7.2kcal/mol). Additionally, molecular dynamics simulation and principal component analysis supported the stability of the peptide-receptor complex. Overall, the present findings will take a step forward in rational drug design through rapid identification and screening of therapeutic peptides.

Ensemble Tree Model for Long-Term Rockburst Prediction in Incomplete Datasets

The occurrence of rockburst can seriously impact the construction and production of deep underground engineering. To prevent rockburst, machine learning (ML) models have been widely employed to predict rockburst based on some related variables. However, due to the costs and complicated geological conditions, complete datasets to evaluate rockburst cannot always be obtained in rock engineering. To fill this limitation, this study proposed an ensemble tree model suitable for incomplete datasets, i.e., the histogram gradient boosting tree (HGBT), to build intelligent models for rockburst prediction. Three hundred fourteen rockburst cases were employed to develop the HGBT model. The hunger game search (HGS) algorithm was implemented to optimize the HGBT model. The established HGBT model had an excellent testing performance (accuracy of 88.9%). An incomplete database with missing values was applied to compare the performances of HGBT and other ML models (random forest, artificial neural network, and so on). HGBT received an accuracy of 78.8% in the incomplete database, and its capacity was better than that of other ML models. Additionally, the importance of input variables in the HGBT model was analyzed. Finally, the feasibility of the HGBT model was validated by rockburst cases from Sanshandao Gold Mine, China.

A hybrid super ensemble learning model for the early-stage prediction of diabetes risk

Diabetes mellitus has become a rapidly growing chronic health problem worldwide. There has been a noticeable increase in diabetes cases in the last two decades. Recent advances in ensemble machine learning methods play an important role in the early detection of diabetes mellitus. These methods are both faster and less costly than traditional methods. This study aims to propose a new super ensemble learning model to enable an early diagnosis of diabetes mellitus. Super learner is a cross-validation-based approach that makes better predictions by combining prediction results of more than one machine learning algorithm. The proposed super learner model was created with four base-learners (logistic regression, decision tree, random forest, gradient boosting) and a meta learner (support vector machines) as a result of a case study. Three different dataset were used to measure the robustness of the proposed model. Chi-square was determined as an optimal feature selection technique from five different techniques, and also hyper-parameter settings were made with GridSearch. Finally, the proposed new super learner model achieved to obtain the best accuracy results in the detection of Diabetes mellitus compared to the base-learners for the early-stage diabetes risk prediction (99.6%), PIMA (92%), and diabetes 130-US hospitals (98%) dataset, respectively. This study revealed that super learner algorithms can be effectively used in the detection of diabetes mellitus. Also, obtaining of the high and convincing statistical scores shows the robustness of the proposed super learner model.

Machine-Learning Model for the Prediction of Hypoxaemia during Endoscopic Retrograde Cholangiopancreatography under Monitored Anaesthesia Care.

Hypoxaemia is a significant adverse event during endoscopic retrograde cholangiopancreatography (ERCP) under monitored anaesthesia care (MAC); however, no model has been developed to predict hypoxaemia. We aimed to develop and compare logistic regression (LR) and machine learning (ML) models to predict hypoxaemia during ERCP under MAC. We collected patient data from our institutional ERCP database. The study population was randomly divided into training and test sets (7:3). Models were fit to training data and evaluated on unseen test data. The training set was further split into k-fold (k=5) for tuning hyperparameters, such as feature selection and early stopping. Models were trained over k loops; the i-th fold was set aside as a validation set in the i-th loop. Model performance was measured using area under the curve (AUC). We identified 6114 cases of ERCP under MAC, with a total hypoxaemia rate of 5.9%. The LR model was established by combining eight variables and had a test AUC of 0.693. The ML and LR models were evaluated on 30 independent data splits. The average test AUC for LR was 0.7230, which improved to 0.7336 by adding eight more variables with an l1 regularisation-based selection technique and ensembling the LRs and gradient boosting algorithm (GBM). The high-risk group was discriminated using the GBM ensemble model, with a sensitivity and specificity of 63.6% and 72.2%, respectively. We established GBM ensemble model and LR model for risk prediction, which demonstrated good potential for preventing hypoxaemia during ERCP under MAC.

An Ensemble Model for Water Temperature Prediction in Intensive Aquaculture

An Ensemble Model for Water Temperature Prediction in Intensive Aquaculture

Data-driven ensemble model for probabilistic prediction of debris-flow volume using Bayesian model averaging

Data-driven ensemble model for probabilistic prediction of debris-flow volume using Bayesian model averaging

Multi-trait ensemble genomic prediction and simulations of recurrent selection highlight importance of complex trait genetic architecture for long-term genetic gains in wheat

AbstractCereal crop breeders have achieved considerable genetic gain in genetically complex traits, such as grain yield, while maintaining genetic diversity. However, focus on selection for yield has negatively impacted other important traits. To better understand multi-trait selection within a breeding context, and how it might be optimized, we analysed genotypic and phenotypic data from a genetically diverse, 16-founder wheat multi-parent advanced generation inter-cross population. Compared to single-trait models, multi-trait ensemble genomic prediction models increased prediction accuracy for almost 90 % of traits, improving grain yield prediction accuracy by 3–52 %. For complex traits, non-parametric models (Random Forest) also outperformed simplified, additive models (LASSO), increasing grain yield prediction accuracy by 10–36 %. Simulations of recurrent genomic selection then showed that sustained greater forward prediction accuracy optimized long-term genetic gains. Simulations of selection on grain yield found indirect responses in related traits, involving optimized antagonistic trait relationships. We found multi-trait selection indices could effectively optimize undesirable relationships, such as the trade-off between grain yield and protein content, or combine traits of interest, such as yield and weed competitive ability. Simulations of phenotypic selection found that including Random Forest rather than LASSO genetic models, and multi-trait rather than single-trait models as the true genetic model accelerated and extended long-term genetic gain whilst maintaining genetic diversity. These results (i) suggest important roles of pleiotropy and epistasis in the wider context of wheat breeding programmes, and (ii) provide insights into mechanisms for continued genetic gain in a limited genepool and optimization of multiple traits for crop improvement.

Multilevel ensemble model for prediction of gender and age of blackbuck using pugmarks

Multilevel ensemble model for prediction of gender and age of blackbuck using pugmarks

Stacking ensemble learning models for daily runoff prediction using 1D and 2D CNNs

In recent years, applications of convolutional neural networks (CNNs) to runoff prediction have received some attention due to their excellent feature extraction capabilities. However, existing studies are still limited since merely either 1D or 2D CNNs are developed to predict runoff. In this study, a stacking ensemble learning model for daily runoff prediction based on different types of 1D and 2D CNNs is proposed and applied to the Quinebaug River Basin, Connecticut, USA. The structure of the CNN models is developed with reference to the classic LeNet5 network. Especially, the predictors are reconstructed into 1D vectors and 2D matrices with 10-, 20- and 30-day time steps. Totally 18 member models are constructed through selecting 3 representative 1D and 2D CNN models with 3 time steps. The simple average method (SAM) is used to integrate different CNN member models. The results show that the performance of the same-type SAM based on either 1D or 2D CNNs improves because it can counteract the effects of both positive and negative predicted values by the member models to some extent. Furthermore, the mixed-type SAM models based on both 1D and 2D CNN member models can further improve the prediction accuracy. The optimal model SAM15 consists of two 1D CNNs and four 2D CNNs. Compared with the optimal CNN member models, SAM15 reduces the validation RMSE by about 13% and improves the validation R and NSE by about 3% and 7%, respectively. This study highlights that the proposed stacking ensemble learning model can improve the daily runoff prediction accuracy through integration of the nonlinear fitting ability of 1D and 2D CNN member models.

An optimized ensemble prediction model using AutoML based on soft voting classifier for network intrusion detection

Traditional ML based IDS cannot handle high-speed and ever-evolving attacks. Furthermore, these traditional IDS face several common challenges, such as processing large amounts of network profiling data, low accuracy and detection rates, and data imbalance. Therefore, there is an immense demand for a security system for multi-purpose network applications to mitigate intrusion in network profiling data to protect users’ privacy and property across the Internet. This study presents an OE-IDS model using AutoML based on a soft voting method for detecting intrusion in the network environment. The proposed study uses an AutoML framework to select optimal supervised classifiers for developing an optimal ensemble strategy using a soft voting method to maximize accuracy and minimize false alarm rates. Furthermore, our proposed study uses different sampling methods to extend data to solve the problem of data imbalance and minority classes. In addition, our proposed OE-IDS is validated using two benchmark datasets, such as UNSWNB15 and CICIDS2017, to achieve a higher detection rate. Both original and re-sampled data samples are considered to validate the proposed OE-IDS to ensure the generalizability and stability of the proposed OE-IDS model. The performance assessment shows that the proposed OE-IDS model detects intrusions with high accuracy in the network intrusion environment. The detection rate of the OE-IDS model based on reduced features set for UNSWNB15 and CICIDS2017 is 0.98% and 0.973%, respectively. Moreover, our proposed model is compared with conventional ML-based IDS; it is found that our proposed OE-IDS has produced higher accuracy and better detection rate.

An ensemble model considering health index based classification for remaining useful life prediction

AbstractAccurate prediction of remaining useful life (RUL) plays an important role in the formulation of maintenance strategies. However, due to the diversity of the failure mode of equipment, there are significant differences between the degradation data, which greatly affects the accuracy of RUL prediction. In this case, an ensemble prediction model considering health index‐based (HI‐based) classification is proposed in this paper. Firstly, the stacked autoencoder (SAE) is employed to construct the HI. Then, the time window is used to sequentially process the HI sequence, so that many data segments with the same length can be achieved. To differentiate the data with the similar degradation process, K‐means and Xgboost are selected to construct offline and online data classification models respectively. Finally, according to the results of the data classification, the ensemble model that integrates multiple machine learning methods are separately trained and then adaptively used for RUL prediction. In addition, integrating multiple methods helps to improve the generalization ability of the model. The NASA C‐MAPSS dataset is applied to verify the effectiveness of the proposed method, and the results show that the proposed method achieves a higher prediction accuracy and shorter computational time than other existing prediction models.

A stacked ensemble model for automatic stroke prediction using only raw electrocardiogram

Stroke is one of the main causes of death and disability in elderly people worldwide. Early diagnosis of this disease is desirable. An electrocardiogram (ECG) signal is a powerful tool in the diagnosis of stroke. Many recent studies are using ECG signals in the diagnosis of stroke. This study is done with the aim to develop an efficient classification model which can be used in the diagnosis of stroke disease using ECG features. This study uses ECG data collected from 71 different subjects of which 35 were stroke patients and 36 were normal patients. This study proposes a stacked ensemble model which is built by stacking three different convolutional neural networks (CNN) models. The raw ECG signals are used as input to the model for training and testing. The result shows that the proposed model is capable of predicting stroke with an accuracy of 99.7%. F1-score, precision and recall are 99.69%, 99.67% and 99.71% respectively. Hence, this study reports that with the proposed model, ECG can be used as an aid in the diagnosis of stroke disease with high efficiency.

Optimized ensemble machine learning model for software bugs prediction

Optimized ensemble machine learning model for software bugs prediction

Extending machine learning prediction capabilities by explainable AI in financial time series prediction

Prediction with higher accuracy is vital for stock market prediction. Recently, considerable amount of effort has been poured into employing machine learning (ML) techniques for successfully predicting stock market price direction. No matter how successful the proposed prediction model is, it can be argued that there occur two major drawbacks for further increasing the prediction accuracy. The first one can be referred as the black box nature of ML techniques, in other words inference from the predictions cannot be explained. Furthermore, due to the complex characteristics of the predicted time series, no matter how sophisticated techniques are employed, it would be very difficult to achieve a marginal increase in accuracy that would meaningfully offset the additional computational burden it brings in. For these two reasons, instead of chasing incremental improvements in accuracy, we propose utilizing an “eXplainable Artificial Intelligence” (XAI) approach which can be employed for assessing the reliability of the predictions hence allowing decision maker to abstain from poor decisions which are responsible for declining overall prediction performance. If there would be a measure of how sure the prediction model is on any prediction, the predictions with a relatively higher reliability could be used to make a decision while lower quality decisions could be avoided. In this study, a novel two-stage stacking ensemble model for stock market direction prediction based on ML, empirical mode decomposition (EMD) and XAI is proposed. Our experiments have shown that, proposed prediction model supported with local interpretable model-agnostic explanations (LIME) achieved the highest accuracy of 0.9913 when only the most trusted predictions have been considered on KOSPI dataset and analogous successful results have been obtained from five other major stock market indices.

Studies on Parameters Affecting Temperature of Liquid Steel and Prediction Using Modified AdaBoost.RT Algorithm Ensemble Extreme Learning Machine

The present work aimed to develop a predictive model for the end temperature of liquid steel in advance to support the smooth functioning of a vacuum tank degasser (VTD). An ensemble model that combines extreme learning machine (ELM) with a self-adaptive AdaBoost.RT algorithm was established for the regression problem. Based on analyzing the energy equilibrium of the VTD system, the factors were determined for predicting the end temperature of liquid steel. To establish a hybrid ensemble prediction model, an ELM algorithm was selected as the ensemble predictor due to its strong performance and robustness, and a modification of the AdaBoost.RT algorithm is proposed to overcome the drawback of the original AdaBoost.RT by embedding statistical theory to dynamically self-adjust the threshold value. For efficient VTD operations, an ensemble model that combines ELM with the self-adaptive AdaBoost.RT algorithm was established to model the end temperature of liquid steel. The proposed approach was analyzed and validated on actual production data derived from a steelmaking workshop in Baosteel. The experimental results reveal that the proposed model can improve the generalization performance, and the accuracy of the model is feasible for the secondary steel refining process. In addition, a polynomial equation is obtained from the ensemble predictive model for calculating the value of the end temperature. The predicted results are in good agreement with the actual data with <1.7% error.