Ensemble Regression Method Research Articles

Meta-Learning Guided Weight Optimization for Enhanced Solar Radiation Forecasting and Sustainable Energy Management with VotingRegressor

Solar radiation prediction plays a crucial role in renewable energy management, impacting various decision-making processes aimed at optimizing the utilization of solar resources and promoting sustainability. Ensemble regression methods, notably VotingRegressor, have emerged as promising tools for accurate solar radiation forecasting. By integrating predictions from multiple base estimators, ensemble methods have the potential to capture intricate patterns inherent in solar radiation data. However, achieving optimal predictive performance with ensemble methods heavily relies on the careful weighting assigned to each base estimator, presenting a significant challenge. In this study, a novel approach is presented to enhance solar radiation prediction by utilizing meta-learning techniques to optimize the weighting mechanism in the VotingRegressor ensemble. Meta-learning, a subfield of machine learning focusing on learning algorithms across different tasks, provides a systematic framework for learning to learn. This enables models to adapt and generalize more effectively to new datasets and tasks. Our proposed methodology demonstrated significant improvements, with the VotingRegressor with meta-learning techniques achieving an RMSE of 8.7343, an MAE of 5.42145, and an R² of 0.991913. These results mitigate the need for manual weight tuning and improve the adaptability of the VotingRegressor to varying solar radiation conditions, ultimately contributing to the sustainability of renewable energy systems. The methodology involves a comprehensive exploration of meta-learning techniques, encompassing gradient-based optimization, reinforcement learning, and Bayesian optimization.

Machine Learning-Assisted Determination of C6H14 Mole Fraction From Molecular Emissions of Laser-Induced Hexane-Air Plasmas.

Laser-induced plasmas of materials containing hydrocarbons present strong carbon molecular emission features. Using these emissions to build models relating changes in spectral features to a physical parameter of the system, such as hydrocarbon content, can be difficult because of the dynamic complexity of the spectral features and temperature disequilibrium between molecular species. This study presents machine learning models trained to quantify the mole fraction of hexane in hexane-air plasmas from CN Violet and C2 Swan spectral features. Ensemble regression methods provide the most accurate predictions with root mean squared error on the order 10-2. Artificial neural network regressions produce predictions with superlative sensitivity, exhibiting detection limits as low as 0.008. These foundational models can be further refined with more advanced data to quantify the presence of carbon species in complex plasma environments, such as high-speed reacting flows.

Enhanced Restaurant Rating Predictions: A Study of Linear and Ensemble Regression Methods

In this study, we investigate the application of various regression techniques to predict restaurant ratings based on a comprehensive Zomato dataset. The dataset encompasses 51,717 entries with features such as restaurant type, location, cuisine, and cost estimates. We employed four regression models: Linear Regression, CatBoost Regressor, Random Forest Regressor, and Decision Tree Regressor, to evaluate their performance in predicting ratings. The models were assessed using the R-squared (R²) metric to determine their effectiveness. The Random Forest Regressor achieved the highest R² score of 0.875, indicating its superior predictive capability. The CatBoost Regressor also performed well, demonstrating the effectiveness of gradient boosting methods. The study provides insights into the performance of various regression models and highlights the advantages of ensemble methods for accurate restaurant rating predictions.

Software effort estimation using stacked ensembled techniques and proposed stacking ensemble using principal component regression as super learner

As the size and complexity of projects grows, estimates are increasingly used, especially in the agile community. Software development cannot begin without first conducting thorough planning and estimation. Estimating how much work a project will take is a common first step in the software development life cycle. By employing ensemble techniques, we integrate multiple learning algorithms to build a more accurate predictive model. The core elements of our proposed stacked ensemble strategy include Decision Tree, Principal Components Regression, Random Forest, NeuralNet, GLMNET, XGBoost, Earth, and Support Vector Machine. Moreover, we augment the model’s performance by incorporating a blend of these foundational algorithms with other ensemble regression methods. Extensive testing in the suggested research work with a number of Super Learners demonstrates that Regression is the best technique for judging effort. The evaluation of the different estimators involved the use of various metrics, including Mean Absolute Error, Root Mean Squared Error, Mean Squared Error, Percentage of Close Approximations within 25% of the True Values (PRED (25)), R-Squared Coefficients, Precision, Recall, and F1-Score. The proposed method yields more trustworthy predicted performance than either single-model approaches or stacked ensembles. Effort estimation serves as the foundation for the rest of the project management process.

New Two-Level Machine Learning Method for Evaluating the Real Characteristics of Objects

A new two-level ensemble regression method, as well as its modifications and application in applied problems, are considered. The key feature of the method is its focus on constructing an ensemble of predictors that approximate the target variable well and, at the same time, consist of algorithms that, if possible, differ from each other in terms of the calculated predictions. The ensemble with the indicated properties at the first stage is constructed through the optimization of a special functional, whose choice is theoretically substantiated in this study. At the second stage, a collective solution is calculated based on the forecasts formed by this ensemble. In addition, some heuristic modifications are described that have a positive effect on the quality of the forecast in applied problems. The effectiveness of the method is confirmed by the results obtained for specific applied problems.

Ensemble regression based on polynomial regression-based decision tree and its application in the in-situ data of tunnel boring machine

Regression is an important branch of engineering data mining tasks, aiming to establish a regression model to predict the output of interest based on the input variables. To meet the requirements of different missions, the engineering system usually changes its operation status so that the regression relationship between the output and input variables changes. In this paper, two ensemble regression methods are proposed based on polynomial regression and decision tree, in which sample space partition is used to improve the prediction accuracy and ensemble strategy is used to improve the performance robustness of the regression model. The first ensemble regression method (named PRB) is developed under the framework of bagging. The second ensemble regression method (named PRF) is similar to the first one, but feature randomness is introduced. At each node of the polynomial regression-based decision tree, the polynomial regression error is used to select the best splitting feature. The experiments on a series of mathematical functions and engineering datasets indicate that the proposed ensemble regression methods outperform the polynomial regression-based decision tree, the polynomial regression method, and the random forest method in most experiments. The proposed ensemble regression methods are applied to model the dataset of a tunnel boring machine, aiming to predict the earth pressure based on the operation parameters of the cutterhead. The results indicate that the proposed two ensemble regression methods produce more accurate prediction results, and the PRF method performs best in most experiments.

Optimization of Stock Price Time Series Prediction Model based on Karhunen-Loève Expansion and Information Gain Weighted Integrated Regression

Time series prediction model plays an important role in stock price prediction, such as ARIMA, LSTM neural network. However, due to the need for stationary assumption of time series itself and the problems of high dimension and high noise, the common time series prediction methods have limitations. Based on this, this paper propose a framework for the optimization of the stock price time series prediction model. The proposed method uses the intra-day price as the auxiliary variable and obtains the function feature information based on Karhunen-Loève expansion. Considering that the feature variables after dimension reduction still have problems such as information loss and irrelevant noise. This paper use data enhancement method to improve the effective information of feature variables and reduce the influence of irrelevant noise. Then, since the potential model structure between the characteristic variable and the residual sequence is unknown, this paper develop a weighted ensemble regression method based on information gain to balance the variance and deviation of the prediction model, thereby improving the prediction accuracy. The actual data analysis results show that the proposed method can greatly improve the fitting accuracy of ARIMA and LSTM neural networks for stock prices. Finally, the optimization framework can also be used for the prediction of average temperature, air quality and port cargo flow.

Survival Prediction of Glioma Patients from Integrated Radiology and Pathology Images Using Machine Learning Ensemble Regression Methods

Gliomas are tumors of the central nervous system, which usually start within the glial cells of the brain or the spinal cord. These are extremely migratory and diffusive tumors, which quickly expand to the surrounding regions in the brain. There are different grades of gliomas, hinting about their growth patterns and aggressiveness and potential response to the treatment. As part of routine clinical procedure for gliomas, both radiology images (rad), such as multiparametric MR images, and digital pathology images (path) from tissue samples are acquired. Each of these data streams are used separately for prediction of the survival outcome of gliomas, however, these images provide complimentary information, which can be used in an integrated way for better prediction. There is a need to develop an image-based method that can utilise the information extracted from these imaging sequences in a synergistic way to predict patients’ outcome and to potentially assist in building comprehensive and patient-centric treatment plans. The objective of this study is to improve survival prediction outcomes of gliomas by integrating radiology and pathology imaging. Multiparametric magnetic resonance imaging (MRI), rad images, and path images of glioma patients were acquired from The Cancer Imaging Archive. Quantitative imaging features were extracted from tumor regions in rad and path images. The features were given as input to an ensemble regression machine learning pipeline, including support vector regression, AdaBoost, gradient boost, and random forest. The performance of the model was evaluated in several configurations, including leave-one-out, five-fold cross-validation, and split-train-test. Moreover, the quantitative performance evaluations were conducted separately in the complete cohort (n = 171), high-grade gliomas (HGGs), n = 75, and low-grade gliomas (LGGs), n = 96. The combined rad and path features outperformed individual feature types in all the configurations and datasets. In leave-one-out configuration, the model comprising both rad and path features was successfully validated on the complete dataset comprising HGFs and LGGs (R=0.84 p=2.2×10−16). The Kaplan–Meier curves generated on the predictions of the proposed model yielded a hazard ratio of 3.314 [95%CI:1.718−6.394], log−rank(P)=2×10−4 on combined rad and path features. Conclusion: The proposed approach emphasizes radiology experts and pathology experts’ clinical workflows by creating prognosticators upon ‘rad’ radiology images and digital pathology ‘path’ images independently, as well as combining the power of both, also through delivering integrated analysis, that can contribute to a collaborative attempt between different departments for administration of patients with gliomas.

A novel bagged tree ensemble regression method with multiple correlation coefficients to predict the train body vibrations using rail inspection data

Prediction of the train body vibrations induced by the train running is desirable and useful to ensure comfortable service, reliable, safe, and secure operation of railway systems. By using daily measurement data from GJ-5 rail detection vehicle, this paper presents a novel prediction algorithm, which is based on bagged tree ensemble regression with multiple correlation coefficients. To obtain the valuable data sets from a large amount of inspection data, an approach of multiple correlation coefficients is used for the data pre-processing. Then the prediction model of train body vibrations is established by combining regression tree algorithm and bagged ensemble algorithm. By training the valuable data sets, the prediction results are calculated by the bagged tree ensemble regression method. Finally, the proposed method is evaluated with experimental data and the traditional method. The experimental results show that the proposed method not only has higher accuracy but also can effectively reduce the number of the data sets, the accuracy is up to 98% and the number of valuable training data sets is reduced by 78.3%. The new method proposed in the paper can accurately predict the vibration status of the train body without installing any new sensors and other monitoring equipment on the train, which can reduce maintenance costs and prevent potential safety risks.

Optimized machine learning approaches for the prediction of viscoelastic behavior of modified asphalt binders

Additives are commonly used in pavement engineering to improve the original bitumen's rheological and mechanical characteristics to meet severe loading and climatic condition requirements. To select the optimum dosage of an additive for modifying the original bitumen, it is essential to predict the viscoelastic behavior of modified bitumens, which can be performed by implementing the constitutive viscoelastic parameters of the complex shear modulus (G*) and phase angle (δ). In this work, a comprehensive experimental database consisting of the results of the frequency sweep mode of a dynamic shear rheometer (DSR) at seven test temperatures (−22 ~ 22 °C) was used. Prediction models for the viscoelastic behavior of bitumen modified with different dosages of crumb rubber, styrene–butadienestyrene (SBS), and polyphosphoric acid (PPA) were developed by optimizing and applying different machine learning approaches, including Artificial Neural Networks (ANN), Robust Linear Regression, Linear Support Vector Regression, Decision Tree Regression, Gaussian Process Regression (GPR), and Ensemble Regression, on the data. By comparing the various studied model outputs in terms of performance measurements, such as the correlation of coefficients, relative root mean square error, scatter index, relative error, and Nash-Sutcliffe efficiency coefficient, it was determined that the Ensemble Regression method has the highest performance in predictions.

Least square based ensemble deep learning for inertia tensor identification of combined spacecraft

The high accurate identification of inertia tensor of combined spacecraft, which is composed of a servicing spacecraft and a target, is necessary to perform attitude control. Due to the uncertainty of the operating environments of combined spacecraft, the measurement noise of the angular rate may be very complex and will seriously influence the identification accuracy. This paper proposes a least square based weighted ensemble deep learning method to realize a highly accurate identification for the inertia tensor of combined spacecraft in complex operating environments. In this method, a single deep neural network regression model is firstly constructed as an individual model for the ensemble deep learning, and then is trained by enough training data and a designed training strategy. After obtaining a certain number of accurate and diverse single models, all the outputs of single models are combined by several linear functions. A least square based ensemble method for regression is developed to automatically determine the optimal weights of these linear functions. The identification performance of the proposed ensemble deep learning method is tested using two testing sets with different ranks of measurement noises and is compared with other identification methods. The results demonstrate that the proposed ensemble deep learning method has a significant advantage compared with other methods for identifying inertia tensor.

Data-Driven-Based Optimization for Power System Var-Voltage Sequential Control

This paper proposes a data-driven-based optimization method for var-voltage sequential control (V2SC). First, power system var-voltage control characteristics, defined as the function between the reactive power injections and the bus voltages, are approximated by the var-voltage sensitivities (V2S). Then, V2S is estimated online using the noise-assisted ensemble regression method. Subsequently, an optimal model is proposed for V2SC based on the V2S estimation. To avoid frequent back and forth control actions, a hysteresis control strategy is employed. The performance of the proposed method is statistically validated in the 8-generator 36-node system and the Nordic32 system with test data measured from real power systems.

A semi-empirical approach for the estimation of land-surface emissivity from satellite data based on spectral index fusion using ensemble regression

ABSTRACTLand Surface Emissivity (LSE) is a key parameter in the thermal remote sensing, with several important applications, most notably in Land Surface Temperature (LST) estimation. This paper presents a semi-empirical method of LSE estimation from remote sensing data based on a fusion of spectral indices using the ensemble regression methods. The performance of the proposed method for Moderate Resolution Imaging Spectroradiometer (MODIS), Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER), Operational Land Imager (OLI) and Thermal Infrared Sensor (TIRS) data was evaluated and compared with other semi-empirical methods developed for these sensors. The proposed method was designed in four stages. In the first stage, the reflectance of non-thermal bands and emissivity of thermal bands were simulated for different classes using the ASTER spectral library and the spectral response function of each sensor. In the second stage, the dataset to be used for the training of ensemble regression was arranged by calculating a number of spectral indices, which constitute the feature space along with non-thermal bands. In the third stage, the regression between emissivity of thermal bands of each sensor and the features extracted in the second stage was derived by the use of bagging, boosting and Random Forest (RF) regression methods. In the final stage Using Normalized Difference Vegetation Index (NDVI) values, the image was categorized into three classes including vegetation, non-vegetation and mixture areas using conditions NDVI > 0.5, NDVI < 0.2 and 0.2 ≤ NDVI ≤ 0.5, respectively. The non-vegetation class was then categorized to soil, rock, and man-made classes using land use map. The spectral indices of these classes were then calculated, and the corresponding model trained in the third stage was used to estimate the LSE for that band. The results of LSE estimations were compared with the standard product of each sensor. Due to the lack of standard product for Landsat-8, the ASTER product was used as a substitute. For better analysis, the proposed method was also evaluated with other semi-empirical methods developed for MODIS, ASTER and OLI/TIRS sensors. This evaluation showed that the lowest Root Mean Square Error (RMSE) values for OLI/TIRS bands 10 and 11 are 0.0070 and 0.0075 obtained, respectively, by bagging and RF regression methods. For ASTER bands 13 and 14, the lowest RMSE values of 0.0078 and 0.0077 are both obtained by RF regression. For MODIS bands 31 and 32, the lowest RMSE values are 0.0053 and 0.0049 and obtained by boosting method. A comparison between the proposed method and other semi-empirical methods provided for these sensors demonstrated the ability of the method to improve the RMSE by up to 0.5%. Regarding the higher accuracy and applicability of the proposed method, it can serve as an effective and efficient means of estimating LSE using remote sensing data.

MTBR: Multi-Target Boosting for Regression

Gradient boosting method has been successfully used for single target prediction problems. In real world applications, however, problems involving the prediction of multiple target attributes are often of interest. In this paper, a multi-target boosting method for regression problems, named as MTBR, is proposed. Although MTBR builds one model for each target attribute separately, all the target attributes are utilized when building each model. In each boosting iteration, the base learner, the regression tree in particular, is learned by selecting the best models from all the target attributes. We also introduce a novel knowledge transfer approach. That is, the tree structure learned from one target attribute, representing a way to partition the feature space, is used to predict another target attribute. Experiments with six data sets compare MTBR to other ensemble regression methods, and prove the effectiveness of MTBR in leveraging the knowledge of multiple target attributes and improving the model accuracy.

Short-term traffic volume prediction by ensemble learning in concept drifting environments

Because of the rapid changes in traffic conditions caused by various circumstances, such as road construction and traffic jams, the distribution of the traffic volume data changes over time. The performances of traditional traffic volume prediction methods, with fixed model types and parameter settings, suffer from gradual degradation during these concept drift processes. In this paper, a novel incremental regression framework under the concept drifting environment is proposed, with ensemble learning as the major solution for updating the distribution representation. First, we transform the regression problem of traffic volume forecasting into a binary classification problem. Second, loss functions for incremental and ensemble learning are constructed based on this transformation. Finally, the incremental learning of the regression function is formulated as stepwise updating of the decision hyperplane. The experimental results show that our method is more stable and accurate than the existing incremental and ensemble regression methods.

A tree ensemble-based two-stage model for advanced-stage colorectal cancer survival prediction

Classification techniques have widely been applied to cancer survival prediction for predicting survival or death of patients. However, little attention has been paid to patients who are predicted to die. In this work, we consider survival prediction as a two-stage task, where the first stage is to predict whether the outcome is survival or not, and the second stage is to predict the remaining lifespan for patients whose predicted outcome is death. To this end, we propose a two-stage machine learning model to enhance cancer survival prediction. At the first stage, a tree-based imbalanced ensemble classification method is proposed for classification of the survivability of advanced-stage cancer patients. At the second stage, a selective ensemble regression method is proposed for survival time prediction, where a priori knowledge is adopted for feature selection and the mean proportion of error interval is proposed for selecting base learners. Extensive computational studies performed on colorectal cancer data from SEER database demonstrate that the proposed two-stage model can achieve a more accurate prediction compared to the one-stage regression model. The results show that the proposed classification approach can effectively handle the imbalanced survivability data, and the proposed regression method outperforms several state-of-the-art regression models.

Least squares kernel ensemble regression in Reproducing Kernel Hilbert Space

Abstract Ensemble regression method shows better performance than single regression since ensemble regression method can combine several single regression methods together to improve accuracy and stability of a single regressor. In this paper, we propose a novel kernel ensemble regression method by minimizing total least square loss in multiple Reproducing Kernel Hilbert Spaces (RKHSs). Base kernel regressors are co-optimized and weighted to form an ensemble regressor. In this way, the problem of finding suitable kernel types and their parameters in base kernel regressor is solved in the ensemble regression framework. Experimental results on several datasets, such as artificial datasets, UCI regression and classification datasets, show that our proposed approach achieves the lowest regression loss among comparative regression methods such as ridge regression, support vector regression (SVR), gradient boosting, decision tree regression and random forest.

Co-regularized kernel ensemble regression

In this paper, co-regularized kernel ensemble regression scheme is brought forward. In the scheme, multiple kernel regressors are absorbed into a unified ensemble regression framework simultaneously and co-regularized by minimizing total loss of ensembles in Reproducing Kernel Hilbert Space. In this way, one kernel regressor with more accurate fitting precession on data can automatically obtain bigger weight, which leads to a better overall ensemble performance. Compared with several single and ensemble regression methods such as Gradient Boosting, Tree Regression, Support Vector Regression, Ridge Regression and Random Forest, our proposed method can achieve best performances of regression and classification tasks on several UCI datasets.

Noise Effect and Noise-Assisted Ensemble Regression in Power System Online Sensitivity Identification

Recently developed data acquisition equipment and data processing methods have ignited the possibility of power system online sensitivity identification (OSI). Despite the existing OSI algorithms, practical issues such as data collinearity and the noise effect on the identification algorithm must be considered to realize OSI in real-power systems. In this study, the negative and positive aspects of noise to OSI are first studied. Then, under the data collinearity condition and by making use of the positive aspects of noise, a noise-assisted ensemble regression method is proposed to simultaneously solve the data collinearity problem and manage the negative aspects of noise. Moreover, the proposed method is proven equivalent to one of the most effective measures, the norm-2 regularization method, to address the collinearity problem, and therefore provides satisfactory OSI results. The proposed method is tested in an 8-generator 36-node system with original operations data from a real-power system, and the results validate its effectiveness.

Identifying outliers of non-Gaussian groundwater state data based on ensemble estimation for long-term trends

A hydrogeological dataset often includes substantial deviations that need to be inspected. In the present study, three outlier identification methods – the three sigma rule (3σ), inter quantile range (IQR), and median absolute deviation (MAD) – that take advantage of the ensemble regression method are proposed by considering non-Gaussian characteristics of groundwater data. For validation purposes, the performance of the methods is compared using simulated and actual groundwater data with a few hypothetical conditions. In the validations using simulated data, all of the proposed methods reasonably identify outliers at a 5% outlier level; whereas, only the IQR method performs well for identifying outliers at a 30% outlier level. When applying the methods to real groundwater data, the outlier identification performance of the IQR method is found to be superior to the other two methods. However, the IQR method shows limitation by identifying excessive false outliers, which may be overcome by its joint application with other methods (for example, the 3σ rule and MAD methods). The proposed methods can be also applied as potential tools for the detection of future anomalies by model training based on currently available data.