Ensemble Support Vector Regression Research Articles

A Novel Surrogate-Assisted Multi-Objective Well Control Parameter Optimization Method Based on Selective Ensembles

Multi-objective optimization algorithms are crucial for addressing real-world problems, particularly with regard to optimizing well control parameters, which are often computationally expensive due to their reliance on numerical simulations. Surrogate-assisted models help to reduce this computational burden, but their effectiveness depends on the quality of the surrogates, which can be affected by candidate dimension and noise. This study proposes a novel surrogate-assisted multi-objective optimization framework (MOO-SESA) that combines selective ensemble support-vector regression with NSGA-II. The framework’s uniqueness lies in its adaptive selection of a diverse subset of surrogates, established prior to iteration, to enhance accuracy, robustness, and computational efficiency. To our knowledge, this is the first instance in which selective ensemble techniques with multi-objective optimization have been applied to reservoir well control problems. Through employing an ensemble strategy for improving the quality of the surrogate model, MOO-SESA demonstrated superior well control scenarios and faster convergence compared to traditional surrogate-assisted models when applied to the SPE10 and Egg reservoir models.

A general procedure for evaluating models and ensemble Support Vector Regression

In practice, we may want to discover if there is a relationship between a response variable as a function of the predictor variables. Multiple linear regression (MLR) is a popular tool for such purpose. When the relationship is nonlinear, nonparametric regression methods such as local linear regression, smoothing splines, and support vector regression (SVR) provide flexible alternatives to MLR. How do we compare the performance of these methods and choose an appropriate one for use? In this research, we propose a general procedure to evaluate the performance of different regression methods for use in large data. We also propose an ensemble SVR for regression analysis. The proposed methods are applied to address research questions using the Research and Development Survey, conducted by the National Center for Health Statistics.

New secondary decomposition ensemble support vector regression for monthly drought index forecasting

Drought prediction on a monthly timescale is conducive to the timely identification of drought characteristics and provides a reference for drought mitigation. Current research is generally based on drought indices on timescales of three months or more for drought prediction, whereas the presence of highly nonlinear and drastic changes in the characteristics of drought indices leads to suboptimal drought prediction for drought index sequences on monthly timescales. In this study, a secondary decomposition model based on extreme-point symmetric mode decomposition (ESMD), variational mode decomposition (VMD), and support vector regression (SVR) was investigated to predict a monthly-scale drought index. To evaluate the predictive ability of the model, it was applied to 100 meteorological stations in the Pearl River Basin of China and compared and analyzed using four comparative models. The results showed that the model proposed in this study has better performance index values, and the prediction accuracies of all stations are improved compared with the four comparative models, with the most significant improvement in the Nash–Sutcliffe efficiency (NSE) index value, followed by the root mean square error (RMSE) and mean absolute error (MAE) index values, and the smallest improvement in the Wilmot consistency index (WIA) index value. The monthly-scale drought quadratic decomposition (ESMD-VMD) combined prediction model established in this study provides a novel method for drought prediction.

Optimized Ensemble Support Vector Regression Models for Predicting Stock Prices with Multiple Kernels

Optimized Ensemble Support Vector Regression Models for Predicting Stock Prices with Multiple Kernels

Optimal weight support vector regression ensemble with cluster-based subsampling for electricity price forecasting

Accurate prediction of short-term electricity price is the key to obtain economic benefit and also an important index of power system planning and management. Support vector regression (SVR) based ensemble works have gained remarkable achievements in terms of high accuracy and steady performance, but they are highly dependent on data representativeness and have a high computational complexity O (k * N 3) of data samples and parameter selection. To further improve the data representativeness and reduce its computational complexity, this paper develops a new approach to forecast electricity price via optimal weighted ensemble. In the model, the cluster-based subsampling algorithm is proposed to categorize the inputs being seasonally decomposed into several groups, and representative data are drawn from each group in a certain proportion to ensure that each subset trained with SVR has the same representativeness and features. Moreover, the optimal weighted combination method is presented to assign weights to the sub-SVRs to obtain the optimal support vector regression ensemble model (OWSSVRE). The experimental results show that the improved support vector regression ensemble model with the same features and representativeness of the subset has better performance in electricity price forecasting. As a result, it is suitable to support decision making in the energy and other sectors.

Hybrid model of support vector regression and innovative gunner optimization algorithm for estimating ski-jump spillway scour depth

Scour hole that occurs downstream of the hydraulic structures threatens the safety and stability of the hydraulic structures. The scour around the structures is a complex and important hydraulic phenomenon; hence, it requires a data extensive research for the accurate estimation of scour depth. Although many analytical models are available for scour depth estimation, they suffer from huge limitations. In this research, the support vector regression (SVR) model and SVR ensemble with the metaheuristic algorithm of innovative gunner (SVR-AIG) models have been developed for accurate prediction of scour depth downstream of the ski-jump spillways. Field measurements including head and discharge intensity are used for developing the models. The performances of the models are compared using root mean square error (RMSE), mean average error (MAE), and correlation coefficient (CC) criteria and some statistical plots. The results showed that the hybrid SVR-AIG-based estimations (with CC = 0.987, 0.991, RMSE = 2.839, 1.987, and MAE = 2.247, 1.201) are more accurate than the SVR standalone model estimations (with CC = 0.942, 0.975, RMSE = 5.686, 4.040, and MAE = 4.114, 3.201) at the training and testing phases. This study is an important reference for analyzing the high capability of the AIG as an optimization tool in improving scour estimations of a standalone model. Also, this algorithm eliminates the trial-and-error procedure to optimize the internal parameters during the model development.Graphical abstract

Unified whale optimization algorithm based multi-kernel SVR ensemble learning for wind speed forecasting

Support vector regression (SVR) is widely used in the field of wind speed forecasting because of its excellent nonlinear learning ability. However, the drawback of SVR is the model selection problem, which has the high complexity O(K×m3) including kernel function selection and parameter selection. To solve this problem, this paper proposes a multi-kernel SVR ensemble (MKSVRE) model based on unified optimization and whale optimization algorithm (WOA), where the MKSVRE model is used to solve the kernel function selection problem, and the unified optimization and the WOA are used to solve the parameter selection problem. The proposed model provides an alternative without the need to specifically select a kernel function and thus enhances the adaptability of SVR to diverse data. In addition, the unified optimization takes into account the interactions between models and achieves a global parameter selection. The proposed model is tested by simulations on wind speed data from Shandong Province, China. By comparing the prediction results of the proposed model, the single kernel SVR models, the models before and after optimization, and six other models, the effectiveness of the proposed model is confirmed.

State-of-health estimation for lithium-ion batteries based on historical dependency of charging data and ensemble SVR

Accurate estimation of State-of-Health (SOH) is very important for the safe and reliable operation of lithium-ion batteries. Considering that the historical dependency of charging data could reflect the internal electrochemical reaction of the battery, a new SOH estimation method is proposed. Firstly, a data pre-processing method is developed to resample the voltage data of the constant current charging stage with a predefined fixed number of samples. It can suppress the measurement noise and facilitate calculating the difference of voltage curves under different aging levels. Secondly, a new health indicator (HI) is proposed. It includes two types of features, one is accumulated voltage of different intervals and the other is charging capacity, they are used to reflect the non-linear changes of the charging voltage and changes of the charging time with the battery aging respectively. In addition, considering the cell inconsistency, an Ensemble Support Vector Regression (ESVR) model is put forward to establish the relationship between HI and battery SOH. Finally, two kinds of open-source battery data are tested and the results show that the method developed in the paper could get high-precision SOH estimation results and the HI is robust to the battery type and cell inconsistency.

Semi-supervised ensemble support vector regression based soft sensor for key quality variable estimation of nonlinear industrial processes with limited labeled data

Soft sensor technique has become a promising solution to enable real-time estimations of difficult-to-measure quality variables in industrial processes. However, traditional soft sensor models cannot always function well due to two challenging issues. First, labeled data are usually expensive to obtain in many real-world applications, thus leading to unsatisfactory performance for traditional supervised soft sensors. Meanwhile, the information behind abundant unlabeled data is not fully exploited. Second, it is very common for soft sensors to encounter the modeling uncertainty resulting from the diversity of training data, model hyperparameters, and optimization parameters. Therefore, a new soft sensor method called semi-supervised ensemble support vector regression (SSESVR) is proposed by combining semi-supervised learning with ensemble learning. The SSESVR method first formulates the estimation of pseudo-labels as a multi-learner pseudo-labeling optimization problem and then solve it through evolutionary approach, thus extending the labeled training set using satisfactory pseudo-labeled data. Further, by considering multimodal perturbation mechanism, a two-level ensemble architecture is employed to enable efficient cooperation of semi-supervised and ensemble learning framework. Two case studies are conducted to verify the effectiveness and superiority of the proposed SSESVR approach.

KESVR: An Ensemble Model for Drug Response Prediction in Precision Medicine Using Cancer Cell Lines Gene Expression.

Background: Cancer cell lines are frequently used in research as in-vitro tumor models. Genomic data and large-scale drug screening have accelerated the right drug selection for cancer patients. Accuracy in drug response prediction is crucial for success. Due to data-type diversity and big data volume, few methods can integrative and efficiently find the principal low-dimensional manifold of the high-dimensional cancer multi-omics data to predict drug response in precision medicine. Method: A novelty k-means Ensemble Support Vector Regression (kESVR) is developed to predict each drug response values for single patient based on cell-line gene expression data. The kESVR is a blend of supervised and unsupervised learning methods and is entirely data driven. It utilizes embedded clustering (Principal Component Analysis and k-means clustering) and local regression (Support Vector Regression) to predict drug response and obtain the global pattern while overcoming missing data and outliers’ noise. Results: We compared the efficiency and accuracy of kESVR to 4 standard machine learning regression models: (1) simple linear regression, (2) support vector regression (3) random forest (quantile regression forest) and (4) back propagation neural network. Our results, which based on drug response across 610 cancer cells from Cancer Cell Line Encyclopedia (CCLE) and Cancer Therapeutics Response Portal (CTRP v2), proved to have the highest accuracy (smallest mean squared error (MSE) measure). We next compared kESVR with existing 17 drug response prediction models based a varied range of methods such as regression, Bayesian inference, matrix factorization and deep learning. After ranking the 18 models based on their accuracy of prediction, kESVR ranks first (best performing) in majority (74%) of the time. As for the remaining (26%) cases, kESVR still ranked in the top five performing models. Conclusion: In this paper we introduce a novel model (kESVR) for drug response prediction using high dimensional cell-line gene expression data. This model outperforms current existing prediction models in terms of prediction accuracy and speed and overcomes overfitting. This can be used in future to develop a robust drug response prediction system for cancer patients using the cancer cell-lines guidance and multi-omics data.

Sensitivity analysis and ensemble artificial intelligence-based model for short-term prediction of NO2 concentration

In this study, in the first step, three scenarios with different input combinations are created to implement a sensitivity analysis for hourly NO2 prediction in Columbus City, Ohio. Three classes of inputs including concentration-related data (NO2 concentration at previous time steps and NO2 concentration in the suburban monitoring station), meteorology (wind speed, wind direction, and temperature), and traffic-related data (traffic count, hour of the day, and day of the week) are applied to create three scenarios. Also, the support vector regression methodology is employed to perform the sensitivity analysis. Dominant variables determined in the sensitivity analysis are applied as inputs to three models called feed-forward neural network, support vector regression, as well as classification and regression tree. In the last step, ensemble techniques including simple linear averaging, weighted linear averaging, and nonlinear support vector regression ensemble are proposed to improve the performance of sole models. The results indicate that, in the urban area, in addition to NO2 variations in the previous time step, other variables such as hourly traffic count in freeway loop, suburban NO2 concentration, and hour of the day can affect the NO2 concentration. Further, the values of determination coefficient for the individual models, namely classification and regression tree and feed-forward neural network, are 67 and 81% that the ensemble technique as a post-processing approach enhances the performance of them up to 19% and 5% in the verification steps, respectively.

Predicting the printed circuit board cycle time of surface-mount-technology production lines using a symbiotic organism search-based support vector regression ensemble

This paper presents a symbiotic organism search (SOS)-based support vector regression (SVR) ensemble for predicting the printed circuit board (PCB) cycle time of surface-mount-technology (SMT) production lines. Being able to predict the PCB cycle time accurately is essential for optimising the SMT production schedule. Although a machine simulator can be reliably used for single-type PCB production, it is time-consuming and often inaccurate for the simulator to be applied for highly mixed orders in multiple flexible SMT production lines. Due to the dynamic changes in both PCB orders and SMT production lines, there is a diverse set of samples, but the size of similar samples is relatively small. An SVR model is therefore used to estimate the PCB cycle time, and the SOS algorithm is employed to optimise the SVR parameters. We assume that uncertainties during the assembly process can be captured by the characteristics of PCB and SMT lines, which are utilised as features to train the SVR model. To enhance the performance of the prediction accuracy, an SOS-SVR ensemble is proposed. Experiments based on datasets collected from a leading global electronics manufacturer confirm the efficiency of the proposed approach compared to industrial solutions currently in place and other machine learning methods.

A novel two-step procedure for tourism demand forecasting

ABSTRACT Tourism demand forecasting is a critical process in the planning of tourism utilities. In recent years, Internet search indices have been popularly used as tourism demand indicators. However, due to the complex relationship between tourism demand and the search index and the vast amounts of search engine data, the traditional econometric and artificial intelligence models could not be enough to complete the prediction task. Under this scenario, this paper proposes a novel two-step method to improve tourism demand prediction accuracy. Firstly, a double-boosting algorithm is proposed to select the keywords and their lags from the potential relevant high-dimensional search queries. Second, the ensemble Support Vector Regression (SVR) based Deep belief Network (DBN) approach is adopted to capture the possible non-linear relationship and to improve the forecasting performance through deep learning combination. The empirical results demonstrate that this procedure significantly outperforms other benchmark models when forecasting monthly Hong Kong tourist arrivals.

Support Vector Regression Ensemble for Effective Modeling of Magnetic Ordering Temperature of Doped Manganite in Magnetic Refrigeration

Magnetic refrigeration technology (MRT) is an affordable and efficient refrigeration system that can conveniently replace the present conventional gas compression and expansion (GCE) system of refrigeration. Apart from the fact that GCE refrigeration system has reached its thermodynamic limit and releases ozone depleting gases that is harmful to the environment, its non-compactness and lower energy efficiency are of serious concerns that remain unaddressed. Utilization of solid magnetic refrigerant in MRT aids its compactness and enhances environmental friendliness system of refrigeration. Manganite refrigerants have recently attracted attention from experimental and modeling perspectives due to its stability, cost-effectiveness as well as tunable magnetic properties as compared to the existing metal gadolinium refrigerant. However, magnetic ordering temperature of manganite refrigerants plays a crucial role in the implementation of MRT at ambient condition. Support vector regression (SVR) ensemble models with gravitational search algorithm (GSA) hyper-parameter optimization are hereby proposed for modeling the magnetic ordering temperature of manganite using ionic radii and dopant concentrations as descriptors. Ensemble model (EMS) developed using average of the outputs of five different SVR base estimators performs better than the ensemble model (EMM) developed using individual outputs without averaging with performance improvement of 25.4%. Similarly, EMS performs better than the conventional SVR-based model as well as GSA-optimized SVR-based model (GSVR). The proposed ensemble models also outperform other existing models in the literature. The results of this research work will not only serve to circumvent the experimental challenges of magnetic ordering temperature measurement but also further promotes environmental-friendly system of refrigeration.

Intelligent Quality of Service Routing in Software-Defined Satellite Networking

Software-Defined Satellite Networking (SDSN) has emerged as a new paradigm that offers the programmability required to dynamically manage a satellite network. However, the characteristics of SDSN bring several important issues to be solved in the routing scheme. First, interface protocols of the controller incur nonnegligible overhead, which affects the general traffic. Second, more than one controller is required to ensure the reliability of SDSN, which means that they should cooperate with each other to realize the routing function. Third, how to make full use of the advantage of SDSN's centralized management for routing algorithm is worth studying. In this paper, we first propose an SDSN architecture to realize flexible centralized management. Based on this architecture, Intelligent Quality of Service (QoS) Routing scheme is proposed and evaluated. Intelligent QoS Routing (IQR) scheme is composed of overhead balancing strategy, controller cooperation strategy, and IQR algorithm. Overhead balancing strategy effectively reduces the impact of overhead on general traffic, which improves the performance of IQR algorithm. Controller cooperation strategy adopts the ensemble Support Vector Regression (SVR) from artificial intelligence tools to enhance the consistency of different controllers' network views under low frequency situation, which will provide a more accurate reference for IQR algorithm and other management strategies. IQR algorithm takes full advantage of SDSN's centralized control and rich data to offer fine-grained QoS guarantee intelligently. Simulation results show that IQR scheme achieves better overhead balancing performance, more effective controller cooperation, and better QoS guarantee compared with the existing methods.

Improving TIGGE Precipitation Forecasts Using an SVR Ensemble Approach in the Huaihe River Basin

Recently, the use of the numerical rainfall forecast has become a common approach to improve the lead time of streamflow forecasts for flood control and reservoir regulation. The control forecasts of five operational global prediction systems from different centers were evaluated against the observed data by a series of area-weighted verification and classification metrics during May to September 2015–2017 in six subcatchments of the Xixian Catchment in the Huaihe River Basin. According to the demand of flood control safety, four different ensemble methods were adopted to reduce the forecast errors of the datasets, especially the errors of missing alarm (MA), which may be detrimental to reservoir regulation and flood control. The results indicate that the raw forecast datasets have large missing alarm errors (MEs) and cannot be directly applied to the extension of flood forecasting lead time. Although the ensemble methods can improve the performance of rainfall forecasts, the missing alarm error is still large, leading to a huge hazard in flood control. To improve the lead time of the flood forecast, as well as avert the risk from rainfall prediction, a new ensemble method was proposed on the basis of support vector regression (SVR). Compared to the other methods, the new method has a better ability in reducing the ME of the forecasts. More specifically, with the use of the new method, the lead time of flood forecasts can be prolonged to at least 3 d without great risk in flood control, which corresponds to the aim of flood prevention and disaster reduction.

Subsampled support vector regression ensemble for short term electric load forecasting

Accurate prediction of short-term electric load is critical for power system planning and operation. However, integration of the point estimation into the power system is constrained by its uncertainty nature and low interpretability for confidence level. For this propose, this study derives and tests methods to model and forecast short term load point estimation and its confidence interval length by using Subsampled support vector regression ensemble (SSVRE). To improve the computational accuracy and efficiency, a subsampling strategy is designed for the programming implementation of the support vector regression (SVR) learning process. This subsampling strategy ensures that each individual SVR ensemble has enough diversity. Then, for model selection, we present a novel swarm optimization learning based on all the individual SVR ensembles. The advantage of swarm coordination learning is that we can ensure that each individual SVR ensemble has enough strength for forecasting the short term load data. Theoretically, the latest research shows that formal statistical inference procedures can be determined for small size subsamples based ensemble. In practice, a subset of small size subsamples is employed for the speeding-up of SVR learning process. Accordingly, the results indicate the better performance and lower uncertainty of SSVRE model in forecasting short term electric load.

State of health prediction for lithium-ion batteries using multiple-view feature fusion and support vector regression ensemble

Lithium-ion batteries have been widely used in many electronic systems. Accurately estimating the state of health (SOH) of a lithium-ion battery is important for ensuring its safety and reliability. Among the various kinds of methods for predicting the SOH of lithium-ion batteries, machine-learning-based methods are the most popular. However, two common critical problems in machine-learning-based methods are extracting discriminative features and effectively utilizing the extracted features. In this study, we focused on solving these two issues. First, a sliding-window-based feature extraction technology (SWBFE) was designed to effectively extract features from different views in the discharge process of lithium-ion batteries. Second, we developed a multiple-view feature fusion with a support vector regression (SVR) ensemble strategy (MVFF-ESVR) for enhancing the performance in fusing multiple extracted features. The basic idea of MVFF-ESVR is to transform the feature-level fusion problem into a decision-level fusion problem. More specifically, for each feature, an SVR was modeled on the corresponding training set, and the AdaBoost and Stacking algorithms were utilized to incorporate multiple trained SVRs for generating two ensemble SVR models. By combining SWBFE with MVFF-ESVR, we further implemented two predictors, namely, Ada-TargetSOH and Sta-TargetSOH, for robust prediction of lithium-ion battery SOH. To evaluate the efficacy of the proposed predictors, we applied Ada-TargetSOH and Sta-TargetSOH on three types of lithium-ion battery datasets. The experimental results have demonstrated that our predictors outperform other existing lithium-ion battery SOH predictors.

Price forecasting using an ACO-based support vector regression ensemble in cloud manufacturing

Abstract In a cloud manufacturing environment, deviation of a service price from its value and cost makes on-demand price forecasting a challenging task for the service providers. The main objective of this paper is to present taxonomy of Value Measures and Metrics (VMMs) for pricing decisions over the product life cycle, e.g. the design, manufacturing, and service stages. Furthermore, a parametric pricing approach is proposed to formulate pricing variables, which represent pricing factors and are calculated in terms of VMMs, as well as a regression relation between the pricing variables and price. An Ant Colony Optimization Algorithm (ACO)-based Support Vector Regression (SVR) ensemble is developed to forecast a price. We demonstrate the effectiveness of the proposed methodology with the real-world data of an organization in China. The experimental results show that the proposed method achieves significant generalization performance with the best mean squared error (MSE) and reliable results in randomness of ensemble learning. Thus, the proposed pricing method provides a way to make viable prices for service providers.

Predicting short-term stock prices using ensemble methods and online data sources

With the ubiquity of the Internet, platforms such as: Google, Wikipedia and the like can provide insights pertaining to firms’ financial performance as well as capture the collective interest of traders through search trends, number of web page visitors and/or financial news sentiment. Information emanating from these platforms can significantly affect, or be affected by, changes in the stock market. The overarching goal of this paper is to develop a financial expert system that incorporates these features to predict short term stock prices. Our expert system is comprised of two main modules: a knowledge base and an artificial intelligence (AI) platform. The “knowledge base” for our expert system captures: (a) historical stock prices; (b) several well-known technical indicators; (c) counts and sentiment scores of published news articles for a given stock; (d) trends in Google searches for the given stock ticker; and (e) number of unique visitors for pertinent Wikipedia pages. Once the data is collected, we use a structured approach for data preparation. Then, the AI platform trains four machine learning ensemble methods: (a) a neural network regression ensemble; (b) a support vector regression ensemble; (c) a boosted regression tree; and (d) a random forest regression. In the cross-validation phase, the AI platform picks the “best” ensemble for a given stock. To evaluate the efficacy of our expert system, we first present a case study based on the Citi Group stock ($C) with data collected from 01/01/2013 - 12/31/2016. We show the expert system can predict the 1-day ahead $C stock price with a mean absolute percent error (MAPE) ≤ 1.50% and the 1–10 day ahead with a MAPE ≤ 1.89%, which is better than the reported results in the literature. We show that the use of features extracted from online sources does not substitute the traditional financial metrics, but rather supplements them to improve upon the prediction performance of machine learning based methods. To highlight the utility and generalizability of our expert system, we predict the 1-day ahead price of 19 additional stocks from different industries, volatilities and growth patterns. We report an overall mean for the MAPE statistic of 1.07% across our five different machine learning models, including a MAPE of under 0.75% for 18 of the 19 stocks for the best ensemble (boosted regression tree).