Least Squares Support Vector Regression Model Research Articles

A dynamic ensemble selection approach to developing softcomputing models for two-phase flow metering

This paper presents a dynamic ensemble selection approach to developing least squares support vector regression (LSSVR) models for the flow metering of gas-liquid two-phase CO2. Ensemble models based on flow pattern recognition and dynamic ensemble selection (FPR-DES) are established for the measurement of total mass flowrate and gas volume fraction of two-phase CO2, respectively. The input variables of ensemble models are obtained from a Coriolis flowmeter and a differential-pressure transducer installed on a horizontal test section. Experimental tests were conducted with liquid CO2 mass flowrate ranging from 200 kg/h to 3100 kg/h and gas volume fraction between 3.1% and 88.4%. Performance comparisons between the proposed FPR-DES based ensemble model, bagging based ensemble model and the single LSSVR model are undertaken with experimental data under stratified, bubbly and mist flow conditions. Experimental results suggest that the FPR-DES based ensemble model outperforms the other two models with maximum errors of ±1% and ±10% for the total CO2 mass flowrate and gas volume fraction, respectively.

Using Internet Search Trends and Historical Trading Data for Predicting Stock Markets by the Least Squares Support Vector Regression Model.

Historical trading data, which are inevitably associated with the framework of causality both financially and theoretically, were widely used to predict stock market values. With the popularity of social networking and Internet search tools, information collection ways have been diversified. Instead of only theoretical causality in forecasting, the importance of data relations has raised. Thus, the aim of this study was to investigate performances of forecasting stock markets by data from Google Trends, historical trading data (HTD), and hybrid data. The keywords employed for Google Trends are collected from three different ways including users' definitions (GTU), trending searches of Google Trends (GTTS), and tweets (GTT) correspondingly. The hybrid data include Internet search trends from Google Trends and historical trading data. In addition, the correlation-based feature selection (CFS) technique is used to select independent variables, and one-step ahead policy is adopted by the least squares support vector regression (LSSVR) for predicting stock markets. Numerical experiments indicate that using hybrid data can provide more accurate forecasting results than using single historical trading data or data from Google Trends. Thus, using hybrid data of Internet search trends and historical trading data by LSSVR models is a promising alternative for forecasting stock markets.

Monthly long-term rainfall estimation in Central India using M5Tree, MARS, LSSVR, ANN and GEP models

This study investigates the performance of the M5Tree model, multivariate adaptive regression spline, least square support vector regression (LSSVR), gene expressing programming (GEP) and artificial neural networks methods in estimating monthly long-term rainfall. Data from 61 rain stations in Madhya Pradesh and Chhattisgarh states, Central India, were used in the applications. Geographical inputs and periodicity were used as inputs to the models, and the methods were compared with each other according to the determination coefficient (R2), mean absolute errors (MAE) and root-mean-square errors (RMSE). LSSVR was found to be the best model with RMSE = 13.93 mm, MAE = 9.52 mm and R2 = 0.995 while the GEP provided the worst results with RMSE = 36.74 mm, MAE = 29.89 mm and R2 = 0.955 in prediction of long-term rainfall in the test stage. The lowest RMSE (5.53 mm) and MAE (3.89 mm) were obtained for the Rajnandgaon Station, while the Raigarh Station provided the worst accuracy (RMSE = 31.8 mm and MAE = 21.56 mm) for LSSVR model.

Efficient design of gain-flattened multi-pump Raman fiber amplifiers using least squares support vector regression

An efficient method to design the broadband gain-flattened Raman fiber amplifier with multiple pumps is proposed based on least squares support vector regression (LS-SVR). A multi-input multi-output LS-SVR model is introduced to replace the complicated solving process of the nonlinear coupled Raman amplification equation. The proposed approach contains two stages: offline training stage and online optimization stage. During the offline stage, the LS-SVR model is trained. Owing to the good generalization capability of LS-SVR, the net gain spectrum can be directly and accurately obtained when inputting any combination of the pump wavelength and power to the well-trained model. During the online stage, we incorporate the LS-SVR model into the particle swarm optimization algorithm to find the optimal pump configuration. The design results demonstrate that the proposed method greatly shortens the computation time and enhances the efficiency of the pump parameter optimization for Raman fiber amplifier design.

Predicting Vehicle Sales by Sentiment Analysis of Twitter Data and Stock Market Values

Owning to the booming of social media, making comments or expressing opinions about merchandises online becomes easier than before. Data from social media might be one of the essential inputs for forecasting sales of vehicles. Besides, some other effects, such as stock market values, have influences on purchasing power of vehicles. In this paper, both multivariate regression models with social media data and stock market values and time series models are employed to predict monthly total vehicle sales. The least squares support vector regression (LSSVR) models are used to deal with multivariate regression data. Three types of data, namely sentiment scores of tweets, stock market values, and hybrid data, are employed in this paper to forecast monthly total vehicle sales in USA. The hybrid data contain both sentiment scores of tweets and stock market values. In addition, seasonal factors of monthly total vehicle sales are employed to deseasonalizing both monthly total vehicle sales and three types of input data. The time series models include the naive model, the exponential smoothing model, the autoregressive integrated moving average model, the seasonal autoregressive integrated moving average model, and backpropagation neural networks and LSSVR with time series models. The numerical results indicate that using hybrid data with deseasonalizing procedures by the LSSVR models can obtain more accurate results than other models with different data. Thus, both social media data and stock values are essential to forecast monthly total vehicle sales; and deseasonalizing procedures can improve forecasting accuracy in predicting monthly total vehicle sales.

Whole flow field performance prediction by impeller parameters of centrifugal pumps using support vector regression

The relationship of multiple impeller parameters and performance indices is difficult to describe because of some unknown hydrodynamic phenomena. Modeling of performance indices of the whole flow field from impeller parameters often encounters some challenges, especially lower prediction accuracy in relatively small and large flow points, dependence on designers’ experience and time-consuming designing process. In this work, the least squares support vector regression (LSSVR) method is proposed to predict multiple pump performance indices of the whole flow field. To describe the performance more completely, the powder, the head, and the efficiency indices are chosen as the model outputs. Additionally, to improve the prediction accuracy and reduce the manufacture difficulty, nine impeller parameters and the flow rate are selected as the model inputs. With the LSSVR model, the complex nonlinearity relationship between multiple impeller parameters and performance indices can be described approximately. Moreover, the LSSVR model and the computational fluid dynamics numerical simulation model are applied to predict the powder, the head, and the efficiency of an actual centrifugal mine pump in the whole flow field. Compared with the performance test results, the superiority of the proposed method is demonstrated in terms of more accurate prediction performance and faster designing process.

Pan evaporation modeling using four different heuristic approaches

Evaporation plays important roles in regional water resources management, climate change and agricultural production. This study investigates the abilities of fuzzy genetic (FG), least square support vector regression (LSSVR), multivariate adaptive regression spline (MARS), M5 model tree (M5Tree) and multiple linear regression (MLR) in estimating daily pan evaporation (Ep). Daily climatic data, air temperature (Ta), surface temperature (Ts), wind speed (Ws), relative humidity (RH) and sunshine hours (Hs) at eight stations in the Dongting Lake Basin, China are used for model development and validation. The first part of this study focuses on testing the model accuracies at each station using local input and output data. The results show that LSSVR and FG models with more input variables perform better than the MARS, M5Tree and MLR models in predicting daily Ep at most stations with respect to mean absolute errors (MAE), root mean square errors (RMSE) and determination coefficient (R2). In the second part of this study, the models are tested using cross-validation method in two different applications. The daily Ep of Yueyang station is estimated using the input and output data of Jingzhou and Changsha, respectively. Comparisons of the models indicate that the FG, LSSVR and MARS models outperform the M5Tree model, Ts, Hs and Ta are major influencing factors and adding Ws or RH into model inputs significantly improve the model performances. The overall results indicate that above models can be successfully used for estimating daily Ep using local input and output data while the FG and LSSVR generally perform better than the other models without local input and outputs.

Novel sparse LSSVR models in primal weight space for robust system identification with outliers

Recent demands from big data applications have strongly motivated a successful sparse formulation of the least squares support vector regression (LSSVR) model in primal weight space. Such an approach, which has been called fixed-size LSSVR (FS-LSSVR), is built upon an approximation of the nonlinear feature mapping based on Nyström method to overcome the usual memory constraints and high computational costs of the standard non-sparse LSSVR model. Despite this advance, an important modeling issue remains unaddressed by the FS-LSSVR model. As in the standard LSSVR model, the performance of FS-LSSVR model is greatly degraded when estimation data is corrupted with non-Gaussian noise or outliers. Bearing this major issue in mind, we introduce two robust variants of the FS-LSSVR model based on M-estimation framework and the weighted least squares method. The proposed approaches, henceforth called Robust FS-LSSVR (RFS-LSSVR) and Reweighted Robust FS-LSSVR (R2FS-LSSVR) models, produce solutions that are simultaneously robust to outliers and sparse, making use of only a small sample of training patterns as prototype vectors. We evaluate the performances of both algorithms in benchmarking nonlinear system identification problems with synthetic and real-world datasets (including a large scale dataset) corresponding to SISO and MIMO systems, whose estimation outputs are contaminated with outliers. The obtained results indicate that our proposed approaches consistently outperforms existing robust models designed in dual space (e.g. W-LSSVR and IR-LSSVR models), specially as the amount of outliers in the data increases.

Modelling of runoff and sediment yield using ANN, LS-SVR, REPTree and M5 models

AbstractIn this study, the performance evaluation of five machine learning models, namely, ANNLM, ANNSCG, least square-support vector regression (LS-SVR), reduced error pruning tree (REPTree) and M5, was carried out for predicting runoff and sediment in the Pokhariya watershed, India using hydro-meteorological variables as input. The input variables were selected using the trial-and-error procedure which represents the hydrological process in the watershed. The seven input variables to all the models comprised a combination of rainfall, average temperature, relative humidity, pan evaporation, sunshine duration, solar radiation and wind speed. The monthly runoff and sediment yield data were used to calibrate and validate all models for the years 2000 to 2008. Evaluation of models' performances were carried out using four statistical indices, i.e., Nash–Sutcliffe coefficient (NSE), coefficient of determination (R2), percent bias (PBIAS) and RMSE-observations standard deviation ratio (RSR). Comparative analysis showed that the ANNLM model marginally outperformed the LS-SVR model and all the other models investigated during calibration and validation for runoff modelling whereas the LS-SVR model surpassed the artificial neural networks (ANN) model and other models for sediment yield modelling. Moreover, M5 model tree is better in simulating sediment yield and runoff than its near counterpart, the REPTree model, and marginally inferior when compared to LS-SVR and ANN models.

An enhanced chiller FDD strategy based on the combination of the LSSVR-DE model and EWMA control charts

Development of the fault detection and diagnosis (FDD) for chiller systems is very important for improving the equipment reliability and saving energy consumption. The results of FDD performance are strongly dependent on the accuracy of chiller models. Since the accuracy of the chiller models depends on the indefinite model parameters which are normally chosen by experiments or experiences, an accurate chiller model is difficult to build. Therefore, optimization of model parameters is very useful to increase the accuracy of chiller models. This paper presents a new FDD strategy for centrifugal chillers of building air-conditioning systems, which is the combination between the nonlinear least squares support vector regression (LSSVR) based on the differential evolution (DE) algorithm and the exponentially weighted moving average (EWMA) control charts. In this strategy, the nonlinear LSSVR, which is a reformulation of SVR model with better generalization performances, is adopted to develop the reference feature parameter models in a typical non-linear chiller system. The DE algorithm which is a real-coding optimal algorithm with powerful global searching capacity is employed to enhance the accuracy of LSSVR models. The exponentially weighted moving average (EWMA) control charts are introduced to improve the fault detection capability as well as to reduce the Type II errors in a t-statistics-based way. Six typical faults of the chiller from the real-time experimental data of ASHRAE RP-1043 project are chosen to validate proposed FDD methods. Comprehensive comparisons between the proposed method and two similarly previous studies are performed. The comparison results show that the proposed method has achieved significant improvement in accuracy and reliability, especially at low severity levels. The proposed DE-LSSVR-EWMA strategy is robust for fault detection and diagnosis in centrifugal chiller systems.

Enhancing Long-Term Streamflow Forecasting and Predicting using Periodicity Data Component: Application of Artificial Intelligence

Streamflow forecasting and predicting are significant concern for several applications of water resources and management including flood management, determination of river water potentials, environmental flow analysis, and agriculture and hydro-power generation. Forecasting and predicting of monthly streamflows are investigated by using three heuristic regression techniques, least square support vector regression (LSSVR), multivariate adaptive regression splines (MARS) and M5 Model Tree (M5-Tree). Data from four different stations, Besiri and Malabadi located in Turkey, Hit and Baghdad located in Iraq, are used in the analysis. Cross validation method is employed in the applications. In the first stage of the study, the heuristic regression models are compared with each other and multiple linear regression (MLR) in forecasting one month ahead streamflow of each station, individually. In the second stage, the models are evaluated and compared in predicting streamflow of one station using data of nearby station. The research investigated also the influence of the periodicity component (month number of the year) as an external sub-set in modeling long-term streamflow. In both stages, the comparison results indicate that the LSSVR model generally performs superior to the MARS, M5-Tree and MLR models. In addition, it is seen that adding periodicity as input to the models significantly increase their accuracy in forecasting and predicting monthly streamflows in both stages of the study.

Modeling reference evapotranspiration using three different heuristic regression approaches

Modeling reference evapotranspiration (ET0) is important in reservoir management, planning regional water resources and evaluation of drinking-water supplies. The study investigates the ability of three different heuristic regression approaches, least square support vector regression (LSSVR), multivariate adaptive regression splines (MARS) and M5 Model Tree (M5Tree) in modeling ET0. The first part of the study focused on testing the accuracy of the LSSVR, MARS and M5Tree models in estimating the ET0 data of Antalya and Isparta stations located in Mediterranean Region of Turkey. Cross-validation method was utilized in the applications. The LSSVR models were observed to be better than the MARS and M5Tree models in estimating ET0 of Antalya and Isparta stations with local input and output data. The accuracy of the applied methods was investigated in estimation of ET0 using air temperature, solar radiation, relative humidity and wind speed inputs from nearby station in the second part of the study (cross-station application without local input data). The results showed that the MARS models provided better accuracy than the LSSVR and M5Tree models with respect to SI, mean absolute error (MAE) and determination coefficient (R2). In the third part of the study, the accuracy of the applied models was investigated in ET0 estimation using input and output data from nearby station. The results showed that the M5Tree models outperformed the other models with respect to SI, MAE and R2. The overall results showed that the LSSVR could be successfully used in estimating ET0 by using local input and output data. In case of without local inputs, however, the MARS model performed better than the LSSVR and M5Tree models while the M5Tree was observed to be the best alternative for estimating ET0 in the absence of local input and output data.

On conditional risk estimation considering model risk

Usually, parametric procedures used for conditional variance modelling are associated with model risk. Model risk may affect the volatility and conditional value at risk estimation process either due to estimation or misspecification risks. Hence, non-parametric artificial intelligence models can be considered as alternative models given that they do not rely on an explicit form of the volatility. In this paper, we consider the least-squares support vector regression (LS-SVR), weighted LS-SVR and Fixed size LS-SVR models in order to handle the problem of conditional risk estimation taking into account issues of model risk. A simulation study and a real application show the performance of proposed volatility and VaR models.

Object tracking based on Kalman particle filter with LSSVR

In visual tracking field, traditional Kalman particle filter often suffers from the accuracy loss when estimating the target. To alleviate this problem, we propose a novel object tracking method with the fusion of the extended Kalman particle filter (EKPF) and the least squares support vector regression (LSSVR). First, the observation value of Kalman filter is acquired with the cues of color and motion features. The importance probability density function is generated by extended Kalman filter (EKF), which makes the distribution of particles approximately to the posterior probability. And then, a weighted plan is used to determine the weighted coefficient of LSSVR model, the robustness and sparseness of LSSVR modeling will thereby be enhanced. The updated EKF features of tested samples served as training samples to establish the dynamic LSSVR model real-time in the next frame. Finally, the LSSVR is used to calibrate the tracking results of Kalman particle filter, such that the tracking object will always follow the correct motion trajectory. The experimental results show that our method performs favorably against traditional Kalman particle filter with real-time performance and strong robustness.

Application of least squares support vector regression and linear multiple regression for modeling removal of methyl orange onto tin oxide nanoparticles loaded on activated carbon and activated carbon prepared from Pistacia atlantica wood

Two novel and eco friendly adsorbents namely tin oxide nanoparticles loaded on activated carbon (SnO2-NP-AC) and activated carbon prepared from wood tree Pistacia atlantica (AC-PAW) were used for the rapid removal and fast adsorption of methyl orange (MO) from the aqueous phase. The dependency of MO removal with various adsorption influential parameters was well modeled and optimized using multiple linear regressions (MLR) and least squares support vector regression (LSSVR). The optimal parameters for the LSSVR model were found based on γ value of 0.76 and σ2 of 0.15. For testing the data set, the mean square error (MSE) values of 0.0010 and the coefficient of determination (R2) values of 0.976 were obtained for LSSVR model, and the MSE value of 0.0037 and the R2 value of 0.897 were obtained for the MLR model. The adsorption equilibrium and kinetic data was found to be well fitted and in good agreement with Langmuir isotherm model and second-order equation and intra-particle diffusion models respectively. The small amount of the proposed SnO2-NP-AC and AC-PAW (0.015g and 0.08g) is applicable for successful rapid removal of methyl orange (>95%). The maximum adsorption capacity for SnO2-NP-AC and AC-PAW was 250mgg−1 and 125mgg−1 respectively.

Solar power output forecasting using evolutionary seasonal decomposition least-square support vector regression

Renewable power output is an important factor in scheduling and for improving balanced area control performance. This investigation develops an evolutionary seasonal decomposition least-square support vector regression (ESDLS-SVR) to forecast monthly solar power output. The construction of the ESDLS-SVR uses seasonal decomposition and least-square support vector regression (LS-SVR). Genetic algorithms (GA) are used simultaneously to select the parameters of the LS-SVR. Monthly solar power output data from Taiwan Power Company are used. Empirical results indicate that the proposed forecasting system demonstrates a superior performance in terms of forecasting accuracy. A comparative study has been introduced showing that the ESDLS-SVR model performance is better than autoregressive integrated moving average (ARIMA), seasonal autoregressive integrated moving average (SARIMA), generalized regression neural network (GRNN) and LS-SVR models.

Prediction of Pollutant Emissions of Biomass Flames Through Digital Imaging, Contourlet Transform, and Support Vector Regression Modeling

This paper presents a method for the prediction of NO x emissions in a biomass combustion process through the combination of flame radical imaging, contourlet transform and Zernike moment (CTZM), and least squares support vector regression (LS-SVR) modeling. A novel feature extraction technique based on the CTZM algorithm is developed. The contourlet transform provides the multiscale decomposition for flame radical images and the selected operator based on Zernike moments is designed to provide the well-defined structure for the images. The resulted image features are a variable structure, which is originated from the CTZM. Finally, the variable features of the images of four flame radicals (OH*, CN*, CH*, and $\text{C}^{\ast }_{2}$ ) are defined. The relationship between the variable features of radical images and NO x emissions is established through radial basis function network modeling, SVR modeling, and the LS-SVR modeling. A comparison between the three modeling approaches shows that the LS-SVR model outperforms the other two methods in terms of root-mean-square error and mean relative error criteria. In addition, the structure of the image features has a significant impact on the performance of the prediction models. The test results obtained on a biomass-gas fired test rig show the effectiveness of the proposed technical approach for the prediction of NO x emissions.

Streamflow Forecasting and Estimation Using Least Square Support Vector Regression and Adaptive Neuro-Fuzzy Embedded Fuzzy c-means Clustering

This paper investigates the ability of least square support vector regression (LSSVR) and adaptive neuro-fuzzy embedded fuzzy c-means clustering (ANFIS-FCM) in forecasting and estimation of monthly streamflows. In the first part of the study, the LSSVR and ANFIS-FCM models were tested in 1-month ahead streamflow forecasting by using cross-validation method. Monthly streamflow data belonging to two stations, Besiri Station on Garzan Stream and Baykan Station on Bitlis Stream, in Dicle Basin of Turkey were used. The LSSVR and ANFIS-FCM results were compared with autoregressive moving average (ARMA) models. It was found that the LSSVR models performed better than the ANFIS-FCM and ARMA models in 1-month ahead streamflow forecasting. The ANFIS-FCM models are also found to be better than the ARMA models. The effect of periodicity on forecasting performance of the LSSVR models was also investigated. Adding periodicity component as input to the LSSVR models significantly improved the models’ accuracy in forecasting. In the second part of the study, the accuracy of the LSSVR and ANFIS-FCM models was tested in streamflow estimation using data from nearby stream. Based on the results, the LSSVR was found to be better than the ANFIS-FCM and successfully used in estimating monthly streamflows by using nearby station data.

A hybrid grid-GA-based LSSVR learning paradigm for crude oil price forecasting

In order to effectively model crude oil spot price with inherently high complexity, a hybrid learning paradigm integrating least squares support vector regression (LSSVR) with a hybrid optimization searching approach for the parameters selection in the LSSVR [consisting of grid method and genetic algorithm (GA)], i.e., a hybrid grid-GA-based LSSVR model, is proposed in this study. In the proposed hybrid learning paradigm, the grid method, a simple but efficient searching method, is first applied to roughly but rapidly determine the proper boundaries of the parameters in the LSSVR; then, the GA, an effective and powerful intelligent searching algorithm, is further implemented to select the most suitable parameters. For illustration and verification, the proposed learning paradigm is used to predict the crude oil spot prices of the West Texas Intermediate and the Brent markets. The empirical results demonstrate that the proposed hybrid grid-GA-based LSSVR learning paradigm can outperform its benchmarking models (including some popular forecasting techniques and similar LSSVRs with other parameter searching algorithms) in terms of both prediction accuracy and time-savings, indicating that it can be utilized as one effective forecasting tool for crude oil price with high volatility and irregularity.

A novel hybrid FA-Based LSSVR learning paradigm for hydropower consumption forecasting

Due to the nonlinearity and nonstationary of hydropower market data, a novel hybrid learning paradigm is proposed to predict hydropower consumption, by incorporating firefly algorithm (FA) into least square support vector regression (LSSVR), i.e., FA-based LSSVR model. In the novel model, the powerful and effective artificial intelligence (AI) technique, i.e., LSSVR, is employed to forecast hydropower consumption. Furthermore, a promising AI optimization tool, i.e., FA, is especially introduced to address the crucial but difficult task of parameters determination in LSSVR (e.g., hyper and kernel function parameters). With the Chinese hydropower consumption as sample data, the empirical study has statistically confirmed the superiority of the novel FA-based LSSVR model to other benchmark models (including existing popular traditional econometric models, AI models and similar hybrid LSSVRs with other popular parameter searching tools), in terms of level and directional accuracy. The empirical results also imply that the hybrid FA-based LSSVR learning paradigm with powerful forecasting tool and parameters optimization method can be employed as an effective forecasting tool for not only hydropower consumption but also other complex data.