Local Support Vector Regression Research Articles

A general procedure for evaluating models and ensemble Support Vector Regression

In practice, we may want to discover if there is a relationship between a response variable as a function of the predictor variables. Multiple linear regression (MLR) is a popular tool for such purpose. When the relationship is nonlinear, nonparametric regression methods such as local linear regression, smoothing splines, and support vector regression (SVR) provide flexible alternatives to MLR. How do we compare the performance of these methods and choose an appropriate one for use? In this research, we propose a general procedure to evaluate the performance of different regression methods for use in large data. We also propose an ensemble SVR for regression analysis. The proposed methods are applied to address research questions using the Research and Development Survey, conducted by the National Center for Health Statistics.

Measurement Error Prediction of Power Metering Equipment Using Improved Local Outlier Factor and Kernel Support Vector Regression

The measurement error evaluation of power metering equipment (PME) is significant for the instrument design and accurate metering of electric energy, especially under extreme environmental stresses. However, actual measurement error assessment is often disturbed by the environmental noise and insufficient input information. To address this problem, an improved local outlier factor (ILOF) method is first presented to detect potential outliers. And an optimized distance function and adaptive threshold constraint method based on box plot are used to improve the outlier detection performance of ILOF. Next, an error prediction method, namely kernel support vector regression (KSVR), is presented to fuse measurement error and multiple extreme environmental stresses by using the proposed kernel approach. Integrating the ILOF and KSVR, examples from the extreme environmental region demonstrate that the proposed evaluation framework has a higher assessment performance. Compared with several state-of-art prediction methods, our framework has profound outlier identification and error prediction performance under small sample conditions.

High Accuracy Behavior Prediction of Nonlinear Dynamic System with Semi-Parametric Model-Based Signal Separation

The behavior prediction of nonlinear dynamic system is a challenging problem, especially when the system includes many independent subsystems. The observations from the complex dynamic system are the result of the interaction of multiple dynamic subsystems, which results in a loss of predictability. In this paper, semi-parametric model-based signal separation technique, in which validity function with penalizing is used to estimate the component number of the Gaussian mixture model (GMM) for every hidden source signal, is adopted to separate the observations of complex nonlinear dynamic system in order to improve its predictability. Then local support vector regression (SVR) technique is used to model the separated observations and make prediction. Finally, the prediction results are remixed as the original observation prediction or the behavior prediction of the complex nonlinear dynamic system. The experimental results show that the proposed method can separate the observation of the complex dynamic system robustly, improve the prediction accuracy substantially and perform better than the other comparison methods.

Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models

SummaryIn many supervised learning applications, understanding and visualizing the effects of the predictor variables on the predicted response is of paramount importance. A shortcoming of black box supervised learning models (e.g. complex trees, neural networks, boosted trees, random forests, nearest neighbours, local kernel-weighted methods and support vector regression) in this regard is their lack of interpretability or transparency. Partial dependence plots, which are the most popular approach for visualizing the effects of the predictors with black box supervised learning models, can produce erroneous results if the predictors are strongly correlated, because they require extrapolation of the response at predictor values that are far outside the multivariate envelope of the training data. As an alternative to partial dependence plots, we present a new visualization approach that we term accumulated local effects plots, which do not require this unreliable extrapolation with correlated predictors. Moreover, accumulated local effects plots are far less computationally expensive than partial dependence plots. We also provide an R package ALEPlot as supplementary material to implement our proposed method.

Decision trees using local support vector regression models for large datasets

ABSTRACTOur proposed decision trees using local support vector regression models (tSVR, rtSVR) aim to efficiently handle the regression task for large datasets. The learning algorithm tSVR of regression models is done by two main steps. The first one is to construct a decision tree regressor for partitioning the full training dataset into k terminal-nodes (subsets), followed which the second one is to learn the SVR model from each terminal-node to predict the data locally in a parallel way on multi-core computers. The algorithm rtSVR learns the random forest of decision trees with local SVR models for improving the prediction correctness against the tSVR model alone. The performance analysis shows that our algorithms tSVR, rtSVR are efficient in terms of the algorithmic complexity and the generalization ability compared to the classical SVR. The experimental results on five large datasets from UCI repository showed that proposed tSVR and rtSVR algorithms are faster than the standard SVR in training the non-linear regression model from large datasets while achieving the high correctness in the prediction. Typically, the average training time of tSVR and rtSVR are 1282.66 and 482.29 times faster than the standard SVR; Furthermore, tSVR and rtSVR improve 59.43%, 63.70% of the relative prediction correctness compared to the standard SVR.

Forecasting wildfire disease on tobacco: toward developing a high-accuracy prediction model for disease index using local climate factors and support vector regression

Tobacco wildfire disease is common globally, and climate change may increase the risk of outbreaks. Therefore, there is an urgent need to establish an effective climate model to forecast the occurrence of wildfire disease. To design such a model, we collected data for 40 wildfire disease indices via tobacco field surveys and data for 15 climate factors of Guiyang County in China from 2012 to 2016. First, we built multiple linear regression (MLR), stepwise linear regression (SLR) and support vector regression (SVR) models using three climate features (precipitation, mean daily temperature and sunshine duration), and we could not find an effective model. Second, we built three corresponding models using expanded 15 climate features and an in-house WDEM method (the worst descriptor elimination multi-roundly), and the independent test results showed that the best SVR model had not only a higher predictive accuracy ( $$ {Q}_{ext}^2 $$ = 0.94) but also a better stability. Finally, we further evaluated the biological significance of their retained climate features and the single-factor effects of the best model according to the interpretability analysis, and our results indicated that (1) the three climate factors (minimum value of wind velocity, daily range of temperature and daily pressure) strongly affected the occurrence of wildfire disease; (2) the ranges of relative humidity and sunshine hours were negatively correlated with the occurrence of wildfire disease, while daily mean vapour pressure was positively correlated with the occurrence of the disease. Our work enables a useful theoretical prediction for wildfire disease, especially in terms of climate-related predictions.

Prediction of hourly PM2.5 using a space-time support vector regression model

Real-time air quality prediction has been an active field of research in atmospheric environmental science. The existing methods of machine learning are widely used to predict pollutant concentrations because of their enhanced ability to handle complex non-linear relationships. However, because pollutant concentration data, as typical geospatial data, also exhibit spatial heterogeneity and spatial dependence, they may violate the assumptions of independent and identically distributed random variables in most of the machine learning methods. As a result, a space-time support vector regression model is proposed to predict hourly PM2.5 concentrations. First, to address spatial heterogeneity, spatial clustering is executed to divide the study area into several homogeneous or quasi-homogeneous subareas. To handle spatial dependence, a Gauss vector weight function is then developed to determine spatial autocorrelation variables as part of the input features. Finally, a local support vector regression model with spatial autocorrelation variables is established for each subarea. Experimental data on PM2.5 concentrations in Beijing are used to verify whether the results of the proposed model are superior to those of other methods.

PM2.5 forecasting with hybrid LSE model‐based approach

SummaryPM2.5 time series have the features of non‐stationary and nonlinear. Existing forecasting methods for PM2.5 cannot achieve high accuracy for they have ignored the potential characteristics of PM2.5 time series. Aiming at this problem, a hybrid approach using local mean decomposition and Support Vector Regression (SVR)‐Elman (LSE) is firstly proposed in this paper to analyse 5days ahead PM2.5 concentrations for forecasting in Wuhan, China: (1) the meaningful PF1‐PF5 components are extracted from original PM2.5 time series by local mean decomposition; (2) the first high‐frequency product function is managed by using the SVR model, such that the relationship between PM2.5 and other air quality data can be revealed accurately; (3) the other components are trained by Elman model with the sliding window method. Experimental results show that, compared with multiple linear regression, autoregressive integrated moving average, BP neural network, and SVR models, the proposed hybrid LSE model‐based approach exhibits the best performance in terms of R2, MAE, MAPE, RMSE, while it is applied for forecasting in real datasets. Copyright © 2016 John Wiley & Sons, Ltd.

Multi-model adaptive soft sensor modeling method using local learning and online support vector regression for nonlinear time-variant batch processes

Batch processes are often characterized by inherent nonlinearity, multiplicity of operating phases, and batch-to-batch variations, which poses great challenges for accurate and reliable online prediction of soft sensor. Especially, the soft sensor built with old data may encounter performance deterioration due to a failure of capturing the time-variant behaviors of batch processes, thus adaptive strategies are necessary. Unfortunately, conventional adaptive soft sensors cannot efficiently account for the within-batch as well as between-batch time-variant changes in batch process characteristics, which results in poor prediction accuracy. Therefore, a novel multi-model adaptive soft sensor modeling method is proposed based on the local learning framework and online support vector regression (OSVR) for nonlinear time-variant batch processes. First, a batch process is identified with a set of local domains and then the localized OSVR models are built for all isolated domains. Further, the estimation for a query data is obtained by adaptively combining multiple local models that perform best on the similar samples to the query point. The proposed multi-model OSVR (MOSVR) method provides four types of adaptation strategies: (i) adaptive combination based on Bayesian ensemble learning; (ii) online offset compensation; (iii) incremental updating of local models; and (iv) database updating. The effectiveness of the MOSVR approach and its superiority over traditional adaptive soft sensors in dealing with the within-batch and between-batch shifting dynamics is demonstrated through a simulated fed-batch penicillin fermentation process as well as an industrial fed-batch chlortetracycline fermentation process.

Grey relational grade in local support vector regression for financial time series prediction

Support vector regression (SVR) has often been applied in the prediction of financial time series with many characteristics. On account of much time consumption of global SVR, local machines are carried out to accelerate the computation. In this paper, we introduce local grey SVR (LG-SVR) integrated grey relational grade with local SVR for financial time series forecasting. Pattern search method and leave-one-out errors are adopted for model selection. Experimental results of three real financial time series prediction demonstrate that LG-SVR can speed up computing speed and improve prediction accuracy.

Image Quality Enhancement for Single-Image Super Resolution Based on Local Similarities and Support Vector Regression

In reconstruction-based super resolution, a high-resolution image is estimated using multiple low-resolution images with sub-pixel misalignments. Therefore, when only one low-resolution image is available, it is generally difficult to obtain a favorable image. This letter proposes a method for overcoming this difficulty for single- image super resolution. In our method, after interpolating pixel values at sub-pixel locations on a patch-by-patch basis by support vector regression, in which learning samples are collected within the given image based on local similarities, we solve the regularized reconstruction problem with a sufficient number of constraints. Evaluation experiments were performed for artificial and natural images, and the obtained high-resolution images indicate the high-frequency components favorably along with improved PSNRs.

Electric Load Forecasting Based on Locally Weighted Support Vector Regression

The forecasting of electricity demand has become one of the major research fields in electrical engineering. Accurately estimated forecasts are essential part of an efficient power system planning and operation. In this paper, a modified version of the support vector regression (SVR) is presented to solve the load forecasting problem. The proposed model is derived by modifying the risk function of the SVR algorithm with the use of locally weighted regression (LWR) while keeping the regularization term in its original form. In addition, the weighted distance algorithm based on the Mahalanobis distance for optimizing the weighting function's bandwidth is proposed to improve the accuracy of the algorithm. The performance of the new model is evaluated with two real-world datasets, and compared with the local SVR and some published models using the same datasets. The results show that the proposed model exhibits superior performance compare to that of LWR, local SVR, and other published models.

A hybrid model coupled with singular spectrum analysis for daily rainfall prediction

A hybrid model integrating artificial neural networks and support vector regression was developed for daily rainfall prediction. In the modeling process, singular spectrum analysis was first adopted to decompose the raw rainfall data. Fuzzy C-means clustering was then used to split the training set into three crisp subsets which may be associated with low-, medium- and high-intensity rainfall. Two local artificial neural network models were involved in training and predicting low- and medium-intensity subsets whereas a local support vector regression model was applied to the high-intensity subset. A conventional artificial neural network model was selected as the benchmark. The artificial neural network with the singular spectrum analysis was developed for the purpose of examining the singular spectrum analysis technique. The models were applied to two daily rainfall series from China at 1-day-, 2-day- and 3-day-ahead forecasting horizons. Results showed that the hybrid support vector regression model performed the best. The singular spectrum analysis model also exhibited considerable accuracy in rainfall forecasting. Also, two methods to filter reconstructed components of singular spectrum analysis, supervised and unsupervised approaches, were compared. The unsupervised method appeared more effective where nonlinear dependence between model inputs and output can be considered.

Microarray missing values imputation methods: Critical analysis review

Gene expression data often contain missing expression values. For the purpose of conducting an effective clustering analysis and since many algorithms for gene expression data analysis require a complete matrix of gene array values, choosing the most effective missing value estimation method is necessary. In this paper, the most commonly used imputation methods from literature are critically reviewed and analyzed to explain the proper use, weakness and point the observations on each published method. From the conducted analysis, we conclude that the Local Least Square (LLS) and Support Vector Regression (SVR) algorithms have achieved the best performances. SVR can be considered as a complement algorithm for LLS especially when applied to noisy data. However, both algorithms suffer from some deficiencies presented in choosing the value of Number of Selected Genes (K) and the appropriate kernel function. To overcome these drawbacks, the need for new method that automatically chooses the parameters of the function and it also has an appropriate computational complexity is imperative.

Prediction of chemical toxicity with local support vector regression and activity-specific kernels

We propose a new kernel, based on 2-D structural chemical similarity, that integrates activity-specific information from the training data, and a new approach to applicability domain estimation that takes feature significances and activity distributions into consideration. The new kernel provides superior results than the well-established Tanimoto kernel, and activity-sensitive feature selection enhances prediction quality. Validation of local support vector regression models based on this kernel has been preformed with three publicly available datasets from the DSSTox project. One of them (Fathead Minnow Acute Toxicity) has been already modelled by other groups, and serves as a benchmark dataset, the other two (Maximum Recommended Therapeutic Dose, IRIS Lifetime Cancer Risk) have been modelled for the first time according to the knowledge of the authors. For all three models predictive accuracies increase with the prediction confidences that indicate the applicability domain. Depending on the confidence cutoff for acceptable predictions we were able to achieve > 90% predictions within 1 log unit of the experimental data for all datasets.

River stage prediction based on a distributed support vector regression

An accurate and timely prediction of river flow flooding can provide time for the authorities to take pertinent flood protection measures such as evacuation. Various data-derived models including LR (linear regression), NNM (the nearest-neighbor method) ANN (artificial neural network) and SVR (support vector regression), have been successfully applied to water level prediction. Of them, SVR is particularly highly valued, because it has the advantage over many data-derived models in overcoming overfitting of training data. However, SVR is computationally time-consuming when used to solve large-size problems. In the context of river flow prediction, equipped with LR model as a benchmark and genetic algorithm-based ANN (ANN-GA) and NNM as counterparts, a novel distributed SVR (D-SVR) model is proposed in this study. It implements a local approximation to training data because partitioned original training data are independently fitted by each local SVR model. ANN-GA and LR models are also used to help determine input variables. A two-step GA algorithm is employed to find the optimal triplets ( C , ε , σ ) for D-SVR model. The validation results reveal that the proposed D-SVR model can carry out the river flow prediction better in comparison with others, and dramatically reduce the training time compared with the conventional SVR model. The pivotal factor contributing to the performance of D-SVR may be that it implements a local approximation method and the principle of structural risk minimization.

Fuzzy Weighted Support Vector Regression With a Fuzzy Partition

The problem of the traditional support vector regression (SVR) approach, referred to as the global SVR approach, is the incapability of interpreting local behavior of the estimated models. An approach called the local SVR approach was proposed in the literature to cope with this problem. Although the local SVR approach can indeed model local behavior of models better than the global SVR approach does, the local SVR approach still has the problem of boundary effects, which may generate a large bias at the boundary and also need more time to calculate. In this paper, the fuzzy weighted SVR with a fuzzy partition is proposed. Because the concept of locally weighted regression is not used in the proposed approach, the boundary effects will not appear. The proposed method first employs the fuzzy c-mean clustering algorithm to split training data into several training subsets. Then, the local-regression models (LRMs) are independently obtained by the SVR approach for each training subset. Finally, those LRMs are combined by a fuzzy weighted mechanism to form the output. Experimental results show that the proposed approach needs less computational time than the local SVR approach and can have more accurate results than the local/global SVR approaches does.