Regression Datasets Research Articles

Kalman filter-based method for Online Sequential Extreme Learning Machine for regression problems

In this paper, a new sequential learning algorithm is constructed by combining the Online Sequential Extreme Learning Machine (OS-ELM) and Kalman filter regression. The Kalman Online Sequential Extreme Learning Machine (KOSELM) handles the problem of multicollinearity of the OS-ELM, which can generate poor predictions and unstable models. The KOSELM learns the training data one-by-one or chunk-by-chunk by adjusting the variance of the output weights through the Kalman filter. The performance of the proposed algorithm has been validated on benchmark regression datasets, and the results show that KOSELM can achieve a higher learning accuracy than OS-ELM and its related extensions. A statistical validation for the differences of the accuracy for all algorithms is performed, and the results confirm that KOSELM has better stability than ReOS-ELM, TOSELM and LS-IELM.

A probabilistic learning algorithm for robust modeling using neural networks with random weights

Robust modeling approaches have received considerable attention due to its practical value to deal with the presence of outliers in data. This paper proposes a probabilistic robust learning algorithm for neural networks with random weights (NNRWs) to improve the modeling performance. The robust NNRW model is trained by optimizing a hybrid regularization loss function according to the sparsity of outliers and compressive sensing theory. The well-known expectation maximization (EM) algorithm is employed to implement our proposed algorithm under some assumptions on noise distribution. Experimental results on function approximation as well as UCI data sets for regression and classification demonstrate that the proposed algorithm is promising with good potential for real world applications.

Very sparse LSSVM reductions for large-scale data.

Least squares support vector machines (LSSVMs) have been widely applied for classification and regression with comparable performance with SVMs. The LSSVM model lacks sparsity and is unable to handle large-scale data due to computational and memory constraints. A primal fixed-size LSSVM (PFS-LSSVM) introduce sparsity using Nyström approximation with a set of prototype vectors (PVs). The PFS-LSSVM model solves an overdetermined system of linear equations in the primal. However, this solution is not the sparsest. We investigate the sparsity-error tradeoff by introducing a second level of sparsity. This is done by means of L0 -norm-based reductions by iteratively sparsifying LSSVM and PFS-LSSVM models. The exact choice of the cardinality for the initial PV set is not important then as the final model is highly sparse. The proposed method overcomes the problem of memory constraints and high computational costs resulting in highly sparse reductions to LSSVM models. The approximations of the two models allow to scale the models to large-scale datasets. Experiments on real-world classification and regression data sets from the UCI repository illustrate that these approaches achieve sparse models without a significant tradeoff in errors.

Multi-task proximal support vector machine

With the explosive growth of the use of imagery, visual recognition plays an important role in many applications and attracts increasing research attention. Given several related tasks, single-task learning learns each task separately and ignores the relationships among these tasks. Different from single-task learning, multi-task learning can explore more information to learn all tasks jointly by using relationships among these tasks. In this paper, we propose a novel multi-task learning model based on the proximal support vector machine. The proximal support vector machine uses the large-margin idea as does the standard support vector machines but with looser constraints and much lower computational cost. Our multi-task proximal support vector machine inherits the merits of the proximal support vector machine and achieves better performance compared with other popular multi-task learning models. Experiments are conducted on several multi-task learning datasets, including two classification datasets and one regression dataset. All results demonstrate the effectiveness and efficiency of our proposed multi-task proximal support vector machine.

Combining a relaxed EM algorithm with Occam’s razor for Bayesian variable selection in high-dimensional regression

We address the problem of Bayesian variable selection for high-dimensional linear regression. We consider a generative model that uses a spike-and-slab-like prior distribution obtained by multiplying a deterministic binary vector, which traduces the sparsity of the problem, with a random Gaussian parameter vector. The originality of the work is to consider inference through relaxing the model and using a type-II log-likelihood maximization based on an EM algorithm. Model selection is performed afterwards relying on Occam’s razor and on a path of models found by the EM algorithm. Numerical comparisons between our method, called spinyReg, and state-of-the-art high-dimensional variable selection algorithms (such as lasso, adaptive lasso, stability selection or spike-and-slab procedures) are reported. Competitive variable selection results and predictive performances are achieved on both simulated and real benchmark data sets. An original regression data set involving the prediction of the number of visitors of the Orsay museum in Paris using bike-sharing system data is also introduced, illustrating the efficiency of the proposed approach. The R package spinyReg implementing the method proposed in this paper is available on CRAN.

Helping predictive analytics interpretation using regression trees and clustering perturbation

Regression trees are helpful tools for decision support and predictive analytics, due to their simple structure and the ease with which they can be obtained from data. Nonetheless, when applied to non-trivial datasets, they tend to grow according to the complexity of the data, becoming difficult to interpret. This difficulty can be overcome by clustering the dataset and representing the regression tree of each cluster independently. In order to help create predictive models that are more comprehensible, we propose in this work a clustering perturbation method to reduce the size of the regression tree obtained from each cluster. A prototype has been developed and tested on several regression datasets.

Effect of credit disbursement on sectoral output of the Indian economy

This paper examines the state-wise differential effect of credit on agriculture, industries and the services sector of the Indian economy using panel regression approach and data sets panelled over 15 of the largest states for a period ranging between 2001 and 2010. The Least Square Dummy Variable (LSDV) estimation approach is employed to capture the state-wise effect of credit on sectoral output. The results of the study reveal that credit has a statistically significant and positive effect on the output of agriculture, industry and services sector of the economy. The results further reveal that in the agricultural sector, irrigation intensity has a significant and positive effect on output. With regard to the industrial sector, the number of workers and working capital are found to have significant and positive impact on output.

A novel multiple instance learning method based on extreme learning machine.

Since real-world data sets usually contain large instances, it is meaningful to develop efficient and effective multiple instance learning (MIL) algorithm. As a learning paradigm, MIL is different from traditional supervised learning that handles the classification of bags comprising unlabeled instances. In this paper, a novel efficient method based on extreme learning machine (ELM) is proposed to address MIL problem. First, the most qualified instance is selected in each bag through a single hidden layer feedforward network (SLFN) whose input and output weights are both initialed randomly, and the single selected instance is used to represent every bag. Second, the modified ELM model is trained by using the selected instances to update the output weights. Experiments on several benchmark data sets and multiple instance regression data sets show that the ELM-MIL achieves good performance; moreover, it runs several times or even hundreds of times faster than other similar MIL algorithms.

Soft-sensing in complex chemical process based on a sample clustering extreme learning machine model

In actual chemical processes, the fact that some essential variables cannot be directly measured makes the production quality out-of-control and even results in large economic losses. In this study, a novel sample clustering extreme learning machine (SC-ELM) model is developed to achieve timely and accurate measurement. SC-ELM is a fast training algorithm with an excellent generalization performance, and the combined sample clustering approach solves the non-optimal input weights of ELM. The network structure is designed by a fast leave-one-out cross-validation (FLOO-CV) method. Meanwhile, the validity of SC-ELM model is firstly tested by two classical regression datasets. With the comparison of other ELM models, SC-ELM is proved to be an effective model in both modeling accuracy and network structure. Then, SC-ELM is applied in measuring the quality index of a high-density polyethylene (HDPE) process running in a chemical plant, and the experiment results demonstrate that SC-ELM model can achieve quality estimation with higher measuring accuracy and less training time.

Fast cross validation for regularized extreme learning machine

A method for fast l-fold cross validation is proposed for the regularized extreme learning machine (RELM). The computational time of fast l-fold cross validation increases as the fold number decreases, which is opposite to that of naive l-fold cross validation. As opposed to naive l-fold cross validation, fast l-fold cross validation takes the advantage in terms of computational time, especially for the large fold number such as l > 20. To corroborate the efficacy and feasibility of fast l-fold cross validation, experiments on five benchmark regression data sets are evaluated.

Parsimonious extreme learning machine using recursive orthogonal least squares.

Novel constructive and destructive parsimonious extreme learning machines (CP- and DP-ELM) are proposed in this paper. By virtue of the proposed ELMs, parsimonious structure and excellent generalization of multiinput-multioutput single hidden-layer feedforward networks (SLFNs) are obtained. The proposed ELMs are developed by innovative decomposition of the recursive orthogonal least squares procedure into sequential partial orthogonalization (SPO). The salient features of the proposed approaches are as follows: 1) Initial hidden nodes are randomly generated by the ELM methodology and recursively orthogonalized into an upper triangular matrix with dramatic reduction in matrix size; 2) the constructive SPO in the CP-ELM focuses on the partial matrix with the subcolumn of the selected regressor including nonzeros as the first column while the destructive SPO in the DP-ELM operates on the partial matrix including elements determined by the removed regressor; 3) termination criteria for CP- and DP-ELM are simplified by the additional residual error reduction method; and 4) the output weights of the SLFN need not be solved in the model selection procedure and is derived from the final upper triangular equation by backward substitution. Both single- and multi-output real-world regression data sets are used to verify the effectiveness and superiority of the CP- and DP-ELM in terms of parsimonious architecture and generalization accuracy. Innovative applications to nonlinear time-series modeling demonstrate superior identification results.

Ordinal regression by a generalized force-based model.

This paper introduces a new instance-based algorithm for multiclass classification problems where the classes have a natural order. The proposed algorithm extends the state-of-the-art gravitational models by generalizing the scaling behavior of the class-pattern interaction force. Like the other gravitational models, the proposed algorithm classifies new patterns by comparing the magnitude of the force that each class exerts on a given pattern. To address ordinal problems, the algorithm assumes that, given a pattern, the forces associated to each class follow a unimodal distribution. For this reason, a weight matrix that allows to modify the metric in the attributes space and a vector of parameters that allows to modify the force law for each class have been introduced in the model definition. Furthermore, a probabilistic formulation of the error function allows the estimation of the model parameters using global and local optimization procedures toward minimization of the errors and penalization of the non unimodal outputs. One of the strengths of the model is its competitive grade of interpretability which is a requisite in most of real applications. The proposed algorithm is compared to other well-known ordinal regression algorithms on discretized regression datasets and real ordinal regression datasets. Experimental results demonstrate that the proposed algorithm can achieve competitive generalization performance and it is validated using nonparametric statistical tests.

Confidence-weighted extreme learning machine for regression problems

Abstract Based on Gaussian margin machine (GMM) and extreme learning machine (ELM), confidence-weighted ELM (CW-ELM) is proposed to provide point forecasts and confidence intervals. CW-ELM maintains a multivariate normal distribution over the output weight vector. It is applied to seek the least informative distribution from those that keep the targets within the forecast confidence intervals. For simplicity, the covariance matrix is assumed to be diagonal. The simplified problem of CW-ELM is approximately solved by using Leave-One-Out-Incremental ELM (LOO-IELM) and the interior point method. Our experimental results on both synthetic and real-world regression datasets demonstrate that CW-ELM has better performance than Bayesian ELM and Gaussian process regression.

A robust least squares support vector machine for regression and classification with noise

Least squares support vector machines (LS-SVMs) are sensitive to outliers or noise in the training dataset. Weighted least squares support vector machines (WLS-SVMs) can partly overcome this shortcoming by assigning different weights to different training samples. However, it is a difficult task for WLS-SVMs to set the weights of the training samples, which greatly influences the robustness of WLS-SVMs. In order to avoid setting weights, in this paper, a novel robust LS-SVM (RLS-SVM) is presented based on the truncated least squares loss function for regression and classification with noise. Based on its equivalent model, we theoretically analyze the reason why the robustness of RLS-SVM is higher than that of LS-SVMs and WLS-SVMs. In order to solve the proposed RLS-SVM, we propose an iterative algorithm based on the concave–convex procedure (CCCP) and the Newton algorithm. The statistical tests of the experimental results conducted on fourteen benchmark regression datasets and ten benchmark classification datasets show that compared with LS-SVMs, WLS-SVMs and iteratively reweighted LS-SVM (IRLS-SVM), the proposed RLS-SVM significantly reduces the effect of the noise in the training dataset and provides superior robustness.

A new online data imputation method based on general regression auto associative neural network

In this paper we proposed online, offline and semi-online data imputation models based on the four auto associative neural networks. The online model employs mean imputation followed by general regression auto associative neural network (GRAANN). The offline methods include mean imputation followed by particle swarm optimization based auto associative neural network (PSOAANN); mean imputation followed by particle swarm optimization based auto associative wavelet neural network (PSOAAWNN) and the semi-online method involving mean imputation followed by radial basis function auto associative neural network (RBFAANN). We compared the performance of these hybrid models with that of mean imputation and a hybrid imputation method viz, K-means and multi-layer perceptron (MLP) of Ankaiah and Ravi (2011) [65]. We tested the effectiveness of these models on four benchmark classification and four benchmark regression datasets; three bankruptcy prediction datasets and one credit scoring datasets under 10-fold cross-validation testing. From the experiments, we observed that the GRAANN yielded better imputation for the missing values than the rest of the models. We confirmed this by performing the Wilcoxon signed rank test to test the statistical significance between the methods proposed. It turned out that GRAANN outperformed other models in most of the datasets.

Online probabilistic learning for fuzzy inference system

Online learning is a key methodology for expert systems to gracefully cope with dynamic environments. In the context of neuro-fuzzy systems, research efforts have been directed toward developing online learning methods that can update both system structure and parameters on the fly. However, the current online learning approaches often rely on heuristic methods that lack a formal statistical basis and exhibit limited scalability in the face of large data stream. In light of these issues, we develop a new Sequential Probabilistic Learning for Adaptive Fuzzy Inference System (SPLAFIS) that synergizes the Bayesian Adaptive Resonance Theory (BART) and Rule-Wise Decoupled Extended Kalman Filter (RDEKF) to generate the rule base structure and refine its parameters, respectively. The marriage of the BART and RDEKF methods, both of which are built upon the maximum a posteriori (MAP) principle rooted in the Bayes’ rule, offers a comprehensive probabilistic treatment and an efficient way for online structural and parameter learning suitable for large, dynamic data stream. To manage the model complexity without sacrificing its predictive accuracy, SPLAFIS also includes a simple procedure to prune inconsequential rules that have little contribution over time. The predictive accuracy, structural simplicity, and scalability of the proposed model have been exemplified in empirical studies using chaotic time series, stock index, and large nonlinear regression datasets.

Dynamical Granular Support Vector Regression Machine

PDF HTML阅读 XML下载 导出引用 引用提醒 动态粒度支持向量回归机 DOI: 10.3724/SP.J.1001.2013.04472 作者: 作者单位: 作者简介: 通讯作者: 中图分类号: 基金项目: 国家自然科学基金(60975035,61273291);山西省回国留学人员科研基金(2012008);山西省研究生教育创新项目(20133001) Dynamical Granular Support Vector Regression Machine Author: Affiliation: Fund Project: 摘要 | 图/表 | 访问统计 | 参考文献 | 相似文献 | 引证文献 | 资源附件 | 文章评论 摘要:粒度支持向量机(granular support vector machine,简称GSVM)可以有效提高支持向量机(support vectormachine,简称SVM)的学习效率,但由于经典GSVM 通常将粒用个别样本替代,且粒划和学习在不同空间进行,因而不可避免地改变了原始数据分布,从而可能导致泛化能力降低.针对这一问题,通过引入动态层次粒划的方法,设计了动态粒度支持向量回归(dynamical granular support vector regression,简称DGSVR)模型.该方法首先将训练样本映射到高维空间,使得在低维样本空间无法直接得到的分布信息显示出来,并在该特征空间中进行初始粒划.然后,通过衡量样本粒与当前回归超平面的距离,找到含有较多回归信息的粒,并通过计算其半径和密度进行深层次的动态粒划.如此循环迭代,直到没有信息粒需要进行深层粒划时为止.最后,通过动态粒划过程得到的不同层次的粒进行回归训练,在有效压缩训练集的同时,尽可能地使含有重要信息的样本在最终训练集中保留下来.在基准函数数据集及UCI 上的回归数据集上的实验结果表明,DGSVR 方法能够以较快的速度完成动态粒划的过程并收敛,在保持较高训练效率的同时可有效提高传统粒度支持向量回归机(granular support vector regression machine,简称GSVR)的泛化性能. Abstract:Although granular support vector machine (GSVM) can improve the learning speed, the generalization performance may be decreased because the original data distribution will be changed inevitably by two reasons: (1) A granule is usually replaced by individual datum; (2) Granulation and learning are carried out in different spaces. To address this problem, this study presents a granular support vector regression (SVR) model based on dynamical granulation, namely DGSVR, by using the dynamical hierarchical granulation method. With DGSVR, the original data are mapped into the high-dimensional space by mercer kernel to reveal the distribution features implicit in original sample space, and the data are divided into some granules initially. Then, some granules are obtained with important regression information by measuring the distances of granules and regression hyperplane. By computing the radius and density of granules, the deep dynamical granulation process executes until there are no informational granules need to be granulated. Finally, those granules in different granulation levels are extracted and trained by SVR. The experimental results on benchmark function datasets and UCI regression datasets demonstrate that the DGSVR model can quickly finish the dynamical granulation process and is convergent. It concludes this model can improve the generalization performance and achieve high learning efficiency at the same time. 参考文献 相似文献 引证文献

Adaptive Linear and Normalized Combination of Radial Basis Function Networks for Function Approximation and Regression

This paper presents a novel adaptive linear and normalized combination (ALNC) method that can be used to combine the component radial basis function networks (RBFNs) to implement better function approximation and regression tasks. The optimization of the fusion weights is obtained by solving a constrained quadratic programming problem. According to the instantaneous errors generated by the component RBFNs, the ALNC is able to perform the selective ensemble of multiple leaners by adaptively adjusting the fusion weights from one instance to another. The results of the experiments on eight synthetic function approximation and six benchmark regression data sets show that the ALNC method can effectively help the ensemble system achieve a higher accuracy (measured in terms of mean-squared error) and the better fidelity (characterized by normalized correlation coefficient) of approximation, in relation to the popular simple average, weighted average, and the Bagging methods.

Estimating bootstrap and Bayesian prediction intervals for constituent load rating curves

[1] Assessment of constituent loads in rivers is essential to evaluate water quality of streams and estuaries; however, uncertainty in load estimation may be large and must be considered and communicated together with estimates. In this comparative study, the usefulness of two existing methods (bootstrap and Bayesian inference) to assess uncertainty in constituent loads estimated with an improved eight-parameter rating curve is demonstrated. Bootstrap prediction intervals and Bayesian credible intervals were estimated for daily and monthly loads obtained with a rating curve applied to routine monitoring sampling data sets of nitrate (NO3-N), reactive phosphorus (RP), and total phosphorus (TP) of the Duck River, in Tasmania (Australia). Predicted loads and prediction intervals were compared to benchmark loads obtained by an independent, high frequency monitoring program. The eight-parameter rating curve resulted in better prediction of NO3-N and TP than RP loads. Both inference methods successfully generated prediction intervals. The bracketing frequency (i.e., the fraction of prediction intervals that comprised benchmark loads) of bootstrap prediction intervals was 50–65% of daily or monthly benchmark loads. Bracketing frequency of Bayesian credible intervals was consistently higher, and included 74–85% of benchmark daily loads and 80% or more of benchmark monthly loads. Both methods proved to be robust to the presence of an artificial outlier. Prediction intervals were affected by the distribution of the regression error, hence they reflected uncertainty in the regression data set and limitations in the rating curve formulation. They did not account for other sources of uncertainty, i.e., they were still conservative predictions of load uncertainty.

Meta-ELM: ELM with ELM hidden nodes

Extreme Leaning Machine (ELM) simply randomly assigns input weights and biases, ineluctably leading to certain stochastic behaviors and reducing generalization performance. In this paper, we propose a meta-learning model of ELM, called Meta-ELM. The Meta-ELM architecture consists of several base ELMs and one top ELM. Therefore, the Meta-ELM learning proceeds in two stages. First, each base ELM is trained on a subset of the training data. Then, the top ELM is learned with the base ELMs as hidden nodes. Theoretical analysis and experimental results on a few artificial and benchmark regression datasets show that the proposed Meta-ELM model is feasible and effective.