Empirical Risk Minimization Principle Research Articles

Online Learning from Data Streams with Varying Feature Spaces

We study the problem of online learning with varying feature spaces. The problem is challenging because, unlike traditional online learning problems, varying feature spaces can introduce new features or stop having some features without following a pattern. Other existing methods such as online streaming feature selection (Wu et al. 2013), online learning from trapezoidal data streams (Zhang et al. 2016), and learning with feature evolvable streams (Hou, Zhang, and Zhou 2017) are not capable to learn from arbitrarily varying feature spaces because they make assumptions about the feature space dynamics. In this paper, we propose a novel online learning algorithm OLVF to learn from data with arbitrarily varying feature spaces. The OLVF algorithm learns to classify the feature spaces and the instances from feature spaces simultaneously. To classify an instance, the algorithm dynamically projects the instance classifier and the training instance onto their shared feature subspace. The feature space classifier predicts the projection confidences for a given feature space. The instance classifier will be updated by following the empirical risk minimization principle and the strength of the constraints will be scaled by the projection confidences. Afterwards, a feature sparsity method is applied to reduce the model complexity. Experiments on 10 datasets with varying feature spaces have been conducted to demonstrate the performance of the proposed OLVF algorithm. Moreover, experiments with trapezoidal data streams on the same datasets have been conducted to show that OLVF performs better than the state-of-the-art learning algorithm (Zhang et al. 2016).

An efficient regularized K-nearest neighbor structural twin support vector machine

K-nearest neighbor based structural twin support vector machine (KNN-STSVM) performs better than structural twin support vector machine (S-TSVM). It applies the intra-class KNN method, and different weights are given to the samples in one class to strengthen the structural information. For the other class, the redundant constraints are deleted by the inter-class KNN method to speed up the training process. However, the empirical risk minimization principle is implemented in the KNN-STSVM, so it easily leads to over-fitting and reduces the prediction accuracy of the classifier. To enhance the generalization ability of the classifier, we propose an efficient regularized K-nearest neighbor structural twin support vector machine, called RKNN-STSVM, by introducing a regularization term into the objective function. So there are two parts in the objective function, one of which is to maximize the margin between the two parallel hyper-planes, and the other one is to minimize the training errors of two classes of samples. Therefore the structural risk minimization principle is implemented in our RKNN-STSVM. Besides, a fast DCDM algorithm is introduced to handle relatively large-scale problems more efficiently. Comprehensive experimental results on twenty-seven benchmark datasets and two popular image datasets demonstrate the efficiency of our proposed RKNN-STSVM.

Robust training of neural network via minimizing robust estimates of the average of loss functions

The paper suggests an extended version of principle of empirical risk minimization and principle of smoothly winsorized averages minimization for robust neural networks learning. It’s based on using of M-averaging and WM-averaging functions instead of the arithmetic mean for empirical risk estimation. These approaches generalize robust algorithms based on using median and quantiles for estimation of mean losses. An iteratively reweighted schema for minimization of differentiable robust estimations of the averages of loss functions is proposed. This schema allows to use weighted version of traditional back-propagation algorithms for neural networks learning in presence of outliers.

Support vector regression based metamodeling for structural reliability analysis

Various metamodeling approaches e.g. polynomial response surface method artificial neural network, Kriging method etc. have been emerged as an effective alternative for solving computationally challenging complex reliability analysis problems involving finite element response analysis. However, such approaches are primarily based on the principle of empirical risk minimization. The support vector machine for regression i.e. the support vector regression (SVR) which is based on structural risk minimization has revealed improved abilities of response approximation using small sample learning. The implementation of SVR model requires to optimize a loss function involving the loss function parameter, regularization parameter and also kernel function parameter(s) to tackle nonlinear problems. The success of SVR largely depends on proper choice of such parameters. A simple yet effective algorithm by solving an optimization sub-problem to minimize the mean square error value obtained by cross-validation method is investigated in the present study to construct SVR model for structural reliability analysis. The effectiveness of the algorithm is demonstrated numerically by comparing various computed statistical metrics obtained by the most accurate direct Monte Carlo Simulation (MCS) technique. The performance of SVR based metamodel to estimate the reliability is also studied by comparing the results with the direct MCS based results.

A Classification Learning Research based on Discriminative Knowledge-Leverage Transfer

Current transfer learning models study the source data for future target inferences within a major view, the whole source data should be used to explore the shared knowledge structure. However, human resources are constrained, the source domain data is collected as a whole in the real scene. However, this is not realistic, this data is associated with the target domain. A generalized empirical risk minimization model (GERM) is proposed in this article with discriminative knowledge-leverage (KL). The empirical risk minimization (ERM) principle is extended to the transfer learning setting. The theoretical upper bound of generalized ERM (GERM) is given for the practical discriminative transfer learning. The subset of the source domain data can be automatically selected in the model, and the source domain data is associated with the target domain. It can solve with only some knowledge of the source domain being available, thus it can avoid the negative transfer effect which is caused by the whole source domain dataset in the real scene. Simulation results show that the proposed algorithm is better than the traditional transfer learning algorithm in simulation data sets and real data sets.

On the ERM Principle With Networked Data

Networked data, in which every training example involves two objects and may share some common objects with others, is used in many machine learning tasks such as learning to rank and link prediction. A challenge of learning from networked examples is that target values are not known for some pairs of objects. In this case, neither the classical i.i.d. assumption nor techniques based on complete U-statistics can be used. Most existing theoretical results of this problem only deal with the classical empirical risk minimization (ERM) principle that always weights every example equally, but this strategy leads to unsatisfactory bounds. We consider general weighted ERM and show new universal risk bounds for this problem. These new bounds naturally define an optimization problem which leads to appropriate weights for networked examples. Though this optimization problem is not convex in general, we devise a new fully polynomial-time approximation scheme (FPTAS) to solve it.

An effective way to integrate ε-support vector regression with gradients

ε-support vector regression (ε-SVR), as a direct implementation of the structural risk minimization principle rather than empirical risk minimization principle, is a new regression method with good generalization ability and can efficiently solve small-sample learning problems. In this work, through incorporating gradient information into the traditional ε-SVR, the gradient-enhanced ε-SVR (GESVR) is developed. The efficiency of GESVR is compared with the traditional ε-SVR by employing analytical function fitting, compared with the gradient-enhanced least square support vector regression (GELSSVR) by using two real-life examples, and tested in a scenario where the exact gradient information is unknown. The results show that GESVR provides more accurate prediction results than the traditional ε-SVR model, and outperforms GELSSVR in some real-life cases.

On the principle of empirical risk minimization based on averaging aggregation functions

An extended version of the principle of empirical risk minimization is proposed. It is based on the application of averaging aggregation functions, rather than arithmetic means, to compute empirical risk. This is justified if the distribution of losses has outliers or is substantially distorted, which results in that the risk estimate becomes biased from the very beginning. In this case, for optimizing parameters, a robust estimate of the mean risk should be used. Such estimates can be constructed by using averaging aggregation functions, which are the solutions of the problem of minimizing the function of penalty for deviation from the mean value. An iterative reweighting scheme for numerically solving the problem of empirical risk minimization is proposed. Illustrative examples of the construction of a robust procedure for estimating parameters in the linear regression problem and in the problem of linearly separating two classes based on the application of an averaging mean function, which replaces the α-quantile, are given.

Robust energy-based least squares twin support vector machines

Twin support vector machine (TSVM), least squares TSVM (LSTSVM) and energy-based LSTSVM (ELS-TSVM) satisfy only empirical risk minimization principle. Moreover, the matrices in their formulations are always positive semi-definite. To overcome these problems, we propose in this paper a robust energy-based least squares twin support vector machine algorithm, called RELS-TSVM for short. Unlike TSVM, LSTSVM and ELS-TSVM, our RELS-TSVM maximizes the margin with a positive definite matrix formulation and implements the structural risk minimization principle which embodies the marrow of statistical learning theory. Furthermore, RELS-TSVM utilizes energy parameters to reduce the effect of noise and outliers. Experimental results on several synthetic and real-world benchmark datasets show that RELS-TSVM not only yields better classification performance but also has a lower training time compared to ELS-TSVM, LSPTSVM, LSTSVM, TBSVM and TSVM.

An efficient implicit regularized Lagrangian twin support vector regression

Twin support vector regression (TSVR) and Lagrangian TSVR (LTSVR) satisfy only empirical risk minimization principle. Moreover, the matrices in their formulations are always positive semi-definite. To overcome these problems, we propose an efficient implicit Lagrangian formulation for the dual regularized twin support vector regression, called IRLTSVR for short. By introducing a regularization term to each objective function, the optimization problems in our IRLTSVR are positive definite and implement the structural risk minimization principle. Moreover, the 1-norm of the vector of slack variable is replaced with 2-norm to make the objective functions strongly convex. Our IRLTSVR solves two systems of linear equations instead of solving two quadratic programming problems (QPPs) in TSVR and one large QPP in SVR, which makes the learning speed of IRLTSVR faster than TSVR and SVR. Particularly, we compare three implementations of IRLTSVR with existing approaches. Computational results on several synthetic and real-world benchmark datasets clearly indicate the effectiveness and applicability of the IRLTSVR in comparison to SVR, TSVR and LTSVR.

Optimizing graph-based patterns to extract biomedical events from the literature.

In BioNLP-ST 2013We participated in the BioNLP 2013 shared tasks on event extraction. Our extraction method is based on the search for an approximate subgraph isomorphism between key context dependencies of events and graphs of input sentences. Our system was able to address both the GENIA (GE) task focusing on 13 molecular biology related event types and the Cancer Genetics (CG) task targeting a challenging group of 40 cancer biology related event types with varying arguments concerning 18 kinds of biological entities. In addition to adapting our system to the two tasks, we also attempted to integrate semantics into the graph matching scheme using a distributional similarity model for more events, and evaluated the event extraction impact of using paths of all possible lengths as key context dependencies beyond using only the shortest paths in our system. We achieved a 46.38% F-score in the CG task (ranking 3rd) and a 48.93% F-score in the GE task (ranking 4th).After BioNLP-ST 2013We explored three ways to further extend our event extraction system in our previously published work: (1) We allow non-essential nodes to be skipped, and incorporated a node skipping penalty into the subgraph distance function of our approximate subgraph matching algorithm. (2) Instead of assigning a unified subgraph distance threshold to all patterns of an event type, we learned a customized threshold for each pattern. (3) We implemented the well-known Empirical Risk Minimization (ERM) principle to optimize the event pattern set by balancing prediction errors on training data against regularization. When evaluated on the official GE task test data, these extensions help to improve the extraction precision from 62% to 65%. However, the overall F-score stays equivalent to the previous performance due to a 1% drop in recall.

Support Vector Machine Ensemble Based on Choquet Integral for Financial Distress Prediction

Due to the radical change in both Chinese and global economic environment, it is essential to develop a practical model to predict financial distress. The support vector machine (SVM), a new outstanding learning machine based on the statistical learning theory, embodying the principle of structural risk minimization instead of empirical risk minimization principle, is a promising method for such financial distress prediction. However, to some extent, the performance of single classifier depends on the sample's pattern characteristics and each single classifier has its own uncertainty. Using the ensemble methods to predict financial distress becomes a rising trend in this field. This research puts forward a SVM ensemble based on the Choquet integral for financial distress prediction in which Bagging algorithm is used to generate new training sets. The proposed ensemble method can be expressed as "Choquet + Bagging + SVMs". With real data from Chinese listed companies, an experiment is carried out to compare the performance of single classifiers with the proposed ensemble method. Empirical results indicate that the proposed ensemble of SVMs based on the Choquet integral for financial distress prediction has higher average accuracy and stability than single SVM classifiers.

A Novel Twin Support Vector Regression

Twin support vector regression (TSVR), as an effective regression machine, solves a pair of smaller-sized quadratic programming problems (QPPs) rather than a single large one as in the classical support vector regression (SVR), which makes the learning speed of TSVR approximately 4 times faster than that of the conventional SVR. However, the empirical risk minimization principle is implemented in TSVR, which reduces its generalization ability to a certain extent. In order to improve the prediction accuracy and stability of algorithm, we propose a novel TSVR for the regression problem by introducing a regularization term into the objective function, which ensures the new algorithm implements the structural risk minimization principle instead of the empirical risk minimization principle. Moreover, the up- and down-bound functions obtained in our algorithm are as parallel as possible. Thus it ensures that our proposed algorithm yields lower prediction error and lower standard deviation in theory. The experimental results on one artificial dataset and six benchmark datasets indicate the feasibility and validity of our novel TSVR.

Querying Discriminative and Representative Samples for Batch Mode Active Learning

Empirical risk minimization (ERM) provides a principled guideline for many machine learning and data mining algorithms. Under the ERM principle, one minimizes an upper bound of the true risk, which is approximated by the summation of empirical risk and the complexity of the candidate classifier class. To guarantee a satisfactory learning performance, ERM requires that the training data are i.i.d. sampled from the unknown source distribution. However, this may not be the case in active learning, where one selects the most informative samples to label, and these data may not follow the source distribution. In this article, we generalize the ERM principle to the active learning setting. We derive a novel form of upper bound for the true risk in the active learning setting; by minimizing this upper bound, we develop a practical batch mode active learning method. The proposed formulation involves a nonconvex integer programming optimization problem. We solve it efficiently by an alternating optimization method. Our method is shown to query the most informative samples while preserving the source distribution as much as possible, thus identifying the most uncertain and representative queries. We further extend our method to multiclass active learning by introducing novel pseudolabels in the multiclass case and developing an efficient algorithm. Experiments on benchmark datasets and real-world applications demonstrate the superior performance of our proposed method compared to state-of-the-art methods.

Learning Bounds of ERM Principle for Sequences of Time-Dependent Samples

Many generalization results in learning theory are established under the assumption that samples are independent and identically distributed (i.i.d.). However, numerous learning tasks in practical applications involve the time-dependent data. In this paper, we propose a theoretical framework to analyze the generalization performance of the empirical risk minimization (ERM) principle for sequences of time-dependent samples (TDS). In particular, we first present the generalization bound of ERM principle for TDS. By introducing some auxiliary quantities, we also give a further analysis of the generalization properties and the asymptotical behaviors of ERM principle for TDS.

Estimation of Reservoir Porosity and Water Saturation Based on Seismic Attributes Using Support Vector Regression Approach

Porosity and fluid saturation distributions are crucial properties of hydrocarbon reservoirs and are involved in almost all calculations related to reservoir and production. True measurements of these parameters derived from laboratory measurements, are only available at the isolated localities of a reservoir and also are expensive and time-consuming. Therefore, employing other methodologies which have stiffness, simplicity, and cheapness is needful. Support Vector Regression approach is a moderately novel method for doing functional estimation in regression problems. Contrary to conventional neural networks which minimize the error on the training data by the use of usual Empirical Risk Minimization principle, Support Vector Regression minimizes an upper bound on the anticipated risk by means of the Structural Risk Minimization principle. This difference which is the destination in statistical learning causes greater ability of this approach for generalization tasks. In this study, first, appropriate seismic attributes which have an underlying dependency with reservoir porosity and water saturation are extracted. Subsequently, a non-linear support vector regression algorithm is utilized to obtain quantitative formulation between porosity and water saturation parameters and selected seismic attributes. For an undrilled reservoir, in which there are no sufficient core and log data, it is moderately possible to characterize hydrocarbon bearing formation by means of this method. • We use SVR to predict porosity and water saturation from seismic attributes. • This methodology was done on a small data set. • Input selection was done through the use of multi regression analysis. • The best SVR models were obtained from the RBF kernel function.

An improved ν-twin support vector machine

Twin support vector machine (TSVM) is regarded as a milestone in the development of the powerful SVM. It finds two nonparallel planes by resolving a pair of smaller-sized quadratic programming problems rather than a single large one, which makes the learning speed of TSVM approximately four times faster than that of the standard SVM. However, the empirical risk minimization principle is implemented in the TSVM, so it easily leads to the over-fitting problem and reduces the prediction accuracy of the classifier. ν-TSVM, as a variant of TSVM, also implements the empirical risk minimization principle. To enhance the generalization ability of the classifier, we propose an improved ν-TSVM by introducing a regularization term into the objective function, so there are two parts in the objective function, one of which is to maximize the margin between the two parallel hyper-planes, and the other one is to minimize the training errors of two classes of samples. Therefore the structural risk minimization principle is implemented in our improved ν-TSVM. Numerical experiments on one artificial dataset and nine benchmark datasets show that our improved ν-TSVM yields better generalization performance than SVM, ν-SVM, and ν-TSVM. Moreover, numerical experiments with different proportions of outliers demonstrate that our improved ν-TSVM is robust and stable. Finally, we apply our improved ν-TSVM to two BCI competition datasets, and also obtain better prediction accuracy.

Improved Extreme Learning Machine and Its Application in Image Quality Assessment

Extreme learning machine (ELM) is a new class of single-hidden layer feedforward neural network (SLFN), which is simple in theory and fast in implementation. Zong et al. propose a weighted extreme learning machine for learning data with imbalanced class distribution, which maintains the advantages from original ELM. However, the current reported ELM and its improved version are only based on the empirical risk minimization principle, which may suffer from overfitting. To solve the overfitting troubles, in this paper, we incorporate the structural risk minimization principle into the (weighted) ELM, and propose a modified (weighted) extreme learning machine (M-ELM and M-WELM). Experimental results show that our proposed M-WELM outperforms the current reported extreme learning machine algorithm in image quality assessment.

Identifying organic-rich Marcellus Shale lithofacies by support vector machine classifier in the Appalachian basin

Unconventional shale reservoirs as the result of extremely low matrix permeability, higher potential gas productivity requires not only sufficient gas-in-place, but also a high concentration of brittle minerals (silica and/or carbonate) that is amenable to hydraulic fracturing. Shale lithofacies is primarily defined by mineral composition and organic matter richness, and its representation as a 3-D model has advantages in recognizing productive zones of shale-gas reservoirs, designing horizontal wells and stimulation strategy, and aiding in understanding depositional process of organic-rich shale. A challenging and key step is to effectively recognize shale lithofacies from well conventional logs, where the relationship is very complex and nonlinear. In the recognition of shale lithofacies, the application of support vector machine (SVM), which underlies statistical learning theory and structural risk minimization principle, is superior to the traditional empirical risk minimization principle employed by artificial neural network (ANN). We propose SVM classifier combined with learning algorithms, such as grid searching, genetic algorithm and particle swarm optimization, and various kernel functions the approach to identify Marcellus Shale lithofacies. Compared with ANN classifiers, the experimental results of SVM classifiers showed higher cross-validation accuracy, better stability and less computational time cost. The SVM classifier with radius basis function as kernel worked best as it is trained by particle swarm optimization. The lithofacies predicted using the SVM classifier are used to build a 3-D Marcellus Shale lithofacies model, which assists in identifying higher productive zones, especially with thermal maturity and natural fractures.

Fuzzy classifier based support vector regression framework for Poisson ratio determination

Abstract Poisson ratio is considered as one of the most important rock mechanical properties of hydrocarbon reservoirs. Determination of this parameter through laboratory measurement is time, cost, and labor intensive. Furthermore, laboratory measurements do not provide continuous data along the reservoir intervals. Hence, a fast, accurate, and inexpensive way of determining Poisson ratio which produces continuous data over the whole reservoir interval is desirable. For this purpose, support vector regression (SVR) method based on statistical learning theory (SLT) was employed as a supervised learning algorithm to estimate Poisson ratio from conventional well log data. SVR is capable of accurately extracting the implicit knowledge contained in conventional well logs and converting the gained knowledge into Poisson ratio data. Structural risk minimization (SRM) principle which is embedded in the SVR structure in addition to empirical risk minimization (EMR) principle provides a robust model for finding quantitative formulation between conventional well log data and Poisson ratio. Although satisfying results were obtained from an individual SVR model, it had flaws of overestimation in low Poisson ratios and underestimation in high Poisson ratios. These errors were eliminated through implementation of fuzzy classifier based SVR (FCBSVR). The FCBSVR significantly improved accuracy of the final prediction. This strategy was successfully applied to data from carbonate reservoir rocks of an Iranian Oil Field. Results indicated that SVR predicted Poisson ratio values are in good agreement with measured values.