Regularization Hyperparameter Research Articles

A heuristic method for parameter selection in LS-SVM: Application to time series prediction

Abstract Least Squares Support Vector Machines (LS-SVM) are the state of the art in kernel methods for regression. These models have been successfully applied for time series modelling and prediction. A critical issue for the performance of these models is the choice of the kernel parameters and the hyperparameters which define the function to be minimized. In this paper a heuristic method for setting both the σ parameter of the Gaussian kernel and the regularization hyperparameter based on information extracted from the time series to be modelled is presented and evaluated.

Recursive Bayesian Recurrent Neural Networks for Time-Series Modeling

This paper develops a probabilistic approach to recursive second-order training of recurrent neural networks (RNNs) for improved time-series modeling. A general recursive Bayesian Levenberg-Marquardt algorithm is derived to sequentially update the weights and the covariance (Hessian) matrix. The main strengths of the approach are a principled handling of the regularization hyperparameters that leads to better generalization, and stable numerical performance. The framework involves the adaptation of a noise hyperparameter and local weight prior hyperparameters, which represent the noise in the data and the uncertainties in the model parameters. Experimental investigations using artificial and real-world data sets show that RNNs equipped with the proposed approach outperform standard real-time recurrent learning and extended Kalman training algorithms for recurrent networks, as well as other contemporary nonlinear neural models, on time-series modeling.

A New Solution Path Algorithm in Support Vector Regression

In this paper, regularization path algorithms were proposed as a novel approach to the model selection problem by exploring the path of possibly all solutions with respect to some regularization hyperparameter in an efficient way. This approach was later extended to a support vector regression (SVR) model called epsilon-SVR. However, the method requires that the error parameter epsilon be set a priori. This is only possible if the desired accuracy of the approximation can be specified in advance. In this paper, we analyze the solution space for epsilon-SVR and propose a new solution path algorithm, called epsilon-path algorithm, which traces the solution path with respect to the hyperparameter epsilon rather than lambda. Although both two solution path algorithms possess the desirable piecewise linearity property, our epsilon-path algorithm overcomes some limitations of the original lambda-path algorithm and has more advantages. It is thus more appealing for practical use.

Classification model selection via bilevel programming

Support vector machines and related classification models require the solution of convex optimization problems that have one or more regularization hyper-parameters. Typically, the hyper-parameters are selected to minimize the cross-validated estimates of the out-of-sample classification error of the model. This cross-validation optimization problem can be formulated as a bilevel program in which the outer-level objective minimizes the average number of misclassified points across the cross-validation folds, subject to inner-level constraints such that the classification functions for each fold are (exactly or nearly) optimal for the selected hyper-parameters. Feature selection is included in the bilevel program in the form of bound constraints in the weights. The resulting bilevel problem is converted to a mathematical program with linear equilibrium constraints, which is solved using state-of-the-art optimization methods. This approach is significantly more versatile than commonly used grid search procedures, enabling, in particular, the use of models with many hyper-parameters. Numerical results demonstrate the practicality of this approach for model selection in machine learning.

Temporal image reconstruction in electrical impedance tomography

Electrical impedance tomography (EIT) calculates images of the body from body impedance measurements. While the spatial resolution of these images is relatively low, the temporal resolution of EIT data can be high. Most EIT reconstruction algorithms solve each data frame independently, although Kalman filter algorithms track the image changes across frames. This paper proposes a new approach which directly accounts for correlations between images in successive data frames. Image reconstruction is posed in terms of an augmented image and measurement vector , which concatenate the values from the d previous and future frames. Image reconstruction is then based on an augmented regularization matrix , which accounts for a model of both the spatial and temporal correlations between image elements. Results are compared for reconstruction algorithms based on independent frames, Kalman filters and the proposed approach. For low values of the regularization hyperparameter, the proposed approach performs similarly to independent frames, but for higher hyperparameter values, it uses adjacent frame data to reduce reconstructed image noise.

Penalized discriminant analysis of [15O]-water PET brain images with prediction error selection of smoothness and regularization hyperparameters.

We propose a flexible, comprehensive approach for analysis of [15O]-water positron emission tomography (PET) brain images using a penalized version of linear discriminant analysis (PDA). We applied it to scans from 20 subjects (eight scans/subject) performing a finger movement task and analyzed: 1) two classes to obtain a covariance-normalized baseline-activation image, and 2) eight classes for the mean within subject temporal structure which contained baseline-activation and time-dependent changes in a two-dimensional canonical subspace. We imposed spatial smoothness on the resulting image(s) by expanding it in five tensor-product B-spline (TPS) bases of varying smoothness, and further regularized with a ridge-type penalty on the noise covariance matrix. The discrimination approach of PDA provides a probabilistic framework within which prediction error (PE) estimates are derived. We used these to optimize over TPS bases and a ridge hyperparameter (expressed as equivalent degrees of freedom, EDF). We obtained unbiased, low variance PE estimates using modern resampling tools (.632+ Bootstrap and cross validation), and compared PDA of 1) TPS-projected, mean-normalized and unnormalized scans and 2) mean-normalized scans with and without additional presmoothing. By examining the tradeoffs between PE and EDF, as a function of basis selection and image smoothing we demonstrate the utility of PDA, the PE framework, and the relationship between singular value decomposition and smooth TPS bases in the analysis of functional neuroimages.