Least-squares Loss Research Articles

Rank-based variable selection

This note considers variable selection in the robust linear model via R-estimates. The proposed rank-based approach is a generalisation of the penalised least-squares estimators where we replace the least-squares loss function with Jaeckel's (1972) dispersion function. Our rank-based method is robust to outliers in the errors and has roots in traditional non-parametric statistics for simple location-shift problems. We establish the theoretical properties of our estimators which ensure desirable asymptotic behaviour of setting coefficient estimates to zero for unimportant variables and consistently estimating coefficients for important variables. Numerical studies indicate that the rank-based methods perform well for both light- and heavy-tailed error distributions.

Multi-kernel regularized classifiers

A family of classification algorithms generated from Tikhonov regularization schemes are considered. They involve multi-kernel spaces and general convex loss functions. Our main purpose is to provide satisfactory estimates for the excess misclassification error of these multi-kernel regularized classifiers when the loss functions achieve the zero value. The error analysis consists of two parts: regularization error and sample error. Allowing multi-kernels in the algorithm improves the regularization error and approximation error, which is one advantage of the multi-kernel setting. For a general loss function, we show how to bound the regularization error by the approximation in some weighted L q spaces. For the sample error, we use a projection operator. The projection in connection with the decay of the regularization error enables us to improve convergence rates in the literature even for the one-kernel schemes and special loss functions: least-square loss and hinge loss for support vector machine soft margin classifiers. Existence of the optimization problem for the regularization scheme associated with multi-kernels is verified when the kernel functions are continuous with respect to the index set. Concrete examples, including Gaussian kernels with flexible variances and probability distributions with some noise conditions, are used to illustrate the general theory.

Three-mode partitioning

The three-mode partitioning model is a clustering model for three-way three-mode data sets that implies a simultaneous partitioning of all three modes involved in the data. In the associated data analysis, a data array is approximated by a model array that can be represented by a three-mode partitioning model of a prespecified rank, minimizing a least squares loss function in terms of differences between data and model. Algorithms have been proposed for this minimization, but their performance is not yet clear. A framework for alternating least-squares methods is described in order to offset the performance problem. Furthermore, a number of both existing and novel algorithms are discussed within this framework. An extensive simulation study is reported in which these algorithms are evaluated and compared according to sensitivity to local optima. The recovery of the truth underlying the data is investigated in order to assess the optimal estimates. The ordering of the algorithms with respect to performance in finding the optimal solution appears to change as compared to the results obtained from the simulation study when a collection of four empirical data sets have been used. This finding is attributed to violations of the implicit stochastic model underlying both the least-squares loss function and the simulation study. Support for the latter attribution is found in a second simulation study.

Combinatorial individual differences scaling within the city-block metric

A new method is proposed for conducting individual differences scaling within the city-block metric that does not rely on gradient- or subgradient-based optimization. Instead, a combinatorial optimization scheme is utilized for identifying object coordinates minimizing the least-squares loss function. The illustrative application of combinatorial individual differences scaling within the city-block metric to schematic face stimuli suggests that the new method offers a promising alternative to gradient-based attempts for fitting city-block scaling models, which suffer from the well-documented difficulty of local minima.

Learning Rates of Least-Square Regularized Regression

This paper considers the regularized learning algorithm associated with the least-square loss and reproducing kernel Hilbert spaces. The target is the error analysis for the regression problem in learning theory. A novel regularization approach is presented, which yields satisfactory learning rates. The rates depend on the approximation property and on the capacity of the reproducing kernel Hilbert space measured by covering numbers. When the kernel is C∞ and the regression function lies in the corresponding reproducing kernel Hilbert space, the rate is mζ with ζ arbitrarily close to 1, regardless of the variance of the bounded probability distribution.

Ecologically meaningful transformations for ordination of species data.

This paper examines how to obtain species biplots in unconstrained or constrained ordination without resorting to the Euclidean distance [used in principal-component analysis (PCA) and redundancy analysis (RDA)] or the chi-square distance [preserved in correspondence analysis (CA) and canonical correspondence analysis (CCA)] which are not always appropriate for the analysis of community composition data. To achieve this goal, transformations are proposed for species data tables. They allow ecologists to use ordination methods such as PCA and RDA, which are Euclidean-based, for the analysis of community data, while circumventing the problems associated with the Euclidean distance, and avoiding CA and CCA which present problems of their own in some cases. This allows the use of the original (transformed) species data in RDA carried out to test for relationships with explanatory variables (i.e. environmental variables, or factors of a multifactorial analysis-of-variance model); ecologists can then draw biplots displaying the relationships of the species to the explanatory variables. Another application allows the use of species data in other methods of multivariate data analysis which optimize a least-squares loss function; an example is K-means partitioning.

An Evaluation of Two Algorithms for Hierarchical Classes Analysis

In this procedure, a least-squares loss function in terms of discrepancies between D and M is minimized. The present paper describes the original hierarchical classes algorithm proposed by De Boeck and Rosenberg (1988), which is based on an alternating greedy heuristic, and proposes a new algorithm, based on an alternating branch-and-bound procedure. An extensive simulation study is reported in which both algorithms are evaluated and compared according to goodness-of-fit to the data and goodness-of-recovery of the underlying true structure. Furthermore, three heuristics for selecting models of different ranks for a given D are presented and compared. The simulation results show that the new algorithm yields models with slightly higher goodness-of-fit and goodness-of-recovery values.

Optimal scaling by alternating length-constrained nonnegative least squares, with application to distance-based analysis

An important feature of distance-based principal components analysis, is that the variables can be optimally transformed. For monotone spline transformation, a nonnegative least-squares problem with a length constraint has to be solved in each iteration. As an alternative algorithm to Lawson and Hanson (1974), we propose the Alternating Length-Constrained Non-Negative Least-Squares (ALC-NNLS) algorithm, which minimizes the nonnegative least-squares loss function over the parameters under a length constraint, by alternatingly minimizing over one parameter while keeping the others fixed. Several properties of the new algorithm are discussed. A Monte Carlo study is presented which shows that for most cases in distance-based principal components analysis, ALC-NNLS performs as good as the method of Lawson and Hanson or sometimes even better in terms of the quality of the solution.

Graph-theoretic representations for proximity matrices through strongly-anti-Robinson or circular strongly-anti-Robinson matrices

There are various optimization strategies for approximating, through the minimization of a least-squares loss function, a given symmetric proximity matrix by a sum of matrices each subject to some collection of order constraints on its entries. We extend these approaches to include components in the approximating sum that satisfy what are called the strongly-anti-Robinson (SAR) or circular strongly-anti-Robinson (CSAR) restrictions. A matrix that is SAR or CSAR is representable by a particular graph-theoretic structure, where each matrix entry is reproducible from certain minimum path lengths in the graph. One published proximity matrix is used extensively to illustrate the types of approximation that ensue when the SAR or CSAR constraints are imposed.

A Gauss-Newton-like optimization algorithm for "weighted" nonlinear least-squares problems

The Gauss-Newton algorithm is often used to minimize a nonlinear least-squares loss function instead of the original Newton-Raphson algorithm. The main reason is the fact that only first-order derivatives are needed to construct the Jacobian matrix. Some applications as, for instance multivariable system identification, give rise to weighted nonlinear least-squares problems for which it can become quite hard to obtain an analytical expression of the Jacobian matrix. To overcome that struggle, a pseudo-Jacobian matrix is introduced, which leaves the stationary points untouched and can be calculated analytically. Moreover, by slightly changing the pseudo-Jacobian matrix, a better approximation of the Hessian can be obtained resulting in faster convergence.

Ambiguity Resolution for Satellite Doppler Positioning Systems

The implementation of satellite-based Doppler positioning systems frequently requires the recovery of transmitter position from a single pass of Doppler data. The least-squares approach to the problem yieds conjugate solutions on either side of the satellite subtrack. It is important to develop a procedure for choosing the proper solution which is correct in a high percentage of cases. A test for ambiguity resolution which is the most powerful in the sense that it maximizes the probability of a correct decision is derived. When systematic error sources are properly included in the least-squares reduction process to yield an optimal solution the test reduces to choosing the solution which provides the smaller valuation of the least-squares loss function. When systematic error sources are ignored in the least-squares reduction, the most powerful test is a quadratic form compasison with the weighting matrix of the quadratic form obtained by computing the pseudoinverse of a reduced-rank square matrix. A formula for computing the power of the most powerful test is provided. Numerical examples are included in which the power of the test is computed for situations that are relevant to the design of a satellite-aided search and rescue system.