Iteratively Reweighted Least Squares Research Articles

Improved learning algorithms for mixture of experts in multiclass classification

Mixture of experts (ME) is a modular neural network architecture for supervised learning. A double-loop Expectation-Maximization (EM) algorithm has been introduced to the ME architecture for adjusting the parameters and the iteratively reweighted least squares (IRLS) algorithm is used to perform maximization in the inner loop [Jordan, M.I., Jacobs, R.A. (1994). Hierarchical mixture of experts and the EM algorithm, Neural Computation, 6(2), 181–214]. However, it is reported in literature that the IRLS algorithm is of instability and the ME architecture trained by the EM algorithm, where IRLS algorithm is used in the inner loop, often produces the poor performance in multiclass classification. In this paper, the reason of this instability is explored. We find out that due to an implicitly imposed incorrect assumption on parameter independence in multiclass classification, an incomplete Hessian matrix is used in that IRLS algorithm. Based on this finding, we apply the Newton–Raphson method to the inner loop of the EM algorithm in the case of multiclass classification, where the exact Hessian matrix is adopted. To tackle the expensive computation of the Hessian matrix and its inverse, we propose an approximation to the Newton–Raphson algorithm based on a so-called generalized Bernoulli density. The Newton–Raphson algorithm and its approximation have been applied to synthetic data, benchmark, and real-world multiclass classification tasks. For comparison, the IRLS algorithm and a quasi-Newton algorithm called BFGS have also been applied to the same tasks. Simulation results have shown that the use of the proposed learning algorithms avoids the instability problem and makes the ME architecture produce good performance in multiclass classification. In particular, our approximation algorithm leads to fast learning. In addition, the limitation of our approximation algorithm is also empirically investigated in this paper.

Investigation on the relationship between analytical precision and concentration with iteratively reweighted least-squares linear regression method

It is very important to investigate the relationship between analytical precision and concentration for quality control and assessment of analytical results. A set of analytical data of trace elements in soil and water deposits (Chinese certified reference materials) was studied for the relationship between the analytical precision of collaboration trials (i.e. reproducibility) and the concentration of the analytes. Iteratively reweighted least-squares (IRLS) linear regression, a robust method, was employed in this study. The linear relationship between the standard deviation and concentration was successfully established. On comparison with the conventional least-squares (LS) method , the results of IRLS linear regression are superior to those of LS.

Efficient and fast iterative reweighted least-squaresnonrecursive filters

An efficient and fast technique for designing Lp approximation filters using the iterative reweighted least-squares (IRLS) algorithm is proposed. This technique introduces an extra frequency response which implicitly includes the weighting function such that the filter coefficients can be obtained with O(N2) complexity.

Iteratively reweighted least-squares state estimation through Givens Rotations

This paper presents a robust iteratively reweighted least squares (IRLS) method for power system state estimation (PSSE). An orthogonal implementation through Givens Rotations to solve estimators based on nonquadratic criteria is introduced. The nonquadratic criteria are defined based on the robust statistical theory. The weights associated with the measurements include the effects of measurement variances and the location where each measurement is taken. Bad data identification is a byproduct of the proposed method. After bad data identification, no reprocessing is required in order to obtain the correct estimates. The method is validated by using the 14-bus, 30-bus and 118-bus IEEE test systems, in addition to a 340-bus realistic system.

Nonlinear Regression, Quasi Likelihood, and Overdispersion in Generalized Linear Models

The aim of this article is to reconsider the methods for handling of overdispersion in generalized linear models proposed by McCullagh and Nelder. Our starting point will be a nonlinear regression model with normal errors, specified by a mean function, a variance function and a matrix of covariates. Structurally, this model is very similar to a generalized linear model, except that a common dispersion (or squared scale) parameter is a natural ingredient. In this context, we discuss the estimation method known as IRLS (iteratively reweighted least squares) or quasi likelihood. For generalized linear models, this method coincides with maximum-likelihood. We discuss the proposals made by McCullagh and Nelder for situations where such models fail due to overdispersion. For many such models (in particular for discrete responses), the idea of an overdispersion parameter does not make much sense at first sight. Our approach is based on approximation by a nonlinear regression model. In particular, we are interested in the validity of approximate F-tests for removal of model terms and approximate T-distribution-based confidence intervals.

Non-linear inversion using general measures of data misfit and model structure

Abstract We investigate the use of general, non-l 2 measures of data misfit and model structure in the solution of the non-linear inverse problem. Of particular interest are robust measures of data misfit, and measures of model structure which enable piecewise-constant models to be constructed. General measures can be incorporated into traditional linearized, iterative solutions to the non-linear problem through the use of an iteratively reweighted least-squares (IRLS) algorithm. We show how such an algorithm can be used to solve the linear inverse problem when general measures of misfit and structure are considered. The magnetic stripe example of Parker (1994) is used as an illustration. This example also emphasizes the benefits of using a robust measure of misfit when outliers are present in the data. We then show how the IRLS algorithm can be used within a linearized, iterative solution to the non-linear problem. The relevant procedure contains two iterative loops which can be combined in a number of ways. We present two possibilities. The first involves a line search to determine the most appropriate value of the trade-off parameter and the complete solution, via the IRLS algorithm, of the linearized inverse problem for each value of the trade-off parameter. In the second approach, a schedule of prescribed values for the trade-off parameter is used and the iterations required by the IRLS algorithm are combined with those for the linearized, iterative inversion procedure. These two variations are then applied to the 1-D inversion of both synthetic and field time-domain electromagnetic data.

Hybrid ℓ1/ℓ2 minimization with applications to tomography

Least squares or [Formula: see text] solutions of seismic inversion and tomography problems tend to be very sensitive to data points with large errors. The [Formula: see text] minimization for 1 ≤ p < 2 gives more robust solutions, but usually with higher computational cost. Iteratively reweighted least squares (IRLS) gives efficient approximate solutions to these [Formula: see text] problems. We apply IRLS to a hybrid [Formula: see text] minimization problem that behaves like an [Formula: see text] fit for small residuals and like an [Formula: see text] fit for large residuals. The smooth transition from [Formula: see text] to [Formula: see text] behavior is controlled by a parameter that we choose using an estimate of the standard deviation of the data error. For linear problems of full rank, the hybrid objective function has a unique minimum, and IRLS can be proven to converge to it. We obtain a robust efficient method. For nonlinear problems, a version of the Gauss‐Newton algorithm can be applied. Synthetic crosswell tomography examples and a field‐data VSP tomography example demonstrate the improvement of the hybrid method over least squares when there are outliers in the data.

Letter: Doppler optimised compression filters for both aperiodical and periodical sequences

AbstractThe problem of designing Spread Spectrum (SS) radar detection filters that minimise sidelobe (SL) levels in both Doppler and range co‐ordinates is considered and a new algorithm for the Doppler optimised mismatched filter is presented. The Iterative Reweighted Least Square (IRLS) procedure for minimax filter coefficients calculation is applied. A special attention has been paid to the analysis of Code Division Multiple Access (CDMA) SS communication systems. The computer simulation results obtained show that SL levels rejection using a new algorithm is very high, both for aperiodical and periodical sequences.

Estimation of Residual Valve Gradient from Incomplete Data with Outliers

AbstractAn important indicator for the long‐term recovery after valve replacement surgery is postoperative valve gradient. This information is available only for patients received catheterization or echocardiogram postoperatively. It is plausible that sicker patients are more inclined to undergo these postoperative procedures and their valve gradients tend to be higher. Under this situation, ignoring the missing values and using sample mean based on the available information as an estimate of the whole study population leads to overestimation. Regression estimator is a reasonable choice to eliminate this bias if independent (explanatory) variables closely associated with both residual valve gradient and non‐response mechanism can be identified. Using a series of patients receiving St. Jude Medical prosthetic valves, we found that valve area index can be used as an independent variable in the regression estimator. Two digressions from the standard assumptions used in linear regression, heteroscedastic trend of the error term and outliers were found in the data set. Iteratively reweighted least square (IRLS) was adopted to handle heteroscedasticity. Influence function approach was used to evaluate the sensitivity of outliers in regression estimator. Under an equal response rate mechanism, IRLS not only solves the problem of heteroscedasticity, but is also less sensitive to outliers.

Least absolute deviation (LAD) image matching

The robust estimator properties of the L,-norm or least absolute deviation (LAD) is shown to provide better subpixel matching accuracy in the presence of outlier points than the least squares method widely employed for image matching applications. Two LAD algorithms are compared with each other and with the least squares (LS) method and the iteratively reweighted least squares (IRLS) method. Results indicate that the Barrodale-Roberts LAD algorithm can be used advantageously in conjunction with or in place of the IRLS and LS algorithms.

NO x emission modelling using the iteratively reweighted least-square procedures

NO x emission dispatching is a method to reduce the amount of oxide of nitrogen produced by thermal generating units to meet given power demands. This requires adequate models relating NO x emissions to the active power generation of the unit. Least-squares estimation procedures work best when measurement errors are Gaussian. For the non-Gaussi an errors and Gaussian errors, the iteratively reweighted least-squares (IRWLS) procedure with optimized weight functions and their associated weight constants can be used to estimate, refine and fine tune parameters of identified models. The form of the emissions model is restricted to those with monotonically increasing derivatives due to subsequent minimization process requirements. We explore the application of some transformations to the form of model chosen. Moreover, emission models are developed using the IRWLS procedure. The results obtained using the iteratively reweighted least-square procedures are compared with the least squares.

A Globally Convergent Method for $l_p $ Problems

The $l_p $-norm discrete estimation problem $\min_{x \in \Re^n } ||b - A^T x||_p^p $ is troublesome when p is close to unity because the objective function approaches a nonsmooth form as p converges to one. This paper presents an efficient algorithm for solving $l_p $-norm problems for all $1 \leq p < 2$. When $p = 1$ it is essentially the method presented by T. F. Coleman and Y. Li [Math. Programming, 56 (1992), pp. 189–222], which is a globally and quadratically convergent algorithm under some nondegeneracy assumptions. The existing iteratively reweighted least-squares (IRLS) method can be obtained from the new approach by updating some dual multipliers in a special fashion. The new method is globally convergent, and it is superlinearly convergent when there is no zero residual at the solution. At each iteration the main computational cost of the new method is the same as that of the IRLS method: solving a reweighted least-squares problem. Numerical experiments indicate that this method is significantly faster than popular iteratively reweighted least-squares methods when p is close or equal to one.

Non-oscillatory and Non-diffusive Solution of Convection Problems by the Iteratively Reweighted Least-Squares Finite Element Method

In this paper we first introduce the least-squares finite element method (LSFEM) for two-dimensional steady-state pure convection problems with smooth solutions and compare the LSFEM with other finite element methods. We prove that the LSFEM has the same stability estimate as the original equation; i.e., the LSFEM has better control of the streamline derivative than the streamline upwinding Petrov-Galerkin method. Numerical convergence rates are given to show that the LSFEM is almost optimal. Then we use this LSFEM as a framework to develop an iteratively reweighted least-squares finite element method (IRLSFEM) to obtain non-oscillatory and non-diffusive solutions for problems with contact discontinuities. This new method produces a highly accurate numerical solution that has a sharp discontinuity in one element. A number of examples solved by using triangular and bilinear elements are presented to show that the method can convect contact discontinuities without error.

Load forecasting via suboptimal seasonal autoregressive models and iteratively reweighted least squares estimation

Suboptimal least squares or iteratively reweighted least-squares (IRWLS) procedures for estimating the parameters of a seasonal multiplicative autoregressive (AR) model encountered during power system load forecasting are proposed. The method involves using an interactive computer environment to estimate the parameters of a seasonal multiplicative AR process. The method comprises five major computational steps. The first determines the order of the seasonal multiplicative AR process, and the second uses the least squares or the IRWLS to estimate the optimal nonseasonal AR model parameters. In the third step one obtains the intermediate series by back forecast, which is followed by using the least squares or the IRWLS to estimate the optimal seasonal AR parameters. The final step uses the estimated parameters to forecast future load. The method is applied to predict the Nova Scotia Power Corporation's 168 lead time hourly load. The results obtained are documented and compared with results based on the Box and Jenkins method. >

Design of anSFunction for Robust Regression Using Iteratively Reweighted Least Squares

Abstract We develop a set of 5 functions for robust regression using the technique of iteratively reweighted least squares (IRLS). Together with a set of weight functions, function rreg is simple to understand and provides great flexibility for IRLS methods. This article focuses on the programming strategies adopted to achieve the twin goals of power and simplicity.

L/sub 1/ and L/sub infinity / minimization via a variant of Karmarkar's algorithm

Simple iterative algorithms are presented for L/sub 1/ and L/sub infinity / minimization (regression) based on a variant of Karmarkar's linear programming algorithm. Although these algorithms are based on entirely different theoretical principles to the popular IRLS (iteratively reweighted least squares) algorithm, they have almost identical matrix operations. Also presented are the results of a Monte Carlo study comparing the numerical convergence properties of the Karmarkar algorithm for L/sub 1/ minimization to those of an IRLS and a simplex algorithm. The test problem involves L/sub 1/ estimation of AR (autoregressive) model parameters. The Karmarkar algorithm outperformed IRLS by achieving higher numerical accuracy in fewer iterations. Techniques for reducing the computational cost per iteration of the Karmarkar L/sub 1/ algorithm are discussed. >

Fast Ip solution of large, sparse, linear systems: Application to seismic travel time tomography

Least-squares inversion tends to be sensitive to small changes in the assumptions; in the statistical lexicon, it is not “robust.” Least-squares solutions of noisy, ill-conditioned systems often exhibit unphysical, high-frequency oscillations which have to be damped or smoothed in some fashion. The iteratively reweighted least squares (IRLS) algorithm provides a means of computing approximate l p solutions (1 ⩽ p). We will show that IRLS can be combined with the preconditioned conjugate gradient algorithm in order to solve large, sparse, rectangular systems of linear, algebraic equations very efficiently. Further, for p ≅ 1, the resulting algorithm is extremely stable; it seems to give reasonably smooth solutions without minute adjustment of the parameters and does not become unstable if an overly optimistic error estimate is used. It also has the ability to reject, or at least significantly diminish, the influence of outliers in the data and does not require the use of damping. In addition, we will show how to efficiently include diagonal weighting of the observations and parameters in order to incorporate a priori information into the inversion. This combined inversion algorithm has wide applicability in “real-world” inverse theory, parameter estimation, filtering, etc., where data are noisy and not uniformly reliable. Numerical examples of the l 1 inversion of noisy seismic travel time data will be shown.

On the connection between IRLS and Gauss' method for l&amp;lt;inf&amp;gt;1&amp;lt;/inf&amp;gt;inversion: Comments on "Fast algorithms for l&amp;lt;inf&amp;gt;p&amp;lt;/inf&amp;gt;deconvolution"

We discuss the relation between the iteratively reweighted least squares (IRLS) method described by Yarlagadda et al. and the standard linear programming approach to computing l <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> inverses. In particular, we discuss why these methods may give significantly different results.

Maximum Likelihood Computations with Repeated Measures: Application of the EM Algorithm

Abstract The purpose of this article is to consider the use of the EM algorithm (Dempster, Laird, and Rubin 1977) for both maximum likelihood (ML) and restricted maximum likelihood (REML) estimation in a general repeated measures setting using a multivariate normal data model with linear mean and covariance structure (Anderson 1973). Several models and methods of analysis have been proposed in recent years for repeated measures data; Ware (1985) presented an overview. Because the EM algorithm is a general-purpose, iterative method for computing ML estimates with incomplete data, it has often been used in this particular setting (Dempster et al. 1977; Andrade and Helms 1984; Jennrich and Schluchter 1985). There are two apparently different approaches to using the EM algorithm in this setting. In one application, each experimental unit is observed under a standard protocol specifying measurements at each of n occasions (or under n conditions), and incompleteness implies that the number of measurements actually collected on each unit is less than the requisite n for at least some units. In this circumstance, incompleteness may be modeled if one regards the measurements actually collected as the observed data, the conceptual set of n measurements on each individual as the complete data, and the unobserved data as the missing measurements on those units with fewer than n observations. Application of the EM algorithm in this setting [referred to as “missing data” in Dempster et al. (1977) and “incomplete data” in Jennrich and Schluchter (1985)] was discussed by Orchard and Woodbury (1972), Beale and Little (1975), and Jennrich and Schluchter (1985). One drawback of this approach in the longitudinal data setting is that the multivariate model with linear mean and covariance structure does not, in general, possess closed-form solutions even with complete data (Anderson 1973; Szatrowski 1980). Thus implementing the EM algorithm requires either an iterative M step within each EM iteration or the use of a generalized EM (GEM) algorithm that requires only that the complete data likelihood be increased rather than maximized at each M step. A second drawback is that this approach requires specification of the covariates for both the observed and the missing observations. If the covariates are unknown for the missing observations, arbitrary values must be specified, which may affect the rate but not the final point of convergence (Jennrich and Schluchter 1985). The second application of the EM algorithm arises naturally when we use mixed models to analyze serial measurements. In this setting, the incomplete data are modeled quite differently. The observed data are as before, that is, the measurements actually collected on each unit. The complete data, however, consist of the observed data plus the unobservable random parameters and error terms specified in the mixed model. Thus the missing data (the random parameters and error terms) would not be viewed as data in the traditional statistical sense. Laird and Ware (1982) and Andrade and Helms (1984) took this approach. This article shows that the latter approach is more general and encompasses the missing-data approach as a special case. This result has several important applications. First, it means that EM algorithms encoded for models with random effects can also be used for multivariate normal models with arbitrary covariance structure and missing data. Second, this approach avoids specification of covariates for missing observations. Finally, use of the general formulation means that closed-form solutions for the complete data maximization will exist for a much broader class of models, enabling one to avoid use of GEM or iterations within each M step. For a certain class of multivariate growth curve models with random effects structure (Reinsel 1982), closed-form solutions exist for both ML and REML estimates of the mean and covariance parameters. Formulas for these closed-form solutions are given that are applicable whenever the solution is not on the boundary. The choice of starting values for the EM iterations is important, since the EM algorithm will not, in general, converge from arbitrary starting values to the closed-form solution (if it exists) in one iteration. Several possibilities for starting values are given. The rate of convergence of the EM algorithm is generally linear. The actual speed of convergence in two data examples is shown to depend heavily on both the actual data set and the assumed structure for the covariance matrix. We discuss two methods for accelerating convergence, which we find are most useful when the covariance matrix is assumed to have a random effects structure. When the covariance matrix is assumed to be arbitrary, the EM iterations reduce to familiar iteratively reweighted least squares (IRLS) computations. The EM algorithm has the unusual property in this setting that when all of the data are complete (no missing observations), the iterations are still IRLS, but the rate of convergence changes from linear to quadratic.

L&amp;lt;inf&amp;gt;1&amp;lt;/inf&amp;gt;deconvolution and its application to seismic signal processing

This correspondence reviews fast iterative reweighted least squares (IRLS) and residual steepest descent (RSD) algorithms for L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</inf> , <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1 \leq p \leq 2</tex> , deconvolution. The timing aspects and the implementation of the IRLS algorithm on an array processor are discussed. The effectiveness of L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> deconvolution and its insensitivity to noise bursts are illustrated using simple synthetic as well as complex seismic data. Finally, it is shown that the L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</inf> prediction filters in general need not be stable, and that L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> solutions predict the possibility of nonminimum phase aspects of a given set of data.