Constrained Least-squares Problem Research Articles

Modeling Control Valves in Water Distribution Systems Using a Continuous State Formulation

AbstractControl valves are commonly used for the operation of water distribution systems. Modeling these devices typically requires that their operating states are known, or that a computationally expensive search is undertaken over all possible operating states. This paper presents a novel method of modeling control valves (including flow control, pressure sustaining, pressure reducing and check valves) in extended-period simulations of water distribution systems. Instead of the normal discrete control problem formulation, it is approached with the Karush-Kuhn-Tucker equations for an optimization problem with constraints. The proposed method does not prerequire the operating state (open, closed, active) of each valve to be determined, as this is done implicitly. Pipe and valve flow rates and nodal heads are determined by: (1) minimizing deviations from targets at control valves, and (2) satisfy the state equations (conservation of mass and energy) by solving a constrained least-squares problem. Sensitivi...

A robust combined trust region–line search exact penalty projected structured scheme for constrained nonlinear least squares

We present a combined trust region–line search structured algorithm for solving constrained nonlinear least-squares problems. The approach is based on adaptive structured scheme due to Mahdavi-Amiri and Bartels of the exact penalty method of Coleman and Conn for nonlinearly constrained optimization problems. The structured adaptation also makes use of the ideas of Nocedal and Overton for handling quasi-Newton updates of projected Hessians. For robustness, we propose a new penalty parameter updating strategy and a specific line search strategy within the trust region, taking account of the least-squares objective and special structured considerations for the approximate projected least-squares Hessians. Our added usage of a second-order correction step in the global phase, combined trust region–line search strategy, and, of course, the new adaptive penalty parameter updating strategy helps in speeding up the global iterations and reaching the asymptotic phase confidently. We discuss the details of our implementation and provide comparative results of the testing of our programs and three nonlinear programing codes from KNITRO on test problems (both small and large residual) from Hock and Schittkowski, Lukšan and Vlček, Biegler et al. and some randomly generated ones due to Bartels and Mahdavi-Amiri. The numerical results obtained by our approach showed a clear local two-step superlinear convergence rate, and as compared to the results obtained by Mahdavi-Amiri and Bartels, the best methods tested in the collection of Hock and Schittkowski, three methods in KNITRO, a recent nonlinear least-squares method of Bidabadi and Mahdavi-Amiri and two algorithms provided by fmincon in MATLAB, would indeed confirm the practical significance of our adaptive penalty update scheme, combined trust region–line search strategy, and special structured considerations for the approximate projected least-squares Hessians.

Interpolation methods and the accuracy of lattice-Boltzmann mesh refinement

A lattice-Boltzmann model to solve the equivalent of the Navier–Stokes equations on adaptively refined grids is presented. A method for transferring information across interfaces between different grid resolutions was developed following established techniques for finite-volume representations. This new approach relies on a space–time interpolation and solving constrained least-squares problems to ensure conservation. The effectiveness of this method at maintaining the second order accuracy of lattice-Boltzmann is demonstrated through a series of benchmark simulations and detailed mesh refinement studies. These results exhibit smaller solution errors and improved convergence when compared with similar approaches relying only on spatial interpolation. Examples highlighting the mesh adaptivity of this method are also provided.

A two-step superlinearly convergent projected structured BFGS method for constrained nonlinear least squares

We present an exact penalty approach for solving constrained non-linear least-squares problems, when the projected structured Hessian is approximated by a projected version of the structured BFGS formula and prove its local two-step Q-superlinear convergence. The numerical results obtained in an extensive comparative testing experiment confirm the approach to be reliable and efficient.

Operations research and optimization (ORO)

We have recently proposed a structured algorithm for solving constrained nonlinear least-squares problems and established its local two-step Q-superlinear convergence rate. The approach is based on an earlier adaptive structured scheme due to Mahdavi-Amiri and Bartels of the exact penalty method. The structured adaptation makes use of the ideas of Nocedal and Overton for handling quasi-Newton updates of projected Hessians and adapts a structuring scheme due to Engels and Martinez. For robustness, we have employed a specific nonsmooth line search strategy, taking account of the least-squares objective. Numerical results also confirm the practical relevance of our special considerations for the inherent structure of the least squares. Here, we establish global convergence of the proposed algorithm using a weaker condition than the one used by the exact penalty method of Coleman and Conn for general nonlinear programs.

Linearized Alternating Direction Method of Multipliers for Constrained Linear Least-Squares Problem

Abstract.The alternating direction method of multipliers (ADMM) is applied to a constrained linear least-squares problem, where the objective function is a sum of two least-squares terms and there are box constraints. The original problem is decomposed into two easier least-squares subproblems at each iteration, and to speed up the inner iteration we linearize the relevant subproblem whenever it has no known closed-form solution. We prove the convergence of the resulting algorithm, and apply it to solve some image deblurring problems. Its efficiency is demonstrated, in comparison with Newton-type methods.

Multi-candidate missing data imputation for robust speech recognition

The application of Missing Data Techniques (MDT) to increase the noise robustness of HMM/GMM-based large vocabulary speech recognizers is hampered by a large computational burden. The likelihood evaluations imply solving many constrained least squares (CLSQ) optimization problems. As an alternative, researchers have proposed frontend MDT or have made oversimplifying independence assumptions for the backend acoustic model. In this article, we propose a fast Multi-Candidate (MC) approach that solves the per-Gaussian CLSQ problems approximately by selecting the best from a small set of candidate solutions, which are generated as the MDT solutions on a reduced set of cluster Gaussians. Experiments show that the MC MDT runs equally fast as the uncompensated recognizer while achieving the accuracy of the full backend optimization approach. The experiments also show that exploiting the more accurate acoustic model of the backend does pay off in terms of accuracy when compared to frontend MDT.

An alternating direction algorithm for matrix completion with nonnegative factors

This paper introduces an algorithm for the nonnegative matrix factorization-and-completion problem, which aims to find nonnegative low-rank matrices X and Y so that the product XY approximates a nonnegative data matrix M whose elements are partially known (to a certain accuracy). This problem aggregates two existing problems: (i) nonnegative matrix factorization where all entries of M are given, and (ii) low-rank matrix completion where nonnegativity is not required. By taking the advantages of both nonnegativity and low-rankness, one can generally obtain superior results than those of just using one of the two properties. We propose to solve the non-convex constrained least-squares problem using an algorithm based on the classic alternating direction augmented Lagrangian method. Preliminary convergence properties of the algorithm and numerical simulation results are presented. Compared to a recent algorithm for nonnegative matrix factorization, the proposed algorithm produces factorizations of similar quality using only about half of the matrix entries. On tasks of recovering incomplete grayscale and hyperspectral images, the proposed algorithm yields overall better qualities than those produced by two recent matrix-completion algorithms that do not exploit nonnegativity.

Smoothed state estimates under abrupt changes using sum-of-norms regularization

The presence of abrupt changes, such as impulsive and load disturbances, commonly occur in applications, but make the state estimation problem considerably more difficult than in the standard setting with Gaussian process disturbance. Abrupt changes often introduce a jump in the state, and the problem is therefore readily and often treated by change detection techniques. In this paper, we take a different approach. The state smoothing problem for linear state space models is here formulated as a constrained least-squares problem with sum-of-norms regularization, a generalization of ℓ1-regularization. This novel formulation can be seen as a convex relaxation of the well known generalized likelihood ratio method by Willsky and Jones. Another nice property of the suggested formulation is that it only has one tuning parameter, the regularization constant which is used to trade off fit and the number of jumps. Good practical choices of this parameter along with an extension to nonlinear state space models are given.

Constrained Nonlinear Least Squares: A Superlinearly Convergent Projected Structured Secant Method

Numerical solution of nonlinear least-squares problems is an important computational task in science and engineering. Effective algorithms have been developed for solving nonlinear least squares problems. The structured secant method is a class of efficient methods developed in recent years for optimization problems in which the Hessian of the objective function has some special structure. A primary and typical application of the structured secant method is to solve the nonlinear least squares problems. We present an exact penalty method for solving constrained nonlinear leastsquares problems, when the structured projected Hessian is approximated by a projected version of the structured BFGS formula and give its local two-step Q-superlinear convergence. For robustness, we employ a special nonsmooth line search strategy, taking account of the least squares objective. We discuss the comparative results of the testing of our programs and three nonlinear programming codes from KNITRO on some randomly generated test problems due to Bartels and MahdaviAmiri. Numerical results also confirm the practical relevance of our special considerations for the inherent structure of the least squares.

MIMO Power Control for High-Density Servers in an Enclosure

Power control is becoming a key challenge for effectively operating a modern data center. In addition to reducing operating costs, precisely controlling power consumption is an essential way to avoid system failures caused by power capacity overload or overheating due to increasing high server density. Control-theoretic techniques have recently shown a lot of promise for power management because of their better control performance and theoretical guarantees on control accuracy and system stability. However, existing work oversimplifies the problem by controlling a single server independently from others. As a result, at the enclosure level where multiple high-density servers are correlated by common workloads and share common power supplies, power cannot be shared to improve application performance. In this paper, we propose an enclosure-level power controller that shifts power among servers based on their performance needs, while controlling the total power of the enclosure to be lower than a constraint. Our controller features a rigorous design based on an optimal Multi-Input-Multi-Output (MIMO) control theory. We present detailed control problem formulation and transformation to a standard constrained least-squares problem, as well as stability analysis in the face of significant workload variations. We then conduct extensive experiments on a physical testbed to compare our controller with three state-of-the-art controllers: a heuristic-based MIMO control solution, a Single-Input-Single-Output (SISO) control solution, and an improved SISO controller with simple power shifting among servers. Our empirical results demonstrate that our controller outperforms all the three baselines by having more accurate power control and up to 11.8 percent better benchmark performance.

A NEW METHOD FOR THE SYNTHESIS OF NON-UNIFORM LINEAR ARRAYS WITH SHAPED POWER PATTERNS

Antenna arrays with shaped power patterns have many applications in communications and radars. Many antenna array synthesis techniques for shaped patterns have been developed in the past years, and most of them deal only with uniformly spaced arrays. In this paper, a new method is proposed for the synthesis of nonuniform linear antenna arrays with shaped power patterns. The proposed synthesis method consists of three steps. First, we flnd a satisfactory power pattern for the required radiation characteristics by solving a constrained least-squares problem which is obtained with the help of non-redundant representation of squared magnitude of a linear array factor. Then, we factorize the polynomial associated with the power pattern by using polynomial rooting, and consequently obtain the corresponding fleld patterns. Finally, the forward-backward matrix pencil method is used to obtain a nonuniform linear array with optimized excitation magnitudes, phases and locations for a speciflc choice of fleld patterns. The synthesized array has a smaller number of elements than the one with uniformly spaced elements for the same

Nonnegative least-squares image deblurring: improved gradient projection approaches

The least-squares approach to image deblurring leads to an ill-posed problem. The addition of the nonnegativity constraint, when appropriate, does not provide regularization, even if, as far as we know, a thorough investigation of the ill-posedness of the resulting constrained least-squares problem has still to be done. Iterative methods, converging to nonnegative least-squares solutions, have been proposed. Some of them have the ‘semi-convergence’ property, i.e. early stopping of the iteration provides ‘regularized’ solutions. In this paper we consider two of these methods: the projected Landweber (PL) method and the iterative image space reconstruction algorithm (ISRA). Even if they work well in many instances, they are not frequently used in practice because, in general, they require a large number of iterations before providing a sensible solution. Therefore, the main purpose of this paper is to refresh these methods by increasing their efficiency. Starting from the remark that PL and ISRA require only the computation of the gradient of the functional, we propose the application to these algorithms of special acceleration techniques that have been recently developed in the area of the gradient methods. In particular, we propose the application of efficient step-length selection rules and line-search strategies. Moreover, remarking that ISRA is a scaled gradient algorithm, we evaluate its behaviour in comparison with a recent scaled gradient projection (SGP) method for image deblurring. Numerical experiments demonstrate that the accelerated methods still exhibit the semi-convergence property, with a considerable gain both in the number of iterations and in the computational time; in particular, SGP appears definitely the most efficient one.

The least squares problem of the matrix equation A 1 X 1 B 1 T + A 2 X 2 B 2 T = T

The matrix least squares (LS) problem minX |AXBT − T|F is trivial and its solution can be simply formulated in terms of the generalized inverse of A and B. Its generalized problem minX1,X2|A1X1B1T + A2X2B2T − T|F can also be regarded as the constrained LS problem minX=diag (X1,X2)|AXBT − T|F with A = [A1, A2] and B = [B1, B2]. The authors transform T to \( \tilde T \) such that minX1,X2 |A1X1B1T + A2X2B2T − T|F is equivalent to minX=diag(X1,X2) |AXBT − \( \tilde T \)|F whose solutions are included in the solution set of unconstrained problem minX |AXBT − \( \tilde T \)|F. So the general solutions of minX1,X2 |A1X1B1T + A2X2B2T − T|F are reconstructed by selecting the parameter matrix in that of minX |AXBT − \( \tilde T \)|F.

A reduced Newton method for constrained linear least-squares problems

We propose an iterative method that solves constrained linear least-squares problems by formulating them as nonlinear systems of equations and applying the Newton scheme. The method reduces the size of the linear system to be solved at each iteration by considering only a subset of the unknown variables. Hence the linear system can be solved more efficiently. We prove that the method is locally quadratic convergent. Applications to image deblurring problems show that our method gives better restored images than those obtained by projecting or scaling the solution into the dynamic range.

Robust approaches to remote calibration of a transmitting array

We consider the problem of estimating the gains and phases of the RF channels of a M-element transmitting array, based on a calibration procedure where M orthogonal signals are sent through M orthogonal beams and received on a single antenna. The received data vector obeys a linear model of the type y = A Fg + n where A is an unknown complex scalar accounting for propagation loss and g is the vector of unknown complex gains. In order to improve the performance of the least-squares (LS) estimator at low signal to noise ratio (SNR), we propose to exploit knowledge of the nominal value of g , viz g ¯ . Towards this end, two approaches are presented. First, a Bayesian approach is advocated where A and g are considered as random variables, with a non-informative prior distribution for A and a Gaussian prior distribution for g . The posterior distributions of the unknown random variables are derived and a Gibbs sampling strategy is presented that enables one to generate samples distributed according to these posterior distributions, leading to the minimum mean-square error (MMSE) estimator. A second approach consists in solving a constrained least-squares problem in which h = A g is constrained to be close to a scaled version of g ¯ . This second approach yields a closed-form solution, which amounts to a linear combination of g ¯ and the LS estimator. Numerical simulations show that the two new estimators significantly outperform the conventional LS estimator, especially at low SNR.

Random projections for the nonnegative least-squares problem

Constrained least-squares regression problems, such as the Nonnegative Least Squares (NNLS) problem, where the variables are restricted to take only nonnegative values, often arise in applications. Motivated by the recent development of the fast Johnson–Lindestrauss transform, we present a fast random projection type approximation algorithm for the NNLS problem. Our algorithm employs a randomized Hadamard transform to construct a much smaller NNLS problem and solves this smaller problem using a standard NNLS solver. We prove that our approach finds a nonnegative solution vector that, with high probability, is close to the optimum nonnegative solution in a relative error approximation sense. We experimentally evaluate our approach on a large collection of term-document data and verify that it does offer considerable speedups without a significant loss in accuracy. Our analysis is based on a novel random projection type result that might be of independent interest. In particular, given a tall and thin matrix Φ ∈ R n × d ( n ≫ d ) and a vector y ∈ R d , we prove that the Euclidean length of Φ y can be estimated very accurately by the Euclidean length of Φ ∼ y , where Φ ∼ consists of a small subset of (appropriately rescaled) rows of Φ .

Expansion of weighted pseudoinverse matrices with positive definite weights into matrix power products. Iterative methods

Weighted pseudoinverse matrices with positive definite weights are expanded into matrix power products with negative exponents and arbitrary positive parameters. These expansions are used to develop and analyze iterative methods for evaluating weighted pseudoinverse matrices and weighted normal pseudosolutions and solving constrained least-squares problems.

Series expansion of weighted pseudoinverse matrices and iterative methods for calculating weighted pseudoinverse matrices and weighted normal pseudosolutions

Weighted pseudoinverse matrices are expanded into matrix power series with negative exponents and arbitrary positive parameters. Based on this expansion, iterative methods for evaluating weighted pseudoinverse matrices and weighted normal pseudosolutions are designed and analyzed. The iterative methods for weighted normal pseudosolutions are extended to solving constrained least-squares problems.

Adaptive Algorithm for Constrained Least-Squares Problems

This paper is concerned with the implementation and testing of an algorithm for solving constrained least-squares problems. The algorithm is an adaptation to the least-squares case of sequential quadratic programming (SQP) trust-region methods for solving general constrained optimization problems. At each iteration, our local quadratic subproblem includes the use of the Gauss–Newton approximation but also encompasses a structured secant approximation along with tests of when to use this approximation. This method has been tested on a selection of standard problems. The results indicate that, for least-squares problems, the approach taken here is a viable alternative to standard general optimization methods such as the Byrd–Omojokun trust-region method and the Powell damped BFGS line search method.