Constrained Least-squares Problem Research Articles

An augmented Lagrangian approach for cardinality constrained minimization applied to variable selection problems

To solve convex constrained minimization problems, that also include a cardinality constraint, we propose an augmented Lagrangian scheme combined with alternating projection ideas. Optimization problems that involve a cardinality constraint are NP-hard mathematical programs and typically very hard to solve approximately. Our approach takes advantage of a recently developed and analyzed continuous formulation that relaxes the cardinality constraint. Based on that formulation, we solve a sequence of smooth convex constrained minimization problems, for which we use projected gradient-type methods. In our setting, the convex constraint region can be written as the intersection of a finite collection of convex sets that are easy and inexpensive to project. We apply our approach to a variety of over and under determined constrained linear least-squares problems, with both synthetic and real data that arise in variable selection, and demonstrate its effectiveness.

Variable Elimination in Model Predictive Control Based on K-SVD and QR Factorization

For linearly constrained least-squares problems that depend on a vector of parameters, this article proposes techniques for reducing the number of involved optimization variables. After first eliminating equality constraints in a numerically robust way by QR factorization, we propose a technique based on singular value decomposition (SVD) and unsupervised learning, that we call <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$K$</tex-math></inline-formula> -SVD, and neural classifiers to automatically partition the set of parameter vectors in <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$K$</tex-math></inline-formula> nonlinear regions in which the original problem is approximated by using a smaller set of variables. For the special case of parametric constrained least-squares problems that arise from model predictive control (MPC) formulations, we propose a novel and very efficient QR factorization method for eliminating equality constraints. Together with SVD or <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$K$</tex-math></inline-formula> -SVD, the method provides a numerically robust alternative to standard condensing and move blocking, and to other complexity reduction methods for MPC based on basis functions. We show the good performance of the proposed techniques in numerical tests and in a problem of linearized MPC of a nonlinear benchmark process.

Localizability With Range-Difference Measurements: Numerical Computation and Error Bound Analysis

This paper studies the localization problem using noisy range-difference measurements, or equivalently time difference of arrival (TDOA) measurements. There is a reference sensor, and for each other sensor, the TDOA measurement is obtained with respect to the reference one. By minimizing the sum of squared errors, a nonconvex constrained least squares (CLS) problem is formulated. In this work, we focus on devising an algorithm to seek the global minimizer of the CLS problem, hoping that the numerical solution meets some precision requirement in terms of relative error. Based on the Lagrange multiplier method, we first branch the feasible Lagrange multiplier set into several subsets and develop a workflow in terms of if-then-else control structure to seek the global minimizer by searching for the optimal Lagrange multiplier. The execution order is carefully organized so that it is in line with the general principle of putting the flow that one normally understands to be executed first. We then dive into detailed searching methods in different cases and conduct computational error analysis, giving the error bound on the Lagrange multiplier, when we search for it, to meet the precision requirement on an approximate solution. Based on the above achievements, a programmable global minimizer seeking algorithm is proposed for the CLS problem. Simulations and experimental tests on a public dataset demonstrate the effectiveness of the proposed algorithm.

Feasible Sequential Convex Programming With Inexact Restoration for Multistage Ascent Trajectory Optimization

This paper presents a feasible sequential convex programming method to solve the multistage ascent trajectory optimization problem. The proposed method is based on the inexact restoration technique, in which a more feasible intermediate iterate is first produced by solving a constrained least-squares problem, and then a more optimal iterate is generated by solving a convex programming problem constructed around the newly found feasible solution. By virtue of the inexact restoration idea, the proposed method prevents the common artificial infeasibility issue and can provide the intermediate iterate as a feasible suboptimal solution if the algorithmic procedure terminates before convergence. In addition, the Picard iteration-based convexification and Chebyshev polynomial-based discretization methods are employed in the proposed method, given their benefits in terms of robustness, efficiency, and solution accuracy. Numerical simulations for a minimum-time launch ascent problem are conducted, and the results show that the proposed method exhibits better practical performance than other sequential convex programming methods.

SAR Coregistration by Robust Selection of Extended Targets and Iterative Outlier Cancellation

This letter extends the constrained least-squares (CLS) optimization method developed to coregister multitemporal synthetic aperture radar (SAR) images affected by a joint rotation effect and range/azimuth shifts enforcing the absence of zooming effects. To take advantage of the structural information extracted from the scene, the method starts with a detection stage that identifies extended targets/areas in the images. The selected tie-points allow the CLS problem to be reformulated to find its (initial) solution based on a robust subset of image blocks. Then, the mean square error (MSE) of each equation evaluated from the initial solution allows to implement an iterative cancellation procedure to further skim the CLS equation set. The effectiveness of the proposed procedure is validated on real SAR data in comparison with the standard CLS.

General approach to precise deformable mirror control.

We develop a simple and effective control method for accurate control of deformable mirrors (DMs). For a desired DM surface profile and using batches of observed surface profile data, the proposed method adaptively determines both a DM model (influence matrix) and control actions that produce the desired surface profile with good accuracy. In the first iteration, the developed method estimates a DM influence matrix by solving a multivariable least-squares problem. This matrix is then used to compute the control actions by solving a constrained least-squares problem. Then, the computed actions are randomly perturbed and applied to the DM to generate a new batch of surface profile data. The new data batch is used to estimate a new influence matrix that is then used to re-compute control actions. This procedure is repeated until convergence is achieved. The method is experimentally tested on a Boston Micromachines DM with 140 micro-electronic-mechanical-system actuators. Our experimental results show that the developed control approach can achieve accurate correction despite significant DM nonlinearities. Using only a few control iterations, the developed method is able to produce a surface profile root-mean-square error that varies from 5 - 30 [nm] for most of the tested Zernike wave-front modes without using direct feedback control. These results can additionally be improved by using larger data batches and more iterations or by combining the developed approach with feedback control. Finally, as we experimentally demonstrate, the developed method can be used to estimate a DM model that can effectively be used for a single-step open-loop DM control.

Alternating Polynomial Reconstruction Method for Hyperbolic Conservation Laws

We propose a new multi-moment numerical solver for hyperbolic conservation laws by using the alternating polynomial reconstruction approach. Unlike existing multi-moment schemes, our approach updates model variables by implementing two polynomial reconstructions alternately. First, Hermite interpolation reconstructs the solution within the cell by matching the point-based variables containing both physical values and their spatial derivatives. Then the reconstructed solution is updated by the Euler method. Second, we solve a constrained least-squares problem to correct the updated solution to preserve the conservation laws. Our method enjoys the advantages of a compact numerical stencil and high-order accuracy. Fourier analysis also indicates that our method allows a larger CFL number compared with many other high-order schemes. By adding a proper amount of artificial viscosity, shock waves and other discontinuities can also be computed accurately and sharply without solving an approximated Riemann problem.

Design of Integrated Autonomous Driving Control System That Incorporates Chassis Controllers for Improving Path Tracking Performance and Vehicle Stability

This paper describes an integrated autonomous driving (AD) control system for an autonomous vehicle with four independent in-wheel motors (IWMs). The system consists of two parts: the AD controller and the chassis controller. These elements are functionally integrated to improve vehicle stability and path tracking performance. The vehicle is assumed to employ an IWM independently at each wheel. The AD controller implements longitudinal/lateral path tracking using proportional-integral(PI) control and adaptive model predictive control. The chassis controller is composed of two lateral control units: the active front steering (AFS) control and the torque vectoring (TV) control. Jointly, they find the yaw moment to maintain vehicle stability using sliding mode control; AFS is prioritized over TV to enhance safety margin and energy saving. Then, the command yaw moment is optimally distributed to each wheel by solving a constrained least-squares problem. Validation was performed using simulation in a double lane change scenario. The simulation results show that the integrated AD control system of this paper significantly improves the path tracking capability and vehicle stability in comparison with other control systems.

Seismic wavefield redatuming with regularized multi-dimensional deconvolution

In seismic imaging the aim is to obtain an image of the subsurface using reflection data. The reflection data are generated using sound waves and the sources and receivers are placed at the surface. The target zone, for example an oil or gas reservoir, lies relatively deep in the subsurface below several layers. The area above the target zone is called the overburden. This overburden will have an imprint on the image. Wavefield redatuming is an approach that removes the imprint of the overburden on the image by creating so-called virtual sources and receivers above the target zone. The virtual sources are obtained by determining the impulse response, or Green’s function, in the subsurface. The impulse response is obtained by deconvolving all up- and downgoing wavefields at the desired location. In this paper, we pose this deconvolution problem as a constrained least-squares problem. We describe the constraints that are involved in the deconvolution and show that they are associated with orthogonal projection operators. We show different optimization strategies to solve the constrained least-squares problem and provide an explicit relation between them, showing that they are in a sense equivalent. We show that the constrained least-squares problem remains ill-posed and that additional regularization has to be provided. We show that Tikhonov regularization leads to improved resolution and a stable optimization procedure, but that we cannot estimate the correct regularization parameter using standard parameter selection methods. We also show that the constrained least-squares can be posed in such a way that additional nonlinear regularization is possible.

Data Snooping for the Equality Constrained Nonlinear Gauss–Helmert Model Using Sensitivity Analysis

To develop a universal-outliers processing algorithm under the conditions with equality constraints, the equality-constrained nonlinear Gauss–Helmert (GH) model, which contains the equality-constrained Gauss–Markov (GM) and errors-in-variables (EIV) models as special cases, is selected as the research object in this paper. The least squares solution for the nonlinear GH model with equality constraints is obtained using the Euler–Lagrange approach, and then, it is equivalently formulated as the standard constrained least squares (CLS) problem. To construct the test statistics for the outliers detection, a distinctive sensitivity analysis approach is introduced into this CLS problem. The local sensitivity of the weighted sum of squared residuals to the perturbations of observations in the CLS problem is discussed, and then, the local test statistics are constructed based on these sensitivity indicators. To verify the performance of the sensitivity-based test statistics, the proposed data-snooping algorithm for the equality-constrained nonlinear GH model is applied to a three-dimensional (3D) symmetric similarity transformation. The computational results of the simulated and real examples manifest that the proposed data-snooping algorithm using the sensitivity-based test statistics can effectually decrease the negative impact of the outliers and derive reliable parameters. It should be pointed out that the new algorithm is applicable in various kinds of equality-constrained least squares and total least squares problems.

A discrete least squares collocation method for two-dimensional nonlinear time-dependent partial differential equations

In this paper, we develop regularized discrete least squares collocation and finite volume methods for solving two-dimensional nonlinear time-dependent partial differential equations on irregular domains. The solution is approximated using tensor product cubic spline basis functions defined on a background rectangular (interpolation) mesh, which leads to high spatial accuracy and straightforward implementation, and establishes a solid base for extending the computational framework to three-dimensional problems. A semi-implicit time-stepping method is employed to transform the nonlinear partial differential equation into a linear boundary value problem. A key finding of our study is that the newly proposed mesh-free finite volume method based on circular control volumes reduces to the collocation method as the radius limits to zero. Both methods produce a large constrained least-squares problem that must be solved at each time step in the advancement of the solution. We have found that regularization yields a relatively well-conditioned system that can be solved accurately using QR factorization. An extensive numerical investigation is performed to illustrate the effectiveness of the present methods, including the application of the new method to a coupled system of time-fractional partial differential equations having different fractional indices in different (irregularly shaped) regions of the solution domain.

A regularized interior-point method for constrained linear least squares

We propose an infeasible interior-point algorithm for constrained linear least-squares problems based on the primal-dual regularization of convex programs of Friedlander and Orban. Regularization allows us to dispense with the assumption that the active gradients are linearly independent. At each iteration, a linear system with a symmetric quasi-definite (SQD) matrix is solved. This matrix is shown to be uniformly bounded and nonsingular. While the linear system may be solved using sparse LDL factorization, we observe that other approaches may be used. In particular, we build on the connection between SQD linear systems and unconstrained linear least-squares problems to solve the linear system with sparse QR factorization. We establish conditions under which a solution of the original, constrained least-squares problem is recovered. We report computational experience with the sparse QR factorization and illustrate the potential advantages of our approach.

Constrained Ulam Dynamic Mode Decomposition: Approximation of the Perron-Frobenius Operator for Deterministic and Stochastic Systems

Dynamical systems described by ordinary and stochastic differential equations can be analyzed through the eigen-decomposition of the Perron–Frobenius (PF) and Koopman transfer operators. While the Koopman operator may be approximated by data-driven techniques, e.g., extended dynamic mode decomposition (EDMD), the approximation of the PF operator uses a single-pass Monte Carlo approach in Ulam’s method, which requires a sufficiently long time step. This letter proposes a finite-dimensional approximation technique for the PF operator that uses multi-pass Monte Carlo data to pose and solve a constrained EDMD-like least-squares problem to approximate the PF operator on a finite-dimensional basis. The basis functions used to project the PF operator are the characteristic functions of the state-space partitions. The results are analyzed theoretically and illustrated using deterministic and time-homogeneous stochastic systems.

Estimating uterine source current during contractions using magnetomyography measurements.

Understanding the uterine source of the electrophysiological activity of contractions during pregnancy is of scientific interest and potential clinical applications. In this work, we propose a method to estimate uterine source currents from magnetomyography (MMG) temporal course measurements on the abdominal surface. In particular, we develop a linear forward model, based on the quasistatic Maxwell’s equations and a realistic four-compartment volume conductor, relating the magnetic fields to the source currents on the uterine surface through a lead-field matrix. To compute the lead-field matrix, we use a finite element method that considers the anisotropic property of the myometrium. We estimate the source currents by minimizing a constrained least-squares problem to solve the non-uniqueness issue of the inverse problem. Because we lack the ground truth of the source current, we propose to predict the intrauterine pressure from our estimated source currents by using an absolute-value-based method and compare the result with real abdominal deflection recorded during contractile activity. We test the feasibility of the lead-field matrix by displaying the lead fields that are generated by putative source currents at different locations in the myometrium: cervix and fundus, left and right, front and back. We then illustrate our method by using three synthetic MMG data sets, which are generated using our previously developed multiscale model of uterine contractions, and three real MMG data sets, one of which has simultaneous real abdominal deflection measurements. The numerical results demonstrate the ability of our method to capture the local contractile activity of human uterus during pregnancy. Moreover, the predicted intrauterine pressure is in fair agreement with the real abdominal deflection with respect to the timing of uterine contractions.

A Convex Formulation for Binary Tomography

Binary tomography is concerned with the recovery of binary images from a few of their projections (i.e., sums of the pixel values along various directions). To reconstruct an image from noisy projection data, one can pose it as a constrained least-squares problem. As the constraints are non-convex, many approaches for solving it rely on either relaxing the constraints or heuristics. In this paper we propose a novel convex formulation, based on the Lagrange dual of the constrained least-squares problem. The resulting problem is a generalized LASSO problem which can be solved efficiently. It is a relaxation in the sense that it can only be guaranteed to give a feasible solution; not necessarily the optimal one. In exhaustive experiments on small images (2x2, 3x3, 4x4) we find, however, that if the problem has a unique solution, our dual approach finds it. In case of multiple solutions, our approach finds the commonalities between the solutions. Further experiments on realistic numerical phantoms and an experiment on X-ray dataset show that our method compares favourably to Total Variation and DART.

Iterative Constrained Least Squares for Robust Constant Modulus Beamforming

This paper addresses robust beamforming in the presence of model uncertainty for constant modulus (CM) signals that are frequently encountered in radar and communication systems. Instead of employing the commonly used minimum variance approach, we utilize a CM criterion that is different from the conventional techniques for blind equalization. The proposed robust beamformer minimizes the CM objective function while constraining the magnitude response of any steering vector within a spherical uncertainty region to exceed unity. We develop two algorithms to solve the resultant nonconvex and nonsmooth optimization problem. The first is subgradient projection method, which is simple but converges slowly and is sensitive to initialization. By introducing auxiliary phase variables, our second scheme reformulates the original nonsmooth problem into a differentiable one, that can be solved by an iterative constrained least squares (ICLS) procedure. At each iteration of the ICLS, a least squares (LS) problem with a second-order cone constraint is solved. More important, a closed-form solution for this constrained LS problem, which has a low computational complexity, is derived. The ICLS has fast convergence and is not sensitive to the initial value. Simulation results demonstrate that the proposed CM beamformer achieves a substantial performance gain compared with several representative robust beamformers.

Using a spectral scaling structured BFGS method for constrained nonlinear least squares

In this paper, we use a spectral scaled structured BFGS formula for approximating projected Hessian matrices in an exact penalty approach for solving constrained nonlinear least-squares problems. We show this spectral scaling formula has a good self-correcting property. The reported numerical results show that the use of the spectral scaling structured BFGS method outperforms the standard structured BFGS method.

CNN-Based Projected Gradient Descent for Consistent CT Image Reconstruction.

We present a new image reconstruction method that replaces the projector in a projected gradient descent (PGD) with a convolutional neural network (CNN). Recently, CNNs trained as image-to-image regressors have been successfully used to solve inverse problems in imaging. However, unlike existing iterative image reconstruction algorithms, these CNN-based approaches usually lack a feedback mechanism to enforce that the reconstructed image is consistent with the measurements. We propose a relaxed version of PGD wherein gradient descent enforces measurement consistency, while a CNN recursively projects the solution closer to the space of desired reconstruction images. We show that this algorithm is guaranteed to converge and, under certain conditions, converges to a local minimum of a non-convex inverse problem. Finally, we propose a simple scheme to train the CNN to act like a projector. Our experiments on sparse-view computed-tomography reconstruction show an improvement over total variation-based regularization, dictionary learning, and a state-of-the-art deep learning-based direct reconstruction technique.

Comments on “A Square-Root-Free Matrix Decomposition Method for Energy-Efficient Least Square Computation on Embedded Systems”

A square-root-free matrix QR decomposition (QRD) scheme, dubbed QDR decomposition (QDRD), was rederived by Ren <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> based on a scheme by Björck in order to simplify computations when solving least-squares (LS) problems on embedded systems. The QDRD scheme aims at eliminating both the square-root and division operations in the QRD normalization and backward substitution steps in LS computations. It is claimed by Ren <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> (F. Ren <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al.</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IEEE Embedded Syst. Lett.</i> , vol. 6, no. 4, pp. 73–76) that the LS solution only requires finding the directions of the orthogonal basis of the matrix in question, regardless of the normalization of their Euclidean norms. Multiple-input multiple-output (MIMO) detection problems have been named as potential applications that benefit from this. While this is true for unconstrained LS problems, we conversely show here that constrained LS problems such as MIMO detection still require computing the norms of the orthogonal basis to produce the correct result.

Push recovery of a quadruped robot on challenging terrains

SUMMARYLegged robots may become unstable when subjected to unexpected disturbances such as external pushes and environmental irregularities mostly while moving on natural terrains. To enhance the mobility performance, legged robots should be able to keep or restore their balanced configuration when a sudden disturbance is exerted. The aim of this article is to design a controller for a quadruped robot to restore its balanced configuration despite exerting external pushes. This is achieved based on developing a full-dynamics model of the robot moving over even and uneven terrains. The proposed controller is based on a PD module which calculates the required accelerations for restoring the robot equilibrium. However, these accelerations may make the robot unstable and also cause the slippage of stance feet. Therefore, an optimization algorithm is used to compute the maximum admissible accelerations. The constraints of the optimization problem are the conditions which guarantee the robot stability and the stance feet slippage avoidance. The optimization algorithm is transformed into a linear constrained least-squares problem to be solved in real-time. The main contributions of this article are the development of a push recovery algorithm for quadruped robots and also the introduction of an appropriate condition which guarantees the stability of the robot even on uneven terrains. This stability condition is developed based on a full-dynamics model of the robot. The proposed algorithm is applied on an 18-DOF quadruped robot when the robot is standing over both even and uneven terrains. The obtained results show that the robot can successfully restore its balanced configuration by precise adjustment of the position and orientation of its main body while a massive external disturbance is exerted.