Quadratic Penalty Term Research Articles

Linearized alternating direction method with parallel splitting and adaptive penalty for separable convex programs in machine learning

Many problems in machine learning and other fields can be (re)for-mulated as linearly constrained separable convex programs. In most of the cases, there are multiple blocks of variables. However, the traditional alternating direction method (ADM) and its linearized version (LADM, obtained by linearizing the quadratic penalty term) are for the two-block case and cannot be naively generalized to solve the multi-block case. So there is great demand on extending the ADM based methods for the multi-block case. In this paper, we propose LADM with parallel splitting and adaptive penalty (LADMPSAP) to solve multi-block separable convex programs efficiently. When all the component objective functions have bounded subgradients, we obtain convergence results that are stronger than those of ADM and LADM, e.g., allowing the penalty parameter to be unbounded and proving the sufficient and necessary conditions} for global convergence. We further propose a simple optimality measure and reveal the convergence rate of LADMPSAP in an ergodic sense. For programs with extra convex set constraints, with refined parameter estimation we devise a practical version of LADMPSAP for faster convergence. Finally, we generalize LADMPSAP to handle programs with more difficult objective functions by linearizing part of the objective function as well. LADMPSAP is particularly suitable for sparse representation and low-rank recovery problems because its subproblems have closed form solutions and the sparsity and low-rankness of the iterates can be preserved during the iteration. It is also highly parallelizable and hence fits for parallel or distributed computing. Numerical experiments testify to the advantages of LADMPSAP in speed and numerical accuracy.

Fast X-ray CT image reconstruction using a linearized augmented Lagrangian method with ordered subsets.

Augmented Lagrangian (AL) methods for solving convex optimization problems with linear constraints are attractive for imaging applications with composite cost functions due to the empirical fast convergence rate under weak conditions. However, for problems such as X-ray computed tomography (CT) image reconstruction, where the inner least-squares problem is challenging and requires iterations, AL methods can be slow. This paper focuses on solving regularized (weighted) least-squares problems using a linearized variant of AL methods that replaces the quadratic AL penalty term in the scaled augmented Lagrangian with its separable quadratic surrogate function, leading to a simpler ordered-subsets (OS) accelerable splitting-based algorithm, OS-LALM. To further accelerate the proposed algorithm, we use a second-order recursive system analysis to design a deterministic downward continuation approach that avoids tedious parameter tuning and provides fast convergence. Experimental results show that the proposed algorithm significantly accelerates the convergence of X-ray CT image reconstruction with negligible overhead and can reduce OS artifacts when using many subsets.

A Proximal Dual Consensus ADMM Method for Multi-Agent Constrained Optimization

This paper considers a convex optimization problem with a globally coupled linear equality constraint and local polyhedron constraints and develops efficient distributed optimization methods. The considered problem has many engineering applications. Due to the polyhedron constraints, agents in the existing methods have to deal with polyhedron constrained subproblems at each iteration. One of the key challenges is that projection onto a polyhedron set is not trivial, which prohibits the agents from solving these subproblems efficiently. In this paper, based on the alternating direction method of multipliers (ADMM), we propose a new distributed optimization method, called proximal dual consensus ADMM (PDC-ADMM). The PDC-ADMM transforms the polyhedron constraints as quadratic penalty terms in the subproblems, making the subproblems efficiently solvable and consequently reducing the overall computational overhead of the agents. In addition, we propose a randomized PDC-ADMM which can deal with time-varying networks with randomly ON/OFF agents and communication errors, and an inexact (randomized) PDC-ADMM for low-complexity computations. We show that the proposed distributed methods converge to the optimal solution set almost surely and have a O (1/k) ergodic convergence rate in the mean. Numerical results show that the proposed methods offer significantly lower computation time than the existing distributed ADMM method in solving a linearly constrained LASSO problem.

Material identification in frequency-domain coupled acoustic-structure interaction using an error in constitutive equation functional

This work focuses on the inverse identification of linear elastic material parameters in the context of frequency-domain, coupled acoustic-structure interaction. The approach postulates the inverse problem as an optimization problem where the solution is obtained by minimizing a modified error in constitutive equation (MECE) functional. The MECE functional measures the discrepancy in the constitutive equations that connect kinematically admissible strains and dynamically admissible stresses while incorporating the measurement data as an additional quadratic penalty term. The method is formulated generally for the recovery of material properties in a coupled acoustic-structure system using solid displacement and/or fluid pressure measurement data. Numerical results demonstrate that the proposed methodology can identify the spatial distribution of elastic moduli from partial and noisy measurements taken in either the fluid or solid subdomains.

Large scale parameter estimation problems in frequency-domain elastodynamics using an error in constitutive equation functional

This paper presents the formulation and implementation of an Error in Constitutive Equations (ECE) method suitable for large-scale inverse identification of linear elastic material properties in the context of steady-state elastodynamics. In ECE-based methods, the inverse problem is postulated as an optimization problem in which the cost functional measures the discrepancy in the constitutive equations that connect kinematically admissible strains and dynamically admissible stresses. Furthermore, in a more recent modality of this methodology introduced by Feissel and Allix [17], referred to as the Modified ECE (MECE), the measured data is incorporated into the formulation as a quadratic penalty term. We show that a simple and efficient continuation scheme for the penalty term, suggested by the theory of quadratic penalty methods, can significantly accelerate the convergence of the MECE algorithm. Furthermore, a (block) successive over-relaxation (SOR) technique is introduced, enabling the use of existing parallel finite element codes with minimal modification to solve the coupled system of equations that arises from the optimality conditions in MECE methods. Our numerical results demonstrate that the proposed methodology can successfully reconstruct the spatial distribution of elastic material parameters from partial and noisy measurements in as few as ten iterations in a 2D example and fifty in a 3D example. We show (through numerical experiments) that the proposed continuation scheme can improve the rate of convergence of MECE methods by at least an order of magnitude versus the alternative of using a fixed penalty parameter. Furthermore, the proposed block SOR strategy coupled with existing parallel solvers produces a computationally efficient MECE method that can be used for large scale materials identification problems, as demonstrated on a 3D example involving about 400,000 unknown moduli. Finally, our numerical results suggest that the proposed MECE approach can be significantly faster than the conventional approach of L2 minimization using quasi-Newton methods.

An energy functional approach for inverse characterization of material properties in elastodynamics

We present an inverse problem methodology based on the Error in Constitutive Equations (ECE) approach for the identification of material properties in the context of frequency-domain elastodynamics. In the ECE approach, we define a cost functional based on an energy norm that connects a set of kinematically admissible displacements and a set of dynamically admissible stresses. The set of kinematically admissible displacements is composed of fields that satisfy essential boundary conditions and possess sufficient regularity (i.e. smoothness). The set of dynamically admissible stresses is composed of fields that satisfy conservation of linear momentum and natural (i.e. traction) boundary conditions. The inverse problem is solved by finding material properties along with admissible displacement and stress fields such that the ECE functional is minimized. Experimental data is introduced in the formulation as a quadratic penalty term added to the ECE functional. The talk will focus on the reconstruction of elastic and viscoelastic properties in heterogeneous materials in the context of frequency-domain dynamics. Our findings indicate that ECE methods provide faster and more accurate results than conventional least-squares minimization. We will show numerical and experimental results that demonstrate the salient features of the method.

Trust Region-Based Optimization for Maximum Mutual Information Estimation of HMMs in Speech Recognition

In this paper, we have proposed two novel optimization methods for discriminative training (DT) of hidden Markov models (HMMs) in speech recognition based on an efficient global optimization algorithm used to solve the so-called trust region (TR) problem, where a quadratic function is minimized under a spherical constraint. In the first method, maximum mutual information estimation (MMIE) of Gaussian mixture HMMs is formulated as a standard TR problem so that the efficient global optimization method can be used in each iteration to maximize the auxiliary function of discriminative training for speech recognition. In the second method, we propose to construct a new auxiliary function for DT of HMMs by adding a quadratic penalty term. The new auxiliary function is constructed to serve as first-order approximation as well as lower bound of the original discriminative objective function within a locality constraint. Due to the lower-bound property, the found optimal point of the new auxiliary function is guaranteed to improve the original discriminative objective function until it converges to a local optimum or stationary point of the objective function. Both TR-based optimization methods have been investigated on two standard large-vocabulary continuous speech recognition tasks, using the WSJ0 and Switchboard databases. Experimental results have shown that the proposed TR methods outperform the conventional EBW method in terms of convergence behavior as well as recognition performance.

Nonlinear Regularization Path for Quadratic Loss Support Vector Machines

Regularization path algorithms have been proposed to deal with model selection problem in several machine learning approaches. These algorithms allow computation of the entire path of solutions for every value of regularization parameter using the fact that their solution paths have piecewise linear form. In this paper, we extend the applicability of regularization path algorithm to a class of learning machines that have quadratic loss and quadratic penalty term. This class contains several important learning machines such as squared hinge loss support vector machine (SVM) and modified Huber loss SVM. We first show that the solution paths of this class of learning machines have piecewise nonlinear form, and piecewise segments between two breakpoints are characterized by a class of rational functions. Then we develop an algorithm that can efficiently follow the piecewise nonlinear path by solving these rational equations. To solve these rational equations, we use rational approximation technique with quadratic convergence rate, and thus, our algorithm can follow the nonlinear path much more precisely than existing approaches such as predictor-corrector type nonlinear-path approximation. We show the algorithm performance on some artificial and real data sets.

SU‐E‐I‐33: Low‐Dose CBCT by Iterative Image Reconstruction Using Non‐Local Edge‐Preserving Prior

Purpose: Statistical iterative image reconstruction (SIR) algorithms are effective for reconstructing high‐quality low‐dose cone beam computed tomography (CBCT). The performance of SIR strongly depends on the prior/penalty term. In this work, we develop and evaluate an edge‐preserving penalty term in SIR that incorporates high‐order neighbors. Methods: CBCT projection data of a CatPhan 600 phantom was acquired on a Varian Acuity simulator (Varian Medical Systems, Palo Alto, CA) using two protocols: 80mA/12ms/projeciton for high dose and 10mA/10ms/projection for low dose. Low‐dose CBCT images were reconstructed by minimizing an objective function based on the penalized weighted least‐squares (PWLS) criterion. Conventionally, the penalty term is used to enforce the smoothness constraint within a local neighborhood (e.g. up to third‐order neighbors). In this work, we propose a non‐local anisotropic quadratic penalty term that incorporates higher‐order neighbors. The relative contribution from different neighbors is controlled by the absolute difference between the selected neighbors. For neighbors associated with larger differences, equivalence between them will be discouraged. Contrast‐to‐noise ratio (CNR) at different regions of interest (ROIs) is calculated to quantitatively evaluate the proposed high‐order neighborhood penalty. Results: Noise is substantially reduced in low‐dose CBCT reconstructed using the iterative PWLS algorithm, while the edges are well‐preserved. The CNRs in selected ROIs in PWLS‐reconstructed images are higher than that of the corresponding FDK reconstructed high‐dose images. Using the higher‐order neighbor penalty, the average CNR of the selected ROIs is 24% higher than the image reconstructed by PWLS using the penalty with the conventional local neighbor. Conclusions: A higher‐order neighborhood methodology is proposed for the penalty term in PWLS iterative image reconstruction for enhancement of low‐dose CBCT image quality. PWLS coupled with higher‐order neighbor penalty outperforms the PWLS with conventional local neighbor penalty.This work was partially supported by American Cancer Society Institutional Research Grant (ACS‐IRG‐02‐196).

10.2478/s11814-009-0252-1

A combined run-to-run (R2R) and LQG control method has been proposed for rapid thermal processing (RTP) equipment for run-wise improvement and real-time multivariable control of the temperature uniformity over the wafer surface. The standard LQG objective was modified to include a quadratic penalty term for input deviation from bias values which are updated by an R2R control law. The proposed method has been applied to commercial 12-inch rotating RTP equipment with four pyrometers and ten circular groups of tungsten-halogen lamps for measurements and manipulation of wafer temperatures. The performance of LQG control was evaluated under wafer rotation and found to show quite accurate tracking. For evaluation of the combined control technique, a wafer with seven thermocouples (TC’s) attached along the radial direction has been employed for the TC measurements to be used for R2R control, whereas the pyrometer measurements are fed back for real-time LQG control. It was observed that the temperature uniformity is improved as the run number increases.

The Smooth-Lasso and other ℓ1+ℓ2-penalized methods

We consider a linear regression problem in a high dimensional setting where the number of covariates p can be much larger than the sample size n. In such a situation, one often assumes sparsity of the regression vector, i.e., the regression vector contains many zero components. We propose a Lasso-type estimator β̂Quad (where ‘Quad’ stands for quadratic) which is based on two penalty terms. The first one is the ℓ1 norm of the regression coefficients used to exploit the sparsity of the regression as done by the Lasso estimator, whereas the second is a quadratic penalty term introduced to capture some additional information on the setting of the problem. We detail two special cases: the Elastic-Net β̂EN introduced in [42], which deals with sparse problems where correlations between variables may exist; and the Smooth-Lasso β̂SL, which responds to sparse problems where successive regression coefficients are known to vary slowly (in some situations, this can also be interpreted in terms of correlations between successive variables). From a theoretical point of view, we establish variable selection consistency results and show that β̂Quad achieves a Sparsity Inequality, i.e., a bound in terms of the number of non-zero components of the ‘true’ regression vector. These results are provided under a weaker assumption on the Gram matrix than the one used by the Lasso. In some situations this guarantees a significant improvement over the Lasso. Furthermore, a simulation study is conducted and shows that the S-Lasso β̂SL performs better than known methods as the Lasso, the Elastic-Net β̂EN, and the Fused-Lasso (introduced in [30]) with respect to the estimation accuracy. This is especially the case when the regression vector is ‘smooth’, i.e., when the variations between successive coefficients of the unknown parameter of the regression are small. The study also reveals that the theoretical calibration of the tuning parameters and the one based on 10 fold cross validation imply two S-Lasso solutions with close performance.

Topology optimization of fluid channels with flow rate equality constraints

This note presents topology optimization of fluid channels with flow rate equality constraints. The equality constraints on the specified boundaries are implemented using the lumped Lagrange multiplier method. The quadratic penalty term and cut-off sensitivity are used to maintain the stability of optimization.

Analysis and Regularization of Problems in Diffuse Optical Tomography

In this paper we consider the regularization of the inverse problem of diffuse optical tomography by standard regularization methods with quadratic penalty terms. We therefore investigate in detail the properties of the associated forward operators and derive continuity and differentiability results, which are based on derivation of $W^{1,p}$ regularity of solutions for the governing elliptic boundary value problems. We then show that Tikhonov regularization can be applied for a stable solution and that the standard convergence and convergence rates results hold. Our analysis also allows us to ensure convergence of iterative regularization methods, which are important from a practical point of view.

Application of augmented Lagrangian relaxation to coordinated scheduling of interdependent hydrothermal power and natural gas systems

This study proposes an optimisation model for the coordinated scheduling of interdependent electric power and natural gas transmission systems from a joint operator's viewpoint. The objective is to minimise the coordinated social cost while satisfying network and temporal constraints of the two interdependent systems. The joint operator will coordinate hourly schedules to supply natural gas to loads or generate electric power. The authors consider the application of Lagrangian relaxation (LR) or augmented LR to relax the coupling constraints of the two systems. The Lagrangian dual is decomposed into the security-constrained unit commitment subproblem with the hydro coordination and the natural gas allocation subproblem. The application of LR for solving the coordinated problem could cause oscillations in the dual solution which is due to the non-convex characteristics of the coordinated problem represented by integer variables and network constraints. Moreover, with slight changes in multiplier values, the linear cost function of the natural gas well may result in a cycling behaviour of the gas well output between its max and min limits. To avoid numerical oscillations and improve the solution quality, the augmented LR with a piecewise linear approximation of quadratic penalty terms and the block descent coordination technique are proposed. The authors consider the 6-bus with 7-node and the 118-bus with 14-node systems to verify that the applicability of the proposed method to the coordinated scheduling of electric power and natural gas transmission systems.

Combined run-to-run and LQG control of a 12-inch RTP equipment

A combined run-to-run (R2R) and LQG control method has been proposed for rapid thermal processing (RTP) equipment for run-wise improvement and real-time multivariable control of the temperature uniformity over the wafer surface. The standard LQG objective was modified to include a quadratic penalty term for input deviation from bias values which are updated by an R2R control law. The proposed method has been applied to commercial 12-inch rotating RTP equipment with four pyrometers and ten circular groups of tungsten-halogen lamps for measurements and manipulation of wafer temperatures. The performance of LQG control was evaluated under wafer rotation and found to show quite accurate tracking. For evaluation of the combined control technique, a wafer with seven thermocouples (TC’s) attached along the radial direction has been employed for the TC measurements to be used for R2R control, whereas the pyrometer measurements are fed back for real-time LQG control. It was observed that the temperature uniformity is improved as the run number increases.

An observation‐space formulation of variational assimilation using a restricted preconditioned conjugate gradient algorithm

AbstractWe consider parameter estimation problems involving a set of m physical observations, where an unknown vector of n parameters is defined as the solution of a nonlinear least‐squares problem. We assume that the problem is regularized by a quadratic penalty term. When solution techniques based on successive linearization are considered, as in the incremental four‐dimensional variational (4D‐Var) techniques for data assimilation, a sequence of linear systems with particular structure has to be solved. We exhibit a subspace of dimension m that contains the solution of these linear systems, and derive a variant of the conjugate gradient algorithm that is more efficient in terms of memory and computational costs than its standard form, when m is smaller than n. The new algorithm, which we call the Restricted Preconditioned Conjugate Gradient (RPCG), can be viewed as an alternative to the so‐called Physical‐space Statistical Analysis System (PSAS) algorithm, which is another approach to solve the linear problem. In addition, we show that the non‐monotone and somehow chaotic behaviour of PSAS algorithm when viewed in the model space, experimentally reported by some authors, can be fully suppressed in RPCG.Moreover, since preconditioning and re‐orthogonalization of residuals vectors are often used in practice to accelerate convergence in high‐dimension data assimilation, we show how to reformulate these techniques within subspaces of dimension m in RPCG. Numerical experiments are reported, on an idealized data assimilation system based on the heat equation, that clearly show the effectiveness of our algorithm for large‐scale problems. Copyright © 2009 Royal Meteorological Society

A dimension-reduced method of sensitivity analysis for stochastic user equilibrium assignment model

This paper gives a new dimension-reduced method of sensitivity analysis for perturbed stochastic user equilibrium assignment (SUEA) model based on the relation between its Lagrange function and logarithmic barrier function combined with a Courant quadratic penalty term. The advantage of this method is of smaller dimension than general sensitivity analysis and reducing complexity. Firstly, it presents the dimension-reduced sensitivity results of the general nonlinear programming perturbation problem and the improved results when the objective or constraint functions are not twice continuously differentiable. Then it proves the corresponding conclusion of SUEA with smooth or non-smooth cost functions by the method of converting constraint conditions and decision variables. Finally, two corresponding examples (smooth and non-smooth) are given to illustrate the feasibility of this method.

Simultaneous constraining of model and data smoothness for regularization of geophysical inverse problems

Summary In this paper, a new approach is introduced to solve ill-posed linear inverse problems in geophysics. Our method combines classical quadratic regularization and data smoothing by imposing constraints on model and data smoothness simultaneously. When imposing a quadratic penalty term in the data space to control smoothness of the data predicted by classical zero-order regularization, the method leads to a direct regularization in standard form, which is simple to be implemented and ensures that the estimated model is smooth. In addition, by enforcing Tikhonov's predicted data to be sparse in a wavelet domain, the idea leads to an efficient regularization algorithm with two superior properties. First, the algorithm ensures the smoothness of the estimated model while substantially preserving the edges of it, so, it is well suited for recovering piecewise smooth/constant models. Second, parsimony of wavelets on the columns of the forward operator and existence of a fast wavelet transform algorithm provide an efficient sparse representation of the forward operator matrix. The reduced size of the forward operator makes the solution of large-scale problems straightforward, because during the inversion process, only sparse matrices need to be stored, which reduces the memory required. Additionally, all matrix–vector multiplications are carried out in sparse form, reducing CPU time. Applications on both synthetic and real 1-D seismic-velocity estimation experiments illustrate the idea. The performance of the method is compared with that of classical quadratic regularization, total-variation regularization and a two-step, wavelet-based, inversion method.

Unit Commitment by Separable Augmented Lagrangian Relaxation

The non-separable quadratic penalty terms create an inherent difficulty when applying the standard augmented Lagrangian relaxation(ALR) method for decomposing the unit commitment problem into independent subproblems. This paper presents a separable augmented Lagrangian relaxation method for solving the unit commitment problem. The proposed method is designed to have a separable structure by introducing the quadratic terms with additional auxiliary terms in the augmented Lagrangian function. Numerical results are presented to validate the effectiveness of the proposed method.

Revisit of Linear-Quadratic Optimal Control

The classical finite-dimensional linear-quadratic optimal control problem is revisited. A new linear-quadratic control problem with linear state penalty terms but without quadratic state penalty terms, is introduced. An optimal control exists and the closed-form optimal solution is given. It is remarkable that feedback action plays no role and state information does not feature in the optimal control. The optimal cost function, rather than being quadratic, is linear in the initial state.