Inexact Augmented Lagrangian Method Research Articles

A new inexact gradient descent method with applications to nonsmooth convex optimization

The paper proposes and develops a novel inexact gradient method (IGD) for minimizing C 1 -smooth functions with Lipschitzian gradients, i.e. for problems of C 1 , 1 optimization. We show that the sequence of gradients generated by IGD converges to zero. The convergence of iterates to stationary points is guaranteed under the Kurdyka-Łojasiewicz (KL) property of the objective function with convergence rates depending on the KL exponent. The newly developed IGD is applied to designing two novel gradient-based methods of nonsmooth convex optimization such as the inexact proximal point methods (GIPPM) and the inexact augmented Lagrangian method (GIALM) for convex programs with linear equality constraints. These two methods inherit global convergence properties from IGD and are confirmed by numerical experiments to have practical advantages over some well-known algorithms of nonsmooth convex optimization.

Inexact augmented Lagrangian method-based full-waveform inversion with randomized singular value decomposition

Abstract The main advantage of full-waveform inversion (FWI) is the ability to obtain useful subsurface structure information, such as velocity and density, from complex seismic data. We have developed a novel inversion algorithm to improve the capability of FWI to achieve high-resolution imaging, even under complex conditions caused by random noise contamination, initial model dependence, or the selection of parameters to be estimated. Our algorithm considers an effective image processing and dimension reduction tool, randomized singular value decomposition-weighted truncated nuclear norm regularization, for embedding FWI to achieve high-resolution imaging results. This algorithm obtains a truncated matrix approximating the original matrix by reducing the rank of the velocity increment matrix, thus achieving the truncation of noisy data, with the truncation range controlled by weighted truncated nuclear norm regularization. Subsequently, we employ an inexact augmented Lagrangian method algorithm in the optimization to compress the solution space range, thus relaxing the dependence of FWI and randomized singular value decomposition-weighted truncated nuclear norm regularization on the initial model and accelerating the convergence rate of the objective function. We tested on one set of synthetic data, and the results show that, compared with traditional FWI, our method can more effectively suppress the impact of random noise, thus obtaining higher resolution and more accurate subsurface model information. This work indicates that the combination of randomized singular value decomposition-weighted truncated nuclear norm regularization and FWI is an effective imaging strategy that can help to solve the challenges faced by traditional FWI.

Zero-One Composite Optimization: Lyapunov Exact Penalty and a Globally Convergent Inexact Augmented Lagrangian Method

We consider the problem of minimizing the sum of a smooth function and a composition of a zero-one loss function with a linear operator, namely the zero-one composite optimization problem (0/1-COP). It has a vast body of applications, including the support vector machine (SVM), calcium dynamics fitting (CDF), one-bit compressive sensing (1-bCS), and so on. However, it remains challenging to design a globally convergent algorithm for the original model of 0/1-COP because of the nonconvex and discontinuous zero-one loss function. This paper aims to develop an inexact augmented Lagrangian method (IALM), in which the generated whole sequence converges to a local minimizer of 0/1-COP under reasonable assumptions. In the iteration process, IALM performs minimization on a Lyapunov function with an adaptively adjusted multiplier. The involved Lyapunov penalty subproblem is shown to admit the exact penalty theorem for 0/1-COP, provided that the multiplier is optimal in the sense of the proximal-type stationarity. An efficient zero-one Bregman alternating linearized minimization algorithm is also designed to achieve an approximate solution of the underlying subproblem in finite steps. Numerical experiments for handling SVM, CDF, and 1-bCS demonstrate the satisfactory performance of the proposed method in terms of solution accuracy and time efficiency. Funding: This work was supported by the Fundamental Research Funds for the Central Universities [Grant 2022YJS099] and the National Natural Science Foundation of China [Grants 12131004 and 12271022].

A linear algebra perspective on the random multi-block ADMM: the QP case

Embedding randomization procedures in the Alternating Direction Method of Multipliers (ADMM) has recently attracted an increasing amount of interest as a remedy to the fact that the direct multi-block generalization of ADMM is not necessarily convergent. Even if, in practice, the introduction of such techniques could mitigate the diverging behaviour of the multi-block extension of ADMM, from the theoretical point of view, it can ensure just the convergence in expectation, which may not be a good indicator of its robustness and efficiency. In this work, analysing the strongly convex quadratic programming case from a linear algebra perspective, we interpret the block Gauss–Seidel sweep performed by the multi-block ADMM in the context of the inexact Augmented Lagrangian Method. Using the proposed analysis, we are able to outline an alternative technique to those present in the literature which, supported from stronger theoretical guarantees, is able to ensure the convergence of the multi-block generalization of the ADMM method.

Iteration-Complexity of First-Order Augmented Lagrangian Methods for Convex Conic Programming

In this paper we consider a class of convex conic programming. In particular, we first propose an inexact augmented Lagrangian (I-AL) method that resembles the classical I-AL method for solving this problem, in which the augmented Lagrangian subproblems are solved approximately by a variant of Nesterov’s optimal first-order method. We show that the total number of first-order iterations of the proposed I-AL method for finding an -KKT solution is at most . We then propose an adaptively regularized I-AL method and show that it achieves a first-order iteration complexity , which significantly improves existing complexity bounds achieved by first-order I-AL methods for finding an -KKT solution. Our complexity analysis of the I-AL methods is based on a sharp analysis of the inexact proximal point algorithm (PPA) and the connection between the I-AL methods and inexact PPA. It is vastly different from existing complexity analyses of the first-order I-AL methods in the literature, which typically regard the I-AL methods as an inexact dual gradient method.

An Inexact Augmented Lagrangian Method for Second-Order Cone Programming with Applications

In this paper, we adopt the augmented Lagrangian method (ALM) to solve convex quadratic second-order cone programming problems (SOCPs). Fruitful results on the efficiency of the ALM have been estab...

Zero norm based analysis model for image smoothing and reconstruction

The sparsity-based approaches have demonstrated promising performance in image processing. In this paper, for better preservation of the salient edge structures of images, we propose an ℓ 0 + ℓ 2-norm based analysis model， which requires solving a challenging non-separable ℓ 0-norm related minimization problem, and we also propose an inexact augmented Lagrangian method with proven convergence to a local minimum. Extensive experiments in image smoothing, including texture removal and context smoothing, show that our method achieves better visual results over various sparsity-based models and the CNN method. Also, experiments on sparse view CT reconstruction further validate the advantage of the proposed method.

Augmented Lagrangian optimization under fixed-point arithmetic

In this paper, we propose an inexact Augmented Lagrangian Method (ALM) for the optimization of convex and nonsmooth objective functions subject to linear equality constraints and box constraints where errors are due to fixed-point data. To prevent data overflow we also introduce a projection operation in the multiplier update. We analyze theoretically the proposed algorithm and provide convergence rate results and bounds on the accuracy of the optimal solution. Since iterative methods are often needed to solve the primal subproblem in ALM, we also propose an early stopping criterion that is simple to implement on embedded platforms, can be used for problems that are not strongly convex, and guarantees the precision of the primal update. To the best of our knowledge, this is the first fixed-point ALM that can handle non-smooth problems, data overflow, and can efficiently and systematically utilize iterative solvers in the primal update. Numerical simulation studies on a logistic regression problem are presented that illustrate the proposed method.

An inexact augmented Lagrangian method for nonlinear dispersion-curve inversion using Dix-type global linear approximation

Dispersion-curve inversion of Rayleigh waves to infer subsurface shear-wave velocity is a long-standing problem in seismology. Due to nonlinearity and ill-posedness, sophisticated regularization techniques are required to solve the problem for a stable velocity model. We have formulated the problem as a minimization problem with nonlinear operator constraint and then solve it by using an inexact augmented Lagrangian method, taking advantage of the Haney-Tsai Dix-type relation (a global linear approximation of the nonlinear forward operator). This replaces the original regularized nonlinear problem with iterative minimization of a more tractable regularized linear problem followed by a nonlinear update of the phase velocity (data) in which the update can be performed accurately with any forward modeling engine, for example, the finite-element method. The algorithm allows discretizing the medium with thin layers (for the finite-element method) and thus omitting the layer thicknesses from the unknowns and also allows incorporating arbitrary regularizations to shape the desired velocity model. In this research, we use total variation regularization to retrieve the shear-wave velocity model. We use two synthetic and two real data examples to illustrate the performance of the inversion algorithm with total variation regularization. We find that the method is fast and stable, and it converges to the solution of the original nonlinear problem.

On the complexity of an augmented Lagrangian method for nonconvex optimization

Abstract In this paper we study the worst-case complexity of an inexact augmented Lagrangian method for nonconvex constrained problems. Assuming that the penalty parameters are bounded we prove a complexity bound of $\mathcal{O}(|\log (\epsilon )|)$ outer iterations for the referred algorithm to generate an $\epsilon$-approximate KKT point for $\epsilon \in (0,1)$. When the penalty parameters are unbounded we prove an outer iteration complexity bound of $\mathcal{O}(\epsilon ^{-2/(\alpha -1)} )$, where $\alpha>1$ controls the rate of increase of the penalty parameters. For linearly constrained problems these bounds yield to evaluation complexity bounds of $\mathcal{O}(|\log (\epsilon )|^{2}\epsilon ^{-2})$ and $\mathcal{O}(\epsilon ^{- (\frac{2(2+\alpha )}{\alpha -1}+2 )})$, respectively, when appropriate first-order methods ($p=1$) are used to approximately solve the unconstrained subproblems at each iteration. In the case of problems having only linear equality constraints the latter bounds are improved to $\mathcal{O}(|\log (\epsilon )|^{2}\epsilon ^{-(p+1)/p})$ and $\mathcal{O}(\epsilon ^{-(\frac{4}{\alpha -1}+\frac{p+1}{p})})$, respectively, when appropriate $p$-order methods ($p\geq 2$) are used as inner solvers.

A Discriminative Projection and Representation-Based Classification Framework for Face Recognition

The sparse representation-based classifier (SRC) has been developed and verified as having great potential for real-world face recognition. In this paper, we propose a discriminative projection and representation-based classification (DPRC) method to enhance the discriminant ability of the SRC. The proposed method first obtains a discriminative projection matrix not only maximizing the ratio of the distance within interclass over the distance within intraclass, but also minimizing the linear approximation error within intraclass. Then it maps the original data onto the discriminative space, and adopts an SRC method to obtain the final solution. An inexact augmented Lagrangian method of multiplier is proposed for finding the optimal representation vector in our framework, and a proximal alternating minimization method is adopted to the iteration subproblems of the proposed method. The proposed method is proven to have the subsequence convergence property. Experimental results on Yale, ORL, and AR face image databases demonstrate that, compared with some existing feature extraction methods based on the SRC, the proposed DPRC method is more efficient.

An adaptive primal-dual framework for nonsmooth convex minimization

We propose a new self-adaptive and double-loop smoothing algorithm to solve composite, nonsmooth, and constrained convex optimization problems. Our algorithm is based on Nesterov’s smoothing technique via general Bregman distance functions. It self-adaptively selects the number of iterations in the inner loop to achieve a desired complexity bound without requiring to set the accuracy a priori as in variants of augmented Lagrangian methods (ALM). We prove $$\mathcal {O}\left( \frac{1}{k}\right) $$ -convergence rate on the last iterate of the outer sequence for both unconstrained and constrained settings in contrast to ergodic rates which are common in ALM as well as alternating direction method-of-multipliers literature. Compared to existing inexact ALM or quadratic penalty methods, our analysis does not rely on the worst-case bounds of the subproblem solved by the inner loop. Therefore, our algorithm can be viewed as a restarting technique applied to the ASGARD method in Tran-Dinh et al. (SIAM J Optim 28(1):96–134, 2018) but with rigorous theoretical guarantees or as an inexact ALM with explicit inner loop termination rules and adaptive parameters. Our algorithm only requires to initialize the parameters once, and automatically updates them during the iteration process without tuning. We illustrate the superiority of our methods via several examples as compared to the state-of-the-art.

An inexact augmented Lagrangian method for computing strongly orthogonal decompositions of tensors

A strongly orthogonal decomposition of a tensor is a rank one tensor decomposition with the two component vectors in each mode of any two rank one tensors are either colinear or orthogonal. A strongly orthogonal decomposition with few number of rank one tensors is favorable in applications, which can be represented by a matrix-tensor multiplication with orthogonal factor matrices and a sparse tensor; and such a decomposition with the minimum number of rank one tensors is a strongly orthogonal rank decomposition. Any tensor has a strongly orthogonal rank decomposition. In this article, computing a strongly orthogonal rank decomposition is equivalently reformulated as solving an optimization problem. Different from the ill-posedness of the usual optimization reformulation for the tensor rank decomposition problem, the optimization reformulation of the strongly orthogonal rank decomposition of a tensor is well-posed. Each feasible solution of the optimization problem gives a strongly orthogonal decomposition of the tensor; and a global optimizer gives a strongly orthogonal rank decomposition, which is however difficult to compute. An inexact augmented Lagrangian method is proposed to solve the optimization problem. The augmented Lagrangian subproblem is solved by a proximal alternating minimization method, with the advantage that each subproblem has a closed formula solution and the factor matrices are kept orthogonal during the iteration. Thus, the algorithm always can return a feasible solution and thus a strongly orthogonal decomposition for any given tensor. Global convergence of this algorithm to a critical point is established without any further assumption. Extensive numerical experiments are conducted, and show that the proposed algorithm is quite promising in both efficiency and accuracy.

Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming

Augmented Lagrangian method (ALM) has been popularly used for solving constrained optimization problems. Practically, subproblems for updating primal variables in the framework of ALM usually can only be solved inexactly. The convergence and local convergence speed of ALM have been extensively studied. However, the global convergence rate of the inexact ALM is still open for problems with nonlinear inequality constraints. In this paper, we work on general convex programs with both equality and inequality constraints. For these problems, we establish the global convergence rate of the inexact ALM and estimate its iteration complexity in terms of the number of gradient evaluations to produce a primal and/or primal-dual solution with a specified accuracy. We first establish an ergodic convergence rate result of the inexact ALM that uses constant penalty parameters or geometrically increasing penalty parameters. Based on the convergence rate result, we then apply Nesterov’s optimal first-order method on each primal subproblem and estimate the iteration complexity of the inexact ALM. We show that if the objective is convex, then $$O(\varepsilon ^{-1})$$ gradient evaluations are sufficient to guarantee a primal $$\varepsilon $$ -solution in terms of both primal objective and feasibility violation. If the objective is strongly convex, the result can be improved to $$O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)$$ . To produce a primal-dual $$\varepsilon $$ -solution, more gradient evaluations are needed for convex case, and the number is $$O(\varepsilon ^{-\frac{4}{3}})$$ , while for strongly convex case, the number is still $$O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)$$ . Finally, we establish a nonergodic convergence rate result of the inexact ALM that uses geometrically increasing penalty parameters. This result is established only for the primal problem. We show that the nonergodic iteration complexity result is in the same order as that for the ergodic result. Numerical experiments on quadratically constrained quadratic programming are conducted to compare the performance of the inexact ALM with different settings.

An inexact augmented Lagrangian multiplier method for solving quadratic complementary problems: An adapted algorithmic framework combining specific resolution techniques

In this paper, an inexact augmented Lagrangian multiplier method (ALM) is designed for solving the quadratic complementarity problem (QCP). The primary goal is proposing an algorithm exploring the structures of QCPs and incorporating them into the implementation. In the implementation, an inexact proximal alternating minimization method is employed to solve the augmented Lagrangian subproblems. As a result, all the subproblems are either strongly convex quadratic programs or linear equations with positive semidefinite symmetric coefficient matrices, which in turn can either be solved with explicitly formula solutions, or by an accelerated proximal gradient method, or a conjugate gradient method. Convergence rates of these subproblems are given, and global convergence of the ALM algorithm is established under mild assumptions. Preliminary numerical experiments show that the algorithm is feasible and efficient.

On the convergence of inexact augmented Lagrangian methods for problems with convex constraints

We consider the augmented Lagrangian method (ALM) for constrained optimization problems in the presence of convex inequality and convex abstract constraints. We focus on the case where the Lagrangian sub-problems are solved up to approximate stationary points, with increasing accuracy. We analyze two different criteria of approximate stationarity for the sub-problems and we prove the global convergence to stationary points of ALM in both cases.

Kernel Low-Rank Multitask Learning in Variational Mode Decomposition Domain for Multi-/Hyperspectral Classification

Multitask learning (MTL) has recently yielded impressive results for classification of remotely sensed data due to its ability to incorporate shared information across multiple tasks. However, it remains a challenging issue to achieve robust classification results in the case that the data are from nonlinear subspaces. In this paper, we propose a kernel low-rank MTL (KL-MTL) method to handle multiple features from the 2-D variational mode decomposition (2-D-VMD) domain for multi-/hyperspectral classification. On the one hand, a nonrecursive 2-D-VMD method is applied to extract various features [i.e., intrinsic mode functions (IMFs)] of the original data concurrently. Compared with the existing 2-D empirical mode decomposition, 2-D-VMD has much stronger mathematical foundation and does not need any recursive sifting process. On the other hand, KL-MTL is proposed for classification by taking the extracted IMFs as features of multiple tasks. In KL-MTL, the low-rank representation formulated by nuclear norm can capture global structure of multiple tasks, while the kernel tricks are utilized for nonlinear extension of the low-rank MTL. Moreover, the optimization problem in KL-MTL is solved by the inexact augmented Lagrangian method. Compared with several state-of-the-art feature extraction and classification methods, the experimental results using both multi-/hyperspectral images demonstrate that the proposed method has satisfactory classification performance.

Simultaneous Spatial and Spectral Low-Rank Representation of Hyperspectral Images for Classification

Arising from various environmental and atmos- pheric conditions and sensor interference, spectral variations are inevitable during hyperspectral remote sensing, which degrade the subsequent hyperspectral image analysis significantly. In this paper, we propose simultaneous spatial and spectral low-rank representation (S3LRR) that can effectively suppress the within-class spectral variations for classification purposes. The S3LRR recovers an intrinsic component with the same dimension as the original image, in which both spatial and spectral low-rank priors are adopted to regularize the intrinsic component simultaneously and compensate to each other, together with robust modeling of spectral variations. Compared with existing methods that explore only the spectral low-rank prior, the novel spatial low-rank prior (i.e., low-rank prior in band-wise) can take the spatial structure information of hyperspectral images into account, which has demonstrated to be very useful. Technically, we formulate S3LRR as a constrained convex optimization problem, and solve it using the efficient inexact augmented Lagrangian multiplier method. The resulting intrinsic component is less interfered by within-class spectral variations, and more discriminatory to offer higher classification accuracy. Comprehensive experiments on benchmark data sets demonstrate that the proposed S3LRR improves classification accuracy significantly, which outperforms state-of-the-art methods.

Semi-Supervised Spectral Clustering With Structured Sparsity Regularization

Spectral clustering (SC) is one of the most widely used clustering methods. In this letter, we extend the traditional SC with a semi-supervised manner. Specifically, with the guidance of small amount of supervisory information, we build a matrix with anti-block-diagonal appearance, which is further utilized to regularize the product of the low-dimensional embedding and its transpose. Technically, we formulate the proposed model as a constrained optimization problem. Then, we relax it as a convex problem, which can be efficiently solved with the global convergence guaranteed via the inexact augmented Lagrangian multiplier method. Experimental results over four real-world datasets demonstrate that higher accuracy and normalized mutual information are achieved when compared with state-of-the-art methods.

Small target detection based on reweighted infrared patch‐image model

To further improve the effect of infrared small target detection, a reweighted infrared patch-image model is proposed. First, the authors point out that the nuclear norm in the infrared patch-image model could easily leave some sparse background edges in the target patch-image, leading to an inaccurate background estimation. Then, to overcome this defect, the reweighted nuclear norm is adopted to constrain the background patch-image, which could preserve the background edges better. Considering that some non-target sparse points could not be suppressed by only using l1 norm, the authors introduce the reweighted l1 norm to further enhance the sparsity of target image. Finally, the proposed model is formulated as a reweighted robust principal component analysis problem and solved by the inexact augmented Lagrangian multiplier method. Extensive experiments show that the proposed model outperforms the other six competitive methods in suppressing background clutter and detecting target.