Proximal Method Of Multipliers Research Articles

Generalizations of the proximal method of multipliers in convex optimization

Generalizations of the proximal method of multipliers in convex optimization

Waveform Design for Watermark Framework Based DFRC System With Application on Joint SAR Imaging and Communication

In this article, the watermarking framework for electromagnetic systems is established with nonblind, semiblind, and blind watermarking demodulation processes mitigated to applications like radar detection, synchronization, integrated dual function, etc. Desired for similar advantages and trade-offs of watermarking technology, the dual function radar and communication (DFRC) system is specifically concerned for covert communication and low possibility of interception (LPI) radar sensing. In this respect, we propose a novel DFRC waveform design method where peak sidelobe level (PSL) of autocorrelation function (ACF) is considered as the figure of merit with information embedded via discrete Fourier transform (DFT) watermarking strategy. Meanwhile, the peak-to-average ratio (PAR) and energy constraints are forced to ensure compatibility with current hardware technique. To handle the resulting NP-hard design problems, the proximal method of multipliers (PMM) is employed with the overall computational burden linear with the amount of information per pulse and quadratic with respect to the code length. Finally, numerical and experimental results are provided to evaluate the effectiveness of the proposed DFRC waveform design scheme with application in joint synthetic aperture radar (SAR) imaging and communication.

Solving Stochastic Optimization with Expectation Constraints Efficiently by a Stochastic Augmented Lagrangian-Type Algorithm

This paper considers the problem of minimizing a convex expectation function with a set of inequality convex expectation constraints. We propose a stochastic augmented Lagrangian-type algorithm—namely, the stochastic linearized proximal method of multipliers—to solve this convex stochastic optimization problem. This algorithm can be roughly viewed as a hybrid of stochastic approximation and the traditional proximal method of multipliers. Under mild conditions, we show that this algorithm exhibits [Formula: see text] expected convergence rates for both objective reduction and constraint violation if parameters in the algorithm are properly chosen, where K denotes the number of iterations. Moreover, we show that, with high probability, the algorithm has a [Formula: see text] constraint violation bound and [Formula: see text] objective bound. Numerical results demonstrate that the proposed algorithm is efficient. History: Accepted byAntonio Frangioni, Area Editor for Design & Analysis ofAlgorithms—Continuous. Funding: This work was supported by the National Natural Science Foundation of China [Grants 11971089, 11731013, 12071055, and 11871135]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplementary Information [ https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.1228 ] or is available from the IJOC GitHub software repository ( https://github.com/INFORMSJoC ) at http://dx.doi.org/10.5281/zenodo.6818229 .

Convergence results of two-step inertial proximal point algorithm

This paper proposes a two-point inertial proximal point algorithm to find zero of maximal monotone operators in Hilbert spaces. We obtain weak convergence results and non-asymptotic O(1/n) convergence rate of our proposed algorithm in non-ergodic sense. Applications of our results to various well-known convex optimization methods, such as the proximal method of multipliers and the alternating direction method of multipliers are given. Numerical results are given to demonstrate the accelerating behaviors of our method over other related methods in the literature.

Regrets of proximal method of multipliers for online non-convex optimization with long term constraints

The online optimization problem with non-convex loss functions over a closed convex set, coupled with a set of inequality (possibly non-convex) constraints is a challenging online learning problem. A proximal method of multipliers with quadratic approximations (named as OPMM) is presented to solve this online non-convex optimization with long term constraints. Regrets of the violation of Karush-Kuhn-Tucker conditions of OPMM for solving online non-convex optimization problems are analyzed. Under mild conditions, it is shown that this algorithm exhibits \({{\mathcal {O}}}(T^{-1/8})\) Lagrangian gradient violation regret, \({{\mathcal {O}}}(T^{-1/8})\) constraint violation regret and \({{\mathcal {O}}}(T^{-1/4})\) complementarity residual regret if parameters in the algorithm are properly chosen, where T denotes the number of time periods. For the case that the objective is a convex quadratic function, we demonstrate that the regret of the objective reduction can be established even the feasible set is non-convex. For the case when the constraint functions are convex, if the solution of the subproblem in OPMM is obtained by solving its dual, OPMM is proved to be an implementable projection method for solving the online non-convex optimization problem.

Stochastic Approximation Proximal Method of Multipliers for Convex Stochastic Programming

This paper considers the problem of minimizing a convex expectation function over a closed convex set, coupled with a set of inequality convex expectation constraints. We present a new stochastic approximation proximal method of multipliers to solve this convex stochastic optimization problem. We analyze regrets of the proposed method for solving convex stochastic optimization problems. Under mild conditions, we show that this method exhibits sublinear regret for both objective reduction and constraint violation if parameters in the algorithm are properly chosen. Moreover, we investigate the high probability performance of the proposed method under the standard light-tail assumption. Funding: This work was supported by Fundamental Research Funds for the Central Universities [Grant DUT20LK33]; Dalian High-Level Talent Innovation Program [Grant 2020RD09]; National Natural Science Foundation of China [Grant 11731013, 11871135, 11971089, and 12071055].

Cognitive waveform design with desired spectrum-autocorrelation properties

This paper proposes a novel cognitive waveform design methodology aimed at shaping a specified spectrum and achieving good autocorrelation properties in terms of the spectrum band notch and local autocorrelation peak sidelobe level (A-PSL). To this end, the Pareto optimization framework is introduced that minimizes the lp-norm of multi-objective functions both encompassing the stopband energy and local A-PSL. In addition, along with energy and peak-to-average power ratio constraints to ensure hardware compatibility, the out-of-band (OOB) spectrum energy and the rest autocorrelation sidelobe level are restricted to resist the OOB strong interference (also prevent spectrum leakage) and enhance multiple target detectability, respectively. To tackle the resultant nonconvex nonsmooth (in general NP-hard) design problem, an iterative majorization minimization proximal method of multipliers (MM-PMM) algorithm is developed along with each iteration involving multiple subproblems which are solved through a majorization method or closed-form solutions. In particular, resorting to Wiener-Khintchine theorem, the developed algorithm is efficiently implemented via the fast fourier transform and element-wise arithmetic. Extensive numerical examples highlight that the synthesized waveform shares the desired spectrum-autocorrelation properties and is capable of enhancing the weak target detection and anti-jamming performance in comparison with some counterparts.

A Hessian Inversion-Free Exact Second Order Method for Distributed Consensus Optimization

We consider a standard distributed consensus optimization problem where a set of agents connected over an undirected network minimize the sum of their individual (local) strongly convex costs. Alternating Direction Method of Multipliers (ADMM) and Proximal Method of Multipliers (PMM) have been proved to be effective frameworks for design of exact distributed second order methods (involving calculation of local cost Hessians). However, existing methods involve explicit calculation of local Hessian inverses at each iteration that may be very costly when the dimension of the optimization variable is large. In this article, we develop a novel method, termed Inexact Newton method for Distributed Optimization (INDO), that alleviates the need for Hessian inverse calculation. INDO follows the PMM framework but, unlike existing work, approximates the Newton direction through a generic fixed point method (e.g., Jacobi Overrelaxation) that does not involve Hessian inverses. We prove exact global linear convergence of INDO and provide analytical studies on how the degree of inexactness in the Newton direction calculation affects the overall method's convergence factor. Numerical experiments on several real data sets demonstrate that INDO's speed is on par (or better) as state of the art methods iteration-wise, hence having a comparable communication cost. At the same time, for sufficiently large optimization problem dimensions <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n$</tex-math></inline-formula> (even at <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n$</tex-math></inline-formula> on the order of couple of hundreds), INDO achieves savings in computational cost by at least an order of magnitude.

An Interior Point-Proximal Method of Multipliers for Linear Positive Semi-Definite Programming

In this paper we generalize the Interior Point-Proximal Method of Multipliers (IP-PMM) presented in Pougkakiotis and Gondzio (Comput Optim Appl 78:307–351, 2021. https://doi.org/10.1007/s10589-020-00240-9) for the solution of linear positive Semi-Definite Programming (SDP) problems, allowing inexactness in the solution of the associated Newton systems. In particular, we combine an infeasible Interior Point Method (IPM) with the Proximal Method of Multipliers (PMM) and interpret the algorithm (IP-PMM) as a primal-dual regularized IPM, suitable for solving SDP problems. We apply some iterations of an IPM to each sub-problem of the PMM until a satisfactory solution is found. We then update the PMM parameters, form a new IPM neighbourhood, and repeat this process. Given this framework, we prove polynomial complexity of the algorithm, under mild assumptions, and without requiring exact computations for the Newton directions. We furthermore provide a necessary condition for lack of strong duality, which can be used as a basis for constructing detection mechanisms for identifying pathological cases within IP-PMM.

Accelerated proximal point method for maximally monotone operators

This paper proposes an accelerated proximal point method for maximally monotone operators. The proof is computer-assisted via the performance estimation problem approach. The proximal point method includes various well-known convex optimization methods, such as the proximal method of multipliers and the alternating direction method of multipliers, and thus the proposed acceleration has wide applications. Numerical experiments are presented to demonstrate the accelerating behaviors.

A new preconditioning approach for an interior point‐proximal method of multipliers for linear and convex quadratic programming

AbstractIn this article, we address the efficient numerical solution of linear and quadratic programming problems, often of large scale. With this aim, we devise an infeasible interior point method, blended with the proximal method of multipliers, which in turn results in a primal‐dual regularized interior point method. Application of this method gives rise to a sequence of increasingly ill‐conditioned linear systems which cannot always be solved by factorization methods, due to memory and CPU time restrictions. We propose a novel preconditioning strategy which is based on a suitable sparsification of the normal equations matrix in the linear case, and also constitutes the foundation of a block‐diagonal preconditioner to accelerate MINRES for linear systems arising from the solution of general quadratic programming problems. Numerical results for a range of test problems demonstrate the robustness of the proposed preconditioning strategy, together with its ability to solve linear systems of very large dimension.

An interior point-proximal method of multipliers for convex quadratic programming

In this paper we combine an infeasible Interior Point Method (IPM) with the Proximal Method of Multipliers (PMM). The resulting algorithm (IP-PMM) is interpreted as a primal-dual regularized IPM, suitable for solving linearly constrained convex quadratic programming problems. We apply few iterations of the interior point method to each sub-problem of the proximal method of multipliers. Once a satisfactory solution of the PMM sub-problem is found, we update the PMM parameters, form a new IPM neighbourhood and repeat this process. Given this framework, we prove polynomial complexity of the algorithm, under standard assumptions. To our knowledge, this is the first polynomial complexity result for a primal-dual regularized IPM. The algorithm is guided by the use of a single penalty parameter; that of the logarithmic barrier. In other words, we show that IP-PMM inherits the polynomial complexity of IPMs, as well as the strict convexity of the PMM sub-problems. The updates of the penalty parameter are controlled by IPM, and hence are well-tuned, and do not depend on the problem solved. Furthermore, we study the behavior of the method when it is applied to an infeasible problem, and identify a necessary condition for infeasibility. The latter is used to construct an infeasibility detection mechanism. Subsequently, we provide a robust implementation of the presented algorithm and test it over a set of small to large scale linear and convex quadratic programming problems. The numerical results demonstrate the benefits of using regularization in IPMs as well as the reliability of the method.

A primal-dual dynamical approach to structured convex minimization problems

In this paper we propose a primal-dual dynamical approach to the minimization of a structured convex function consisting of a smooth term, a nonsmooth term, and the composition of another nonsmooth term with a linear continuous operator. In this scope we introduce a dynamical system for which we prove that its trajectories asymptotically converge to a saddle point of the Lagrangian of the underlying convex minimization problem as time tends to infinity. In addition, we provide rates for both the violation of the feasibility condition by the ergodic trajectories and the convergence of the objective function along these ergodic trajectories to its minimal value. Explicit time discretization of the dynamical system results in a numerical algorithm which is a combination of the linearized proximal method of multipliers and the proximal ADMM algorithm.

The rate of convergence of proximal method of multipliers for second-order cone optimization problems

In this paper we consider a proximal method of multipliers (PMM) for a nonlinear second-order cone optimization problem. With the assumptions of constraint nondegeneracy, strict complementarity and second-order sufficient condition, we estimate the local convergence rate of PMM to be linear or superlinear, which depends on the strategy of parameter selection.

The rate of convergence of proximal method of multipliers for nonlinear programming

We analyze the rate of convergence of the proximal method of multipliers for non-convex nonlinear programming problems. First, we prove, under the strict complementarity condition, that the rate of convergence of the proximal method of multipliers is linear and the ratio constant is proportional to 1/c when the ratio is small enough, which implies that the rate of convergence of the proximal method of multipliers is superlinear when the parameter c increases to . Second, we prove that, without strict complementarity condition, the rate of convergence of the proximal method of multipliers is proportional to 1/c when c exceeds a threshold.

Minimum local peak sidelobe level waveform design with correlation and/or spectral constraints

The problem of designing waveforms with specified aperiodic/periodic correlation and spectral properties has received much attention because of its wide applicability in sonar, radar, and communications. In particular, a waveform with low peak sidelobe level (PSL) of autocorrelation can improve the detection performance of a weak target masked by a nearby strong target in range compression radar systems, and that with adaptive spectrum nulls/notches can make the sensing systems not be interfered by other electromagnetic equipment. On the other hand, a constant modulus (CM) waveform can maximize the transmit power efficiency of the system. Whereas, it is difficult to design a CM waveform with minimum local PSL and specified autocorrelation and/or spectral properties due to numerous nonconvex inequality and equality constraints. To develop efficient algorithms for this problem, we first express the waveform autocorrelation in the frequency domain, then use the proximal method of multipliers to tackle the resultant nonconvex and nonsmooth constrained optimization. Numerical examples have shown that the proposed methods can produce the waveforms satisfying these challenging requirements.

The rate of convergence of proximal method of multipliers for nonlinear semidefinite programming

ABSTRACTThe proximal method of multipliers was proposed by Rockafellar [Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math Oper Res. 1976;1:97–116] for solving convex programming and it is a kind of proximal point method applied to convex programming. In this paper, we apply this method for solving nonlinear semidefinite programming problems, in which subproblems have better properties than those from the augmented Lagrange method. We prove that, under the linear independence constraint qualification and the strong second-order sufficiency optimality condition, the rate of convergence of the proximal method of multipliers, for a nonlinear semidefinite programming problem, is linear and the ratio constant is proportional to 1/c, where c is the penalty parameter that exceeds a threshold . Moreover, the rate of convergence of the proximal method of multipliers is superlinear when the parameter c increases to .

The rate of convergence of proximal method of multipliers for equality constrained optimization problems

The proximal method of multipliers was proposed by Rockafellar (Math Oper Res 1:97–116, 1976) for solving convex programming and it is a kind of proximal point method for convex programming. In this paper, we apply this method for equality constrained optimization problems, in which subproblems have better properties than those from the augmented Lagrange method. We prove that, under linear independence constraint qualification and the second-order sufficiency optimality condition, the rate of convergence of the proximal method of multipliers, for the equality constrained optimization problem, is linear and the ratio constant is proportional to 1 / c, where c is the penalty parameter that exceeds a threshold $$c_*> 0$$ . Moreover, the rate of convergence of the proximal method of multipliers is superlinear when the parameter c increases to $$+\infty $$ .

Iteration-complexity of a Rockafellar's proximal method of multipliers for convex programming based on second-order approximations

ABSTRACTThis paper studies the iteration-complexity of a new primal-dual algorithm based on Rockafellar's proximal method of multipliers (PMM) for solving smooth convex programming problems with inequality constraints. In each step, either a step of Rockafellar's PMM for a second-order model of the problem is computed or a relaxed extragradient step is performed. The resulting algorithm is a (large-step) relaxed hybrid proximal extragradient (r-HPE) method of multipliers, which combines Rockafellar's PMM with the r-HPE method. We obtain pointwise and ergodic global convergence rates at the price of solving, at each iteration, quadratic quadratically constrained convex programming problems. These convergence rates are superior to the corresponding pointwise and ergodic currently known for standard proximal-point methods, thanks to the incorporation of second-order information. To the best of our knowledge, this is the first time that the above mentioned rates and results are obtained for solving the smooth convex programming problems with inequality constraints.

A Decentralized Second-Order Method with Exact Linear Convergence Rate for Consensus Optimization

This paper considers decentralized consensus optimization problems where different summands of a global objective function are available at nodes of a network that can communicate with neighbors only. The proximal method of multipliers is considered as a powerful tool that relies on proximal primal descent and dual ascent updates on a suitably defined augmented Lagrangian. The structure of the augmented Lagrangian makes this problem nondecomposable, which precludes distributed implementations. This problem is regularly addressed by the use of the alternating direction method of multipliers. The exact second-order method (ESOM) is introduced here as an alternative that relies on: First, the use of a separable quadratic approximation of the augmented Lagrangian, and second, a truncated Taylor's series to estimate the solution of the first-order condition imposed on the minimization of the quadratic approximation of the augmented Lagrangian. The sequences of primal and dual variables generated by ESOM are shown to converge linearly to their optimal arguments when the aggregate cost function is strongly convex and its gradients are Lipschitz continuous. Numerical results demonstrate advantages of ESOM relative to decentralized alternatives in solving least-squares and logistic regression problems.