General Convex Optimization Problems Research Articles

A geometric integration approach to smooth optimization: foundations of the discrete gradient method

Abstract Discrete gradient methods are geometric integration techniques that can preserve the dissipative structure of gradient flows. Due to the monotonic decay of the function values, they are well suited for general convex and nonconvex optimization problems. Both zero- and first-order algorithms can be derived from the discrete gradient method by selecting different discrete gradients. In this paper, we present a thorough analysis of the discrete gradient method for optimization that provides a solid theoretical foundation. We show that the discrete gradient method is well-posed by proving the existence of iterates for any positive time step, as well as uniqueness in some cases, and propose an efficient method for solving the associated discrete gradient equation. Moreover, we establish an $\text{O}(1/k)$ convergence rate for convex objectives and prove linear convergence if instead the Polyak–Łojasiewicz inequality is satisfied. The analysis is carried out for three discrete gradients—the Gonzalez discrete gradient, the mean value discrete gradient, and the Itoh–Abe discrete gradient—as well as for a randomised Itoh–Abe method. Our theoretical results are illustrated with a variety of numerical experiments, and we furthermore demonstrate that the methods are robust with respect to stiffness.

Evaluation of Power Pattern of Antenna Arrays With Elements Position Errors Using Quadratic Relaxation Programming

This letter presents a new method to accurately and reliably evaluate the bounds of the antenna array power pattern in the presence of elements position errors. The erroneous array power pattern is approximated by its second-order Taylor series expansion, from which a box-constrained quadratic programming problem for tolerance analysis is formulated. This problem is a general mixed convex and non-convex optimization problem, and the convex surrogate version is designed by introducing semidefinite relaxation. Subsequently, convex optimization is adopted to solve the proposed problem. The representative results are reported to give more advanced solutions in comparison to two existing methods.

Optimality conditions in a class of generalized convex optimization problems with the multiple interval-valued objective function

In this paper, a new class of generalized convex optimization problems with a multiple interval-valued objective function is considered. The concepts of B-preinvexity, B-invexity, and the generalized of B-invexity are extended to interval-valued functions. Additionally, several properties of interval-valued B-preinvex and B-invex functions are investigated. The Karush–Kuhn–Tucker necessary optimality conditions for a (weak LU-Pareto) LU-Pareto solution are established for a differentiable vector optimization problem with a multiple interval-valued objective function. Furthermore, the sufficient optimality conditions for both LU and UC order relations are derived for such interval-valued vector optimization problems under appropriate (generalized) B-invexity hypotheses. Suitable examples are provided to illustrate the aforementioned results.

General Hölder Smooth Convergence Rates Follow from Specialized Rates Assuming Growth Bounds

Often in the analysis of first-order methods for both smooth and nonsmooth optimization, assuming the existence of a growth/error bound or KL condition facilitates much stronger convergence analysis. Hence, separate analysis is typically needed for the general case and for the growth bounded cases. We give meta-theorems for deriving general convergence rates from those assuming a growth lower bound. Applying this simple but conceptually powerful tool to the proximal point, subgradient, bundle, dual averaging, gradient descent, Frank–Wolfe and universal accelerated methods immediately recovers their known convergence rates for general convex optimization problems from their specialized rates. New convergence results follow for bundle methods, dual averaging and Frank–Wolfe. Our results can lift any rate based on Hölder continuous gradients and Hölder growth bounds. Moreover, our theory provides simple proofs of optimal convergence lower bounds under Hölder growth from textbook examples without growth bounds.

From Perspective Maps to Epigraphical Projections

The projection onto the epigraph or a level set of a closed proper convex function can be achieved by finding a root of a scalar equation that involves the proximal operator as a function of the proximal parameter. This paper develops the variational analysis of this scalar equation. The approach is based on a study of the variational-analytic properties of general convex optimization problems that are (partial) infimal projections of the sum of the function in question and the perspective map of a convex kernel. When the kernel is the Euclidean norm squared, the solution map corresponds to the proximal map, and thus, the variational properties derived for the general case apply to the proximal case. Properties of the value function and the corresponding solution map—including local Lipschitz continuity, directional differentiability, and semismoothness—are derived. An SC1 optimization framework for computing epigraphical and level-set projections is, thus, established. Numerical experiments on one-norm projection illustrate the effectiveness of the approach as compared with specialized algorithms. Funding: This work was supported by Natural Sciences and Engineering Research Council of Canada (NSERC). M.P. Friedlander was supported by NSERC discovery grant [Grant RGPIN-2017-04461]. T. Hoheisel was supported by NSERC discovery grant [Grant RGPIN-2017-04035]. A. Goodwin’s work was partially supported by an NSERC summer research stipend.

Bregman Proximal Point Algorithm Revisited: A New Inexact Version and Its Inertial Variant

We study a general convex optimization problem, which covers various classic problems in different areas and particularly includes many optimal transport related problems arising in recent years. To solve this problem, we revisit the classic Bregman proximal point algorithm (BPPA) and introduce a new inexact stopping condition for solving the subproblems, which can circumvent the underlying feasibility difficulty often appearing in existing inexact conditions when the problem has a complex feasible set. Our inexact condition also covers several existing inexact conditions as special cases and hence makes our inexact BPPA (iBPPA) more flexible to fit different scenarios in practice. As an application to the standard optimal transport (OT) problem, our iBPPA with the entropic proximal term can bypass some numerical instability issues that usually plague the popular Sinkhorn's algorithm in the OT community, since our iBPPA does not require the proximal parameter to be very small for obtaining an accurate approximate solution. The iteration complexity of $O(1/k)$ and the convergence of the sequence are also established for our iBPPA under some mild conditions. Moreover, inspired by Nesterov's acceleration technique, we develop an inertial variant of our iBPPA, denoted by V-iBPPA, and establish the iteration complexity of $O(1/k^{\lambda})$, where $\lambda\geq1$ is a quadrangle scaling exponent of the kernel function. In particular, when the proximal parameter is a constant and the kernel function is strongly convex with Lipschitz continuous gradient (hence $\lambda=2$), our V-iBPPA achieves a faster rate of $O(1/k^2)$ just as existing accelerated inexact proximal point algorithms. Some preliminary numerical experiments for solving the standard OT problem are conducted to show the convergence behaviors of our iBPPA and V-iBPPA under different inexactness settings. The experiments also empirically verify the potential of our V-iBPPA for improving the convergence speed.

Asymptotic properties of dual averaging algorithm for constrained distributed stochastic optimization

Considering the constrained stochastic optimization problem over a time-varying random network, where the agents are to collectively minimize a sum of objective functions subject to a common constraint set, we investigate asymptotic properties of a distributed algorithm based on dual averaging of gradients. Different from most existing works on distributed dual averaging algorithms that are mainly focused on their non-asymptotic properties, we prove not only almost sure convergence and rate of almost sure convergence, but also asymptotic normality and asymptotic efficiency of the algorithm. Firstly, for general constrained convex optimization problem distributed over a random network, we prove that almost sure consensus can be achieved and the estimates of agents converge to the same optimal point. For the case of linear constrained convex optimization, we show that the mirror map of the averaged dual sequence identifies the active constraints of the optimal solution with probability 1, which helps us to prove the almost sure convergence rate and then establish asymptotic normality of the algorithm. Furthermore, we also verify that the algorithm is asymptotically optimal. To the best of our knowledge, it is the first asymptotic normality result for constrained distributed optimization algorithms. Finally, a numerical example is provided to justify the theoretical analysis.

Additive Schwarz methods for convex optimization with backtracking

This paper presents a novel backtracking strategy for additive Schwarz methods for general convex optimization problems as an acceleration scheme. The proposed backtracking strategy is independent of local solvers, so that it can be applied to any algorithms that can be represented in an abstract framework of additive Schwarz methods. Allowing for adaptive increasing and decreasing of the step size along the iterations, the convergence rate of an algorithm is improved. The improved convergence rate of the algorithm is analyzed rigorously. In addition, combining the proposed backtracking strategy with a momentum acceleration technique, we propose a further accelerated additive Schwarz method. Numerical results for various convex optimization problems such as nonlinear elliptic problems, nonsmooth problems, and nonsharp problems are presented in order to support our theory.

Exact penalties for decomposable convex optimization problems

We consider a general decomposable convex optimization problem. By using right-hand side allocation technique, it can be transformed into a collection of small dimensional optimization problems. The master problem is a convex non-smooth optimization problem. We propose to apply the exact non-smooth penalty method, which gives a solution of the initial problem under some fixed penalty parameter and provides the consistency of lower level problems. The master problem can be solved with a suitable non-smooth optimization method. The simplest of them is the custom subgradient projection method using the divergent series step-size rule without line-search, whose convergence may be, however, rather low. We suggest to enhance its step-size selection by using a two-speed rule. Preliminary results of computational experiments confirm efficiency of this technique.

Stochastic approximation versus sample average approximation for Wasserstein barycenters

In the machine learning and optimization community, there are two main approaches for the convex risk minimization problem, namely the Stochastic Approximation (SA) and the Sample Average Approximation (SAA). In terms of the oracle complexity (required number of stochastic gradient evaluations), both approaches are considered equivalent on average (up to a logarithmic factor). The total complexity depends on a specific problem, however, starting from the work [A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, Robust stochastic approximation approach to stochastic programming, SIAM. J. Opt. 19 (2009), pp. 1574–1609] it was generally accepted that the SA is better than the SAA. We show that for the Wasserstein barycenter problem, this superiority can be inverted. We provide a detailed comparison by stating the complexity bounds for the SA and SAA implementations calculating barycenters defined with respect to optimal transport distances and entropy-regularized optimal transport distances. As a byproduct, we also construct confidence intervals for the barycenter defined with respect to entropy-regularized optimal transport distances in the -norm. The preliminary results are derived for a general convex optimization problem given by the expectation to have other applications besides the Wasserstein barycenter problem.

A proximal ADMM with the Broyden family for convex optimization problems

Alternating direction methods of multipliers (ADMM) have been well studied and effectively used in various application fields. The classical ADMM must solve two subproblems exactly at each iteration. To overcome the difficulty of computing the exact solution of the subproblems, some proximal terms are added to the subproblems. Recently, {{a special proximal ADMM has been studied}} whose regularized matrix in the proximal term is generated by the BFGS update (or limited memory BFGS) at every iteration for a structured quadratic optimization problem. {{The numerical experiments also showed}} that the numbers of iterations were almost same as those by the exact ADMM. In this paper, we propose such a proximal ADMM for more general convex optimization problems, and extend the proximal term by the Broyden family update. We also show the convergence of the proposed method under standard assumptions.

Additive Schwarz Methods for Convex Optimization as Gradient Methods

This paper gives a unified convergence analysis of additive Schwarz methods for general convex optimization problems. Resembling to the fact that additive Schwarz methods for linear problems are preconditioned Richardson methods, we prove that additive Schwarz methods for general convex optimization are in fact gradient methods. Then an abstract framework for convergence analysis of additive Schwarz methods is proposed. The proposed framework applied to linear elliptic problems agrees with the classical theory. We present applications of the proposed framework to various interesting convex optimization problems such as nonlinear elliptic problems, nonsmooth problems, and nonsharp problems.

On Coordinate-Wise Minimization Applied to General Convex Optimization Problems

Abstract In this paper, we theoretically analyze principal properties of coordinate-wise minimization and we answer a few open questions related to it. In particular, we show that for minimizing any differentiable convex function on a given convex polyhedron, there exists a finite set of directions determined only by the polyhedron such that any local minimum with respect to these directions is a global minimum. We prove that the set of directions has a ‘nice’ structure for polyhedra defined by a k-regular matrix, which is a generalization of total unimodularity. We proceed to show that for some simple polyhedra, the number of these directions is polynomial, which subsumes already existing algorithms. We prove that the main result can not be extended for the case when the polyhedron is not known in advance or if the optimization is performed over a general convex set.

Inexact alternating direction methods of multipliers for separable convex optimization

Inexact alternating direction multiplier methods (ADMMs) are developed for solving general separable convex optimization problems with a linear constraint and with an objective that is the sum of smooth and nonsmooth terms. The approach involves linearized subproblems, a back substitution step, and either gradient or accelerated gradient techniques. Global convergence is established. The methods are particularly useful when the ADMM subproblems do not have closed form solution or when the solution of the subproblems is expensive. Numerical experiments based on image reconstruction problems show the effectiveness of the proposed methods.

Sequential characterizations of robust optimal solutions in uncertain convex programs via perturbation approach

By using robust optimization approach, necessary and sufficient sequential optimality conditions without any constraint qualifications for the general convex optimization problem in the face of data uncertainty are given in terms of the ε-subdifferential. A sequential condition involving only the subdifferentials is also derived using a version of the Brøndsted-Rockafellar theorem. Consequently, sequential Lagrange multiplier condition for robust optimal solution of convex optimization problem with cone constraints in the face of data uncertainty is given. It is worth pointing out that there is no compactness assumption of uncertainty set and upper semicontinuity of functions involved in our results.

A strategy of global convergence for the affine scaling algorithm for convex semidefinite programming

The affine scaling algorithm is one of the earliest interior point methods developed for linear programming. This algorithm is simple and elegant in terms of its geometric interpretation, but it is notoriously difficult to prove its convergence. It often requires additional restrictive conditions such as nondegeneracy, specific initial solutions, and/or small step length to guarantee its global convergence. This situation is made worse when it comes to applying the affine scaling idea to the solution of semidefinite optimization problems or more general convex optimization problems. In (Math Program 83(1–3):393–406, 1998), Muramatsu presented an example of linear semidefinite programming, for which the affine scaling algorithm with either short or long step converges to a non-optimal point. This paper aims at developing a strategy that guarantees the global convergence for the affine scaling algorithm in the context of linearly constrained convex semidefinite optimization in a least restrictive manner. We propose a new rule of step size, which is similar to the Armijo rule, and prove that such an affine scaling algorithm is globally convergent in the sense that each accumulation point of the sequence generated by the algorithm is an optimal solution as long as the optimal solution set is nonempty and bounded. The algorithm is least restrictive in the sense that it allows the problem to be degenerate and it may start from any interior feasible point.

Distributed Continuous-Time Algorithms for Nonsmooth Extended Monotropic Optimization Problems

This paper studies distributed algorithms for the nonsmooth extended monotropic optimization problem, which is a general convex optimization problem with a certain separable structure. The consider...

Primal-dual optimization strategies in Huber-L1 optical flow with temporal subspace constraints for non-rigid sequence registration

This work studies the application of Fenchel-Duality principles to general convex optimization problems and their corresponding relaxed versions in the context of optical flow estimation. We derive the associated primal-dual optimization strategies in the problem of Huber-L1 optical flow with temporal consistency for non-rigid sequence registration. Temporal consistency is imposed using a recently proposed approach that characterizes the optical flow using temporal subspace constraints, yielding solutions in a space spanned by a non-rigid orthogonal trajectory basis. The performance of the resulting optical flow methods has been studied in a framework for non-rigid sequence registration evaluation. In addition, we have compared the solution of the different methods in other challenging datasets. We have found that the strategies with the best outcome are among the ways of applying Fenchel-Duality principles that were not considered in previous works for the optical flow model with temporal subspace constraints. Indeed, our experiments have shown the simplest optimization strategy as the best performing one.

Primal-dual convex optimization in large deformation diffeomorphic metric mapping: LDDMM meets robust regularizers

This paper proposes a method for primal-dual convex optimization in variational large deformation diffeomorphic metric mapping problems formulated with robust regularizers and robust image similarity metrics. The method is based on Chambolle and Pock primal-dual algorithm for solving general convex optimization problems. Diagonal preconditioning is used to ensure the convergence of the algorithm to the global minimum. We consider three robust regularizers liable to provide acceptable results in diffeomorphic registration: Huber, V-Huber and total generalized variation. The Huber norm is used in the image similarity term. The primal-dual equations are derived for the stationary and the non-stationary parameterizations of diffeomorphisms. The resulting algorithms have been implemented for running in the GPU using Cuda. For the most memory consuming methods, we have developed a multi-GPU implementation. The GPU implementations allowed us to perform an exhaustive evaluation study in NIREP and LPBA40 databases. The experiments showed that, for all the considered regularizers, the proposed method converges to diffeomorphic solutions while better preserving discontinuities at the boundaries of the objects compared to baseline diffeomorphic registration methods. In most cases, the evaluation showed a competitive performance for the robust regularizers, close to the performance of the baseline diffeomorphic registration methods.

Extended ADMM and BCD for nonseparable convex minimization models with quadratic coupling terms: convergence analysis and insights

In this paper, we establish the convergence of the proximal alternating direction method of multipliers (ADMM) and block coordinate descent (BCD) method for nonseparable minimization models with quadratic coupling terms. The novel convergence results presented in this paper answer several open questions that have been the subject of considerable discussion. We firstly extend the 2-block proximal ADMM to linearly constrained convex optimization with a coupled quadratic objective function, an area where theoretical understanding is currently lacking, and prove that the sequence generated by the proximal ADMM converges in point-wise manner to a primal-dual solution pair. Moreover, we apply randomly permuted ADMM (RPADMM) to nonseparable multi-block convex optimization, and prove its expected convergence for a class of nonseparable quadratic programming problems. When the linear constraint vanishes, the 2-block proximal ADMM and RPADMM reduce to the 2-block cyclic proximal BCD method and randomly permuted BCD (RPBCD). Our study provides the first iterate convergence result for 2-block cyclic proximal BCD without assuming the boundedness of the iterates. We also theoretically establish the expected iterate convergence result concerning multi-block RPBCD for convex quadratic optimization. In addition, we demonstrate that RPBCD may have a worse convergence rate than cyclic proximal BCD for 2-block convex quadratic minimization problems. Although the results on RPADMM and RPBCD are restricted to quadratic minimization models, they provide some interesting insights: (1) random permutation makes ADMM and BCD more robust for multi-block convex minimization problems; (2) cyclic BCD may outperform RPBCD for “nice” problems, and RPBCD should be applied with caution when solving general convex optimization problems especially with a few blocks.