Strong Convexity Research Articles

Проксимальні алгоритми для дворівневих задач опуклої оптимізації

In this paper, problems of bi-level convex minimization in a Hilbert space are considered. The bi-level convex minimization problem is to minimize the first convex function on the set of minima of the second convex function. This setting has many applications, but the implicit constraints generated by the internal problem make it difficult to obtain optimality conditions and construct algorithms. Multilevel optimization problems are formulated in a similar way, the source of which is the operation research problems (optimization according to sequentially specified criteria or lexicographic optimization). Attention is focused on problem solving using two proximal methods. The main theoretical results are theorems on the convergence of methods in various situations. The first of the methods is obtained by combining the penalty function method and the proximal method. Strong convergence is proved in the case of strong convexity of the function of the exterior problem. In the general case, only weak convergence has been proved. The second, the so-called proximal-gradient method, is a combination of one of the variants of the fast proximal-gradient algorithm with the method of penalty functions. The rates of convergence of the proximal-gradient method and its weak convergence are proved.

Dual Space Preconditioning for Gradient Descent

The conditions of relative smoothness and relative strong convexity were recently introduced for the analysis of Bregman gradient methods for convex optimization. We introduce a generalized left-pr...

A Theoretical Revisit to Linear Convergence for Saddle Point Problems

Recently, convex-concave bilinear Saddle Point Problems (SPP) is widely used in lasso problems, Support Vector Machines, game theory, and so on. Previous researches have proposed many methods to solve SPP, and present their convergence rate theoretically. To achieve linear convergence, analysis in those previouse studies requires strong convexity of φ( z ). But, we find the linear convergence can also be achieved even for a general convex but not strongly convex φ( z ). In the article, by exploiting the strong duality of SPP, we propose a new method to solve SPP, and achieve the linear convergence. We present a new general sufficient condition to achieve linear convergence, but do not require the strong convexity of φ( z ). Furthermore, a more efficient method is also proposed, and its convergence rate is analyzed in theoretical. Our analysis shows that the well conditioned φ( z ) is necessary to improve the efficiency of our method. Finally, we conduct extensive empirical studies to evaluate the convergence performance of our methods.

Stability analysis of solutions to equilibrium problems and applications in economics

Purpose The purpose of this paper is to study the Hölder calmness of solutions to equilibrium problems and apply it to economics. Design/methodology/approach The authors obtain the Hölder calmness by using an effective approach. More precisely, under the key assumption of strong convexity, sufficient conditions for the Hölder continuity of solution maps to equilibrium problems are established. Findings A new result in stability analysis for equilibrium problems and applications in economics is archived. Originality/value The authors confirm that the paper has not been published previously, is not under consideration for publication elsewhere and is not being simultaneously submitted elsewhere.

Ill-Posed Nonlinear Optimization Problems and Uniform Accuracy Estimates of Regularization Methods

We investigate the ill-posed problem of minimizing weakly lower semicontinuous functionals on a convex closed set in a Hilbert space. The functionals to be minimized are available with errors. We prove that a necessary condition for the existence of a regularization procedure with a uniform accuracy estimate on the class of weakly lower semicontinuous functionals is the well-posedness of related optimization problems. We also study regularization methods that provide uniform accuracy estimates, linear with respect to the error level in cost functionals. Under appropriate assumptions on the feasible set, we establish a necessary condition for the existence of such regularizing procedures. In the case of unconstrained problems, the obtained condition reduces to the local strong convexity of functionals under minimization.

Strongly Convex Divergences.

We consider a sub-class of the f-divergences satisfying a stronger convexity property, which we refer to as strongly convex, or -convex divergences. We derive new and old relationships, based on convexity arguments, between popular f-divergences.

On stochastic gradient Langevin dynamics with dependent data streams in the logconcave case

We study the problem of sampling from a probability distribution $\pi $ on $\mathbb{R}^{d}$ which has a density w.r.t. the Lebesgue measure known up to a normalization factor $x\mapsto \mathrm{e}^{-U(x)}/\int _{\mathbb{R}^{d}}\mathrm{e}^{-U(y)}\,\mathrm{d}y$. We analyze a sampling method based on the Euler discretization of the Langevin stochastic differential equations under the assumptions that the potential $U$ is continuously differentiable, $\nabla U$ is Lipschitz, and $U$ is strongly concave. We focus on the case where the gradient of the log-density cannot be directly computed but unbiased estimates of the gradient from possibly dependent observations are available. This setting can be seen as a combination of a stochastic approximation (here stochastic gradient) type algorithms with discretized Langevin dynamics. We obtain an upper bound of the Wasserstein-2 distance between the law of the iterates of this algorithm and the target distribution $\pi $ with constants depending explicitly on the Lipschitz and strong convexity constants of the potential and the dimension of the space. Finally, under weaker assumptions on $U$ and its gradient but in the presence of independent observations, we obtain analogous results in Wasserstein-2 distance.

Proximal gradient flow and Douglas–Rachford splitting dynamics: Global exponential stability via integral quadratic constraints

Many large-scale and distributed optimization problems can be brought into a composite form in which the objective function is given by the sum of a smooth term and a nonsmooth regularizer. Such problems can be solved via a proximal gradient method and its variants, thereby generalizing gradient descent to a nonsmooth setup. In this paper, we view proximal algorithms as dynamical systems and leverage techniques from control theory to study their global properties. In particular, for problems with strongly convex objective functions, we utilize the theory of integral quadratic constraints to prove the global exponential stability of the equilibrium points of the differential equations that govern the evolution of proximal gradient and Douglas–Rachford splitting flows. In our analysis, we use the fact that these algorithms can be interpreted as variable-metric gradient methods on the suitable envelopes and exploit structural properties of the nonlinear terms that arise from the gradient of the smooth part of the objective function and the proximal operator associated with the nonsmooth regularizer. We also demonstrate that these envelopes can be obtained from the augmented Lagrangian associated with the original nonsmooth problem and establish conditions for global exponential convergence even in the absence of strong convexity.

Asynchronous Optimization Over Graphs: Linear Convergence Under Error Bound Conditions

We consider convex and nonconvex constrained optimization with a partially separable objective function: Agents minimize the sum of local objective functions, each of which is known only by the associated agent and depends on the variables of that agent and those of a few others. This partitioned setting arises in several applications of practical interest. We propose what is, to the best of our knowledge, the first distributed, asynchronous algorithm with rate guarantees for this class of problems. When the objective function is nonconvex, the algorithm provably converges to a stationary solution at a sublinear rate whereas linear rate is achieved under the renowned Luo-Tseng error bound condition (which is less stringent than strong convexity). Numerical results on matrix completion and LASSO problems show the effectiveness of our method.

Continuous-Time Discounted Mirror Descent Dynamics in Monotone Concave Games

In this paper, we consider concave continuous-kernel games characterized by monotonicity properties and propose discounted mirror descent-type dynamics. We introduce two classes of dynamics whereby the associated mirror map is constructed based on a strongly convex or a Legendre regularizer. Depending on the properties of the regularizer we show that these new dynamics can converge asymptotically in concave games with monotone (negative) pseudo-gradient. Furthermore, we show that when the regularizer enjoys strong convexity, the resulting dynamics can converge even in games with hypo-monotone (negative) pseudo-gradient, which corresponds to a shortage of monotonicity.

Distributed Nesterov Gradient and Heavy-Ball Double Accelerated Asynchronous Optimization.

In this article, we come up with a novel Nesterov gradient and heavy-ball double accelerated distributed synchronous optimization algorithm, called NHDA, and adopt a general asynchronous model to further propose an effective asynchronous algorithm, called ASY-NHDA, for distributed optimization problem over directed graphs, where each agent has access to a local objective function and computes the optimal solution via communicating only with its immediate neighbors. Our goal is to minimize a sum of all local objective functions satisfying strong convexity and Lipschitz continuity. Consider a general asynchronous model, where agents communicate with their immediate neighbors and start a new computation independently, that is, agents can communicate with their neighbors at any time without any coordination and use delayed information from their in-neighbors to compute a new update. Delays are arbitrary, unpredictable, and time-varying but bounded. The theoretical analysis of NHDA is based on analyzing the interaction among the consensus, the gradient tracking, and the optimization processes. As for the analysis of ASY-NHDA, we equivalently transform the asynchronous system into an augmented synchronous system without delays and prove its convergence through using the generalized small gain theorem. The results show that NHDA and ASY-NHDA converge to the optimal solution at a linear convergence as long as the largest step size is positive and less than an explicitly estimated upper bound, and the largest momentum parameter is nonnegative and less than an upper bound. Finally, we demonstrate the advantages of ASY-NHDA through simulations.

Nonconvex Haar-TV denoising

The anisotropic total variation (TV) denoising model suppresses noise for two-dimensional signals that are vertically and horizontally piecewise constant. However, two-dimensional signals may have sparse derivatives in other directions. We propose a modification of the classical anisotropic two-dimensional TV regularizer from a spectral point of view. In the frequency domain, the TV regularizer can be considered as penalizing the high-frequency component of original signals and promoting only low-frequency components. The classical anisotropic TV, which applies l1-norm on vertical and horizontal differences, suppresses high-frequency components of the signals. The proposed operator, named Haar total variation (Haar-TV), penalizes two-dimensional signals that have more varied high-frequency regions. Furthermore, we propose non-convex penalties based on the Haar-TV operator since non-convex penalties can preserve edges and thus enhance the quality of the estimation. We derive a condition that preserves the strong convexity of the total cost function so the global minimizer can be reached.

Regression to the tail: Why the Olympics blow up

The Olympic Games are the largest, highest-profile, and most expensive megaevent hosted by cities and nations. Average sports-related costs of hosting are $12.0 billion. Non-sports-related costs are typically several times that. Every Olympics since 1960 has run over budget, at an average of 172 percent in real terms, the highest overrun on record for any type of megaproject. The paper tests theoretical statistical distributions against empirical data for the costs of the Games, in order to explain the cost risks faced by host cities and nations. It is documented, for the first time, that cost and cost overrun for the Games follow a power-law distribution. Olympic costs are subject to infinite mean and variance, with dire consequences for predictability and planning. We name this phenomenon "regression to the tail": it is only a matter of time until a new extreme event occurs, with an overrun larger than the largest so far, and thus more disruptive and less plannable. The generative mechanism for the Olympic power law is identified as strong convexity prompted by six causal drivers: irreversibility, fixed deadlines, the Blank Check Syndrome, tight coupling, long planning horizons, and an Eternal Beginner Syndrome. The power law explains why the Games are so difficult to plan and manage successfully, and why cities and nations should think twice before hosting. Based on the power law, two heuristics are identified for better decision making on hosting. Finally, the paper develops measures for good practice in planning and managing the Games, including how to mitigate the extreme risks of the Olympic power law.

Strongly Reciprocally p-Convex Functions and Some Inequalities

In this paper, we generalize the concept of strong and reciprocal convexity. Some basic properties and results will be presented for the new class of strongly reciprocally p-convex functions. Furthermore, we will discuss the Hermite–Hadamard-type, Jensen-type, and Fejér-type inequalities for the strongly reciprocally p-convex functions.

A Block Successive Upper-Bound Minimization Method of Multipliers for Linearly Constrained Convex Optimization

Consider the problem of minimizing the sum of a smooth convex function and a separable nonsmooth convex function subject to linear coupling constraints. Problems of this form arise in many contemporary applications, including signal processing, wireless networking, and smart grid provisioning. Motivated by the huge size of these applications, we propose a new class of first-order primal–dual algorithms called the block successive upper-bound minimization method of multipliers (BSUM-M) to solve this family of problems. The BSUM-M updates the primal variable blocks successively by minimizing locally tight upper bounds of the augmented Lagrangian of the original problem, followed by a gradient-type update for the dual variable in closed form. We show that under certain regularity conditions, and when the primal block variables are updated in either a deterministic or a random fashion, the BSUM-M converges to a point in the set of optimal solutions. Moreover, in the absence of linear constraints and under similar conditions as in the previous result, we show that the randomized BSUM-M (which reduces to the randomized block successive upper-bound minimization method) converges at an asymptotically linear rate without relying on strong convexity.

Proximal-Like Incremental Aggregated Gradient Method with Linear Convergence Under Bregman Distance Growth Conditions

We introduce a unified algorithmic framework, called the proximal-like incremental aggregated gradient (PLIAG) method, for minimizing the sum of a convex function that consists of additive relatively smooth convex components and a proper lower semicontinuous convex regularization function over an abstract feasible set whose geometry can be captured by using the domain of a Legendre function. The PLIAG method includes many existing algorithms in the literature as special cases, such as the proximal gradient method, the Bregman proximal gradient method (also called the NoLips algorithm), the incremental aggregated gradient method, the incremental aggregated proximal method, and the proximal incremental aggregated gradient method. It also includes some novel interesting iteration schemes. First, we show that the PLIAG method is globally sublinearly convergent without requiring a growth condition, which extends the sublinear convergence result for the proximal gradient algorithm to incremental aggregated-type first-order methods. Then, by embedding a so-called Bregman distance growth condition into a descent-type lemma to construct a special Lyapunov function, we show that the PLIAG method is globally linearly convergent in terms of both function values and Bregman distances to the optimal solution set, provided that the step size is not greater than some positive constant. The convergence results derived in this paper are all established beyond the standard assumptions in the literature (i.e., without requiring the strong convexity and the Lipschitz gradient continuity of the smooth part of the objective). When specialized to many existing algorithms, our results recover or supplement their convergence results under strictly weaker conditions.

Accelerated Convergence Algorithm for Distributed Constrained Optimization under Time-Varying General Directed Graphs

This paper studies a class of distributed convex optimization problems by a set of agents in which each agent only has access to its own local convex objective function and the estimate of each agent is restricted to both coupling linear constraint and individual box constraints. Our focus is to devise a distributed primal-dual gradient algorithm for working out the problem over a sequence of time-varying general directed graphs. The communications among agents are assumed to be uniformly strongly connected. A column-stochastic mixing matrix and a fixed step-size are applied in the algorithm which exactly steers all the agents to asymptotically converge to a global optimal solution. Based on the standard strong convexity and the smoothness assumptions of the objective functions, we show that the distributed algorithm is capable of driving the whole network to geometrically converge to an optimal solution of the convex optimization problem only if the step-size does not exceed some upper bound. We also give an explicit analysis for the convergence rate of the proposed optimization algorithm. Simulations on economic dispatch problems and demand response problems in power systems are performed to illustrate the effectiveness of the proposed optimization algorithm.

Edge-Based Stochastic Gradient Algorithm for Distributed Optimization

This paper investigates distributed optimization problems where a group of networked nodes collaboratively minimizes the sum of all local objective functions. The local objective function of each node is further set as an average of a finite set of subfunctions. This adjustment is motivated by machine learning problems with large training samples distributed and known privately to individual computational nodes. An augmented Lagrange (AL) stochastic gradient algorithm is presented to address the distributed optimization problem, which is integrated with the factorization of weighted Laplacian and local unbiased stochastic averaging gradient methods. At each iteration, only one randomly selected gradient of a subfunction is evaluated at a node, and a variance-reduced stochastic averaging gradient technique is applied to approximate the gradient of local objective function. Strong convexity of the local subfunction and Lipschitz continuity of its gradient are shown to ensure a linear convergence rate of the proposed algorithm in expectation. Numerical experiments on a logistic regression problem demonstrate the correctness of theoretical results.

On the resolution of misspecified convex optimization and monotone variational inequality problems

We consider a misspecified optimization problem, requiring the minimization of a function $$f(\cdot;\theta ^*)$$ over a closed and convex set X where $$\theta ^*$$ is an unknown vector of parameters that may be learnt by a parallel learning process. Here, we develop coupled schemes that generate iterates $$(x_k,\theta _k)$$ as $$k \rightarrow \infty$$ , then $$x_k \rightarrow x^*$$ , a minimizer of $$f(\cdot;\theta ^*)$$ over X and $$\theta _k \rightarrow \theta ^*$$ . In the first part of the paper, we consider the solution of problems where f is either smooth or nonsmooth. In smooth strongly convex regimes, we demonstrate that such schemes still display a linear rate of convergence, albeit with larger constants. When strong convexity assumptions are weakened, it can be shown that the convergence in function value sees a modification in the canonical convergence rate of $${{{\mathcal {O}}}}(1/K)$$ by an additive factor proportional to $$\Vert \theta _0-\theta ^*\Vert$$ where $$\Vert \theta _0-\theta ^*\Vert$$ represents the initial misspecification in $$\theta ^*$$ . In addition, when the learning problem is assumed to be merely convex but admits a suitable weak-sharpness property, then the convergence rate deteriorates to $${\mathcal {O}}(1/\sqrt{K})$$ . In both convex and strongly convex regimes, diminishing steplength schemes are also provided and are less reliant on the knowledge of problem parameters. Finally, we present an averaging-based subgradient scheme that displays a rate of $${\mathcal {O}}(1/\sqrt{K})+ \mathcal{O}(\|\theta_0-\theta^*\|(1/K))$$ , implying no effect on the canonical rate of $${{{\mathcal {O}}}}(1/\sqrt{K})$$ . In the second part of the paper, we consider the solution of misspecified monotone variational inequality problems, motivated by the need to contend with more general equilibrium problems as well as the possibility of misspecification in the constraints. In this context, we first present a constant steplength misspecified extragradient scheme and prove its asymptotic convergence. This scheme is reliant on problem parameters (such as Lipschitz constants) and leads to a misspecified variant of iterative Tikhonov regularization, an avenue that does not necessitate the knowledge of such constants.

Fixed-Time Stable Gradient Flows: Applications to Continuous-Time Optimization

Continuous-time optimization is currently an active field of research in optimization theory; prior work in this area has yielded useful insights and elegant methods for proving stability and convergence properties of the continuous-time optimization algorithms. This article proposes novel gradient-flow schemes that yield convergence to the optimal point of a convex optimization problem within a fixed time from any given initial condition for unconstrained optimization, constrained optimization, and min-max problems. It is shown that the solution of the modified gradient-flow dynamics exists and is unique under certain regularity conditions on the objective function, while fixed-time convergence to the optimal point is shown via Lyapunov-based analysis. The application of the modified gradient flow to unconstrained optimization problems is studied under the assumption of gradient dominance, a relaxation of strong convexity. Then, a modified Newton's method is presented that exhibits fixed-time convergence under some mild conditions on the objective function. Building upon this method, a novel technique for solving convex optimization problems with linear equality constraints that yields convergence to the optimal point in fixed time is developed. Finally, the general min-max problem is considered, and a modified saddle-point dynamics to obtain the optimal solution in fixed time is developed.