Convex Case Research Articles

EFIX: Exact fixed point methods for distributed optimization

We consider strongly convex distributed consensus optimization over connected networks. EFIX, the proposed method, is derived using quadratic penalty approach. In more detail, we use the standard reformulation—transforming the original problem into a constrained problem in a higher dimensional space—to define a sequence of suitable quadratic penalty subproblems with increasing penalty parameters. For quadratic objectives, the corresponding sequence consists of quadratic penalty subproblems. For generic strongly convex case, the objective function is approximated with a quadratic model and hence the sequence of the resulting penalty subproblems is again quadratic. EFIX is then derived by solving each of the quadratic penalty subproblems via a fixed point (R)-linear solver, e.g., Jacobi Over-Relaxation method. The exact convergence is proved as well as the worst case complexity of order {{mathcal {O}}}(epsilon ^{-1}) for the quadratic case. In the case of strongly convex generic functions, the standard result for penalty methods is obtained. Numerical results indicate that the method is highly competitive with state-of-the-art exact first order methods, requires smaller computational and communication effort, and is robust to the choice of algorithm parameters.

From the Ravine Method to the Nesterov Method and Vice Versa: A Dynamical System Perspective

We revisit the Ravine method of Gelfand and Tsetlin from a dynamical system perspective, study its convergence properties, and highlight its similarities and differences with the Nesterov accelerated gradient method. The two methods are closely related. They can be deduced from each other by reversing the order of the extrapolation and gradient operations in their definitions. They benefit from similar fast convergence of values and convergence of iterates for general convex objective functions. We will also establish the high resolution ODE of the Ravine and Nesterov methods and reveal an additional geometric damping term driven by the Hessian for both methods. This will allow us to prove fast convergence toward zero of the gradients not only for the Ravine method but also for the Nesterov method for the first time. In the strongly convex case, we show linear convergence for the Ravine method at an optimal rate. We also highlight connections to other algorithms resulting from more subtle discretization schemes and finally describe a Ravine version of the proximal-gradient algorithms for general structured smooth + nonsmooth convex optimization problems.

Interior and up to the boundary regularity for the fractional [formula omitted]-Laplacian: The convex case

We establish interior and up to the boundary Hölder regularity estimates for weak solutions of the Dirichlet problem for the fractional g−Laplacian with bounded right hand side and g convex. These are the first regularity results available in the literature for integro-differential equations in the context of fractional Orlicz–Sobolev spaces.

Zeroth-Order Optimization on Subsets of Symmetric Matrices With Application to MPC Tuning

This article provides a zeroth-order optimization framework for nonsmooth and possibly nonconvex cost functions with matrix parameters that are real and symmetric. We provide complexity bounds on the number of iterations required to ensure a given accuracy level for both the convex and nonconvex cases. The derived complexity bounds for the convex case are less conservative than available bounds in the literature since we exploit the symmetric structure of the underlying matrix space. Moreover, the nonconvex complexity bounds are novel for the class of optimization problems that we consider. The utility of the framework is evident in the suite of applications that use symmetric matrices as tuning parameters. Of primary interest here is the challenge of tuning the gain matrices in model predictive controllers, as this is a challenge known to be inhibiting the industrial implementation of these architectures. To demonstrate the framework, we consider the problem of MIMO diesel air-path control and implement the framework iteratively ``in-the-loop'' to reduce tracking error on the output channels. Both simulations and experimental results are included to illustrate the effectiveness of the proposed framework over different engine drive cycles.

Inexact proximal point algorithm for quasiconvex optimization problems on Hadamard manifolds

ABSTRACT In this paper, by using the inexact scalarization proximal methods, we solve quasiconvex multiobjective optimization problems on Hadamard manifolds. Under some assumptions on the multifunction of problem and vector fields, our methods are proved to be convergent to a Pareto critical point of the problem. For the convex case, the sequences generated by the methods converge to a weak Pareto solution.

Convergence analysis of asynchronous stochastic recursive gradient algorithms

Asynchronous stochastic algorithms with variance reduction techniques have been empirically shown to be useful for many large-scale machine learning problems. By making a parallel optimization algorithm asynchronous, one can reduce the synchronization cost and improve the practical efficiency. Recently, the stochastic recursive gradient algorithm has shown superior theoretical performance; however, it is not scalable enough in the current big data era. To make it more practical, we propose a class of asynchronous stochastic recursive gradient methods and analyze them in the shared memory model. The analysis results show that our asynchronous algorithms can linearly converge to the solution in the strongly convex case and complete the iteration faster. In addition, we analyze the “price of asynchrony” and give the sufficient conditions required for linear speedup. To the best of our knowledge, our speedup conditions match the optimal bounds of asynchronous stochastic algorithms known thus far. Finally, we demonstrate our theoretical analyses from a practical implementation on a multicore machine.

Screening for a Reweighted Penalized Conditional Gradient Method

The conditional gradient method (CGM) is widely used in large-scale sparse convex optimization, having a low per iteration computational cost for structured sparse regularizers and a greedy approach for collecting nonzeros. We explore the sparsity acquiring properties of a general penalized CGM (P-CGM) for convex regularizers and a reweighted penalized CGM (RP-CGM) for nonconvex regularizers, replacing the usual convex constraints with gauge-inspired penalties. This generalization does not increase the per-iteration complexity noticeably. Without assuming bounded iterates or using line search, we show O(1/t) convergence of the gap of each subproblem, which measures distance to a stationary point. We couple this with a screening rule which is safe in the convex case, converging to the true support at a rate O(1/(δ 2 )) where δ≥0 measures how close the problem is to degeneracy. In the nonconvex case the screening rule converges to the true support in a finite number of iterations, but is not necessarily safe in the intermediate iterates. In our experiments, we verify the consistency of the method and adjust the aggressiveness of the screening rule by tuning the concavity of the regularizer.

On the Convergence of Stochastic Primal-Dual Hybrid Gradient

In this paper, we analyze the recently proposed stochastic primal-dual hybrid gradient (SPDHG) algorithm and provide new theoretical results. In particular, we prove almost sure convergence of the iterates to a solution and linear convergence with standard step sizes, independent of strong convexity constants. Our assumption for linear convergence is metric subregularity, which is satisfied for smooth and strongly convex problems in addition to many nonsmooth and/or nonstrongly convex problems, such as linear programs, Lasso, and support vector machines. In the general convex case, we prove optimal sublinear rates for the ergodic sequence and for randomly selected iterate, without bounded domain assumptions. We also provide numerical evidence showing that SPDHG with standard step sizes shows favorable and robust practical performance against its specialized strongly convex variant SPDHG-$\mu$ and other state-of-the-art algorithms including variance reduction methods and stochastic dual coordinate ascent.

The pentagram map, Poncelet polygons, and commuting difference operators

The pentagram map takes a planar polygon $P$ to a polygon $P'$ whose vertices are the intersection points of consecutive shortest diagonals of $P$. This map is known to interact nicely with Poncelet polygons, that is, polygons which are simultaneously inscribed in a conic and circumscribed about a conic. A theorem of Schwartz states that if $P$ is a Poncelet polygon, then the image of $P$ under the pentagram map is projectively equivalent to $P$. In the present paper, we show that in the convex case this property characterizes Poncelet polygons: if a convex polygon is projectively equivalent to its pentagram image, then it is Poncelet. The proof is based on the theory of commuting difference operators, as well as on properties of real elliptic curves and theta functions.

Oracle Complexity Separation in Convex Optimization

Many convex optimization problems have structured objective functions written as a sum of functions with different oracle types (e.g., full gradient, coordinate derivative, stochastic gradient) and different arithmetic operations complexity of these oracles. In the strongly convex case, these functions also have different condition numbers that eventually define the iteration complexity of first-order methods and the number of oracle calls required to achieve a given accuracy. Motivated by the desire to call more expensive oracles fewer times, we consider the problem of minimizing the sum of two functions and propose a generic algorithmic framework to separate oracle complexities for each function. The latter means that the oracle for each function is called the number of times that coincide with the oracle complexity for the case when the second function is absent. Our general accelerated framework covers the setting of (strongly) convex objectives, the setting when both parts are given through full coordinate oracle, as well as when one of them is given by coordinate derivative oracle or has the finite-sum structure and is available through stochastic gradient oracle. In the latter two cases, we obtain accelerated random coordinate descent and accelerated variance reduced methods with oracle complexity separation.

Нижние оценки для методов типа условного градиента для задач минимизации гладких сильно выпуклых функций

In this paper, we consider conditional gradient methods. These are methods that use a linear minimization oracle, which, for a given vector $p \in \mathbb{R}^n$, computes the solution of the subproblem $$\arg \min_{x\in X}{\langle p,x \rangle}.$$ There are a variety of conditional gradient methods that have a linear convergence rate in a strongly convex case. However, in all these methods, the dimension of the problem is included in the rate of convergence, which in modern applications can be very large. In this paper, we prove that in the strongly convex case, the convergence rate of the conditional gradient methods in the best case depends on the dimension of the problem $ n $ as $ \widetilde {\Omega} (\sqrt {n}) $. Thus, the conditional gradient methods may turn out to be ineffective for solving strongly convex optimization problems of large dimensions. Also, the application of conditional gradient methods to minimization problems of a quadratic form is considered. The effectiveness of the Frank-Wolfe method for solving the quadratic optimization problem in the convex case on a simplex (PageRank) has already been proved. This work shows that the use of conditional gradient methods to solve the minimization problem of a quadratic form in a strongly convex case is ineffective due to the presence of dimension in the convergence rate of these methods. Therefore, the Shrinking Conditional Gradient method is considered. Its difference from the conditional gradient methods is that it uses a modified linear minimization oracle. The convergence rate of such an algorithm does not depend on dimension. Using the Shrinking Conditional Gradient method the complexity of solving the minimization problem of quadratic form on a $ \infty $-ball is obtained. The resulting evaluation of the method is comparable to the complexity of the gradient method.

Supply Chain Management Optimization and Prediction Model Based on Projected Stochastic Gradient

Supply chain management (SCM) is considered at the forefront of many organizations in the delivery of their products. Various optimization methods are applied in the SCM to improve the efficiency of the process. In this research, the projected stochastic gradient (PSG) method was proposed to increase the efficiency of the SCM analysis. The key objective of an efficient supply chain is to find the best flow patterns for the best products in order to select the suppliers to different customers. Hence, the focus of this research is on developing an efficient multi-echelon supply chain using factors such as cost, time, and risk. In the convex case, the proposed method has the advantage of a weakly convergent sequence of iterates to a point in the set of minimizers with probability one. The developed method achieves strong sequence convergence to the unique optimum, with probability one. The SCM dataset was utilized to assess the proposed method’s performance. The proposed PSG method has the advantage of considering the holding cost in the profit analysis of the company. The results of the developed PSG method are analyzed according to the product’s profit, stock, and demand. The proposed PSG method also provides the prediction of demand to increase profit.

Slater Condition for Tangent Derivatives

Noting that the existing Slater condition, as a fundamental constraint qualification in optimization, is only applicable in the convex setting, we introduce and study the Slater condition for the Bouligand and Clarke tangent derivatives of a general vector-valued function F with respect to a closed convex cone K. Without any assumption, it is proved that the Slater condition for the Clarke (respectively, Bouligand) tangent derivative with respect to K is always stable when the objective function F undergoes small Lipschitz (calm) perturbations. Based on this, we prove that if the Clarke (Bouligand) tangent derivative of F satisfies the Slater condition (with respect to K) then the conic inequality determined by F has a stable metric subregularity when F undergoes small Lipschitz (calm regular) perturbations. In the composite-convexity case, the converse implication is also proved to be true. Moreover, under the Slater condition for the tangent derivative of F, it is proved that the normal cone to the sublevel set of F can be formulated by the subdifferential of F, which improves the corresponding results in either the smooth or convex case. As applications, without any qualification assumption, we improve and generalize formulas for the normal cone to a convex sublevel set by Cabot and Thibault [(2014), Sequential formulae for the normal cone to sublevel sets. Transactions of the American Mathematical Society 366(12):6591–6628]. With the help of these formulas, some new Karush–Kuhn–Tucker optimality conditions are established.

Efficient unconstrained black box optimization

For the unconstrained optimization of black box functions, this paper introduces a new randomized algorithm called VRBBO. In practice, VRBBO matches the quality of other state-of-the-art algorithms for finding, in small and large dimensions, a local minimizer with reasonable accuracy. Although our theory guarantees only local minimizers our heuristic techniques turn VRBBO into an efficient global solver. In very thorough numerical experiments, we found in most cases either a global minimizer, or where this could not be checked, at least a point of similar quality to the best competitive global solvers. For smooth, everywhere defined functions, it is proved that, with probability arbitrarily close to 1, a basic version of our algorithm finds with {{mathcal {O}}}(nvarepsilon ^{-2}) function evaluations a point whose unknown exact gradient 2-norm is below a given threshold varepsilon >0, where n is the dimension. In the smooth convex case, this number improves to {{mathcal {O}}}(nlog varepsilon ^{-1}) and in the smooth (strongly) convex case to {{mathcal {O}}}(nlog nvarepsilon ^{-1}). This matches known recent complexity results for reaching a slightly different goal, namely the expected unknown exact gradient 2-norm is below a given threshold varepsilon >0. Numerical results show that VRBBO is effective and robust in comparison with the state-of-the-art local and global solvers on the unconstrained CUTEst test problems of Gould et al. (Comput Optim Appl 60:545–557, 2014) for optimization and the test problems of Jamil and Yang (Int J Math Model Numer Optim 4:150, 2013) for global optimization with 2–5000 variables.

Accelerated Gradient Descent Learning Over Multiple Access Fading Channels

We consider a distributed learning problem in a wireless network, consisting of <inline-formula> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> distributed edge devices and a parameter server (PS). The objective function is a sum of the edge devices&#x2019; local loss functions, who aim to train a shared model by communicating with the PS over multiple access channels (MAC). This problem has attracted a growing interest in distributed sensing systems, and more recently in federated learning, known as over-the-air computation. In this paper, we develop a novel Accelerated Gradient-descent Multiple Access (AGMA) algorithm that uses momentum-based gradient signals over noisy fading MAC to improve the convergence rate as compared to existing methods. Furthermore, AGMA does not require power control or beamforming to cancel the fading effect, which simplifies the implementation complexity. We analyze AGMA theoretically, and establish a finite-sample bound of the error for both convex and strongly convex loss functions with Lipschitz gradient. For the strongly convex case, we show that AGMA approaches the best-known linear convergence rate as the network increases. For the convex case, we show that AGMA significantly improves the sub-linear convergence rate as compared to existing methods. Finally, we present simulation results using real datasets that demonstrate better performance by AGMA.

On Distributed Nonconvex Optimization: Projected Subgradient Method for Weakly Convex Problems in Networks

The stochastic subgradient method is a widely-used algorithm for solving large-scale optimization problems arising in machine learning. Often these problems are neither smooth nor convex. Recently, Davis et al. [1-2] characterized the convergence of the stochastic subgradient method for the weakly convex case, which encompasses many important applications (e.g., robust phase retrieval, blind deconvolution, biconvex compressive sensing, and dictionary learning). In practice, distributed implementations of the projected stochastic subgradient method (stoDPSM) are used to speed-up risk minimization. In this paper, we propose a distributed implementation of the stochastic subgradient method with a theoretical guarantee. Specifically, we show the global convergence of stoDPSM using the Moreau envelope stationarity measure. Furthermore, under a so-called sharpness condition, we show that deterministic DPSM (with a proper initialization) converges linearly to the sharp minima, using geometrically diminishing step-size. We provide numerical experiments to support our theoretical analysis.

Communication-Efficient Federated Learning Using Censored Heavy Ball Descent

Distributed machine learning enables scalability and computational offloading, but requires significant levels of communication. Consequently, communication efficiency in distributed learning settings is an important consideration, especially when the communications are wireless and battery-driven devices are employed. In this paper we develop a censoring-based heavy ball (CHB) method for distributed learning in a server-worker architecture. Each worker self-censors unless its local gradient is sufficiently different from the previously transmitted one. The significant practical advantages of the HB method for learning problems are well known, but the question of reducing communications has not been addressed. CHB takes advantage of the HB smoothing to eliminate reporting small changes, and provably achieves a linear convergence rate equivalent to that of the classical HB method for smooth and strongly convex objective functions. The convergence guarantee of CHB is theoretically justified for both convex and nonconvex cases. In addition we prove that, under some conditions, at least half of all communications can be eliminated without any impact on convergence rate. Extensive numerical results validate the communication efficiency of CHB on both synthetic and real datasets, for convex, nonconvex, and nondifferentiable cases. Given a target accuracy, CHB can significantly reduce the number of communications compared to existing algorithms, achieving the same accuracy without slowing down the optimization process.

Convergence rates for the Vlasov-Fokker-Planck equation and uniform in time propagation of chaos in non convex cases

We prove the existence of a contraction rate for Vlasov-Fokker-Planck equation in Wasserstein distance, provided the interaction potential is Lipschitz continuous and the confining potential is both (locally) Lipschitz continuous and greater than a quadratic function, thus requiring no convexity conditions. Our strategy relies on coupling methods suggested by A. Eberle [22] adapted to the kinetic setting enabling also to obtain uniform in time propagation of chaos in a non convex setting.

Необходимые условия оптимальности в одной дискретной граничной задаче управления динамикой популяции

One discrete problem of optimal population dynamics control with a controllable initial condition is considered. The process is described by a nonlinear system of Fredholm-type difference equations. An analogue of the L.S. Pontryagin’ maximum principle is proved by imposing a series of smoothness conditions on the right side the equation. The linearized maximum principle and the analogue of the Euler equation are proved for the case of convexity and openness of the control domain, which are first-order necessary conditions for optimality.

Biotechnological Evaluation of Bee Ethology During Honey Collection and Probiotic Properties of Honey

The influence of biotechnological factors, such as purebred families and honey base on the ethology of bees in the process of nectar collection and the species composition of the association of probiotic cultures in freshly pumped honey to bees was found out. The research was conducted using Ukrainian steppe bees. The following most typical external features of their breed, namely the length of the proboscis from 6.34 to 6.44 mm, cubital index in the range of 2.17 ± 0.04 - 2.61 ± 0.05, discoidal displacement 76 - 100 %, the shape of the edge of the wax mirror 68 - 96% of convex cases. The species composition of honey plants in typical conditions of the Forest-Steppe zone of Ukraine, their contribution to the general honey harvest during the whole beekeeping season and the formation of antibacterial defense mechanism of bees based on the association of probiotic cultures in their intestines and freshly pumped honey are analyzed. Data on differentiation of selected cultures from bee honey by cultural-morphological and biochemical properties are given. Probiotic cultures were identified by cultivation on differential media. Thus, cultures of <i>Lactobacillus plantarum and Bifidobacterium bifidum</i> were accumulated on MRS medium (solid and liquid), and <i>Enterococcus faecium</i> was accumulated on tryptone soy medium. It is shown that the quantitative characteristics of honey accumulation in the bee nest depend on a number of biotechnological factors. The main indicators that affect the ethology of bees when collecting nectar are purebred and species composition of the honey base. The results of the research serve as a basis for further studies of the ethology of the Ukrainian steppe breed of bees during honey collection, its popularization and use in large apiaries (100 and more families) according to the breed zoning plan in the Forest-Steppe zone of Ukraine and for the development of new antibacterial, specific for bees, organic products based on probiotic cultures, isolated from honey.