Strong Convexity Assumption Research Articles

Bayesian Federated Learning with Hamiltonian Monte Carlo: Algorithm and Theory

This work introduces a novel and efficient Bayesian federated learning algorithm, namely, the Federated Averaging stochastic Hamiltonian Monte Carlo (FA-HMC), for parameter estimation and uncertainty quantification. We establish rigorous convergence guarantees of FA-HMC on non-iid distributed datasets, under the strong convexity and Hessian smoothness assumptions. Our analysis investigates the effects of parameter space dimension, noise on gradients and momentum, and the frequency of communication (between the central node and local nodes) on the convergence and communication costs of FA-HMC. Beyond that, we establish the tightness of our analysis by showing that the convergence rate cannot be improved even for continuous FA-HMC process. Moreover, extensive empirical studies demonstrate that FA-HMC outperforms the existing Federated Averaging-Langevin Monte Carlo (FA-LD) algorithm. Supplementary materials for this article are available online.

Adaptive, Doubly Optimal No-Regret Learning in Strongly Monotone and Exp-Concave Games with Gradient Feedback

Feasible Online Learning with Gradient Feedback Online gradient descent (OGD) is well-known to be doubly optimal under strong convexity or monotonicity assumptions: (1) in the single-agent setting, it achieves an optimal regret of [Formula: see text] for strongly convex cost functions, and (2) in the multiagent setting of strongly monotone games with each agent employing OGD we obtain last-iterate convergence of the joint action to a unique Nash equilibrium at an optimal rate of [Formula: see text]. Whereas these finite-time guarantees highlight its merits, OGD has the drawback that it requires knowing the strong convexity/monotonicity parameters. In “Adaptive, Doubly Optimal No-Regret Learning in Strongly Monotone and Exp-Concave Games with Gradient Feedback,” M. Jordan, T. Lin, and Z. Zhou design a fully adaptive OGD algorithm, AdaOGD, that does not require a priori knowledge of these parameters. In the single-agent setting, the algorithm achieves [Formula: see text] regret under strong convexity, which is optimal up to a log factor. Further, if each agent employs AdaOGD in strongly monotone games, the joint action converges in a last-iterate sense to a unique Nash equilibrium at a rate of [Formula: see text], again optimal up to log factors. The algorithms are illustrated in a learning version of the classic newsvendor problem, in which, because of lost sales, only (noisy) gradient feedback can be observed. The results immediately yield the first feasible and near-optimal algorithm for both the single-retailer and multiretailer settings.

Exponential Convergence of Primal-Dual Dynamics Under General Conditions and Its Application to Distributed Optimization.

In this article, we establish the local and global exponential convergence of a primal-dual dynamics (PDD) for solving equality-constrained optimization problems without strong convexity and full row rank assumption on the equality constraint matrix. Under the metric subregularity of Karush-Kuhn-Tucker (KKT) mapping, we prove the local exponential convergence of the dynamics. Moreover, we establish the global exponential convergence of the dynamics in an invariant subspace under a technically designed condition which is weaker than strong convexity. As an application, the obtained theoretical results are used to show the exponential convergence of several existing state-of-the-art primal-dual algorithms for solving distributed optimization without strong convexity. Finally, we provide some experiments to demonstrate the effectiveness of our results.

Interior point methods in optimal control

This paper deals with Interior Point Methods (IPMs) for Optimal Control Problems (OCPs) with pure state and mixed constraints. This paper establishes a complete proof of convergence of IPMs for a general class of OCPs. Convergence results are proved for primal variables, namely state and control variables, and for dual variables, namely, the adjoint state, and the constraints multipliers. In addition, the presented convergence result does not rely on a strong convexity assumption. Finally, this paper compares the performances of a primal and a primal-dual implementation of IPMs in optimal control in three examples.

Accelerated Projection Algorithm Based on Smoothing Approximation for Distributed Nonsmooth Optimization

In this paper, a distributed smoothing accelerated projection algorithm (DSAPA) is proposed to address constrained non-smooth convex optimization problems over undirected multi-agent networks in a distributed manner, where the objective function is free of the assumption of Lipschitz gradient or strong convexity. First, based on a distributed exact penalty method, the original optimization problem is translated to a problem of standard assignment without consensus constraints. Then, a novel DSAPA by combining the smoothing approximation with Nesterov's accelerated schemes is proposed. In addition, we provide a systematic analysis to derive an upper bound on the convergence rate in terms of the objective function based on penalty function and to choose the optimal step size accordingly. Our results demonstrate that the proposed DSAPA can reach <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O(\frac{\log (k)}{k})$</tex-math></inline-formula> when the optimal step size is chosen. Finally, the effectiveness and correctness of the proposed algorithm are verified by numerical and practical application examples.

Out-of-sample error estimation for M-estimators with convex penalty

Abstract A generic out-of-sample error estimate is proposed for $M$-estimators regularized with a convex penalty in high-dimensional linear regression where $(\boldsymbol{X},\boldsymbol{y})$ is observed and the dimension $p$ and sample size $n$ are of the same order. The out-of-sample error estimate enjoys a relative error of order $n^{-1/2}$ in a linear model with Gaussian covariates and independent noise, either non-asymptotically when $p/n\le \gamma $ or asymptotically in the high-dimensional asymptotic regime $p/n\to \gamma ^{\prime}\in (0,\infty )$. General differentiable loss functions $\rho $ are allowed provided that the derivative of the loss is 1-Lipschitz; this includes the least-squares loss as well as robust losses such as the Huber loss and its smoothed versions. The validity of the out-of-sample error estimate holds either under a strong convexity assumption, or for the L1-penalized Huber M-estimator and the Lasso under a sparsity assumption and a bound on the number of contaminated observations. For the square loss and in the absence of corruption in the response, the results additionally yield $n^{-1/2}$-consistent estimates of the noise variance and of the generalization error. This generalizes, to arbitrary convex penalty and arbitrary covariance, estimates that were previously known for the Lasso.

A proximal quasi-Newton method based on memoryless modified symmetric rank-one formula

&lt;p style='text-indent:20px;'&gt;We consider proximal gradient methods for minimizing a composite function of a differentiable function and a convex function. To accelerate the general proximal gradient methods, we focus on proximal quasi-Newton type methods based on proximal mappings scaled by quasi-Newton matrices. Although it is usually difficult to compute the scaled proximal mappings, applying the memoryless symmetric rank-one (SR1) formula makes this easier. Since the scaled (quasi-Newton) matrices must be positive definite, we develop an algorithm using the memoryless SR1 formula based on a modified spectral scaling secant condition. We give the subsequential convergence property of the proposed method for general objective functions. In addition, we show the R-linear convergence property of the method under a strong convexity assumption. Finally, some numerical results are reported.&lt;/p&gt;

A Decentralized Primal-Dual Method for Constrained Minimization of a Strongly Convex Function

We propose decentralized primal-dual methods for cooperative multiagent consensus optimization problems over both static and time-varying communication networks, where only local communications are allowed. The objective is to minimize the sum of agent-specific convex functions over conic constraint sets defined by agent-specific nonlinear functions; hence, the optimal consensus decision should lie in the intersection of these private sets. Under the strong convexity assumption, we provide convergence rates for suboptimality, infeasibility, and consensus violation in terms of the number of communications required; examine the effect of underlying network topology on the convergence rates.

SOLO FTRL ALGORITHM FOR PRODUCTION MANAGEMENT WITH TRANSFER PRICES

We consider a firm producing and selling $d$ commodities, and consisting from $n$ production and $m$ sales divisions. The firm manager tries to stimulate the best division performance by sequentially selecting internal commodity prices (transfer prices). In the static problem under general strong convexity and compactness assumptions we show that the SOLO FTRL algorithm of Orabona and Pal (2018), applied to the dual problem, gives the estimates of order $T^{-1/4}$ in the number $T$ of iterations for the optimality gap and feasibility residuals. This algorithm uses only the information on division reactions to current prices. It does not depend on any parameters and requires no information on the production and cost functions. The results of similar nature are obtained for the dynamic problem, where these functions depend on an i.i.d. sequence of random variables. We present two computer experiments with one and two commodities. In the static case the transfer prices and the difference between the supply and demand demonstrate fast stabilization. In the dynamic case the same quantities fluctuate around equilibrium values after a short transition phase.

Regularized sample average approximation for high-dimensional stochastic optimization under low-rankness

This paper concerns a high-dimensional stochastic programming (SP) problem of minimizing a function of expected cost with a matrix argument. To this problem, one of the most widely applied solution paradigms is the sample average approximation (SAA), which uses the average cost over sampled scenarios as a surrogate to approximate the expected cost. Traditional SAA theories require the sample size to grow rapidly when the problem dimensionality increases. Indeed, for a problem of optimizing over a p-by-p matrix, the sample complexity of the SAA is given by \({\widetilde{O}}(1)\cdot \frac{p^2}{\epsilon ^2}\cdot {polylog}(\frac{1}{\epsilon })\) to achieve an \(\epsilon \)-suboptimality gap, for some poly-logarithmic function \({polylog}(\,\cdot \,)\) and some quantity \({\widetilde{O}}(1)\) independent of dimensionality p and sample size n. In contrast, this paper considers a regularized SAA (RSAA) with a low-rankness-inducing penalty. We demonstrate that, when the optimal solution to the SP is of low rank, the sample complexity of RSAA is \({\widetilde{O}}(1)\cdot \frac{p}{\epsilon ^3}\cdot {polylog}(p,\,\frac{1}{\epsilon })\), which is almost linear in p and thus indicates a substantially lower dependence on dimensionality. Therefore, RSAA can be more advantageous than SAA especially for larger scale and higher dimensional problems. Due to the close correspondence between stochastic programming and statistical learning, our results also indicate that high-dimensional low-rank matrix recovery is possible generally beyond a linear model, even if the common assumption of restricted strong convexity is completely absent.

A globally convergent proximal Newton-type method in nonsmooth convex optimization

The paper proposes and justifies a new algorithm of the proximal Newton type to solve a broad class of nonsmooth composite convex optimization problems without strong convexity assumptions. Based on advanced notions and techniques of variational analysis, we establish implementable results on the global convergence of the proposed algorithm as well as its local convergence with superlinear and quadratic rates. For certain structured problems, the obtained local convergence conditions do not require the local Lipschitz continuity of the corresponding Hessian mappings that is a crucial assumption used in the literature to ensure a superlinear convergence of other algorithms of the proximal Newton type. The conducted numerical experiments of solving the \(l_1\) regularized logistic regression model illustrate the possibility of applying the proposed algorithm to deal with practically important problems.

Primal-dual algorithms for multi-agent structured optimization over message-passing architectures with bounded communication delays

We consider algorithms for solving structured convex optimization problems over a network of agents with communication delays. It is assumed that each agent performs its local updates using possibly outdated information from its neighbours under the assumption that the delay with respect to each neighbour is bounded but otherwise arbitrary. The private objective of each agent is represented by the sum of two possibly nonsmooth functions one of which is composed with a linear mapping. The global optimization problem is the aggregate of local cost functions and a common Lipschitz-differentiable function. When the coupling between agents is represented only through the common function, the V\\~u-Condat primal-dual algorithm is studied. In the case when the linear maps introduce additional coupling between agents a new algorithm is developed. Moreover, a randomized variant of this algorithm is presented that allows the agents to wake up at random and independently from one another. The convergence of the proposed algorithms is established under strong convexity assumptions.

An inexact accelerated stochastic ADMM for separable convex optimization

An inexact accelerated stochastic Alternating Direction Method of Multipliers (AS-ADMM) scheme is developed for solving structured separable convex optimization problems with linear constraints. The objective function is the sum of a possibly nonsmooth convex function and a smooth function which is an average of many component convex functions. Problems having this structure often arise in machine learning and data mining applications. AS-ADMM combines the ideas of both ADMM and the stochastic gradient methods using variance reduction techniques. One of the ADMM subproblems employs a linearization technique while a similar linearization could be introduced for the other subproblem. For a specified choice of the algorithm parameters, it is shown that the objective error and the constraint violation are $$\mathcal {O}(1/k)$$ relative to the number of outer iterations k. Under a strong convexity assumption, the expected iterate error converges to zero linearly. A linearized variant of AS-ADMM and incremental sampling strategies are also discussed. Numerical experiments with both stochastic and deterministic ADMM algorithms show that AS-ADMM can be particularly effective for structured optimization arising in big data applications.

Hyperfast second-order local solvers for efficient statistically preconditioned distributed optimization

Statistical preconditioning enables fast methods for distributed large-scale empirical risk minimization problems. In this approach, multiple worker nodes compute gradients in parallel, which are then used by the central node to update the parameter by solving an auxiliary (preconditioned) smaller-scale optimization problem. The recently proposed Statistically Preconditioned Accelerated Gradient (SPAG) method [1] has complexity bounds superior to other such algorithms but requires an exact solution for computationally intensive auxiliary optimization problems at every iteration. In this paper, we propose an Inexact SPAG (InSPAG) and explicitly characterize the accuracy by which the corresponding auxiliary subproblem needs to be solved to guarantee the same convergence rate as the exact method. We build our results by first developing an inexact adaptive accelerated Bregman proximal gradient method for general optimization problems under relative smoothness and strong convexity assumptions, which may be of independent interest. Moreover, we explore the properties of the auxiliary problem in the InSPAG algorithm assuming Lipschitz third-order derivatives and strongly convexity. For such problem class, we develop a linearly convergent Hyperfast second-order method and estimate the total complexity the InSPAG method with hyperfast auxiliary problem solver. Finally, we illustrate the proposed method's practical efficiency by performing large-scale numerical experiments on logistic regression models. To the best of our knowledge, these are the first empirical results on implementing high-order methods on large-scale problems, we work with data where the dimension is of the order of 3 million, and the number of samples is 700 million. • Inexact Statistically Preconditioned Accelerated Gradient Method for large-scale distributed convex empirical risk minimization (ERM) problems. • Hyperfast second-order algorithm for minimizing strongly convex functions with Lipschitz third-order derivatives. • Inexact adaptive accelerated Bregman proximal gradient method for general optimization problems under relative smoothness and strong convexity assumptions. • Empirical evidence for the efficiency of tensor optimization methods for large-scale ERM problems.

On the convergence of a randomized block coordinate descent algorithm for a matrix least squares problem

Randomized block coordinate descent type methods have been demonstrated to be efficient for solving large-scale optimization problems. Linear convergence to the unique solution is established if the objective function is strongly convex. In this paper we propose a randomized block coordinate descent algorithm for solving the matrix least squares problem minX∈Rm×n‖C − AXB‖F with A∈Rp×m, B∈Rn×q, and C∈Rp×q. We prove that the proposed algorithm converges linearly to the unique minimum norm least squares solution (i.e., A†CB†) without the strong convexity assumption. Instead, we only need that B has full row rank. Numerical experiments are given to illustrate the theoretical results.

Alternating minimization methods for strongly convex optimization

Abstract We consider alternating minimization procedures for convex and non-convex optimization problems with the vector of variables divided into several blocks, each block being amenable for minimization with respect to its variables while maintaining other variables blocks constant. In the case of two blocks, we prove a linear convergence rate for an alternating minimization procedure under the Polyak–Łojasiewicz (PL) condition, which can be seen as a relaxation of the strong convexity assumption. Under the strong convexity assumption in the many-blocks setting, we provide an accelerated alternating minimization procedure with linear convergence rate depending on the square root of the condition number as opposed to just the condition number for the non-accelerated method. We also consider the problem of finding an approximate non-negative solution to a linear system of equations A ⁢ x = y {Ax=y} with alternating minimization of Kullback–Leibler (KL) divergence between Ax and y.

The finite sample properties of sparse M-estimators with pseudo-observations

We provide finite sample properties of general regularized statistical criteria in the presence of pseudo-observations. Under the restricted strong convexity assumption of the unpenalized loss function and regularity conditions on the penalty, we derive non-asymptotic error bounds on the regularized M-estimator. This penalized framework with pseudo-observations is then applied to the M-estimation of some usual copula-based models. These theoretical results are supported by an empirical study.

Stochastic proximal splitting algorithm for composite minimization

Supported by the recent contributions in multiple domains, the first-order splitting became algorithms of choice for structured nonsmooth optimization. The large-scale noisy contexts make available stochastic information on the objective function and thus, the extension of proximal gradient schemes to stochastic oracles is heavily based on the tractability of the proximal operator corresponding to nonsmooth component, which has been highly exploited in the literature. However, some questions remained about the complexity of the composite models with proximal untractable terms. In this paper we tackle composite optimization problems, assuming only the access to stochastic information on both smooth and nonsmooth components, with a stochastic proximal first-order scheme with stochastic proximal updates. We provide sublinear \(\mathcal {O}\left( \frac{1}{k} \right) \) convergence rates (in expectation of squared distance to the optimal set) under the strong convexity assumption on the objective function. Also, linear convergence is achieved for convex feasibility problems. The empirical behavior is illustrated by numerical tests on parametric sparse representation models.

On stochastic accelerated gradient with non-strongly convexity

&lt;abstract&gt;&lt;p&gt;In this paper, we consider stochastic approximation algorithms for least-square and logistic regression with no strong-convexity assumption on the convex loss functions. We develop two algorithms with varied step-size motivated by the accelerated gradient algorithm which is initiated for convex stochastic programming. We analyse the developed algorithms that achieve a rate of $ O(1/n^{2}) $ where $ n $ is the number of samples, which is tighter than the best convergence rate $ O(1/n) $ achieved so far on non-strongly-convex stochastic approximation with constant-step-size, for classic supervised learning problems. Our analysis is based on a non-asymptotic analysis of the empirical risk (in expectation) with less assumptions that existing analysis results. It does not require the finite-dimensionality assumption and the Lipschitz condition. We carry out controlled experiments on synthetic and some standard machine learning data sets. Empirical results justify our theoretical analysis and show a faster convergence rate than existing other methods.&lt;/p&gt;&lt;/abstract&gt;

Stability analysis of solutions to equilibrium problems and applications in economics

Purpose The purpose of this paper is to study the Hölder calmness of solutions to equilibrium problems and apply it to economics. Design/methodology/approach The authors obtain the Hölder calmness by using an effective approach. More precisely, under the key assumption of strong convexity, sufficient conditions for the Hölder continuity of solution maps to equilibrium problems are established. Findings A new result in stability analysis for equilibrium problems and applications in economics is archived. Originality/value The authors confirm that the paper has not been published previously, is not under consideration for publication elsewhere and is not being simultaneously submitted elsewhere.