Strong Convexity Research Articles

A projection-free method for solving convex bilevel optimization problems

AbstractWhen faced with multiple minima of an inner-level convex optimization problem, the convex bilevel optimization problem selects an optimal solution which also minimizes an auxiliary outer-level convex objective of interest. Bilevel optimization requires a different approach compared to single-level optimization problems since the set of minimizers for the inner-level objective is not given explicitly. In this paper, we propose a new projection-free conditional gradient method for convex bilevel optimization which requires only a linear optimization oracle over the base domain. We establish $$O(t^{-1/2})$$ O ( t - 1 / 2 ) convergence rate guarantees for our method in terms of both inner- and outer-level objectives, and demonstrate how additional assumptions such as quadratic growth and strong convexity result in accelerated rates of up to $$O(t^{-1})$$ O ( t - 1 ) and $$O(t^{-2/3})$$ O ( t - 2 / 3 ) for inner- and outer-levels respectively. Lastly, we conduct a numerical study to demonstrate the performance of our method.

Statement of Retraction: Convergence in quadratic mean of averaged stochastic gradient algorithms without strong convexity nor bounded gradient

Statement of Retraction: Convergence in quadratic mean of averaged stochastic gradient algorithms without strong convexity nor bounded gradient

Geodesic Convexity of the Symmetric Eigenvalue Problem and Convergence of Steepest Descent

We study the convergence of the Riemannian steepest descent algorithm on the Grassmann manifold for minimizing the block version of the Rayleigh quotient of a symmetric matrix. Even though this problem is non-convex in the Euclidean sense and only very locally convex in the Riemannian sense, we discover a structure for this problem that is similar to geodesic strong convexity, namely, weak-strong convexity. This allows us to apply similar arguments from convex optimization when studying the convergence of the steepest descent algorithm but with initialization conditions that do not depend on the eigengap δ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\delta $$\\end{document}. When δ>0\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\delta >0$$\\end{document}, we prove exponential convergence rates, while otherwise the convergence is algebraic. Additionally, we prove that this problem is geodesically convex in a neighbourhood of the global minimizer of radius O(δ)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\\mathcal {O}}(\\sqrt{\\delta })$$\\end{document}.

Non-asymptotic convergence bounds for modified tamed unadjusted Langevin algorithm in non-convex setting

We consider the problem of sampling from a high-dimensional target distribution πβ on Rd with density proportional to θ↦e−βU(θ) using explicit numerical schemes based on discretising the Langevin stochastic differential equation (SDE). In recent literature, taming has been proposed and studied as a method for ensuring stability of Langevin-based numerical schemes in the case of super-linearly growing drift coefficients for the Langevin SDE. In particular, the Tamed Unadjusted Langevin Algorithm (TULA) was proposed in [2] to sample from such target distributions with the gradient of the potential U being super-linearly growing. However, theoretical guarantees in Wasserstein distances for Langevin-based algorithms have traditionally been derived assuming strong convexity of the potential U. In this paper, we propose a novel taming factor and derive, under a setting with possibly non-convex potential U and super-linearly growing gradient of U, non-asymptotic theoretical bounds in Wasserstein-1 and Wasserstein-2 distances between the law of our algorithm, which we name the modified Tamed Unadjusted Langevin Algorithm (mTULA), and the target distribution πβ. We obtain respective rates of convergence O(λ) and O(λ1/2) in Wasserstein-1 and Wasserstein-2 distances for the discretisation error of mTULA in step size λ. High-dimensional numerical simulations which support our theoretical findings are presented to showcase the applicability of our algorithm.

Bayesian Federated Learning with Hamiltonian Monte Carlo: Algorithm and Theory

This work introduces a novel and efficient Bayesian federated learning algorithm, namely, the Federated Averaging stochastic Hamiltonian Monte Carlo (FA-HMC), for parameter estimation and uncertainty quantification. We establish rigorous convergence guarantees of FA-HMC on non-iid distributed datasets, under the strong convexity and Hessian smoothness assumptions. Our analysis investigates the effects of parameter space dimension, noise on gradients and momentum, and the frequency of communication (between the central node and local nodes) on the convergence and communication costs of FA-HMC. Beyond that, we establish the tightness of our analysis by showing that the convergence rate cannot be improved even for continuous FA-HMC process. Moreover, extensive empirical studies demonstrate that FA-HMC outperforms the existing Federated Averaging-Langevin Monte Carlo (FA-LD) algorithm. Supplementary materials for this article are available online.

Continuation problem of the electromagnetic field in the frequency domain for horizontally layred media

The paper presents analytical expressions for solving the problem of continuation of the electromagnetic field in the frequency domain for a horizontally layered medium. The numerical solution algorithm uses layerwise recalculation of the required quantities, for which analytical expressions are presented in a form that allows calculations and layerwise recalculation without the accumulation of rounding errors. The solution to the continuation problem is obtained as a solution to the cost functional minimization problem. The strong convexity of the functional is proven, which implies the existence of a unique solution to the problem posed.

Non-ergodic convergence rate of an inertial accelerated primal–dual algorithm for saddle point problems

In this paper, we design an inertial accelerated primal–dual algorithm to address the convex–concave saddle point problem, which is formulated as minxmaxyf(x)+〈Kx,y〉−g(y). Remarkably, both functions f and g exhibit a composite structure, combining “nonsmooth” ＋ “smooth” components. Under the assumption of partially strong convexity in the sense that f is convex and g is strongly convex, we introduce a novel inertial accelerated primal–dual algorithm based on Nesterov’s extrapolation. This algorithm can be reduced to two classical accelerated forward–backward methods for unconstrained optimization problem. We show that the proposed algorithm achieves a non-ergodic O(1/k2) convergence rate, where k represents the number of iterations. Several numerical experiments validate the efficiency of our proposed algorithm.

On multiobjective optimization problems involving higher order strong convexity using directional convexificators

On multiobjective optimization problems involving higher order strong convexity using directional convexificators

Efficient First-Order Algorithms for Large-Scale, Non-Smooth Maximum Entropy Models with Application to Wildfire Science.

Maximum entropy (MaxEnt) models are a class of statistical models that use the maximum entropy principle to estimate probability distributions from data. Due to the size of modern data sets, MaxEnt models need efficient optimization algorithms to scale well for big data applications. State-of-the-art algorithms for MaxEnt models, however, were not originally designed to handle big data sets; these algorithms either rely on technical devices that may yield unreliable numerical results, scale poorly, or require smoothness assumptions that many practical MaxEnt models lack. In this paper, we present novel optimization algorithms that overcome the shortcomings of state-of-the-art algorithms for training large-scale, non-smooth MaxEnt models. Our proposed first-order algorithms leverage the Kullback-Leibler divergence to train large-scale and non-smooth MaxEnt models efficiently. For MaxEnt models with discrete probability distribution of n elements built from samples, each containing m features, the stepsize parameter estimation and iterations in our algorithms scale on the order of O(mn) operations and can be trivially parallelized. Moreover, the strong ℓ1 convexity of the Kullback-Leibler divergence allows for larger stepsize parameters, thereby speeding up the convergence rate of our algorithms. To illustrate the efficiency of our novel algorithms, we consider the problem of estimating probabilities of fire occurrences as a function of ecological features in the Western US MTBS-Interagency wildfire data set. Our numerical results show that our algorithms outperform the state of the art by one order of magnitude and yield results that agree with physical models of wildfire occurrence and previous statistical analyses of wildfire drivers.

New Improvements of the Jensen–Mercer Inequality for Strongly Convex Functions with Applications

In this paper, we use the generalized version of convex functions, known as strongly convex functions, to derive improvements to the Jensen–Mercer inequality. We achieve these improvements through the newly discovered characterizations of strongly convex functions, along with some previously known results about strongly convex functions. We are also focused on important applications of the derived results in information theory, deducing estimates for χ-divergence, Kullback–Leibler divergence, Hellinger distance, Bhattacharya distance, Jeffreys distance, and Jensen–Shannon divergence. Additionally, we prove some applications to Mercer-type power means at the end.

Predicting hostility towards women: incel-related factors in a general sample of men.

Hostility towards women is a type of prejudice that can have adverse effects on women and society, but research on predictors of men's hostility towards women is limited. The present study primarily introduced predictors associated with misogynist involuntary celibates (incels), and then investigated whether loneliness, rejection, attractiveness, number of romantic and sexual partners, right-wing authoritarianism, and gaming predicted hostility towards women among a more general sample of men. A total of 473 men (aged 18-35, single, heterosexual, UK residents) recruited via Prolific answered the hostile sexism subscale, the misogyny scale, the self-perceived sexual attractiveness scale, the right-wing authoritarianism scale, the game addiction scale for adolescents, the adult rejection-sensitivity scale, the UCLA loneliness scale, and self-developed questions regarding number of sexual and romantic partners, and time spent gaming. We found a strong positive relationship between right-wing authoritarianism and hostility towards women, as well as a strong convex curvilinear relationship between attractiveness and hostility towards women. The number of sexual partners showed a moderate concave relationship with hostility towards women. We did not find sufficient support for a relationship between gaming and hostility towards women, and there was no support that loneliness, rejection, or romantic partners predicted hostility towards women among a general sample of men. Our study supports right-wing authoritarianism and self-perceived attractiveness as potential strong predictors in understanding men's hostility towards women in the wider community. Pre-registration: https://osf.io/ms3a4.

Strong F\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$\\mathcal {F}$\\end{document}-convexity and concavity and refinements of some classical inequalities

The concept of strong F\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}${\\mathcal {F}}$\\end{document}-convexity is a natural generalization of strong convexity. Although strongly concave functions are rarely mentioned and used, we show that in more effective and specific analysis this concept is very useful, and especially its generalization, namely strong F\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}${\\mathcal {F}}$\\end{document}-concavity. Using this concept, refinements of the Young inequality are given as a model case. A general form of the self-improving property for Jensen type inequalities is presented. We show that a careful choice of control functions for convex or concave functions can give a control over these refinements and produce refinements of the power mean inequalities.

Distributed stochastic optimization algorithm with non-consistent constraints in time-varying unbalanced networks

This study focuses on a class of distributed optimization problems with non-consistent local set constraints in time-varying unbalanced networks. Considering that each agent only has access to local gradient information with random errors, this paper proposes a distributed stochastic gradient projection algorithm. Under some conditions of random errors and a uniformly joint strongly connected topology, it is shown that the local decision states of all agents converge to a common optimal solution with a probability of one, achieving a sublinear convergence rate of O(lnk/k). Notably, under the condition that at least one local function exhibits strong convexity, the algorithm achieves a faster sublinear convergence rate of O(1/k). These theoretical results are validated through numerical simulations. Furthermore, this paper applies the proposed algorithm to solve for the unknown parameters in distributed linear regression problems with incomplete data. Numerical results indicate that the algorithm not only aligns with the convergence properties reported in existing literature but also demonstrates significant superiority in convergence speed. This important discovery not only lays a solid theoretical foundation for the further application and extension of the algorithm but also reveals its immense potential in solving practical problems.

Spectral Super-Resolution via Model-Guided Cross-Fusion Network.

Spectral super-resolution, which reconstructs a hyperspectral image (HSI) from a single red-green-blue (RGB) image, has acquired more and more attention. Recently, convolution neural networks (CNNs) have achieved promising performance. However, they often fail to simultaneously exploit the imaging model of the spectral super-resolution and complex spatial and spectral characteristics of the HSI. To tackle the above problems, we build a novel cross fusion (CF)-based model-guided network (called SSRNet) for spectral super-resolution. In specific, based on the imaging model, we unfold the spectral super-resolution into the HSI prior learning (HPL) module and imaging model guiding (IMG) module. Instead of just modeling one kind of image prior, the HPL module is composed of two subnetworks with different structures, which can effectively learn the complex spatial and spectral priors of the HSI, respectively. Furthermore, a CF strategy is used to establish the connection between the two subnetworks, which further improves the learning performance of the CNN. The IMG module results in solving a strong convex optimization problem, which adaptively optimizes and merges the two features learned by the HPL module by exploiting the imaging model. The two modules are alternately connected to achieve optimal HSI reconstruction performance. Experiments on both the simulated and real data demonstrate that the proposed method can achieve superior spectral reconstruction results with relatively small model size. The code will be available at https://github.com/renweidian.

Convergence rate analysis of the gradient descent–ascent method for convex–concave saddle-point problems

In this paper, we study the gradient descent–ascent method for convex–concave saddle-point problems. We derive a new non-asymptotic global convergence rate in terms of distance to the solution set by using the semidefinite programming performance estimation method. The given convergence rate incorporates most parameters of the problem and it is exact for a large class of strongly convex-strongly concave saddle-point problems for one iteration. We also investigate the algorithm without strong convexity and we provide some necessary and sufficient conditions under which the gradient descent–ascent enjoys linear convergence.

Linear Convergence of Forward-Backward Accelerated Algorithms without Knowledge of the Modulus of Strong Convexity

Linear Convergence of Forward-Backward Accelerated Algorithms without Knowledge of the Modulus of Strong Convexity

On Stability Estimate of Optimal Transport Maps Using m-th Polynomial Convexity

In 1781, Gaspard Monge first proposed the practical problem of relocating building materials while minimizing workers’ effort. Mathematically, the problem can be reiterated as finding a mapping T0 that transforms a random variable (X) following probability measure (μ) into a random variable (Y) following probability measure (ν), with minimal cost. Afterward, it has been widely studied and applied in statistics, machine learning, and economics, which concern the study of “distance” between usually a pair of probability distributions. The focus of this paper is centered on investigating and generalizing stability estimates for optimal transport plans, particularly through the lens of strong polynomial convexity. Building on previous research using plug-in estimators to strengthen the convergence rate of discrete or semi-discrete estimators for optimal transport plans, this paper introduces a novel stability estimate leveraging L-Lipschitz continuity and a paradigmatic methodology based on polynomial convexity, the understanding of which remains limited.

Adaptive, Doubly Optimal No-Regret Learning in Strongly Monotone and Exp-Concave Games with Gradient Feedback

Feasible Online Learning with Gradient Feedback Online gradient descent (OGD) is well-known to be doubly optimal under strong convexity or monotonicity assumptions: (1) in the single-agent setting, it achieves an optimal regret of [Formula: see text] for strongly convex cost functions, and (2) in the multiagent setting of strongly monotone games with each agent employing OGD we obtain last-iterate convergence of the joint action to a unique Nash equilibrium at an optimal rate of [Formula: see text]. Whereas these finite-time guarantees highlight its merits, OGD has the drawback that it requires knowing the strong convexity/monotonicity parameters. In “Adaptive, Doubly Optimal No-Regret Learning in Strongly Monotone and Exp-Concave Games with Gradient Feedback,” M. Jordan, T. Lin, and Z. Zhou design a fully adaptive OGD algorithm, AdaOGD, that does not require a priori knowledge of these parameters. In the single-agent setting, the algorithm achieves [Formula: see text] regret under strong convexity, which is optimal up to a log factor. Further, if each agent employs AdaOGD in strongly monotone games, the joint action converges in a last-iterate sense to a unique Nash equilibrium at a rate of [Formula: see text], again optimal up to log factors. The algorithms are illustrated in a learning version of the classic newsvendor problem, in which, because of lost sales, only (noisy) gradient feedback can be observed. The results immediately yield the first feasible and near-optimal algorithm for both the single-retailer and multiretailer settings.

A distributed algorithm with network‐independent step‐size and event‐triggered mechanism for economic dispatch problem

AbstractThe economic dispatch problem (EDP) poses a significant challenge in energy management for modern power systems, particularly as these systems undergo expansion. This growth escalates the demand for communication resources and increases the risk of communication failures. To address this challenge, we propose a distributed algorithm with network‐independent step sizes and an event‐triggered mechanism, which reduces communication requirements and enhances adaptability. Unlike traditional methods, our algorithm uses network‐independent step sizes derived from each agent's local cost functions, thus eliminating the need for detailed network topology knowledge. The theoretical derivation identifies a range of step size values that depend solely on the objective function's strong convexity and the gradient's Lipschitz continuity. Furthermore, the proposed algorithm is shown to achieve a linear convergence rate, assuming the event triggering threshold criteria are met for linear convergence. Numerical experiments further validate the effectiveness and advantages of our proposed distributed algorithm by demonstrating its ability to maintain good convergence characteristics while reducing communication frequency.

Bernstein–Kantorovich operators, approximation and shape preserving properties

Inspired by the construction of Bernstein and Kantorovich operators, we introduce a family of positive linear operators Kn\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal{K}_n$$\\end{document} preserving the affine functions. Their approximation properties are investigated and compared with similar properties of other operators. We determine the central moments of all orders of Kn\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${{\\mathcal {K}}}_n$$\\end{document} and use them in order to establish Voronovskaja type formulas. A special attention is paid to the shape preserving properties. The operators Kn\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${{\\mathcal {K}}}_n$$\\end{document} preserve monotonicity, convexity, strong convexity and approximate concavity. They have also the property of monotonic convergence under convexity. All the established inequalities involving convex functions can be naturally interpreted in the framework of convex stochastic ordering.