Strong Convexity Research Articles

Heuristic rule for inexact Newton-Landweber iteration with convex penalty terms of nonlinear: ill-posed problems

Inexact Newton regularization methods are a family of prominent regularization methods for solving nonlinear ill-posed problems, which consist of an outer Newton iteration and an inner scheme providing increments by applying the regularization technique to the local linearized equations. In this paper, we propose a heuristic stopping rule for the inexact Newton regularization method, where the inner scheme is defined by Landweber iteration and the strong convex function is incorporated as the penalty term. In contrast to a prior and a posteriori stopping rules, our heuristic rule is purely data driven and does not require the information on noise level, which renders the method feasible when the noise level is unknown or unreliable. Under certain assumptions on the random noise, we establish a new convergence analysis for the inexact Newton-Landweber method under the heuristic rule. The numerical simulations are provided to demonstrate the performance of our heuristic rule.

Demonstration of the strength of strong convexity via Jensen's gap

This paper demonstrates through a numerical experiment that utilization of strongly convex functions strengthens the bound presented for the Jensen gap in [4]. Consequently the improved result enables to present improvements in the bounds obtained for the Hölder and Hermite-Hadamard gaps and proposes such improvements in the results obtained for various entropies and divergences in information theory.

Linear Convergence of ADMM Under Metric Subregularity for Distributed Optimization

Alternating Direction Method of Multipliers (ADMM) has seen much progress in the literature in recent years. Usually, linear convergence of distributed ADMM is proved under either second-order conditions or strong convexity. When both conditions fail, an alternative is expected to play the role. In this paper, it is shown that distributed ADMM can achieve a linear convergence rate by imposing metric subregularity on a defined mapping. Furthermore, it is proved that both second-order conditions and strong convexity imply metric subregularity under reasonable conditions, e.g. the cost functions being twice continuously differentiable in a neighborhood. In addition, nonergodic convergence rates are presented as well for problems under consideration. Finally, simulation results are carried out to illustrate the efficiency of the proposed algorithm.

Local convexity of the TAP free energy and AMP convergence for Z2-synchronization

We study mean-field variational Bayesian inference using the TAP approach, for Z2-synchronization as a prototypical example of a high-dimensional Bayesian model. We show that for any signal strength λ>1 (the weak-recovery threshold), there exists a unique local minimizer of the TAP free energy functional near the mean of the Bayes posterior law. Furthermore, the TAP free energy in a local neighborhood of this minimizer is strongly convex. Consequently, a natural-gradient/mirror-descent algorithm achieves linear convergence to this minimizer from a local initialization, which may be obtained by a constant number of iterations of Approximate Message Passing (AMP). This provides a rigorous foundation for variational inference in high dimensions via minimization of the TAP free energy. We also analyze the finite-sample convergence of AMP, showing that AMP is asymptotically stable at the TAP minimizer for any λ>1, and is linearly convergent to this minimizer from a spectral initialization for sufficiently large λ. Such a guarantee is stronger than results obtainable by state evolution analyses, which only describe a fixed number of AMP iterations in the infinite-sample limit. Our proofs combine the Kac–Rice formula and Sudakov–Fernique Gaussian comparison inequality to analyze the complexity of critical points that satisfy strong convexity and stability conditions within their local neighborhoods.

An accelerated distributed method with inexact model of relative smoothness and strong convexity

AbstractDistributed optimisation methods are widely applied in many systems where agents cooperate with each other to minimise a sum‐type problem over a connected network. An accelerated distributed method based on the inexact model of relative smoothness and strong convexity is introduced by the authors. The authors demonstrate that the proposed method can converge to the optimal solution at the linear rate and achieve the optimal gradient computation complexity and the near optimal communication complexity, where κg denotes the global condition number. Finally, the numerical experiments are provided to validate the theoretical results and further show the efficacy of the proposed method.

Orientation estimation of cryo-EM images using projected gradient descent method

Orientation estimation is an important task in three-dimensional cryo-EM image reconstruction. By applying the common line method, the orientation estimation task can be formulated as a least squares (LS) problem or a least un-squared deviation (LUD) problem with orthogonality constraint. However, the non-convexity of the orthogonality constraint introduces numerical difficulties to the orientation estimation. The conventional approach is to reformulate the orthogonality constrained minimization problem into a semi-definite programming problem using convex relaxation strategies. In this paper, we consider a direct way to solve the constrained minimization problem without relaxation. We focus on the weighted LS problem because the LUD problem can be reformulated into a sequence of weighted LS problems using the iteratively re-weighted LS approach. As a classical approach, the projected gradient descent (PGD) method has been successfully applied to solve the convex constrained minimization problem. We apply the PGD method to the minimization problem with orthogonality constraint and show that the constraint set is a generalized prox-regular set, and it satisfies the norm compatibility condition. We also demonstrate that the objective function of the minimization problem satisfies the restricted strong convexity and the restricted strong smoothness over a constraint set. Therefore, the sequence generated by the PGD method converges when the initial conditions are satisfied. Experimental results show that the PGD method significantly outperforms the semi-definite relaxation methods from a computation standpoint, and the mean square error is almost the same or smaller.

Sublinear and Linear Convergence of Modified ADMM for Distributed Nonconvex Optimization

In this article, we consider distributed nonconvex optimization over an undirected connected network. Each agent can only access to its own local nonconvex cost function and all agents collaborate to minimize the sum of these functions by using local information exchange. We first propose a modified alternating direction method of multipliers (ADMM) algorithm. We show that the proposed algorithm converges to a stationary point with the sublinear rate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(1/T)$</tex-math></inline-formula> if each local cost function is smooth and the algorithm parameters are chosen appropriately. We also show that the proposed algorithm linearly converges to a global optimum under an additional condition that the global cost function satisfies the Polyak–Łojasiewicz condition, which is weaker than the commonly used conditions for showing linear convergence rates including strong convexity. We then propose a distributed linearized ADMM (L-ADMM) algorithm, derived from the modified ADMM algorithm, by linearizing the local cost function at each iteration. We show that the L-ADMM algorithm has the same convergence properties as the modified ADMM algorithm under the same conditions. Numerical simulations are included to verify the correctness and efficiency of the proposed algorithms.

Some new refinements of the Young, Hölder, and Minkowski inequalities

We prove and discuss some new refined Hölder inequalities for any p>1 and also a reversed version for 0< p<1. The key is to use the concepts of superquadraticity, strong convexity, and to first prove the corresponding refinements of the Young and reversed Young inequalities. Refinements of the Minkowski and reversed Minkowski inequalities are also given.

A stochastic averaging gradient algorithm with multi‐step communication for distributed optimization

AbstractThis paper studies distributed convex optimization problems over an undirected network where all nodes cooperate to minimize a sum of local objective functions. Each local objective function is further assumed to be an average of several convex instantaneous functions. By incorporating the stochastic averaging gradient into the distributed first‐order primal‐dual method, a stochastic averaging gradient algorithm with multi‐step communication is proposed to solve the optimization problem. For each node, one randomly selected gradient of an instantaneous function is evaluated per iteration, which effectively reduces the computation cost of the algorithm. Based on conditions of strong convexity and Lipschitz continuity of the local instantaneous functions, a linear convergence rate of the proposed algorithm is guaranteed. Numerical simulations on the logistic regression problem demonstrate the performance of the algorithm and correctness of the theoretical results.

Communication-efficient and Byzantine-robust distributed learning with statistical guarantee

Communication efficiency and robustness are two major issues in modern distributed learning frameworks. This is due to the practical situations where some computing nodes may have limited communication power or may behave adversarial behaviors. To address the two issues simultaneously, this paper develops two communication-efficient and robust distributed learning algorithms for convex problems. Our motivation is based on surrogate likelihood framework and the median and trimmed mean operations. Particularly, the proposed algorithms are provably robust against Byzantine failures, and also achieve optimal statistical rates for strong convex losses and convex (non-smooth) penalties. For typical statistical models such as generalized linear models, our results show that statistical errors dominate optimization errors in finite iterations. Simulated and real data experiments are conducted to demonstrate the numerical performance of our algorithms.

Federated learning for minimizing nonsmooth convex loss functions

Federated learning possesses a distributed learning framework for protecting privacy where local clients need to collaboratively train a shared model via a central server. Many existing methods in the literature of federated learning are analyzed under some smoothness conditions of loss functions and need access to gradient information to perform local optimization algorithms such as stochastic gradient descent or dual average. Some methods even need a strong convexity of the loss function. However, in many situations of real world applications such as readability of texts, first-order gradient information is difficult to obtain, and the strong smoothness and strong convexity of loss functions are not satisfied. We consider methods to overcome such situations. This paper aims at providing an understanding of federated learning in the situation that the loss functions are nonsmooth and gradient information is unavailable, also a strong convexity condition is not needed. Based on Nesterov's zeroth-order (gradient-free) techniques, we propose a zeroth-order stochastic federated learning method. Constant and decreasing step size strategies are considered. Moreover, a new type of approximating sequence is proposed in federated learning for strictly decreasing step sizes. Expected error bounds for the proposed approximating sequence and learning rates of the proposed method are derived under some selection rules of the step sizes and smoothing parameters.

A proximal quasi-Newton method based on memoryless modified symmetric rank-one formula

&lt;p style='text-indent:20px;'&gt;We consider proximal gradient methods for minimizing a composite function of a differentiable function and a convex function. To accelerate the general proximal gradient methods, we focus on proximal quasi-Newton type methods based on proximal mappings scaled by quasi-Newton matrices. Although it is usually difficult to compute the scaled proximal mappings, applying the memoryless symmetric rank-one (SR1) formula makes this easier. Since the scaled (quasi-Newton) matrices must be positive definite, we develop an algorithm using the memoryless SR1 formula based on a modified spectral scaling secant condition. We give the subsequential convergence property of the proposed method for general objective functions. In addition, we show the R-linear convergence property of the method under a strong convexity assumption. Finally, some numerical results are reported.&lt;/p&gt;

Subordination Properties of Certain Operators Concerning Fractional Integral and Libera Integral Operator

The results contained in this paper are the result of a study regarding fractional calculus combined with the classical theory of differential subordination established for analytic complex valued functions. A new operator is introduced by applying the Libera integral operator and fractional integral of order λ for analytic functions. Many subordination properties are obtained for this newly defined operator by using famous lemmas proved by important scientists concerned with geometric function theory, such as Eenigenburg, Hallenbeck, Miller, Mocanu, Nunokawa, Reade, Ruscheweyh and Suffridge. Results regarding strong starlikeness and convexity of order α are also discussed, and an example shows how the outcome of the research can be applied.

Theoretical and numerical comparison of first order algorithms for cocoercive equations and smooth convex optimization

This paper provides a theoretical and numerical comparison of classical first-order splitting methods for solving smooth convex optimization problems and cocoercive equations. From a theoretical point of view, we compare convergence rates of gradient descent, forward-backward, Peaceman-Rachford, and Douglas-Rachford algorithms for minimizing the sum of two smooth convex functions when one of them is strongly convex. A similar comparison is given in the more general cocoercive setting under the presence of strong monotonicity and we observe that the convergence rates in optimization are strictly better than the corresponding rates for cocoercive equations for some algorithms. We obtain improved rates with respect to the literature in several instances by exploiting the structure of our problems. Moreover, we indicate which algorithm has the smallest convergence rate depending on strong convexity and cocoercive parameters. From a numerical point of view, we verify our theoretical results by implementing and comparing previous algorithms in well established signal and image inverse problems involving sparsity. We replace the widely used ℓ1 norm with the Huber loss and we observe that fully proximal-based strategies have numerical and theoretical advantages with respect to methods using gradient steps.

A new class of composite objective multistep estimating sequence techniques

A plethora of problems arising in signal processing, machine learning and statistics can be cast as large-scale optimization problems with a composite objective structure. Such problems are typically solved by utilizing iterative first-order algorithms. In this work, we devise a new accelerated gradient-based estimating sequence technique for solving large-scale optimization problems with composite objective structure. Specifically, we introduce a new class of estimating functions, which are obtained by utilizing both a tight lower bound on the objective function, as well as the gradient mapping technique. Then, using the proposed estimating functions, we construct a class of Composite Objective Multi-step Estimating-sequence Techniques (COMET), which are endowed with an efficient line-search procedure. We prove that our proposed COMET enjoys the accelerated convergence rate, and our newly established convergence results allow for step-size adaptation. Our theoretical findings are supported by extensive computational experiments on various problem types and real-world datasets. Moreover, our numerical results show evidence of the robustness of the proposed method to the imperfect knowledge of the smoothness and strong convexity parameters.

Treatment of thrombosis in KD Patients using tissue plasminogen activator: a single center study

ObjectiveThe most severe complication associated with giant coronary aneurysm in children with Kawasaki disease is ischemic cardiomyopathy (ICM) caused by thrombosis. Addition of tissue plasminogen activator, Alteplase, in the treatment regimen can be an efficient thrombolytic therapy, and therefore can have a significantly positive impact on patients’ quality of life in long term.MethodsTotal four male KD patients with central thromboses in coronary aneurysm were treated in Pediatric Cardiology Department of Shengjing Hospital, China Medical University, from January 2020 to August 2021. These patients received thrombolytic treatments including Alteplase once + Heparin for 1 week followed by continuous oral Warfarin + Aspirin + Clopidogrel.Results4 young male KD patients had coronary aneurysm (CAA) complicated with total 7 occurrences of central thrombosis. These patients were given alteplase and heparin/oral Warfarin + Aspirin + Clopidogrel treatment. 9 days to 2 months later, thromboses were significantly dissolved. The treatment successfully diminished the thrombosis complication.Conclusion1. Pediatric KD patients complicated with coronary aneurysm thrombosis are prone to recurrence of thrombosis. 2. In KD patients complicated with coronary aneurysm thrombosis, treatments described in Method can be used for treating either small thromboses formed less than 1 month with strong echo and convex lumen or large thromboses with mixed strong and weak echo. With these treatments, coronary artery blood flow can be improved or completely recovered. 3. Clinical experiences at our center in treating these KD patients suggest that Alteplase can be considered in thrombolytic treatment beyond the limitation of less than 12 h of thrombosis occurrence.

The existence and uniqueness of solutions for kernel-based system identification

The notion of reproducing kernel Hilbert space (RKHS) has emerged in system identification during the past decade. In the resulting framework, the impulse response estimation problem is formulated as a regularized optimization defined on an infinite-dimensional RKHS consisting of stable impulse responses. The consequent estimation problem is well-defined under the central assumption that the convolution operators restricted to the RKHS are continuous linear functionals. Moreover, according to this assumption, the representer theorem hold, and therefore, the impulse response can be estimated by solving a finite-dimensional program. Thus, the continuity feature plays a significant role in kernel-based system identification. We show that this central assumption is guaranteed to be satisfied in considerably general situations, namely when the input signal is bounded, the kernel is an integrable function, and in the case of continuous-time dynamics, continuous. Furthermore, the strong convexity of the optimization problem and the continuity property of the convolution operators imply that the kernel-based system identification admits a unique solution. Consequently, it follows that kernel-based system identification is a well-defined approach.

De Rham Decomposition Theorem for Strongly Convex Kähler–Berwald Manifolds

De Rham Decomposition Theorem for Strongly Convex Kähler–Berwald Manifolds

A Decentralized Primal-Dual Method for Constrained Minimization of a Strongly Convex Function

We propose decentralized primal-dual methods for cooperative multiagent consensus optimization problems over both static and time-varying communication networks, where only local communications are allowed. The objective is to minimize the sum of agent-specific convex functions over conic constraint sets defined by agent-specific nonlinear functions; hence, the optimal consensus decision should lie in the intersection of these private sets. Under the strong convexity assumption, we provide convergence rates for suboptimality, infeasibility, and consensus violation in terms of the number of communications required; examine the effect of underlying network topology on the convergence rates.

Privacy-Preserving Asynchronous Vertical Federated Learning Algorithms for Multiparty Collaborative Learning.

The privacy-preserving federated learning for vertically partitioned (VP) data has shown promising results as the solution of the emerging multiparty joint modeling application, in which the data holders (such as government branches, private finance, and e-business companies) collaborate throughout the learning process rather than relying on a trusted third party to hold data. However, most of the existing federated learning algorithms for VP data are limited to synchronous computation. To improve the efficiency when the unbalanced computation/communication resources are common among the parties in the federated learning system, it is essential to develop asynchronous training algorithms for VP data while keeping the data privacy. In this article, we propose an asynchronous federated stochastic gradient descent (AFSGD-VP) algorithm and its two variance reduction variants, including stochastic variance reduced gradient (SVRG) and SAGA on the VP data. Moreover, we provide the convergence analyses of AFSGD-VP and its SVRG and SAGA variants under the condition of strong convexity and without any restrictions of staleness. We also discuss their model privacy, data privacy, computational complexities, and communication costs. To the best of our knowledge, AFSGD-VP and its SVRG and SAGA variants are the first asynchronous federated learning algorithms for VP data with theoretical guarantees. Extensive experimental results on a variety of VP datasets not only verify the theoretical results of AFSGD-VP and its SVRG and SAGA variants but also show that our algorithms have much higher efficiency than the corresponding synchronous algorithms.