Linear Convergence Analysis Research Articles

Testing Convergence of Fiscal Policies in Regions of Turkey

The presence of convergence in the fiscal policies implemented at the national level in order to eliminate regional inequalities is an indication that the resources are rationally distributed among the regions and the differences are decreasing. From this point of view, in this study, it is aimed to investigate the convergence of the fiscal policy implemented in the period of 2004:01-2020:12 in Turkey within the scope of the fiscal purpose of taxes. In the study, nonlinear convergence analysis was performed, but linear convergence analysis was used because the linearity hypothesis could not be rejected. According to the findings, there is an absolute convergence between the regions in the fiscal policy implemented in Turkey. However, in terms of regions, the first region is differentiated from other five regions, while the other regions are very close to each other. Regarding this, it may be possible to bring the first region and the other five regions closer to each other by reducing the difference with the arrangements to be made in taxation policies. The obtained findings offer important implications for policy makers.

A unified and tight linear convergence analysis of the relaxed proximal point algorithm

<p style='text-indent:20px;'>Finding a zero of a maximal monotone operator is fundamental in convex optimization and monotone operator theory, and <i>proximal point algorithm</i> (PPA) is a primary method for solving this problem. PPA converges not only globally under fairly mild conditions but also asymptotically at a fast linear rate provided that the underlying inverse operator is Lipschitz continuous at the origin. These nice convergence properties are preserved by a relaxed variant of PPA. Recently, a linear convergence bound was established in [M. Tao, and X. M. Yuan, J. Sci. Comput., 74 (2018), pp. 826-850] for the relaxed PPA, and it was shown that the bound is tight when the relaxation factor <inline-formula><tex-math id="M1">\begin{document}$ \gamma $\end{document}</tex-math></inline-formula> lies in <inline-formula><tex-math id="M2">\begin{document}$ [1,2) $\end{document}</tex-math></inline-formula>. However, for other choices of <inline-formula><tex-math id="M3">\begin{document}$ \gamma $\end{document}</tex-math></inline-formula>, the bound obtained by Tao and Yuan is suboptimal. In this paper, we establish tight linear convergence bounds for any choice of <inline-formula><tex-math id="M4">\begin{document}$ \gamma\in(0,2) $\end{document}</tex-math></inline-formula> using a unified and much simplified analysis. These results sharpen our understandings to the asymptotic behavior of the relaxed PPA and make the whole picture for <inline-formula><tex-math id="M5">\begin{document}$ \gamma\in(0,2) $\end{document}</tex-math></inline-formula> clear.</p>

Adaptive iterative hard thresholding for low-rank matrix recovery and rank-one measurements

In low-rank matrix recovery, many kinds of measurements fail to meet the standard restricted isometry property (RIP), such as rank-one measurements, that is, [A(X)]i=〈Ai,X〉 with rank(Ai)=1, i=1,...,m. Historical iterative hard thresholding sequence for low-rank matrix recovery and rank-one measurements was taken as Xn+1=Ps(Xn−μnPt(A⁎sign(A(Xn)−y))), which introduced the “tail” and “head” approximations Ps and Pt, respectively. In this paper, we remove the term Pt and provide a new iterative hard thresholding algorithm with adaptive step size (abbreviated as AIHT). The linear convergence analysis and stability results on AIHT are established under the ℓ1/ℓ2-RIP. Particularly, we discuss the rank-one Gaussian measurements under the tight upper and lower bounds on E‖A(X)‖1, and provide better convergence rate and sampling complexity. Besides, several empirical experiments are provided to show that AIHT performs better than the historical rank-one iterative hard thresholding method.

Local and Global Convergence of General Burer-Monteiro Tensor Optimizations

Tensor optimization is crucial to massive machine learning and signal processing tasks. In this paper, we consider tensor optimization with a convex and well-conditioned objective function and reformulate it into a nonconvex optimization using the Burer-Monteiro type parameterization. We analyze the local convergence of applying vanilla gradient descent to the factored formulation and establish a local regularity condition under mild assumptions. We also provide a linear convergence analysis of the gradient descent algorithm started in a neighborhood of the true tensor factors. Complementary to the local analysis, this work also characterizes the global geometry of the best rank-one tensor approximation problem and demonstrates that for orthogonally decomposable tensors the problem has no spurious local minima and all saddle points are strict except for the one at zero which is a third-order saddle point.

A unified linear convergence analysis of k-SVD

Eigenvector computation, e.g., k-SVD for finding top-k singular subspaces, is often of central importance to many scientific and engineering tasks. There has been resurgent interest recently in analyzing relevant methods in terms of singular value gap dependence. Particularly, when the gap vanishes, the convergence of k-SVD is considered to be capped by a gap-free sub-linear rate. We argue in this work both theoretically and empirically that this is not necessarily the case, refreshing our understanding on this significant problem. Specifically, we leverage the recently proposed structured gap in a careful analysis to establish a unified linear convergence of k-SVD to one of the ground-truth solutions, regardless of what target matrix and how large target rank k are given. Theoretical results are evaluated and verified by experiments on synthetic or real data.

Quadratic optimization with orthogonality constraint: explicit Łojasiewicz exponent and linear convergence of retraction-based line-search and stochastic variance-reduced gradient methods

The problem of optimizing a quadratic form over an orthogonality constraint (QP-OC for short) is one of the most fundamental matrix optimization problems and arises in many applications. In this paper, we characterize the growth behavior of the objective function around the critical points of the QP-OC problem and demonstrate how such characterization can be used to obtain strong convergence rate results for iterative methods that exploit the manifold structure of the orthogonality constraint (i.e., the Stiefel manifold) to find a critical point of the problem. Specifically, our primary contribution is to show that the Łojasiewicz exponent at any critical point of the QP-OC problem is 1 / 2. Such a result is significant, as it expands the currently very limited repertoire of optimization problems for which the Łojasiewicz exponent is explicitly known. Moreover, it allows us to show, in a unified manner and for the first time, that a large family of retraction-based line-search methods will converge linearly to a critical point of the QP-OC problem. Then, as our secondary contribution, we propose a stochastic variance-reduced gradient (SVRG) method called Stiefel-SVRG for solving the QP-OC problem and present a novel Łojasiewicz inequality-based linear convergence analysis of the method. An important feature of Stiefel-SVRG is that it allows for general retractions and does not require the computation of any vector transport on the Stiefel manifold. As such, it is computationally more advantageous than other recently-proposed SVRG-type algorithms for manifold optimization.

Convergence analysis for hierarchical longitudinal data

Convergence analysis is typically envisaged either from a macro or a micro perspective. However, empirical tests tend to ignore that the two levels are often “nested” in a hierarchy. Building on hierarchical growth curve modelling, we propose an approach to convergence analysis that allows contemporaneous inference on macro and micro-convergence. Compared to the classic linear convergence analysis, the suggested methodology provides a more flexible alternative to model heterogeneity and validate the results for possible Galton's fallacy. We illustrate the approach in two empirical examples, one considering convergence across European regions and countries and the other across Italian firms and regions. In the European case, we find that the evidence of convergence depends on the choice of cross-sectional sample. Evidence on convergence in Italy applies only to part of the temporal sample and, therefore, is not robust to Galton's fallacy. Our analysis returns more robust results on the convergence process and allows better inference for policy intervention. We can envisage that this approach will find increasing applications in the future, as disaggregated data becomes available and heterogeneity becomes an increasingly prominent feature in economic modelling.

Local linear convergence analysis of Primal–Dual splitting methods

In this paper, we study the local linear convergence properties of a versatile class of Primal–Dual splitting methods for minimizing composite non-smooth convex optimization problems. Under the assumption that the non-smooth components of the problem are partly smooth relative to smooth manifolds, we present a unified local convergence analysis framework for these methods. More precisely, in our framework, we first show that (i) the sequences generated by Primal–Dual splitting methods identify a pair of primal and dual smooth manifolds in a finite number of iterations, and then (ii) enter a local linear convergence regime, which is characterized based on the structure of the underlying active smooth manifolds. We also show how our results for Primal–Dual splitting can be specialized to cover existing ones on Forward–Backward splitting and Douglas–Rachford splitting/ADMM (alternating direction methods of multipliers). Moreover, based on these obtained local convergence analysis result, several practical acceleration techniques are discussed. To exemplify the usefulness of the obtained result, we consider several concrete numerical experiments arising from fields including signal/image processing, inverse problems and machine learning. The demonstration not only verifies the local linear convergence behaviour of Primal–Dual splitting methods, but also the insights on how to accelerate them in practice.

Linearly convergent away-step conditional gradient for non-strongly convex functions

We consider the problem of minimizing the sum of a linear function and a composition of a strongly convex function with a linear transformation over a compact polyhedral set. Jaggi and Lacoste-Julien (An affine invariant linear convergence analysis for Frank-Wolfe algorithms. NIPS 2013 Workshop on Greedy Algorithms, Frank-Wolfe and Friends, 2014) show that the conditional gradient method with away steps -- employed on the aforementioned problem without the additional linear term -- has a linear rate of convergence, depending on the so-called pyramidal width of the feasible set. We revisit this result and provide a variant of the algorithm and an analysis based on simple linear programming duality arguments, as well as corresponding error bounds. This new analysis (a) enables the incorporation of the additional linear term, and (b) depends on a new constant, that is explicitly expressed in terms of the problem's parameters and the geometry of the feasible set. This constant replaces the pyramidal width, which is difficult to evaluate.

Asynchronous Distributed ADMM for Large-Scale Optimization—Part II: Linear Convergence Analysis and Numerical Performance

The alternating direction method of multipliers (ADMM) has been recognized as a versatile approach for solving modern large-scale machine learning and signal processing problems efficiently. When the data size and/or the problem dimension is large, a distributed version of ADMM can be used, which is capable of distributing the computation load and the data set to a network of computing nodes. Unfortunately, a direct synchronous implementation of such algorithm does not scale well with the problem size, as the algorithm speed is limited by the slowest computing nodes. To address this issue, in a companion paper, we have proposed an asynchronous distributed ADMM (AD-ADMM) and studied its worst-case convergence conditions. In this paper, we further the study by characterizing the conditions under which the AD-ADMM achieves linear convergence. Our conditions as well as the resulting linear rates reveal the impact that various algorithm parameters, network delay and network size have on the algorithm performance. To demonstrate the superior time efficiency of the proposed AD-ADMM, we test the AD-ADMM on a high-performance computer cluster by solving a large-scale logistic regression problem.

Efficient implementation of Radau collocation methods

In this paper we define an efficient implementation of Runge–Kutta methods of Radau IIA type, which are commonly used when solving stiff ODE-IVPs problems. The proposed implementation relies on an alternative low-rank formulation of the methods, for which a splitting procedure is easily defined. The linear convergence analysis of this splitting procedure exhibits excellent properties, which are confirmed by its performance on a few numerical tests.

Efficient implementation of Gauss collocation and Hamiltonian boundary value methods

In this paper we define an efficient implementation for the family of low-rank energy-conserving Runge-Kutta methods named Hamiltonian Boundary Value Methods (HBVMs), recently defined in the last years. The proposed implementation relies on the particular structure of the Butcher matrix defining such methods, for which we can derive an efficient splitting procedure. The very same procedure turns out to be automatically suited for the efficient implementation of Gauss-Legendre collocation methods, since these methods are a special instance of HBVMs. The linear convergence analysis of the splitting procedure exhibits excellent properties, which are confirmed by a few numerical tests.

GENERAL FRAMEWORK FOR PROXIMAL POINT ALGORITHMS ON (A, η)-MAXIMAL MONOTONICIT FOR NONLINEAR VARIATIONAL INCLUSIONS

General framework for proximal point algorithms based on the notion of (A, <TEX>${\eta}$</TEX>)-maximal monotonicity (also referred to as (A, <TEX>${\eta}$</TEX>)-monotonicity in literature) is developed. Linear convergence analysis for this class of algorithms to the context of solving a general class of nonlinear variational inclusion problems is successfully achieved along with some results on the generalized resolvent corresponding to (A, <TEX>${\eta}$</TEX>)-monotonicity. The obtained results generalize and unify a wide range of investigations readily available in literature.

Linear Convergence Analysis for General Proximal Point Algorithms Involving (H,η)- Mo notonicity Frameworks

Se desarrolla un marco general para el algoritmo de punto proximal generalizado, basado en la nocion de (H,r)- monotonia. Se examina el analisis de convergencia lineal para el algoritmo de punto proximal generalizado en el contexto de la resolucion de una clase de inclusiones no lineales variacional. Los resultados obtenidos generalizan y unifican una amplia gama de problemas en el contexto de lograr la convergencia lineal de los algoritmos punto proximal.

Linear convergence analysis of the use of gradient projection methods on total variation problems

Optimization problems using total variation frequently appear in image analysis models, in which the sharp edges of images are preserved. Direct gradient descent methods usually yield very slow convergence when used for such optimization problems. Recently, many duality-based gradient projection methods have been proposed to accelerate the speed of convergence. In this dual formulation, the cost function of the optimization problem is singular, and the constraint set is not a polyhedral set. In this paper, we establish two inequalities related to projected gradients and show that, under some non-degeneracy conditions, the rate of convergence is linear.

Approximation accuracy, gradient methods, and error bound for structured convex optimization

Convex optimization problems arising in applications, possibly as approximations of intractable problems, are often structured and large scale. When the data are noisy, it is of interest to bound the solution error relative to the (unknown) solution of the original noiseless problem. Related to this is an error bound for the linear convergence analysis of first-order gradient methods for solving these problems. Example applications include compressed sensing, variable selection in regression, TV-regularized image denoising, and sensor network localization.

Super-Relaxed ()-Proximal Point Algorithms, Relaxed ()-Proximal Point Algorithms, Linear Convergence Analysis, and Nonlinear Variational Inclusions

We glance at recent advances to the general theory of maximal (set-valued) monotone mappings and their role demonstrated to examine the convex programming and closely related field of nonlinear variational inequalities. We focus mostly on applications of the super-relaxed ( )-proximal point algorithm to the context of solving a class of nonlinear variational inclusion problems, based on the notion of maximal ( )-monotonicity. Investigations highlighted in this communication are greatly influenced by the celebrated work of Rockafellar (1976), while others have played a significant part as well in generalizing the proximal point algorithm considered by Rockafellar (1976) to the case of the relaxed proximal point algorithm by Eckstein and Bertsekas (1992). Even for the linear convergence analysis for the overrelaxed (or super-relaxed) ( )-proximal point algorithm, the fundamental model for Rockafellar's case does the job. Furthermore, we attempt to explore possibilities of generalizing the Yosida regularization/approximation in light of maximal ( )-monotonicity, and then applying to first-order evolution equations/inclusions.

Recent advances in linear analysis of convergence for splittings for solving ODE problems

In the nineties, Van der Houwen et al. (see, e.g., [P.J. van der Houwen, B.P. Sommeijer, J.J. de Swart, Parallel predictor–corrector methods, J. Comput. Appl. Math. 66 (1996) 53–71; P.J. van der Houwen, J.J.B. de Swart, Triangularly implicit iteration methods for ODE-IVP solvers, SIAM J. Sci. Comput. 18 (1997) 41–55; P.J. van der Houwen, J.J.B. de Swart, Parallel linear system solvers for Runge–Kutta methods, Adv. Comput. Math. 7 (1–2) (1997) 157–181]) introduced a linear analysis of convergence for studying the properties of the iterative solution of the discrete problems generated by implicit methods for ODEs. This linear convergence analysis is here recalled and completed, in order to provide a useful quantitative tool for the analysis of splittings for solving such discrete problems. Indeed, this tool, in its complete form, has been actively used when developing the computational codes BiM and BiMD [L. Brugnano, C. Magherini, The BiM code for the numerical solution of ODEs, J. Comput. Appl. Math. 164–165 (2004) 145–158. Code available at: http://www.math.unifi.it/~brugnano/BiM/index.html; L. Brugnano, C. Magherini, F. Mugnai, Blended implicit methods for the numerical solution of DAE problems, J. Comput. Appl. Math. 189 (2006) 34–50]. Moreover, the framework is extended for the case of special second order problems. Examples of application, aimed to compare different iterative procedures, are also presented.

Convergence of Numerical Method for Multistate Stochastic Dynamic Programming

Convergence of corrections is examined for a predictor-corrector method to solve Bellman equations of multi-state stochastic optimal control in continuous time. Quadratic costs and constrained control are assumed. A heuristically linearized comparison equation makes the nonlinear, discontinuous Bellman equation amenable to linear convergence analysis. Convergence is studied using the Fourier stability method. A uniform mesh ratio type condition for the convergence is results. The results are valid for both Gaussian and Poisson type stochastic noise. The convergence criteria has been extremely useful for solving the larger multi-state problems on vector supercomputers and massively parallel processors