Prove Convergence Rates Research Articles

Convergence Rates of Zeroth Order Gradient Descent for Łojasiewicz Functions

We prove convergence rates of Zeroth-order Gradient Descent (ZGD) algorithms for Łojasiewicz functions. Our results show that for smooth Łojasiewicz functions with Łojasiewicz exponent larger than 0.5 and smaller than 1, the functions values can converge much faster than the (zeroth-order) gradient descent trajectory. Similar results hold for convex nonsmooth Łojasiewicz functions. History: Accepted by Antonio Frangioni, Area Editor for Design & Analysis of Algorithms–Continuous. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.0247 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2023.0247 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Analysis of the discrepancy principle for Tikhonov regularization with oversmoothing penalty term under low order source conditions

We study the application of Tikhonov regularization to ill-posed nonlinear operator equations. The objective of this work is to prove convergence rates for the discrepancy principle under low order source conditions of logarithmic type. We work within the framework of Hilbert scales and extend existing studies on this subject to the oversmoothing case. The latter means that the exact solution of the treated operator equation does not belong to the domain of definition of the penalty term. As a consequence, the Tikhonov functional fails to have a finite value.

Unique continuation for the Lamé system using stabilized finite element methods

We introduce an arbitrary order, stabilized finite element method for solving a unique continuation problem subject to the time-harmonic elastic wave equation with variable coefficients. Based on conditional stability estimates we prove convergence rates for the proposed method which take into account the noise level and the polynomial degree. A series of numerical experiments corroborates our theoretical results and explores additional aspects, e.g. how the quality of the reconstruction depends on the geometry of the involved domains. We find that certain convexity properties are crucial to obtain a good recovery of the wave displacement outside the data domain and that higher polynomial orders can be more efficient but also more sensitive to the ill-conditioned nature of the problem.

Fourier integrator for periodic NLS: low regularity estimates via discrete Bourgain spaces

In this paper, we propose a new scheme for the integration of the periodic nonlinear Schrödinger equation and rigorously prove convergence rates at low regularity. The new integrator has decisive advantages over standard schemes at low regularity. In particular, it is able to handle initial data in H^s for 0 < s\le 1 . The key feature of the integrator is its ability to distinguish between low and medium frequencies in the solution and to treat them differently in the discretization. This new approach requires a well-balanced filtering procedure which is carried out in Fourier space. The convergence analysis of the proposed scheme is based on discrete (in time) Bourgain space estimates which we introduce in this paper. A numerical experiment illustrates the superiority of the new integrator over standard schemes for rough initial data.

Spectral convergence of probability densities for forward problems in uncertainty quantification

The estimation of probability density functions (PDF) using approximate maps is a fundamental building block in computational probability. We consider forward problems in uncertainty quantification: the inputs or the parameters of a deterministic model are random with a known distribution. The scalar quantity of interest is a fixed measurable function of the parameters, and is therefore a random variable as well. Often, the quantity of interest map is not explicitly known and difficult to compute. Hence, the computational problem is to design a good approximation (surrogate model) of the quantity of interest. For the goal of approximating the moments of the quantity of interest, there is a well developed body of research. One widely popular approach is generalized polynomial chaos (gPC) and its many variants, which approximate moments with spectral accuracy. However, it is not clear whether the PDF of the quantity of interest can be approximated with spectral accuracy as well. This result does not follow directly from spectrally accurate moment estimation. In this paper, we prove convergence rates for PDFs using collocation and Galerkin gPC methods with Legendre polynomials in all dimensions. In particular, exponential convergence of the densities is guaranteed for analytic quantities of interest. In one dimension, we provide more refined results with stronger convergence rates, as well as an alternative proof strategy based on optimal-transport techniques.

Convergence Rates of Monotone Schemes for Conservation Laws for Data with Unbounded Total Variation

We prove convergence rates of monotone schemes for conservation laws for Hölder continuous initial data with unbounded total variation, provided that the Hölder exponent of the initial data is greater than nicefrac {1}{2}. For strictly {{,mathrm{Lip},}}^+ stable monotone schemes, we prove convergence for any positive Hölder exponent. Numerical experiments are presented which verify the theory.

A Randomized Distributed Kaczmarz Algorithm and Anomaly Detection

The Kaczmarz algorithm is an iterative method for solving systems of linear equations. We introduce a randomized Kaczmarz algorithm for solving systems of linear equations in a distributed environment, i.e., the equations within the system are distributed over multiple nodes within a network. The modification we introduce is designed for a network with a tree structure that allows for passage of solution estimates between the nodes in the network. We demonstrate that the algorithm converges to the solution, or the solution of minimal norm, when the system is consistent. We also prove convergence rates of the randomized algorithm that depend on the spectral data of the coefficient matrix and the random control probability distribution. In addition, we demonstrate that the randomized algorithm can be used to identify anomalies in the system of equations when the measurements are perturbed by large, sparse noise.

On rational Krylov and reduced basis methods for fractional diffusion

Abstract We establish an equivalence between two classes of methods for solving fractional diffusion problems, namely, Reduced Basis Methods (RBM) and Rational Krylov Methods (RKM). In particular, we demonstrate that several recently proposed RBMs for fractional diffusion can be interpreted as RKMs. This changed point of view allows us to give convergence proofs for some methods where none were previously available. We also propose a new RKM for fractional diffusion problems with poles chosen using the best rational approximation of the function z −s with z ranging over the spectral interval of the spatial discretization matrix. We prove convergence rates for this method and demonstrate numerically that it is competitive with or superior to many methods from the reduced basis, rational Krylov, and direct rational approximation classes. We provide numerical tests for some elliptic fractional diffusion model problems.

Error estimates at low regularity of splitting schemes for NLS

We study a filtered Lie splitting scheme for the cubic nonlinear Schrödinger equation. We establish error estimates at low regularity by using discrete Bourgain spaces. This allows us to handle data in H s H^s with 0 > s > 1 0>s>1 overcoming the standard stability restriction to smooth Sobolev spaces with index s > 1 / 2 s>1/2 . More precisely, we prove convergence rates of order τ s / 2 \tau ^{s/2} in L 2 L^2 at this level of regularity.

Sparse-grid polynomial interpolation approximation and integration for parametric and stochastic elliptic PDEs with lognormal inputs

By combining a certain approximation property in the spatial domain, and weighted 𝓁2-summability of the Hermite polynomial expansion coefficients in the parametric domain obtained in Bachmayr et al. [ESAIM: M2AN 51 (2017) 341–363] and Bachmayr et al. [SIAM J. Numer. Anal. 55 (2017) 2151–2186], we investigate linear non-adaptive methods of fully discrete polynomial interpolation approximation as well as fully discrete weighted quadrature methods of integration for parametric and stochastic elliptic PDEs with lognormal inputs. We construct such methods and prove convergence rates of the approximations by them. The linear non-adaptive methods of fully discrete polynomial interpolation approximation are sparse-grid collocation methods which are certain sums taken over finite nested Smolyak-type indices sets of mixed tensor products of dyadic scale successive differences of spatial approximations of particular solvers, and of successive differences of their parametric Lagrange interpolating polynomials. The Smolyak-type sparse interpolation grids in the parametric domain are constructed from the roots of Hermite polynomials or their improved modifications. Moreover, they generate in a natural way fully discrete weighted quadrature formulas for integration of the solution to parametric and stochastic elliptic PDEs and its linear functionals, and the error of the corresponding integration can be estimated via the error in the Bochner space L1 (ℝ∞, V, γ) norm of the generating methods where γ is the Gaussian probability measure on ℝ∞ and V is the energy space. We also briefly consider similar problems for parametric and stochastic elliptic PDEs with affine inputs, and problems of non-fully discrete polynomial interpolation approximation and integration. In particular, the convergence rates of non-fully discrete polynomial interpolation approximation and integration obtained in this paper significantly improve the known ones.

A fast multilevel iteration method for solving linear ill-posed integral equations

Abstract We propose a new concept of noise level: R ⁢ ( K * ) \mathcal{R}(K^{*}) -noise level for ill-posed linear integral equations in Tikhonov regularization, which extends the range of regularization parameter. This noise level allows us to choose a more suitable regularization parameter. Moreover, we also analyze error estimates of the approximate solution with respect to this noise level. For ill-posed integral equations, finding fast and effective numerical methods is a challenging problem. For this, we formulate a matrix truncated strategy based on multiscale Galerkin method to generate the linear system of Tikhonov regularization for ill-posed linear integral equations, which greatly reduce the computational complexity. To further reduce the computational cost, a fast multilevel iteration method for solving the linear system is established. At the same time, we also prove convergence rates of the approximate solution obtained by this fast method with respect to the R ⁢ ( K * ) \mathcal{R}(K^{*}) -noise level under the balance principle. By numerical results, we show that R ⁢ ( K * ) \mathcal{R}(K^{*}) -noise level is very useful and the proposed method is a fast and effective method, respectively.

Adaptive estimation for some nonparametric instrumental variable models with full independence

The problem of endogeneity in statistics and econometrics is often handled by introducing instrumental variables (IV) which fulfill the mean independence assumption, i.e. the unobservable is mean independent of the instruments. When full independence of IV’s and the unobservable is assumed, nonparametric IV regression models and nonparametric demand models lead to nonlinear integral equations with unknown integral kernels. We prove convergence rates for the mean integrated square error of the iteratively regularized Newton method applied to these problems. Compared to related results we derive stronger convergence results that rely on weaker nonlinearity restrictions. We demonstrate in numerical simulations for a nonparametric IV regression that the method produces better results than the standard model.

Compactness and convergence rates in the combinatorial integral approximation decomposition

The combinatorial integral approximation decomposition splits the optimization of a discrete-valued control into two steps: solving a continuous relaxation of the discrete control problem, and computing a discrete-valued approximation of the relaxed control. Different algorithms exist for the second step to construct piecewise constant discrete-valued approximants that are defined on given decompositions of the domain. It is known that the resulting discrete controls can be constructed such that they converge to a relaxed control in the hbox {weak}^* topology of L^infty if the grid constant of this decomposition is driven to zero. We exploit this insight to formulate a general approximation result for optimization problems, which feature discrete and distributed optimization variables, and which are governed by a compact control-to-state operator. We analyze the topology induced by the grid refinements and prove convergence rates of the control vectors for two problem classes. We use a reconstruction problem from signal processing to demonstrate both the applicability of the method outside the scope of differential equations, the predominant case in the literature, and the effectiveness of the approach.

Chemotaxis on networks: Analysis and numerical approximation

We consider the Keller–Segel model of chemotaxis on one-dimensional networks. Using a variational characterization of solutions, positivity preservation, conservation of mass, and energy estimates, we establish global existence of weak solutions and uniform bounds. This extends related results of Osaki and Yagi to the network context. We then analyze the discretization of the system by finite elements and an implicit time-stepping scheme. Mass lumping and upwinding are used to guarantee the positivity of the solutions on the discrete level. This allows us to deduce uniform bounds for the numerical approximations and to establish order optimal convergence of the discrete approximations to the continuous solution without artificial smoothness requirements. In addition, we prove convergence rates under reasonable assumptions. Some numerical tests are presented to illustrate the theoretical results.

An application of sparse measure valued Bayesian inversion to acoustic sound source identification

In this work we discuss the problem of identifying sound sources from pressure measurements with a Bayesian approach. The acoustics are modelled by the Helmholtz equation and the goal is to get information about the number, strength and position of the sound sources, under the assumption that measurements of the acoustic pressure are noisy. We propose a problem specific prior distribution of the number, the amplitudes and positions of the sound sources and algorithms to compute an approximation of the associated posterior. We also discuss a finite element discretization of the Helmholtz equation for the practical computation and prove convergence rates of the resulting discretized posterior to the true posterior. The theoretical results are illustrated by numerical experiments, which indicate that the proven rates are sharp.

Optimal control of elliptic surface PDEs with pointwise bounds on the state

Abstract We consider a linear–quadratic optimal control problem for elliptic surface partial differential equations (PDEs) with additional state constraints. We approximate the optimization problem by a family of discrete problems and prove convergence rates for the discrete controls and the discrete states. With this we extend results known in the Euclidean setting to the surface case. We present numerical examples confirming our theoretical findings, with measures concentrated in points and measures concentrated on a line.

Relaxing Kink Qualifications and Proving Convergence Rates in Piecewise Smooth Optimization

In the paper [Optim. Methods Softw., 31 (2016), pp. 904--930] we derived first order (KKT) and second order (second order sufficiency condition (SOSC)) optimality conditions for functions defined by evaluation programs involving smooth elementals and absolute values. In that analysis, a key assumption on the local piecewise linearization was the linear independence kink qualification (LIKQ), a generalization of the linear independence constraint qualification (LICQ) known from smooth nonlinear optimization. Firstly, we show here that under LIKQ and SOSC with strict complementarity, the natural algorithm of successive abs-linear minimization with a proximal term (SALMIN) achieves a linear rate of convergence. A version of SALMIN called LiPsMin has already been implemented and tested in [Math. Program., 158 (2016), pp. 383--415; Math. Program. (2018), https://doi.org/10.1007/s10107-018-1273-5]. Secondly, we observe that, even without any kink qualifications, local optimality of the nonlinear objective always requires local optimality of its piecewise linearization, and strict minimality of the latter is in fact equivalent to sharp minimality of the former. Moreover, we show that SALMIN will converge quadratically to such sharp minimizers, where the function exhibits linear growth. These results are independent of the particular function representation, and allow in particular duplications of switching variables and other intermediates. Furthermore, we derive for nonsharp minimizers necessary and sufficient optimality conditions without any kink qualification. Finally, we present numerical results to illustrate the obtained convergence results.

On the regularizing property of stochastic gradient descent

Stochastic gradient descent (SGD) and its variants are among the most successful approaches for solving large-scale optimization problems. At each iteration, SGD employs an unbiased estimator of the full gradient computed from one single randomly selected data point. Hence, it scales well with problem size and is very attractive for handling truly massive dataset, and holds significant potentials for solving large-scale inverse problems. In this work, we rigorously establish its regularizing property under a priori early stopping rule for linear inverse problems, and also prove convergence rates under the canonical sourcewise condition. This is achieved by combining tools from classical regularization theory and stochastic analysis. Further, we analyze its preasymptotic weak and strong convergence behavior, in order to explain the fast initial convergence typically observed in practice. The theoretical findings shed insights into the performance of the algorithm, and are complemented with illustrative numerical experiments.

The quasi-optimality criterion in the linear functional strategy

The linear functional strategy for the regularization of inverse problems is considered. For selecting the regularization parameter therein, we propose the heuristic quasi-optimality principle and some modifications including the smoothness of the linear functionals. We prove convergence rates for the linear functional strategy with these heuristic rules taking into account the smoothness of the solution and the functionals and imposing a structural condition on the noise. Furthermore, we study these noise conditions in both a deterministic and stochastic setup and verify that for mildly-ill-posed problems and Gaussian noise, these conditions are satisfied almost surely, where on the contrary, in the severely-ill-posed case and in a similar setup, the corresponding noise condition fails to hold. Moreover, we propose an aggregation method for adaptively optimizing the parameter choice rule by making use of improved rates for linear functionals. Numerical results indicate that this method yields better results than the standard heuristic rule.

Existence of variational source conditions for nonlinear inverse problems in Banach spaces

Abstract Variational source conditions proved to be useful for deriving convergence rates for Tikhonov’s regularization method and also for other methods. Up to now, such conditions have been verified only for few examples or for situations which can be also handled by classical range-type source conditions. Here we show that for almost every ill-posed inverse problem variational source conditions are satisfied. Whether linear or nonlinear, whether Hilbert or Banach spaces, whether one or multiple solutions, variational source conditions are a universal tool for proving convergence rates.