Reduced Rank Extrapolation Research Articles

Convergence acceleration of iterative sequences for equilibrium chemistry computations

The modeling of thermodynamic equilibria leads to complex nonlinear chemical systems which are often solved with the Newton-Raphson method. But this resolution can lead to a non convergence or an excessive number of iterations due to the very ill-conditioned nature of the problem. In this work, we combine a particular formulation of the equilibrium system called the Positive Continuous Fraction method with two iterative methods, Anderson Acceleration method and Vector extrapolation methods (namely the reduced rank extrapolation and the minimal polynomial extrapolation). The main advantage of this approach is to avoid forming the Jacobian matrix. In addition, a strategy is used to improve the robustness of the Anderson acceleration method which consists in reducing the condition number of matrix of the least squares problem in the implementation of the Anderson acceleration so that the numerical stability can be guaranteed. We compare our numerical results with those obtained with the Newton-Raphson method on the Acid Gallic test and the 1D MoMas benchmark test case and we show the high efficiency of our approach.

Structural sensitivity reanalysis formulations based on the polynomial-type extrapolation methods

This paper presents new structural sensitivity reanalysis formulations based on the polynomial-type extrapolation methods. In these formulations, the displacement vector of the modified structure is expressed in the form of the vector sequences based on the fixed-point iteration method. By using these vector sequences, the minimal polynomial extrapolation (MPE) and the reduced rank extrapolation (RRE) methods calculate the approximate displacement vector of the modified structure by solving reduced linear least-square problems. Based on the definitions of the MPE and RRE methods, two sensitivity reanalysis formulations are derived, in which the first- and second-order sensitivities of the modified structure are obtained by solving a set of the over-determined least-square problems with much smaller size than the complete set of equations of the exact sensitivity analyses. The performance of the proposed sensitivity reanalysis formulations is evaluated by using four structural sensitivity reanalysis problems under multiple modifications in their initial designs. The results obtained from the numerical test problems indicate that the proposed sensitivity reanalysis formulations approximate the first- and second-order sensitivities of the modified structure with a high level of accuracy and they are able to converge to the exact solutions.

A convergence study for reduced rank extrapolation on nonlinear systems

Reduced Rank Extrapolation (RRE) is a polynomial type method used to accelerate the convergence of sequences of vectors {xm}. It is applied successfully in different disciplines of science and engineering in the solution of large and sparse systems of linear and nonlinear equations of very large dimension. If s is the solution to the system of equations x = f(x), first, a vector sequence {xm} is generated via the fixed-point iterative scheme xm+ 1 = f(xm), m = 0,1,…, and next, RRE is applied to this sequence to accelerate its convergence. RRE produces approximations sn, k to s that are of the form $\boldsymbol {s}_{n,k}={\sum }_{i=0}^{k} \gamma _{i} \boldsymbol {x}_{n+i}$ for some scalars γi depending (nonlinearly) on xn, xn+ 1,…,xn+k+ 1 and satisfying ${\sum }_{i=0}^{k} \gamma _{i}=1$ . The convergence properties of RRE when applied in conjunction with linear f(x) have been analyzed in different publications. In this work, we discuss the convergence of the sn, k obtained from RRE with nonlinear f(x) (i) when $n\to \infty $ with fixed k, and (ii) in two so-called cycling modes.

Simple efficient solvers for certain ill-conditioned systems of linear equations, including H(div) problems

Simple, accurate, and efficient iterative methods for the solution of a nearly singular variational problem are described. The systems considered arise, e.g., when seeking to determine the flux of second order elliptic partial differential equations. Each (outer) iteration uses a robust and efficient solver, which may be a direct or iterative method, and only a few (outer) iterations are required to obtain an approximate solution. Reduced rank extrapolation may be applied to speed up the convergence.

A new structural reanalysis approach based on the polynomial-type extrapolation methods

In this study, a new approach based on the polynomial-type extrapolation methods is applied to the approximate structural reanalysis under multiple and large changes in the initial design. In this approach, the sequences of approximate displacement of the modified structure are constructed by using a fixed-point iteration method. These sequences are then further analyzed by two polynomial-type vector extrapolation methods to find the approximate response of the modified structure more accurately, namely, the minimal polynomial extrapolation (MPE) and the reduced rank extrapolation (RRE). Based on a single initial design, the MPE and RRE methods approximate the displacement vector of the modified structure by solving a least-squares problem which is much smaller than the original system of equations of the exact analysis. The accuracy and efficiency of this reanalysis approach is evaluated on three large scale structural reanalysis problems under multiple and large changes in their initial designs. The obtained reanalysis results demonstrate that the MPE and RRE methods not only yield accurate results, but also are computationally efficient.

Minimal polynomial and reduced rank extrapolation methods are related

Minimal Polynomial Extrapolation (MPE) and Reduced Rank Extrapolation (RRE) are two polynomial methods used for accelerating the convergence of sequences of vectors {xm}. They are applied successfully in conjunction with fixed-point iterative schemes in the solution of large and sparse systems of linear and nonlinear equations in different disciplines of science and engineering. Both methods produce approximations sk to the limit or antilimit of {xm} that are of the form sk=źi=0kźixi$\boldsymbol {s}_{k}={\sum }^{k}_{i=0}\gamma _{i}\boldsymbol {x}_{i}$ with źi=0kźi=1${\sum }^{k}_{i=0}\gamma _{i}=1$, for some scalars źi. The way the two methods are derived suggests that they might, somehow, be related to each other; this has not been explored so far, however. In this work, we tackle this issue and show that the vectors skMPE$\boldsymbol {s}_{k}^{\textit {{\tiny {MPE}}}}$ and skRRE$\boldsymbol {s}_{k}^{\textit {{\tiny {RRE}}}}$ produced by the two methods are related in more than one way, and independently of the way the xm are generated. One of our results states that RRE stagnates, in the sense that skRRE=skź1RRE$\boldsymbol {s}_{k}^{\textit {{\tiny {RRE}}}}=\boldsymbol {s}_{k-1}^{\textit {{\tiny {RRE}}}}$, if and only if skMPE$\boldsymbol {s}_{k}^{\textit {{\tiny {MPE}}}}$ does not exist. Another result states that, when skMPE$\boldsymbol {s}_{k}^{\textit {{\tiny {MPE}}}}$ exists, there holds μkskRRE=μkź1skź1RRE+źkskMPEwithμk=μkź1+źk,$$\mu_{k}\boldsymbol{s}_{k}^{\textit{{\tiny{RRE}}}}=\mu_{k-1}\boldsymbol{s}_{k-1}^{\textit{{\tiny{RRE}}}}+ \nu_{k}\boldsymbol{s}_{k}^{\textit{{\tiny{MPE}}}}\quad \text{with}\quad \mu_{k}=\mu_{k-1}+\nu_{k}, $$for some positive scalars μk, μkź1, and źk that depend only on skRRE$\boldsymbol {s}_{k}^{\textit {{\tiny {RRE}}}}$, skź1RRE$\boldsymbol {s}_{k-1}^{\textit {{\tiny {RRE}}}}$, and skMPE$\boldsymbol {s}_{k}^{\textit {{\tiny {MPE}}}}$, respectively. Our results are valid when MPE and RRE are defined in any weighted inner product and the norm induced by it. They also contain as special cases the known results pertaining to the connection between the method of Arnoldi and the method of generalized minimal residuals, two important Krylov subspace methods for solving nonsingular linear systems.

SVD-MPE: An SVD-Based Vector Extrapolation Method of Polynomial Type

An important problem that arises in different areas of science and engineering is that of computing the limits of sequences of vectors , where , N being very large. Such sequences arise, for example, in the solution of systems of linear or nonlinear equations by fixed-point iterative methods, and are simply the required solutions. In most cases of interest, however, these sequences converge to their limits extremely slowly. One practical way to make the sequences converge more quickly is to apply to them vector extrapolation methods. Two types of methods exist in the literature: polynomial type methods and epsilon algorithms. In most applications, the polynomial type methods have proved to be superior convergence accelerators. Three polynomial type methods are known, and these are the minimal polynomial extrapolation (MPE), the reduced rank extrapolation (RRE), and the modified minimal polynomial extrapolation (MMPE). In this work, we develop yet another polynomial type method, which is based on the singular value decomposition, as well as the ideas that lead to MPE. We denote this new method by SVD-MPE. We also design a numerically stable algorithm for its implementation, whose computational cost and storage requirements are minimal. Finally, we illustrate the use of SVD-MPE with numerical examples.

Acceleration of Dominant Transversal Laser Resonator Eigenmode Calculation by Vector Extrapolation Methods

Field tracing provides the flexible and fully vectorial calculation of the dominant transversal resonator mode. Therefore, the eigenvalue problem formulated in a set of coupled, nonlinear operator equations has to be solved. Up to now, only the iterative power method, also known as Fox and Li algorithm, was used for the eigenvector calculation, where serious convergence problems can occur. We accelerate the convergence speed by applying two polynomial-type vector extrapolation methods, namely, the minimal polynomial extrapolation and the reduced rank extrapolation. Numerical examples show that the convergence speed and, consequently, the computational time required for the calculation of the fully vectorial transversal resonator eigenmode, could be reduced by up to 70% compared with the Fox and Li algorithm. Furthermore, it is shown that the calculated eigenmode is in good agreement with the experimental results.

Fast solvers for unsteady thermal fluid structure interaction

SummaryWe consider time‐dependent thermal fluid structure interaction. The respective models are the compressible Navier–Stokes equations and the nonlinear heat equation. A partitioned coupling approach via a Dirichlet–Neumann method and a fixed point iteration is employed. As a reference solver, a previously developed efficient time‐adaptive higher‐order time integration scheme is used.To improve on this, we work on reducing the number of fixed point coupling iterations. Using the idea of extrapolation based on data given from the time integration by deriving such methods for SDIRK2, it is possible to reduce the number of fixed point iterations further by up to a factor of two with linear extrapolation performing better than quadratic. This leads to schemes that can use less than two iterations per time step.Furthermore, widely used vector extrapolation methods for convergence acceleration of the fixed point iteration are tested, namely Aitken relaxation, minimal polynomial extrapolation and reduced rank extrapolation. These have no beneficial effects. Copyright © 2015 John Wiley & Sons, Ltd.

Nonlinear Schwarz iterations with reduced rank extrapolation

Extrapolation methods can be a very effective technique used for accelerating the convergence of vector sequences. In this paper, these methods are used to accelerate the convergence of Schwarz iterative methods for nonlinear problems. A new implementation of the reduced-rank-extrapolation (RRE) method is introduced. Some convergence analysis is presented, and it is shown numerically that certain extrapolation methods can indeed be very effective in accelerating the convergence of Schwarz methods.

Review of two vector extrapolation methods of polynomial type with applications to large-scale problems

AbstractAn important problem that arises in different areas of science and engineering is that of computing limits of sequences of vectors {xn}, where xn∈CN with N very large. Such sequences arise, for example, in the solution of systems of linear or nonlinear equations by fixed-point iterative methods, and limn→∞xn are simply the required solutions. In most cases of interest, these sequences converge to their limits extremely slowly, or even diverge. One practical way to make the sequences {xn} converge more quickly is to apply to them vector extrapolation methods. In this work, we review two polynomial-type vector extrapolation methods that have proved to be very efficient convergence accelerators; namely, the minimal polynomial extrapolation (MPE) and the reduced rank extrapolation (RRE). We discuss their derivation, describe the most accurate and stable algorithms for their implementation along with the effective modes of usage, and present their convergence and stability theory. We also discuss their close connection with the method of Arnoldi and GMRES, two well known Krylov subspace methods for linear systems. Finally, we discuss some of their applications to different large-scale problems, such as solution of large-scale systems of equations, eigenvalue problems, computation of the PageRank of the Google matrix, and summation of vector-valued power series.

On Semi-implicit Splitting Schemes for the Beltrami Color Image Filtering

The Beltrami flow is an efficient nonlinear filter, that was shown to be effective for color image processing. The corresponding anisotropic diffusion operator strongly couples the spectral components. Usually, this flow is implemented by explicit schemes, that are stable only for very small time steps and therefore require many iterations. In this paper we introduce a semi-implicit Crank-Nicolson scheme based on locally one-dimensional (LOD)/additive operator splitting (AOS) for implementing the anisotropic Beltrami operator. The mixed spatial derivatives are treated explicitly, while the non-mixed derivatives are approximated in an implicit manner. In case of constant coefficients, the LOD splitting scheme is proven to be unconditionally stable. Numerical experiments indicate that the proposed scheme is also stable in more general settings. Stability, accuracy, and efficiency of the splitting schemes are tested in applications such as the Beltrami-based scale-space, Beltrami denoising and Beltrami deblurring. In order to further accelerate the convergence of the numerical scheme, the reduced rank extrapolation (RRE) vector extrapolation technique is employed.

Accelerated non-linear finite volume method for diffusion

A non-linear finite volume method with monotone matrix for the diffusion equation is presented. It does not extrapolate the primary variable to Neumann boundaries, as this was previously done in similar methods. This change results in faster convergence. Computation time is significantly shortened further using the reduced rank extrapolation method (RRE), and imposing an upper limit on the number of linear iterations per non-linear step. Second-order accuracy and performance improvement are demonstrated by numerical examples.

A Three-Dimensional Mixed Finite-Element Approximation of the Semiconductor Energy-Transport Equations

The stationary energy-transport equations for semiconductors in three space dimensions are numerically discretized. The physical variables are the electron density, the energy density, and the electric potential. Physically motivated mixed Dirichlet–Neumann boundary conditions are employed. The numerical approximation is based on a hybridized mixed finite-element method with Raviart–Thomas–Nédélec elements, applied to the dual-entropy formulation of the energy-transport model. For the solution of the nonlinear discrete system, a Newton scheme with adaptive potential stepping and two decoupling Gummel-type strategies with reduced rank extrapolation are proposed. Multigate field-effect transistors in two dimensions and three dimensions are numerically simulated.

Vector extrapolation methods with applications to solution of large systems of equations and to PageRank computations

An important problem that arises in different areas of science and engineering is that of computing the limits of sequences of vectors { x n } , where x n ∈ C N with N very large. Such sequences arise, for example, in the solution of systems of linear or nonlinear equations by fixed-point iterative methods, and lim n → ∞ x n are simply the required solutions. In most cases of interest, however, these sequences converge to their limits extremely slowly. One practical way to make the sequences { x n } converge more quickly is to apply to them vector extrapolation methods. In this work, we review two polynomial-type vector extrapolation methods that have proved to be very efficient convergence accelerators; namely, the minimal polynomial extrapolation (MPE) and the reduced rank extrapolation (RRE). We discuss the derivation of these methods, describe the most accurate and stable algorithms for their implementation along with the effective modes of usage in solving systems of equations, nonlinear as well as linear, and present their convergence and stability theory. We also discuss their close connection with the method of Arnoldi and with GMRES, two well-known Krylov subspace methods for linear systems. We show that they can be used very effectively to obtain the dominant eigenvectors of large sparse matrices when the corresponding eigenvalues are known, and provide the relevant theory as well. One such problem is that of computing the PageRank of the Google matrix, which we discuss in detail. In addition, we show that a recent extrapolation method of Kamvar et al. that was proposed for computing the PageRank is very closely related to MPE. We present a generalization of the method of Kamvar et al. along with a very economical algorithm for this generalization. We also provide the missing convergence theory for it.

On the Vector ε-Algorithm for Solving Linear Systems of Equations

The four vector extrapolation methods, minimal polynomial extrapolation, reduced rank extrapolation, modified minimal polynomial extrapolation and the topological epsilon algorithm, when applied to linearly generated vector sequences are Krylov subspace methods and it is known that they are equivalent to some well-known conjugate gradient type methods. However, the vector e-algorithm is an extrapolation method, older than the four extrapolation methods above, and no similar results are known for it. In this paper, a determinantal formula for the vector e-algorithm is given. Then it is shown that, when applied to a linearly generated vector sequence, the algorithm is also a Krylov subspace method and for a class of matrices the method is equivalent to a preconditioned Lanczos method. A new determinantal formula for the CGS is given, and an algebraic comparison between the vector e-algorithm for linear systems and CGS is also given.

Vector extrapolation methods. Applications and numerical comparison

The present paper is a survey of the most popular vector extrapolation methods such as the reduced rank extrapolation (RRE), the minimal polynomial extrapolation (MPE), the modified minimal polynomial extrapolation (MMPE), the vector ε-algorithm (VEA) and the topological ε-algorithm (TEA). Using projectors, we derive a different interpretation of these methods and give some theoretical results. The second aim of this work is to give a numerical comparison of the vector extrapolation methods above when they are used for practical large problems such as linear and nonlinear systems of equations.

Convergence of intermediate rows of minimal polynomial and reduced rank extrapolation tables

Let {xm} m =0/∞ be a vector sequence obtained from a linear fixed point iterative technique in a general inner product space. In two previous papers [6,9] the convergence properties of the minimal polynomial and reduced rank extrapolation methods, as they are applied to the vector sequence above, were analyzed. In particular, asymptotically optimal convergence results pertaining to some of the rows of the tables associated with these two methods were obtained. In the present work we continue this analysis and provide analogous results for the remaining (intermediate) rows of these tables. In particular, when {xm} m =0/∞ is a convergent sequence, the main result of this paper says, roughly speaking, that all of the rows converge, and it also gives the rate of convergence for each row. The results are demonstrated numerically through an example.

Efficient implementation of minimal polynomial and reduced rank extrapolation methods

The minimal polynomial extrapolation (MPE) and reduced rank extrapolation (RRE) are two very effective techniques that have been used in accelerating the convergence of vector sequences, such as those that are obtained from iterative solution of linear and nonlinear systems of equations. Their definitions involve some linear least-squares problems, and this causes difficulties in their numerical implementation. In this work timewise efficient and numerically stable implementations for MPE and RRE are developed. A computer program written in FORTRAN 77 is also appended and applied to some model problems, among them a hypersonic flow problem involving chemical reactions.

Erratum: Correction to “Extrapolation Methods for Vector Sequences”

Previous article Next article Full AccessErratum: Correction to “Extrapolation Methods for Vector Sequences”David A. Smith, William F. Ford, and Avram SidiDavid A. Smith, William F. Ford, and Avram Sidihttps://doi.org/10.1137/1030136PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAbout"Erratum: Correction to “Extrapolation Methods for Vector Sequences”." SIAM Review, 30(4), pp. 623–624[1] Avram Sidi, , William F. Ford and , David A. Smith, Acceleration of convergence of vector sequences, SIAM J. Numer. Anal., 23 (1986), 178–196 10.1137/0723013 87i:65009a 0596.65016 LinkISIGoogle Scholar[2] David A. Smith, , William F. Ford and , Avram Sidi, Extrapolation methods for vector sequences, SIAM Rev., 29 (1987), 199–233 10.1137/1029042 89j:65013a 0622.65003 LinkISIGoogle Scholar[3] Roger C. E. Tan, Implementation of the topological $\epsilon$-algorithm, SIAM J. Sci. Statist. Comput., 9 (1988), 839–848 89i:65049 0654.65005 LinkISIGoogle ScholarKeywordstopological epsilon algorithmvector epsilon algorithm Previous article Next article FiguresRelatedReferencesCited byDetails Shanks and Anderson-type acceleration techniques for systems of nonlinear equations25 August 2021 | IMA Journal of Numerical Analysis, Vol. 42, No. 4 Cross Ref Commentaries and Further Developments1 December 2020 Cross Ref The Simplified Topological $\varepsilon$-Algorithms for Accelerating Sequences in a Vector SpaceClaude Brezinski and Michela Redivo-Zaglia23 September 2014 | SIAM Journal on Scientific Computing, Vol. 36, No. 5AbstractPDF (328 KB)Coupling strategies for biomedical fluid-structure interaction problemsInternational Journal for Numerical Methods in Biomedical Engineering, Vol. 26, No. 3-4 Cross Ref Vector Extrapolation for Strong Coupling Fluid-Structure Interaction Solvers15 January 2009 | Journal of Applied Mechanics, Vol. 76, No. 2 Cross Ref Vector extrapolation methods. Applications and numerical comparisonJournal of Computational and Applied Mathematics, Vol. 122, No. 1-2 Cross Ref Hybrid vector transformationsJournal of Computational and Applied Mathematics, Vol. 81, No. 2 Cross Ref Application of Vector-Valued Rational Approximations to the Matrix Eigenvalue Problem and Connections with Krylov Subspace MethodsAvram Sidi17 July 2006 | SIAM Journal on Matrix Analysis and Applications, Vol. 16, No. 4AbstractPDF (3138 KB)Convergence Acceleration for Iterative Finite-Element MethodsJournal of Engineering Mechanics, Vol. 121, No. 1 Cross Ref A review of Padé methods for the acceleration of convergence of a sequence of vectorsApplied Numerical Mathematics, Vol. 15, No. 2 Cross Ref Quasilinear vector extrapolation methodsLinear Algebra and its Applications, Vol. 190 Cross Ref Development of iterative techniques and extrapolation methods for Drazin inverse solution of consistent or inconsistent singular linear systemsLinear Algebra and its Applications, Vol. 167 Cross Ref Efficient implementation of minimal polynomial and reduced rank extrapolation methodsJournal of Computational and Applied Mathematics, Vol. 36, No. 3 Cross Ref Some results about vector extrapolation methods and related fixed-point iterationsJournal of Computational and Applied Mathematics, Vol. 36, No. 3 Cross Ref Sequences of transformations and triangular recursion schemes, with applications in numerical analysisJournal of Computational and Applied Mathematics, Vol. 34, No. 3 Cross Ref Bibliography Cross Ref Application of vector extrapolation methods to consistent singular linear systemsApplied Numerical Mathematics, Vol. 6, No. 6 Cross Ref Solving Linear Equations by Extrapolation Cross Ref Volume 30, Issue 4| 1988SIAM Review History Submitted:26 October 1987Accepted:02 May 1988Published online:18 July 2006 InformationCopyright © 1988 Society for Industrial and Applied MathematicsKeywordstopological epsilon algorithmvector epsilon algorithmMSC codes65-0265B0565F1065H10PDF Download Article & Publication DataArticle DOI:10.1137/1030136Article page range:pp. 623-624ISSN (print):0036-1445ISSN (online):1095-7200Publisher:Society for Industrial and Applied Mathematics