PFASST-ER: combining the parallel full approximation scheme in space and time with parallelization across the method

In this article, we present the Python framework pySDC for solving collocation problems with spectral deferred correction (SDC) methods and their time-parallel variant PFASST, the parallel full approximation scheme in space and time. pySDC features many implementations of SDC and PFASST, from simple implicit timestepping to high-order implicit-explicit or multi-implicit splitting and multilevel SDCs. The software package comes with many different, preimplemented examples and has seven tutorials to help new users with their first steps. Time parallelism is implemented either in an emulated way for debugging and prototyping or using MPI for benchmarking. The code is fully documented and tested using continuous integration, including most results of previous publications. Here, we describe the structure of the code by taking two different perspectives: those of the user and those of the developer. The first sheds light on the front-end, the examples, and the tutorials, and the second is used to describe the underlying implementation and the data structures. We show three different examples to highlight various aspects of the implementation, the capabilities, and the usage of pySDC. In addition, couplings to the FEniCS framework and PETSc, the latter including spatial parallelism with MPI, are described.

The parallel full approximation scheme in space and time (PFASST) is a parallel-in-time integrator that allows to integrate multiple time-steps simultaneously. It has been shown to extend scaling limits of spatial parallelization strategies when coupled with finite differences, spectral discretizations, or particle methods. In this paper we show how to use PFASST together with a finite element discretization in space. While seemingly straightforward, the appearance of the mass matrix and the need to restrict iterates as well as residuals in space makes this task slightly more intricate. We derive the PFASST algorithm with mass matrices and appropriate prolongation and restriction operators and show numerically that PFASST can, after some initial iterations, gain two orders of accuracy per iteration.

SummaryFor the numerical solution of time‐dependent partial differential equations, time‐parallel methods have recently been shown to provide a promising way to extend prevailing strong‐scaling limits of numerical codes. One of the most complex methods in this field is the “Parallel Full Approximation Scheme in Space and Time” (PFASST). PFASST already shows promising results for many use cases and benchmarks. However, a solid and reliable mathematical foundation is still missing. We show that, under certain assumptions, the PFASST algorithm can be conveniently and rigorously described as a multigrid‐in‐time method. Following this equivalence, first steps towards a comprehensive analysis of PFASST using blockwise local Fourier analysis are taken. The theoretical results are applied to examples of diffusive and advective type.

SummaryFor time‐dependent partial differential equations, parallel‐in‐time integration using the “parallel full approximation scheme in space and time” (PFASST) is a promising way to accelerate existing space‐parallel approaches beyond their scaling limits. Inspired by the classical Parareal method and multigrid ideas, PFASST allows to integrate multiple time steps simultaneously using a space–time hierarchy of spectral deferred correction sweeps. While many use cases and benchmarks exist, a solid and reliable mathematical foundation is still missing. Very recently, however, PFASST for linear problems has been identified as a multigrid method. In this paper, we will use this multigrid formulation and, in particular, PFASST's iteration matrix to show that, in the nonstiff and stiff limit, PFASST indeed is a convergent iterative method. We will provide upper bounds for the spectral radius of the iteration matrix and investigate how PFASST performs for increasing numbers of parallel time steps. Finally, we will demonstrate that the results obtained here indeed relate to actual PFASST runs.

Speeding up computationally expensive problems, such as numerical simulations of large micromagnetic systems, requires efficient use of parallel computing infrastructures. While parallelism across space is commonly exploited in micromagnetics, this strategy performs poorly once a minimum number of degrees of freedom per core is reached. We use magnum.pi, a finite-element micromagnetic simulation software, to investigate the Parallel Full Approximation Scheme in Space and Time (PFASST) as a space- and time-parallel solver for the Landau–Lifshitz–Gilbert equation (LLG). Numerical experiments show that PFASST enables efficient parallel-in-time integration of the LLG, significantly improving the speedup gained from using a given number of cores as well as allowing the code to scale beyond spatial limits.

We present a novel space-time parallel version of the Barnes-Hut tree code pepc using pfasst, the Parallel Full Approximation Scheme in Space and Time. The naive use of increasingly more processors for a fixed-size N-body problem is prone to saturate as soon as the number of unknowns per core becomes too small. To overcome this intrinsic strong-scaling limit, we introduce temporal parallelism on top of pepc's existing hybrid MPI/PThreads spatial decomposition. Here, we use pfasst which is based on a combination of the iterations of the parallel-in-time algorithm parareal with the sweeps of spectral deferred correction (SDC) schemes. By combining these sweeps with multiple space-time discretization levels, pfasst relaxes the theoretical bound on parallel efficiency in parareal. We present results from runs on up to 262,144 cores on the IBM Blue Gene/P installation JUGENE, demonstrating that the space-time parallel code provides speedup beyond the saturation of the purely space-parallel approach.

We present a novel space-time parallel version of the Barnes-Hut tree code PEPC using PFASST, the Parallel Full Approximation Scheme in Space and Time. The naive use of increasingly more processors for a fixed-size N-body problem is prone to saturate as soon as the number of unknowns per core becomes too small. To overcome this intrinsic strong-scaling limit, we introduce temporal parallelism on top of PEPC's existing hybrid MPI/PThreads spatial decomposition. Here, we use PFASST which is based on a combination of the iterations of the parallel-in-time algorithm parareal with the sweeps of spectral deferred correction (SDC) schemes. By combining these sweeps with multiple space-time discretization levels, PFASST relaxes the theoretical bound on parallel efficiency in parareal. We present results from runs on up to 262,144 cores on the IBM Blue Gene/P installation JUGENE, demonstrating that the spacetime parallel code provides speedup beyond the saturation of the purely space-parallel approach.

Time parallelization, also known as PinT (parallel-in-time), is a new research direction for the development of algorithms used for solving very large-scale evolution problems on highly parallel computing architectures. Despite the fact that interesting theoretical work on PinT appeared as early as 1964, it was not until 2004, when processor clock speeds reached their physical limit, that research in PinT took off. A distinctive characteristic of parallelization in time is that information flow only goes forward in time, meaning that time evolution processes seem necessarily to be sequential. Nevertheless, many algorithms have been developed for PinT computations over the past two decades, and they are often grouped into four basic classes according to how the techniques work and are used: shooting-type methods; waveform relaxation methods based on domain decomposition; multigrid methods in space–time; and direct time parallel methods. However, over the past few years, it has been recognized that highly successful PinT algorithms for parabolic problems struggle when applied to hyperbolic problems. We will therefore focus on this important aspect, first by providing a summary of the fundamental differences between parabolic and hyperbolic problems for time parallelization. We then group PinT algorithms into two basic groups. The first group contains four effective PinT techniques for hyperbolic problems: Schwarz waveform relaxation (SWR) with its relation to tent pitching; parallel integral deferred correction; ParaExp; and ParaDiag. While the methods in the first group also work well for parabolic problems, we then present PinT methods specifically designed for parabolic problems in the second group: Parareal; the parallel full approximation scheme in space–time (PFASST); multigrid reduction in time (MGRiT); and space–time multigrid (STMG). We complement our analysis with numerical illustrations using four time-dependent PDEs: the heat equation; the advection–diffusion equation; Burgers’ equation; and the second-order wave equation.

The parallel full approximation scheme in space and time (PFASST) introduced by Emmett and Minion in 2012 is an iterative strategy for the temporal parallelization of ODEs and discretized PDEs. As the name suggests, PFASST is similar in spirit to a space-time FAS multigrid method performed over multiple time-steps in parallel. However, since the original focus of PFASST has been on the performance of the method in terms of time parallelism, the solution of any spatial system arising from the use of implicit or semi-implicit temporal methods within PFASST have simply been assumed to be solved to some desired accuracy completely at each sub-step and each iteration by some unspecified procedure. It hence is natural to investigate how iterative solvers in the spatial dimensions can be interwoven with the PFASST iterations and whether this strategy leads to a more efficient overall approach. This paper presents an initial investigation on the relative performance of different strategies for coupling PFASST iterations with multigrid methods for the implicit treatment of diffusion terms in PDEs. In particular, we compare full accuracy multigrid solves at each sub-step with a small fixed number of multigrid V-cycles. This reduces the cost of each PFASST iteration at the possible expense of a corresponding increase in the number of PFASST iterations needed for convergence. Parallel efficiency of the resulting methods is explored through numerical examples.

While many ideas and proofs of concept for parallel-in-time integration methods exists, the number of large-scale, accessible time-parallel codes is rather small. This is often due to the apparent or subtle complexity of the algorithms and the many pitfalls awaiting developers of parallel numerical software. One example of such a time-parallel code is pySDC, which implements, among others, the parallel full approximation scheme in space and time (PFASST). Inspired by nonlinear multigrid ideas, PFASST allows to integrate multiple time steps simultaneously using a space-time hierarchy of spectral deferred corrections. In this paper, we demonstrate the application of performance analysis tools to the PFASST implementation pySDC. We trace the path we took for this work, show examples of how the tools can be applied, and explain the sometimes surprising findings we encountered. Although focusing only on a single implementation of a particular parallel-in-time integrator, we hope that our results and in particular the way we obtained them are a blueprint for other time-parallel codes.

Task graph-based performance analysis of parallel-in-time methods

A strategy for scheduling the communication between processors in a multi-level parallel-in-time algorithm to reduce blocking communication is presented. The particular time-parallel method examined is the parallel full approximation scheme in space and time (PFASST), which utilizes a hierarchy of spatial and temporal discretization levels. By decomposing the update to initial conditions passed between processors into multiple spatial resolutions, the communication at the finest level can be scheduled to overlap with computation at coarser levels. The potential cost savings is demonstrated with a three dimensional PDE example.

The paper presents a combination of the time-parallel “parallel full approximation scheme in space and time” (PFASST) with a parallel multigrid method (PMG) in space, resulting in a mesh-based solver for the three-dimensional heat equation with a uniquely high degree of efficient concurrency. Parallel scaling tests are reported on the Cray XE6 machine “Monte Rosa” on up to 16,384 cores and on the IBM Blue Gene/Q system “JUQUEEN” on up to 65,536 cores. The efficacy of the combined spatial-and temporal parallelization is shown by demonstrating that using PFASST in addition to PMG significantly extends the strong-scaling limit. Implications of using spatial coarsening strategies in PFASST's multi-level hierarchy in large-scale parallel simulations are discussed.

An experimental comparison of a space-time multigrid method with PFASST for a reaction-diffusion problem

not yet added. Parallel-in-time algorithms are more and more recognized as a promising solution to increase computational concurrency after the maximum speed-up has been achieved from spatial parallelism due to communication overhead when the problem size per processor has become too small in distributed memory high-performance computing architecture. Two popular time-parallel approaches are the Parareal algorithm, first introduced by Lions et al. (2001), and the Parallel Full Approximation Scheme in Space and Time (PFASST) technique of Emmett and Minion (2012). Both methods are based on the use of multi-level time-integration, exploiting a fine and a coarse time integrators in combination with an iterative procedure to achieve parallelism in time. The implementation of these two parallel-in-time algorithms in the Nektar++ spectral/hp element framework is described. The extension of the MPI topology to allow concurrency in time is first presented. Following, the implementation of new drivers for the Parareal and PFASST algorithms is described. Finally, application examples for the 1D advection and the 2D diffusion equations are demonstrated. The least-squares finite element method (LSFEM) emerged as an alternative to classical Galerkin FEM around 30 years ago. Since then, it has successfully found applications in many fields of computational physics, including fluid dynamics. The method possesses some desirable properties – it always leads to symmetric, positive-definite algebraic systems (even for non-self-adjoint operators), it does not necessitate the fulfillment of compatibility (LBB) conditions, and it is generally more stable than its classical counterpart. Conversely, it requires recasting equations to the first order (introducing additional degrees of freedom) and involves a more computationally expensive assembly procedure. This talk will attempt to evaluate whether the benefits of LSFEM outweigh its drawbacks and offer a practical perspective on how it can be employed to efficiently solve incompressible flow problems. We will show an alternative equation splitting scheme, which, in conjunction with LSFEM, leads to an enhanced convergence rate. Finally, we will discuss computational aspects of the method using the author’s high-order code L3STER as a point of reference. In particular, we will show how to structure assembly so that it fully utilizes CPU caches, and how hybrid parallelism can be leveraged for improved scalability.