SciDAC Applications Research Articles

Interoperable mesh components for large-scale, distributed-memory simulations

SciDAC applications have a demonstrated need for advanced software tools to manage the complexities associated with sophisticated geometry, mesh, and field manipulation tasks, particularly as computer architectures move toward the petascale. In this paper, we describe a software component – an abstract data model and programming interface – designed to provide support for parallel unstructured mesh operations. We describe key issues that must be addressed to successfully provide high-performance, distributed-memory unstructured mesh services and highlight some recent research accomplishments in developing new load balancing and MPI-based communication libraries appropriate for leadership class computing. Finally, we give examples of the use of parallel adaptive mesh modification in two SciDAC applications.

Factorization-based sparse solvers and preconditioners

Efficient solution of large-scale, ill-conditioned and highly-indefinite algebraic equations often relies on high quality preconditioners together with iterative solvers. Because of their robustness, the factorization-based algorithms could play a significant role when they are combined with iterative methods, particularly in the development of scalable solvers. We present our recent work in using the direct solver SuperLU code base to develop a new supernode-based ILU preconditioner and a domain-decomposition hybrid solver. Our ILU preconditioner is a modification of the classic ILUTP approach, incorporating a number of techniques to improve robustness and performance, which include new dropping strategies that accommodate the use of supernodal structure in the factored matrix. Our hybrid solver is based on the Schur complement method. We use parallel graph partitioning to obtain hierarchical interface/domain decomposition, and multiple parallel direct solvers to solve the subdomain problems simultaneously, and parallel preconditioned iterative solvers to solve the interface problem. We will demonstrate the effectiveness of our new techniques by applying them to two SciDAC applications, modeling next-generation particle accelerators and fusion devices.

Interoperable mesh and geometry tools for advanced petascale simulations

SciDAC applications have a demonstrated need for advanced software tools to manage the complexities associated with sophisticated geometry, mesh, and field manipulation tasks, particularly as computer architectures move toward the petascale. The Center for Interoperable Technologies for Advanced Petascale Simulations (ITAPS) will deliver interoperable and interchangeable mesh, geometry, and field manipulation services that are of direct use to SciDAC applications. The premise of our technology development goal is to provide such services as libraries that can be used with minimal intrusion into application codes. To develop these technologies, we focus on defining a common data model and data-structure neutral interfaces that unify a number of different services such as mesh generation and improvement, front tracking, adaptive mesh refinement, shape optimization, and solution transfer operations. We highlight the use of several ITAPS services in SciDAC applications.

Enhancing scalability of sparse direct methods

TOPS is providing high-performance, scalable sparse direct solvers, which have had significant impacts on the SciDAC applications, including fusion simulation (CEMM), accelerator modeling (COMPASS), as well as many other mission-critical applications in DOE and elsewhere. Our recent developments have been focusing on new techniques to overcome scalability bottleneck of direct methods, in both time and memory. These include parallelizing symbolic analysis phase and developing linear-complexity sparse factorization methods. The new techniques will make sparse direct methods more widely usable in large 3D simulations on highly-parallel petascale computers.

Optimization in SciDAC applications

We present a brief overview of optimization tools that are being developed for SciDAC applications. We emphasize derivative-free and gradient-based methods since these tools make minimal demands on the user and the application. We discuss the performance of these tools and point out developments that have led to significant improvements in performance. A parameter estimation problem that arises in nuclear fission is used to illustrate the challenges that arise as we attack nonlinear, noisy, computationally-intensive optimization applications.

Implicit solvers for large-scale nonlinear problems

Computational scientists are grappling with increasingly complex, multi-rate applications that couple such physical phenomena as fluid dynamics, electromagnetics, radiation transport, chemical and nuclear reactions, and wave and material propagation in inhomogeneous media. Parallel computers with large storage capacities are paving the way for high-resolution simulations of coupled problems; however, hardware improvements alone will not prove enough to enable simulations based on brute-force algorithmic approaches. To accurately capture nonlinear couplings between dynamically relevant phenomena, often while stepping over rapid adjustments to quasi-equilibria, simulation scientists are increasingly turning to implicit formulations that require a discrete nonlinear system to be solved for each time step or steady state solution. Recent advances in iterative methods have made fully implicit formulations a viable option for solution of these large-scale problems. In this paper, we overview one of the most effective iterative methods, Newton-Krylov, for nonlinear systems and point to software packages with its implementation. We illustrate the method with an example from magnetically confined plasma fusion and briefly survey other areas in which implicit methods have bestowed important advantages, such as allowing high-order temporal integration and providing a pathway to sensitivity analyses and optimization. Lastly, we overview algorithm extensions under development motivated by current SciDAC applications.

Mesh quality improvement for SciDAC applications

Accurate and efficient numerical solution of partial differential equations requires well-formed meshes that are non-inverted, smooth, well-shaped, oriented, and size-adapted. The Mesquite mesh quality improvement toolkit is a software library that applies optimization algorithms to create well-formed meshes via node movement. Mesquite can be run standalone using drivers or called directly from an application code. Mesquite can play an essential role in the SLAC accelerator design program as a component in automatic shape optimization software and in manufacturing defect-correction studies to smoothly deform meshes in response to geometric domain deformations guided by the optimization of design parameters. Mesquite has also been applied to problems in fusion, biology, and propellant burn studies.

Solving large-scale eigenvalue problems in SciDAC applications

Large-scale eigenvalue problems arise in a number of DOE applications. This paper provides an overview of the recent development of eigenvalue computation in the context of two SciDAC applications. We emphasize the importance of Krylov subspace methods, and point out its limitations. We discuss the value of alternative approaches that are more amenable to the use of preconditioners, and report the progress on using the multi-level algebraic sub-structuring techniques to speed up eigenvalue calculation. In addition to methods for linear eigenvalue problems, we also examine new approaches to solving two types of non-linear eigenvalue problems arising from SciDAC applications.

SciDAC advances and applications in computational beam dynamics

SciDAC has had a major impact on computational beam dynamics and the design of particle accelerators. Particle accelerators—which account for half of the facilities in the DOE Office of Science Facilities for the Future of Science 20 Year Outlook—are crucial for US scientific, industrial, and economic competitiveness. Thanks to SciDAC, accelerator design calculations that were once thought impossible are now carried routinely, and new challenging and important calculations are within reach. SciDAC accelerator modeling codes are being used to get the most science out of existing facilities, to produce optimal designs for future facilities, and to explore advanced accelerator concepts that may hold the key to qualitatively new ways of accelerating charged particle beams. In this paper we present highlights from the SciDAC Accelerator Science and Technology (AST) project Beam Dynamics focus area in regard to algorithm development, software development, and applications.

Interoperable geometry and mesh components for SciDAC applications

Software components for representing and evaluating geometry (TSTTG/CGM) and finite element mesh (TSTTM/MOAB), and a higher-level component for relations between the two (TSTTR/LASSO), have been combined with electromagnetic modelling and optimization techniques, to form a SciDAC shape optimization application. The TSTT data model described in this paper allows components involved in the shape optimization application to be coupled at a variety of levels, from coarse black-box coupling (e.g. to generate a model accelerator cavity using TSTTG) to very fine-grained coupling (e.g. smoothing individual mesh elements based in part on geometric surface normals at mesh vertices). Despite this flexibility, the TSTT data model uses only four fundamental data types (entities, sets, tags, and the interface object itself). We elaborate on the design and implementation of effective components in the context of this application, and show how our simple but flexible data model facilitates these efforts.