Locally Optimal Block Preconditioned Conjugate Gradient Research Articles

Massively parallel implementation of iterative eigensolvers in large-scale plane-wave density functional theory

The Kohn-sham density functional theory (DFT) is a powerful method to describe the electronic structures of molecules and solids in condensed matter physics, computational chemistry and materials science. However, large and accurate DFT calculations within plane waves process a cubic-scaling computational complexity, which is usually limited by expensive computation and communication costs. The rapid development of high performance computing (HPC) on leadership supercomputers brings new opportunities for developing plane-wave DFT calculations for large-scale systems. Here, we implement parallel iterative eigensolvers in large-scale plane-wave DFT calculations, including Davidson, locally optimal block preconditioned conjugate gradient (LOBPCG), projected preconditioned conjugate gradient (PPCG) and the Chebyshev subspace iteration (CheFSI) algorithms, and analyze the performance of these algorithms in massively parallel plane-wave computing tasks. We adopt a two-level parallelization strategy that combines the message passing interface (MPI) with open multi-processing (OpenMP) parallel programming to handle data exchange and matrix operations in the construction and diagonalization of large-scale Hamiltonian matrix within plane waves. Numerical results illustrate that these iterative eigensolvers can scale up to 42,592 processing cores with high peak performance of 30% on leadship supercomputers to study the electronic structures of bulk silicon systems containing 10,648 atoms. Program summaryProgram Title: Plane wave density functional theory (PWDFT)CPC Library link to program files:https://doi.org/10.17632/c8v2mx5vn4.1Developer's repository link:https://bitbucket.org/berkeleylab/scalesLicensing provisions: BSD 3-clauseProgramming language: C++Nature of problem: PWDFT is used for electronic structure calculations based on Kohn-Sham density functional theory. The key challenge to address is a constrained energy minimization problem, which can also be formulated as a nonlinear eigenvalue problem. MPI/OpenMP-based approaches are employed to provide multi-core acceleration for the study of the chemical and material properties of larger-scale molecules and solids.Solution method: PWDFT implements self-consistent field (SCF) iterations and direct constrained minimization algorithms with various acceleration strategies. It is written in C++ and offers parallel acceleration based on MPI/OpenMP.

Hybrid eigensolvers for nuclear configuration interaction calculations

We examine and compare several iterative methods for solving large-scale eigenvalue problems arising from nuclear structure calculations. In particular, we discuss the possibility of using block Lanczos method, a Chebyshev filtering based subspace iterations and the residual minimization method accelerated by direct inversion of iterative subspace (RMM-DIIS) and describe how these algorithms compare with the standard Lanczos algorithm and the locally optimal block preconditioned conjugate gradient (LOBPCG) algorithm. Although the RMM-DIIS method does not exhibit rapid convergence when the initial approximations to the desired eigenvectors are not sufficiently accurate, it can be effectively combined with either the block Lanczos or the LOBPCG method to yield a hybrid eigensolver that has several desirable properties. We will describe a few practical issues that need to be addressed to make the hybrid solver efficient and robust.

A mixed precision LOBPCG algorithm

The locally optimal block preconditioned conjugate gradient (LOBPCG) algorithm is a popular approach for computing a few smallest eigenvalues and the corresponding eigenvectors of a large Hermitian positive definite matrix A. In this work, we propose a mixed precision variant of LOBPCG that uses a (sparse) Cholesky factorization of A computed in reduced precision as the preconditioner. To further enhance performance, a mixed precision orthogonalization strategy is proposed. To analyze the impact of reducing precision in the preconditioner on performance, we carry out a rounding error and convergence analysis of PINVIT, a simplified variant of LOBPCG. Our theoretical results predict and our numerical experiments confirm that the impact on convergence remains marginal. In practice, our mixed precision LOBPCG algorithm typically reduces the computation time by a factor of 1.4--2.0 on both CPUs and GPUs.

Signed Graph Metric Learning via Gershgorin Disc Perfect Alignment.

Given a convex and differentiable objective Q(M) for a real symmetric matrix M in the positive definite (PD) cone-used to compute Mahalanobis distances-we propose a fast general metric learning framework that is entirely projection-free. We first assume that M resides in a space S of generalized graph Laplacian matrices corresponding to balanced signed graphs. M ∈ S that is also PD is called a graph metric matrix. Unlike low-rank metric matrices common in the literature, S includes the important diagonal-only matrices as a special case. The key theorem to circumvent full eigen-decomposition and enable fast metric matrix optimization is Gershgorin disc perfect alignment (GDPA): given M ∈ S and diagonal matrix S, where Sii = 1/vi and v is the first eigenvector of M, we prove that Gershgorin disc left-ends of similarity transform B = SMS-1 are perfectly aligned at the smallest eigenvalue λmin. Using this theorem, we replace the PD cone constraint in the metric learning problem with tightest possible linear constraints per iteration, so that the alternating optimization of the diagonal / off-diagonal terms in M can be solved efficiently as linear programs via the Frank-Wolfe method. We update v using Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) with warm start as entries in M are optimized successively. Experiments show that our graph metric optimization is significantly faster than cone-projection schemes, and produces competitive binary classification performance.

NeuralSound

We present a novel learning-based modal sound synthesis approach that includes a mixed vibration solver for modal analysis and a radiation network for acoustic transfer. Our mixed vibration solver consists of a 3D sparse convolution network and a Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) module for iterative optimization. Moreover, we highlight the correlation between a standard numerical vibration solver and our network architecture. Our radiation network predicts the Far-Field Acoustic Transfer maps (FFAT Maps) from the surface vibration of the object. The overall running time of our learning-based approach for most new objects is less than one second on a RTX 3080 Ti GPU while maintaining a high sound quality close to the ground truth solved by standard numerical methods. We also evaluate the numerical and perceptual accuracy of our approach on different objects with various shapes and materials.

A Robust and Efficient Implementation of LOBPCG

Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) is widely used to compute eigenvalues of large sparse symmetric matrices. The algorithm can suffer from numerical instability if it is not implemented with care. This is especially problematic when the number of eigenpairs to be computed is relatively large. In this paper we propose an improved basis selection strategy based on earlier work by Hetmaniuk and Lehoucq as well as a robust convergence criterion which is backward stable to enhance the robustness. We also suggest several algorithmic optimizations that improve performance of practical LOBPCG implementations. Numerical examples confirm that our approach consistently and significantly outperforms previous competing approaches in both stability and speed.

Efficient block preconditioned eigensolvers for linear response time-dependent density functional theory

We present two efficient iterative algorithms for solving the linear response eigenvalue problem arising from the time dependent density functional theory. Although the matrix to be diagonalized is nonsymmetric, it has a special structure that can be exploited to save both memory and floating point operations. In particular, the nonsymmetric eigenvalue problem can be transformed into an eigenvalue problem that involves the product of two matrices M and K. We show that, because MK is self-adjoint with respect to the inner product induced by the matrix K, this product eigenvalue problem can be solved efficiently by a modified Davidson algorithm and a modified locally optimal block preconditioned conjugate gradient (LOBPCG) algorithm that make use of the K-inner product. The solution of the product eigenvalue problem yields one component of the eigenvector associated with the original eigenvalue problem. We show that the other component of the eigenvector can be easily recovered in an inexpensive postprocessing procedure. As a result, the algorithms we present here become more efficient than existing methods that try to approximate both components of the eigenvectors simultaneously. In particular, our numerical experiments demonstrate that the new algorithms presented here consistently outperform the existing state-of-the-art Davidson type solvers by a factor of two in both solution time and storage.

GPU-Accelerated LOBPCG Method with Inexact Null-Space Filtering for Solving Generalized Eigenvalue Problems in Computational Electromagnetics Analysis with Higher-Order FEM

AbstractThis paper presents a GPU-accelerated implementation of the Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) method with an inexact nullspace filtering approach to find eigenvalues in electromagnetics analysis with higher-order FEM. The performance of the proposed approach is verified using the Kepler (Tesla K40c) graphics accelerator, and is compared to the performance of the implementation based on functions from the Intel MKL on the Intel Xeon (E5-2680 v3, 12 threads) central processing unit (CPU) executed in parallel mode. Compared to the CPU reference implementation based on the Intel MKL functions, the proposed GPU-based LOBPCG method with inexact nullspace filtering allowed us to achieve up to 2.9-fold acceleration.

On the performance and energy efficiency of sparse linear algebra on GPUs

In this paper we unveil some performance and energy efficiency frontiers for sparse computations on GPU-based supercomputers. We compare the resource efficiency of different sparse matrix–vector products (SpMV) taken from libraries such as cuSPARSE and MAGMA for GPU and Intel’s MKL for multicore CPUs, and develop a GPU sparse matrix–matrix product (SpMM) implementation that handles the simultaneous multiplication of a sparse matrix with a set of vectors in block-wise fashion. While a typical sparse computation such as the SpMV reaches only a fraction of the peak of current GPUs, we show that the SpMM succeeds in exceeding the memory-bound limitations of the SpMV. We integrate this kernel into a GPU-accelerated Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) eigensolver. LOBPCG is chosen as a benchmark algorithm for this study as it combines an interesting mix of sparse and dense linear algebra operations that is typical for complex simulation applications, and allows for hardware-aware optimizations. In a detailed analysis we compare the performance and energy efficiency against a multi-threaded CPU counterpart. The reported performance and energy efficiency results are indicative of sparse computations on supercomputers.

Preconditioned eigensolvers for large-scale nonlinear Hermitian eigenproblems with variational characterizations. I. Extreme eigenvalues

Efficient computation of extreme eigenvalues of large-scale linear Hermitian eigenproblems can be achieved by preconditioned conjugate gradient (PCG) methods. In this paper, we study PCG methods for computing extreme eigenvalues of nonlinear Hermitian eigenproblems of the form T ( λ ) v = 0 T(\lambda )v=0 that admit a nonlinear variational principle. We investigate some theoretical properties of a basic CG method, including its global and asymptotic convergence. We propose several variants of single-vector and block PCG methods with deflation for computing multiple eigenvalues, and compare them in arithmetic and memory cost. Variable indefinite preconditioning is shown to be effective to accelerate convergence when some desired eigenvalues are not close to the lowest or highest eigenvalue. The efficiency of variants of PCG is illustrated by numerical experiments. Overall, the locally optimal block preconditioned conjugate gradient (LOBPCG) is the most efficient method, as in the linear setting.

Nonsymmetric Preconditioning for Conjugate Gradient and Steepest Descent Methods 1

We numerically analyze the possibility of turning off post-smoothing (relaxation) in geometric multigrid when used as a preconditioner in conjugate gradient linear and eigenvalue solvers for the 3D Laplacian. The geometric Semicoarsening Multigrid (SMG) method is provided by the hypre parallel software package. We solve linear systems using two variants (standard and flexible) of the preconditioned conjugate gradient (PCG) and preconditioned steepest descent (PSD) methods. The eigenvalue problems are solved using the locally optimal block preconditioned conjugate gradient (LOBPCG) method available in hypre through BLOPEX software. We observe that turning off the post-smoothing in SMG dramatically slows down the standard PCG-SMG. For flexible PCG and LOBPCG, our numerical results show that post-smoothing can be avoided, resulting in overall acceleration, due to the high costs of smoothing and relatively insignificant decrease in convergence speed. We numerically demonstrate for linear systems that PSD-SMG and flexible PCG-SMG converge similarly if SMG post-smoothing is off. We experimentally show that the effect of acceleration is independent of memory interconnection. A theoretical justification is provided.

A Robust Numerical Algorithm for Computing Maxwell's Transmission Eigenvalue Problems

We study a robust and efficient eigensolver for computing a few smallest positive eigenvalues of the three-dimensional Maxwell's transmission eigenvalue problem. The discretized governing equations by the Nedelec edge element result in a large-scale quadratic eigenvalue problem (QEP) for which the spectrum contains many zero eigenvalues and the coefficient matrices consist of patterns in the matrix form $XY^{-1}Z$, both of which prevent existing eigenvalue solvers from being efficient. To remedy these difficulties, we rewrite the QEP as a particular nonlinear eigenvalue problem and develop a secant-type iteration, together with an indefinite locally optimal block preconditioned conjugate gradient (LOBPCG) method, to sequentially compute the desired positive eigenvalues. Furthermore, we propose a novel method to solve the linear systems in each iteration of LOBPCG. Intensive numerical experiments show that our proposed method is robust, although the desired real eigenvalues are surrounded by complex eigenv...

Efficient iterative method for solving the Dirac–Kohn–Sham density functional theory

We present for the first time an efficient iterative method to directly solve the four-component Dirac–Kohn–Sham (DKS) density functional theory. Due to the existence of the negative energy continuum in the DKS operator, the existing iterative techniques for solving the Kohn–Sham systems cannot be efficiently applied to solve the DKS systems. The key component of our method is a novel filtering step (F) which acts as a preconditioner in the framework of the locally optimal block preconditioned conjugate gradient (LOBPCG) method. The resulting method, dubbed the LOBPCG-F method, is able to compute the desired eigenvalues and eigenvectors in the positive energy band without computing any state in the negative energy band. The LOBPCG-F method introduces mild extra cost compared to the standard LOBPCG method and can be easily implemented. We demonstrate our method in the pseudopotential framework with a planewave basis set which naturally satisfies the kinetic balance prescription. Numerical results for Pt2, Au2, TlF, and Bi2Se3 indicate that the LOBPCG-F method is a robust and efficient method for investigating the relativistic effect in systems containing heavy elements.

Overlapping Schwarz preconditioned eigensolvers for spectral element discretizations

Model generalized eigenproblems associated with self-adjoint differential operators in nonstandard homogeneous or heterogeneous domains are considered. Their numerical approximation is based on Gauss–Lobatto–Legendre conforming spectral elements defined by Gordon–Hall transfinite mappings. The resulting discrete eigenproblems are solved iteratively with a Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) method, accelerated by an overlapping Schwarz preconditioner. Several numerical tests show the good convergence properties of the proposed preconditioned eigensolver, such as its scalability and quasi-optimality in the discretization parameters, which are analogous to those obtained for overlapping Schwarz preconditioners for linear systems.

New eigensolvers for large-scale nanoscience simulations

We present results for applications to nanosystems of state-of-the-art iterative eigensolvers based on conjugate gradients and variants of Davidson in the context of semi-empirical plane wave electronic structure calculations. We are concerned with the computation of electronic and optical properties of nanosystems using the Energy SCAN method to compute interior eigenstates around the band gap that determine their properties. Numerically, this interior Hermitian eigenvalue problem poses several challenges, with respect to both accuracy and efficiency. All the iterative eigensolvers are seeking the minimal eigenvalues of the folded operator with reference shift in the band-gap. The tested methods include standard conjugate-gradient (CG)-based Rayleigh quotient minimization, Locally optimal block-preconditioned CG (LOBPCG) and two variants of the (Jacobi-)Davidson method: JDQMR and GD+1. Our experimental results indicated that Davidson-type methods are often the fastest.

Block Locally Optimal Preconditioned Eigenvalue Xolvers (BLOPEX) in Hypre and PETSc

We describe our software package Block Locally Optimal Preconditioned Eigenvalue Xolvers (BLOPEX) publicly released recently. BLOPEX is available as a stand-alone serial library, as an external package to PETSc (``Portable, Extensible Toolkit for Scientific Computation'', a general purpose suite of tools for the scalable solution of partial differential equations and related problems developed by Argonne National Laboratory), and is also built into {\it hypre} (``High Performance Preconditioners'', scalable linear solvers package developed by Lawrence Livermore National Laboratory). The present BLOPEX release includes only one solver--the Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) method for symmetric eigenvalue problems. {\it hypre} provides users with advanced high-quality parallel preconditioners for linear systems, in particular, with domain decomposition and multigrid preconditioners. With BLOPEX, the same preconditioners can now be efficiently used for symmetric eigenvalue problems. PETSc facilitates the integration of independently developed application modules with strict attention to component interoperability, and makes BLOPEX extremely easy to compile and use with preconditioners that are available via PETSc. We present the LOBPCG algorithm in BLOPEX for {\it hypre} and PETSc. We demonstrate numerically the scalability of BLOPEX by testing it on a number of distributed and shared memory parallel systems, including a Beowulf system, SUN Fire 880, an AMD dual-core Opteron workstation, and IBM BlueGene/L supercomputer, using PETSc domain decomposition and {\it hypre} multigrid preconditioning. We test BLOPEX on a model problem, the standard 7-point finite-difference approximation of the 3-D Laplacian, with the problem size in the range $10^5-10^8$.

Basis selection in LOBPCG

The purpose of our paper is to discuss basis selection for Knyazev’s locally optimal block preconditioned conjugate gradient (LOBPCG) method. An inappropriate choice of basis can lead to ill-conditioned Gram matrices in the Rayleigh–Ritz analysis that can delay convergence or produce inaccurate eigenpairs. We demonstrate that the choice of basis is not merely related to computing in finite precision arithmetic. We propose a representation that maintains orthogonality of the basis vectors and so has excellent numerical properties.

Multilevel preconditioned iterative eigensolvers for Maxwell eigenvalue problems

We investigate eigensolvers for computing a few of the smallest eigenvalues of a generalized eigenvalue problem resulting from the finite element discretization of the time independent Maxwell equation. Various multilevel preconditioners are employed to improve the convergence and memory consumption of the Jacobi–Davidson algorithm and of the locally optimal block preconditioned conjugate gradient (LOBPCG) method. We present numerical results of very large eigenvalue problems originating from the design of resonant cavities of particle accelerators.

Toward the Optimal Preconditioned Eigensolver: Locally Optimal Block Preconditioned Conjugate Gradient Method

We describe new algorithms of the locally optimal block preconditioned conjugate gradient (LOBPCG) method for symmetric eigenvalue problems, based on a local optimization of a three-term recurrence, and suggest several other new methods. To be able to compare numerically different methods in the class, with different preconditioners, we propose a common system of model tests, using random preconditioners and initial guesses. As the "ideal" control algorithm, we advocate the standard preconditioned conjugate gradient method for finding an eigenvector as an element of the null-space of the corresponding homogeneous system of linear equations under the assumption that the eigenvalue is known. We recommend that every new preconditioned eigensolver be compared with this "ideal" algorithm on our model test problems in terms of the speed of convergence, costs of every iteration, and memory requirements. We provide such comparison for our LOBPCG method. Numerical results establish that our algorithm is practically as efficient as the ``ideal' algorithm when the same preconditioner is used in both methods. We also show numerically that the LOBPCG method provides approximations to first eigenpairs of about the same quality as those by the much more expensive global optimization method on the same generalized block Krylov subspace. We propose a new version of block Davidson's method as a generalization of the LOBPCG method. Finally, direct numerical comparisons with the Jacobi--Davidson method show that our method is more robust and converges almost two times faster.