Superlinear Speedup Research Articles

Multi-GPU RI-HF Energies and Analytic Gradients─Toward High-Throughput Ab Initio Molecular Dynamics.

This article presents an optimized algorithm and implementation for calculating resolution-of-the-identity Hartree-Fock (RI-HF) energies and analytic gradients using multiple graphics processing units (GPUs). The algorithm is especially designed for high throughput ab initio molecular dynamics simulations of small and medium size molecules (10-100 atoms). Key innovations of this work include the exploitation of multi-GPU parallelism and a workload balancing scheme that efficiently distributes computational tasks among GPUs. Our implementation also employs techniques for symmetry utilization, integral screening, and leveraging sparsity to optimize memory usage. Computational results show that the implementation achieves significant performance improvements, including over 3 × speedups in single GPU AIMD throughput compared to previous GPU-accelerated RI-HF and traditional HF methods. Furthermore, utilizing multiple GPUs can provide superlinear speedup when the additional aggregate GPU memory allows for the storage of decompressed three-center integrals.

Understanding superlinear speedup in current HPC architectures

Understanding superlinear speedup in current HPC architectures

Report on the exact methods for finding minimum-sized DFA

Abstract This paper presents four state-of-art methods for the finite-state automaton inference based on a sample of labeled strings. The first algorithm is Exbar, and the next three are mathematical models based on ASP, SAT and SMT theories. The potentiality of using multiprocessor computers in the context of automata inference was our research’s primary goal. In a series of experiments, we showed that our parallelization of the exbar algorithm is the best choice when a multiprocessor system is available. Furthermore, we obtained a superlinear speedup for some of the prepared datasets, achieving almost a 5-fold speedup on the median, using 12 and 24 processes.

A massively parallel implicit 3D unstructured grid solver for computing turbulent flows on latest distributed memory computational architectures

An implicit unstructured grid density-based solver–PRAVAHA based on a parallel variant of the lower upper symmetric Gauss-Seidel (LUSGS) method is developed to compute large-scale engineering problems. A four-layered parallel algorithm is designed to efficiently compute three-dimensional turbulent flows on massively parallel modern multiple instruction multiple data-stream (MIMD) computational hardware. This data parallel approach achieves multiple layers of parallelism including continuity of flow solution, transfer of solution gradients, and calculation of drag/lift/solution residuals, right up to the innermost implicit LUSGS solver sub-routine, which is relatively less explored in the literature. Domain decomposition is performed using the METIS software based on multi-level graph partitioning algorithms. Non-blocking message-passing interface library functions are used to manage inter-processor communication through explicit message passing, efficiently. Super-linear scalability of the parallel solver is established on the current state-of-the-art supercomputing facility, the 838 teraflops PARAM seva on up to 6144 cores. Linear or even super-linear speedup on problems of significant size is observed even on ad-hoc parallel computing platforms like workstations and multi-node clusters, for turbulent flow simulations.

Pruning for image segmentation: Improving computational efficiency for large-scale remote sensing applications

Image segmentation is a fundamental step in object-based image analysis and other workflows. However, high-efficiency remains a challenge, especially for the analysis of large-scale Earth observation images. In recent years, considerable effort has been paid to designing merging criteria, automatic scale selection, and object-specific optimisation. These segmentation methods usually rely on the region-adjacency graph (RAG) model and the nearest neighbour graph (NNG) model, which provide acceptable merging performance. Low efficiency occurs due to many redundant edge weight updates in the RAG model. In this study, we propose a generic dynamic pruning framework to improve the efficiency of existing region-merging-based segmentation algorithms, opening the door for large-scale applications in remote sensing. The proposed pruning framework includes intra-object and inter-object pruning modules for the RAG model. Inter-object pruning divides the RAG model into multiple sub-RAG models to reduce the redundant edge weight updates between adjacent objects. Intra-object pruning iteratively divides the sub-RAG into smaller RAGs. In our experimental analysis, we employ the proposed pruning framework with six region-merging segmentation methods and validate the effectiveness on four 10–20M pixel images and a 100M pixel data set. The pruning framework improves the performance of various segmentation algorithms by reducing computational complexity while maintaining segmentation accuracy. We observed a significant improvement in efficiency, with various achieving super-linear speed-up while maintaining the stability of segmentation accuracy. In single-core mode, the computation time of tested algorithms is enhanced by two to ten times on the four test images. In the multicore mode, speed-up increased up to 40 times with eight CPU cores. The computational cost was reduced by 36.15% to 95.77% in the number of weight updates, which is independent of hardware characteristics. On the large-scale image, two modes achieved speed-ups of 36.07 and 102.74, respectively.

Parallel Computation Using Non-Overlapping Domain Decomposition Coupled with Compact Local Integrated RBF for Navier–Stokes Equations

A non-overlapping domain decomposition-based parallel algorithm coupled with a compact local integrated radial basis function (CLIRBF) method is developed for solving Navier-Stokes equations. For this approach, a problem is divided into subdomains. In each sub-domain, a CLIRBF scheme is applied to solve the Navier-Stokes equations of flows. A relaxation factor is used at the interface between sub-domains to ensure the quick convergence of the present method. The Bitmap termination detection technique is introduced to complete the global termination. The present approach is verified using two fluid flow problems: the lid-driven cavity and the natural convection in concentric annuli flow. The numerical results have demonstrated the efficiency of the present parallel method compared with the corresponding sequential one and other published methods. Especially, super-linear speed-up was achieved for several CPUs. In terms of accuracy, the obtained results are in very good agreement with benchmark results.

A parallel branch-and-bound algorithm with history-based domination and its application to the sequential ordering problem

In this paper, we describe the first parallel Branch-and-Bound (B&B) algorithm with a history-based domination technique. Although history-based domination substantially speeds up a B&B search, it makes parallelization much more challenging. Our algorithm is the first parallel exact algorithm for the Sequential Ordering Problem using a pure B&B approach. To effectively explore the solution space, we have developed three novel parallelization techniques: thread restart, parallel history domination, and history-table memory management. The proposed algorithm was experimentally evaluated using the SOPLIB and TSPLIB benchmarks on multi-core processors. Using 32 threads with a time limit of one hour, the algorithm gives geometric-mean speedups of 72x and 20x on the medium-difficulty SOPLIB and TSPLIB instances, respectively. On the hard instances, it solves 12 instances that the sequential algorithm does not solve, with geometric-mean speedups of 16x on SOPLIB and 32x on TSPLIB. Super-linear speedups up to 366x are seen on 16 instances.

Unified X-space parallelization algorithm for conserved discrete unified gas kinetic scheme

In this paper, the open source multiscale flow solver dugksFoam is optimized with a newly proposed parallelization strategy and conserved algorithm. A novel X-space parallel framework is introduced. In contrast to traditional discrete physical/velocity space decomposition parallelization methods, it can decompose each space in a hybrid manner and efficiently parallelize the computation. For better conservation of the macroscopic physical variables, flow properties in each cell are updated by moments of microscopic variable fluxes, which preserves particle multiscale information under sparse velocity space mesh. Several test cases, which include 2D lid-driven cavity flow in the transition flow regime, hypersonic rarefied flow past a 3D sphere, are carried out to verify the accuracy of the conserved discrete unified gas kinetic scheme to simulate the flows in all flow regimes. A large-scale parallel computational test case of 3D sphere with 101,600 physical mesh cells and 62,744 velocity mesh cells is conducted to measure the parallel efficiency of the conserved dugksFoam code. A significant superlinear speedup of 1631.87 has been achieved on 1120 cores.

Quantum optimization of maximum independent set using Rydberg atom arrays.

Realizing quantum speedup for practically relevant, computationally hard problems is a central challenge in quantum information science. Using Rydberg atom arrays with up to 289 qubits in two spatial dimensions, we experimentally investigate quantum algorithms for solving the maximum independent set problem. We use a hardware-efficient encoding associated with Rydberg blockade, realize closed-loop optimization to test several variational algorithms, and subsequently apply them to systematically explore a class of graphs with programmable connectivity. We find that the problem hardness is controlled by the solution degeneracy and number of local minima, and we experimentally benchmark the quantum algorithm's performance against classical simulated annealing. On the hardest graphs, we observe a superlinear quantum speedup in finding exact solutions in the deep circuit regime and analyze its origins.

Scalability of Nek5000 on High-Performance Computing Clusters Toward Direct Numerical Simulation of Molten Pool Convection

In a postulated severe accident, a molten pool with decay heat can form in the lower head of a reactor pressure vessel, threatening the vessel’s structural integrity. Natural convection in molten pools with extremely high Rayleigh (Ra) number is not yet fully understood as accurate simulation of the intense turbulence remains an outstanding challenge. Various models have been implemented in many studies, such as RANS (Reynolds-averaged Navier–Stokes), LES (large-eddy simulation), and DNS (direct numerical simulation). DNS can provide the most accurate results but at the expense of large computational resources. As the significant development of the HPC (high-performance computing) technology emerges, DNS becomes a more feasible method in molten pool simulations. Nek5000 is an open-source code for the simulation of incompressible flows, which is based on a high-order SEM (spectral element method) discretization strategy. Nek5000 has been performed on many supercomputing clusters, and the parallel performance of benchmarks can be useful for the estimation of computation budgets. In this work, we conducted scalability tests of Nek5000 on four different HPC clusters, namely, JUWELS (Atos Bullsquana X1000), Hawk (HPE Apollo 9000), ARCHER2 (HPE Cray EX), and Beskow (Cray XC40). The reference case is a DNS of molten pool convection in a hemispherical configuration with Ra = 1011, where the computational domain consisted of 391 million grid points. The objectives are (i) to determine if there is strong scalability of Nek5000 for the specific problem on the currently available systems and (ii) to explore the feasibility of obtaining DNS data for much higher Ra. We found super-linear speed-up up to 65536 MPI-rank on Hawk and ARCHER2 systems and around 8000 MPI-rank on JUWELS and Beskow systems. We achieved the best performance with the Hawk system with reasonably good results up to 131072 MPI-rank, which is attributed to the hypercube technique on its interconnection. Given the current HPC technology, it is feasible to obtain DNS data for Ra = 1012, but for cases higher than this, significant improvement in hardware and software HPC technology is necessary.

Super-linear speedup for real-time condition monitoring using image processing and drones

Real-time inspections for the large-scale solar system may take a long time to get the hazard situations for any failures that may take place in the solar panels normal operations, where prior hazards detection is important. Reducing the execution time and improving the system’s performance are the ultimate goals of multiprocessing or multicore systems. Real-time video processing and analysis from two camcorders, thermal and charge-coupling devices (CCD), mounted on a drone compose the embedded system being proposed for solar panels inspection. The inspection method needs more time for capturing and processing the frames and detecting the faulty panels. The system can determine the longitude and latitude of the defect position information in real-time. In this work, we investigate parallel processing for the image processing operations which reduces the processing time for the inspection systems. The results show a super-linear speedup for real-time condition monitoring in large-scale solar systems. Using the multiprocessing module in Python, we execute fault detection algorithms using streamed frames from both video cameras. The experimental results show a super-linear speedup for thermal and CCD video processing, the execution time is efficiently reduced with an average of 3.1 times and 6.3 times using 2 processes and 4 processes respectively.

A Novel Parallel Algorithm with Map Segmentation for Multiple Geographical Feature Label Placement Problem

Multiple geographical feature label placement (MGFLP) is an NP-hard problem that can negatively influence label position accuracy and the computational time of the algorithm. The complexity of such a problem is compounded as the number of features for labeling increases, causing the execution time of the algorithms to grow exponentially. Additionally, in large-scale solutions, the algorithm possibly gets trapped in local minima, which imposes significant challenges in automatic label placement. To address the mentioned challenges, this paper proposes a novel parallel algorithm with the concept of map segmentation which decomposes the problem of multiple geographical feature label placement (MGFLP) to achieve a more intuitive solution. Parallel computing is then utilized to handle each decomposed problem simultaneously on a separate central processing unit (CPU) to speed up the process of label placement. The optimization component of the proposed algorithm is designed based on the hybrid of discrete differential evolution and genetic algorithms. Our results based on real-world datasets confirm the usability and scalability of the algorithm and illustrate its excellent performance. Moreover, the algorithm gained superlinear speedup compared to the previous studies that applied this hybrid algorithm.

An efficient ant colony optimization framework for HPC environments

Combinatorial optimization problems arise in many disciplines, both in the basic sciences and in applied fields such as engineering and economics. One of the most popular combinatorial optimization methods is the Ant Colony Optimization (ACO) metaheuristic. Its parallel nature makes it especially attractive for implementation and execution in High Performance Computing (HPC) environments. Here we present a novel parallel ACO strategy making use of efficient asynchronous decentralized cooperative mechanisms. This strategy seeks to fulfill two objectives: (i) acceleration of the computations by performing the ants’ solution construction in parallel; (ii) convergence improvement through the stimulation of the diversification in the search and the cooperation between different colonies. The two main features of the proposal, decentralization and desynchronization, enable a more effective and efficient response in environments where resources are highly coupled. Examples of such infrastructures include both traditional HPC clusters, and also new distributed environments, such as cloud infrastructures, or even local computer networks. The proposal has been evaluated using the popular Traveling Salesman Problem (TSP), as a well-known NP-hard problem widely used in the literature to test combinatorial optimization methods. An exhaustive evaluation has been carried out using three medium and large size instances from the TSPLIB library, and the experiments show encouraging results with superlinear speedups compared to the sequential algorithm (e.g. speedups of 18 with 16 cores), and a very good scalability (experiments were performed with up to 384 cores improving execution time even at that scale).

Mansard Roofline Model: Reinforcing the Accuracy of the Roofs

Continuous enhancements and diversity in modern multi-core hardware, such as wider and deeper core pipelines and memory subsystems, bring to practice a set of hard-to-solve challenges when modeling their upper-bound capabilities and identifying the main application bottlenecks. Insightful roofline models are widely used for this purpose, but the existing approaches overly abstract the micro-architecture complexity, thus providing unrealistic performance bounds that lead to a misleading characterization of real-world applications. To address this problem, the Mansard Roofline Model (MaRM), proposed in this work, uncovers a minimum set of architectural features that must be considered to provide insightful, but yet accurate and realistic, modeling of performance upper bounds for modern processors. By encapsulating the retirement constraints due to the amount of retirement slots, Reorder-Buffer and Physical Register File sizes, the proposed model accurately models the capabilities of a real platform (average rRMSE of 5.4%) and characterizes 12 application kernels from standard benchmark suites. By following a herein proposed MaRM interpretation methodology and guidelines, speed-ups of up to 5× are obtained when optimizing real-world bioinformatic application, as well as a super-linear speedup of 18.5× when parallelized.

Parallel network component analysis technique for gene regulatory network inference

AbstractThe inference of gene regulatory network has a vital role in understanding the topological order of gene interactions, in addition to how genes are affected by the others genes. One of gene regulatory network techniques is Network Component analysis. The primary drawback of Network Component analysis technique is the intensive computation and time consummation. To avoid these drawbacks, parallel techniques are required. This work presents a parallel technique for gene regulatory network inference; referred as Improved Parallel Computation for Sparse Network Component Analysis (iPSparseNCA) algorithm. To improve the performance of network component analysis technique, iPSparseNCA implements a hybrid parallelism computational model that uses cannon's algorithm for the matrix operations. The performance of iPSparseNCA is measured using different genetic datasets. The computational results indicate that iPSparseNCA achieved high computational speedup, where the achieved speedup reached 1359.85 on 256 processing nodes. These computational results indicate that iPSparseNCA achieved super linear speedup, where, the achieved speedup exceeds the number of used processing nodes. Furthermore, iPSparseNCA has O (N2) time instead of O (M3N2) time for the sequential technique, where M is the number of genes in the dataset and N is the number of the samples.

Hybrid High-Performance Computing Algorithm for Gene Regulatory Network

This paper presents a parallel algorithm for gene regulatory network construction, hereby referred to as H2pcGRN. The construction of gene regulatory network is a vital methodology for investigating the genes interactions' topological order, annotating the genes functionality and demonstrating the regulatory process. One of the approaches for gene regulatory network construction techniques is based on the component analysis method. The main drawbacks of component analysis-based algorithms are its intensive computations that consume time. Despite these drawbacks, this approach is widely applied to infer the regulatory network. Therefore, introducing parallel techniques is indispensable for gene regulatory network inference algorithms. H2pcGRN is a hybrid high performance-computing algorithm for gene regulatory network inference. The proposed algorithm is based on both the hybrid parallelism architecture and the generalized cannon's algorithm. A variety of gene datasets is used for H2pcGRN assessment and evaluation. The experimental results indicated that H2pcGRN achieved super-linear speedup, where its computational speedup reached 570 on 256 processing nodes.

A computationally efficient Branch-and-Bound algorithm for the permutation flow-shop scheduling problem

In this work we propose an efficient branch-and-bound (B&B) algorithm for the permutation flow-shop problem (PFSP) with makespan objective. We present a new node decomposition scheme that combines dynamic branching and lower bound refinement strategies in a computationally efficient way. To alleviate the computational burden of the two-machine bound used in the refinement stage, we propose an online learning-inspired mechanism to predict promising couples of bottleneck machines. The algorithm offers multiple choices for branching and bounding operators and can explore the search tree either sequentially or in parallel on multi-core CPUs. In order to empirically determine the most efficient combination of these components, a series of computational experiments with 600 benchmark instances is performed. A main insight is that the problem size, as well as interactions between branching and bounding operators substantially modify the trade-off between the computational requirements of a lower bound and the achieved tree size reduction. Moreover, we demonstrate that parallel tree search is a key ingredient for the resolution of large problem instances, as strong super-linear speedups can be observed. An overall evaluation using two well-known benchmarks indicates that the proposed approach is superior to previously published B&B algorithms. For the first benchmark we report the exact resolution – within less than 20 minutes – of two instances defined by 500 jobs and 20 machines that remained open for more than 25 years, and for the second a total of 89 improved best-known upper bounds, including proofs of optimality for 74 of them.

A Multi-Branch-and-Bound Binary Parallel Algorithm to Solve the Knapsack Problem 0–1 in a Multicore Cluster

This paper presents a process that is based on sets of parts, where elements are fixed and removed to form different binary branch-and-bound (BB) trees, which in turn are used to build a parallel algorithm called “multi-BB”. These sequential and parallel algorithms calculate the exact solution for the 0–1 knapsack problem. The sequential algorithm solves the instances published by other researchers (and the proposals by Pisinger) to solve the not-so-complex (uncorrelated) class and some problems of the medium-complex (weakly correlated) class. The parallel algorithm solves the problems that cannot be solved with the sequential algorithm of the weakly correlated class in a cluster of multicore processors. The multi-branch-and-bound algorithms obtained parallel efficiencies of approximately 75%, but in some cases, it was possible to obtain a superlinear speedup.

Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inference

Real-time Deep Neural Network (DNN) inference with low-latency requirement has become increasingly important for numerous applications in both cloud computing (e.g., Apple’s Siri) and edge computing (e.g., Google/Waymo’s driverless car). FPGA-based DNN accelerators have demonstrated both superior flexibility and performance; in addition, for real-time inference with low batch size, FPGA is expected to achieve further performance improvement. However, the performance gain from the single-FPGA design is obstructed by the limited on-chip resource. In this paper, we employ multiple FPGAs to cooperatively run DNNs with the objective of achieving super-linear speed-up against single-FPGA design. In implementing such systems, we found two barriers that hinder us from achieving the design goal: (1) the lack of a clear partition scheme for each DNN layer to fully exploit parallelism, and (2) the insufficient bandwidth between the off-chip memory and the accelerator due to the growing size of DNNs. To tackle these issues, we propose a general framework, “Super-LIP”, which can support different kinds of DNNs. In this paper, we take Convolutional Neural Network (CNN) as a vehicle to illustrate Super-LIP. We first formulate an accurate system-level model to support the exploration of best partition schemes. Then, we develop a novel design methodology to effectively alleviate the heavy loads on memory bandwidth by moving traffic from memory bus to inter-FPGA links. We implement Super-LIP based on ZCU102 FPGA boards. Results demonstrate that Super-LIP with 2 FPGAs can achieve 3.48× speedup, compared to the state-of-the-art single-FPGA design. What is more, as the number of FPGAs scales up, the system latency can be further reduced while maintaining high energy efficiency.

An analytic computation-driven algorithm for Decentralized Multicore Systems

In the modern era, increasing numbers of cores per chip are applied for decentralized systems, but there is not any appropriate symbolic computation approach to construct multicore analytic approximation. Thus, it is essential to develop an efficient, simple and unified way for decentralized Adomian decomposition method to increase the potential speed of the multicore systems. In our paper, we present an innovative parallel algorithm of constructing analytic solutions for nonlinear differential system, which based on the Adomian–Rach double decomposition method and Rach’s Adomian polynomials. Based on our algorithm, we further developed a user-friendly Python software package to construct analytic approximations of initial or boundary value problems. Finally, the scope of validity of our Python software package is illustrated by several different types of nonlinear examples. The obtained results demonstrate the effectiveness of our package by compared with exact solution and numeric method, the characteristics of each class of Adomian polynomials and the efficiency of parallel algorithm with multicore processors. We emphasis that the super-linear speedup may happens for the duration of constructing approximate solutions. So, it can be considered as a promising alternative algorithm of decentralized Adomian decomposition method for solving nonlinear problems in science and engineering.