Massive Parallel Computing Research Articles

The Distributed Algorithms for the Lower-bounded k-Center Clustering in Metric Space

Clustering is a fundamental unsupervised machine learning problem. However, due to limited processing memory and CPU power, it is challenging to cluster large-scale data. The distributed methods have received great attention in recent years since large-scale data can be stored and computed on multiple machines. In this paper, we study a variant of the k-center clustering problem, i.e., the lower-bounded k-center clustering problem (denoted as the Lb-k-Cen problem), in the Massively Parallel Computation (MPC) distributed model. The current best distributed result for the Lb-k-Cen problem has several rounds of communication between the coordinator and machines, which may increase the local computation and communication cost of the algorithm for handling large-scale data. To achieve fewer local computation and communication rounds, we use the threshold method and flow network technique, which avoid local computation again in each machine, and can achieve a two rounds (9+ϵ)-approximation algorithm in metric space. Moreover, we also consider the distributed algorithm for the Lb-k-Cen problem in the metric space with bounded doubling dimension, and propose a two rounds (3+ϵ)-approximation algorithm.

Supercomputer framework for reverse engineering firing patterns of neuron populations to identify their synaptic inputs

In this study, we develop new reverse engineering (RE) techniques to identify the organization of the synaptic inputs generating firing patterns of populations of neurons. We tested these techniques in silico to allow rigorous evaluation of their effectiveness, using remarkably extensive parameter searches enabled by massively-parallel computation on supercomputers. We chose spinal motoneurons as our target neural system, since motoneurons process all motor commands and have well-established input-output properties. One set of simulated motoneurons was driven by 300,000+ simulated combinations of excitatory, inhibitory, and neuromodulatory inputs. Our goal was to determine if these firing patterns had sufficient information to allow RE identification of the input combinations. Like other neural systems, the motoneuron input-output system is likely non-unique. This non-uniqueness could potentially limit this RE approach, as many input combinations can produce similar outputs. However, our simulations revealed that firing patterns contained sufficient information to sharply restrict the solution space. Thus, our RE approach successfully generated estimates of the actual simulated patterns of excitation, inhibition, and neuromodulation, with variances accounted for ranging from 75–90%. It was striking that nonlinearities induced in firing patterns by the neuromodulation inputs did not impede RE, but instead generated distinctive features in firing patterns that aided RE. These simulations demonstrate the potential of this form of RE analysis. It is likely that the ever-increasing capacity of supercomputers will allow increasingly accurate RE of neuron inputs from their firing patterns from many neural systems.

Supercomputer framework for reverse engineering firing patterns of neuron populations to identify their synaptic inputs.

In this study, we develop new reverse engineering (RE) techniques to identify the organization of the synaptic inputs generating firing patterns of populations of neurons. We tested these techniques in silico to allow rigorous evaluation of their effectiveness, using remarkably extensive parameter searches enabled by massively-parallel computation on supercomputers. We chose spinal motoneurons as our target neural system, since motoneurons process all motor commands and have well-established input-output properties. One set of simulated motoneurons was driven by 300,000+ simulated combinations of excitatory, inhibitory, and neuromodulatory inputs. Our goal was to determine if these firing patterns had sufficient information to allow RE identification of the input combinations. Like other neural systems, the motoneuron input-output system is likely non-unique. This non-uniqueness could potentially limit this RE approach, as many input combinations can produce similar outputs. However, our simulations revealed that firing patterns contained sufficient information to sharply restrict the solution space. Thus, our RE approach successfully generated estimates of the actual simulated patterns of excitation, inhibition, and neuromodulation, with variances accounted for ranging from 75-90%. It was striking that nonlinearities induced in firing patterns by the neuromodulation inputs did not impede RE, but instead generated distinctive features in firing patterns that aided RE. These simulations demonstrate the potential of this form of RE analysis. It is likely that the ever-increasing capacity of supercomputers will allow increasingly accurate RE of neuron inputs from their firing patterns from many neural systems.

Parallel Acyclic Joins: Optimal Algorithms and Cyclicity Separation

We study equi-join computation in the massively parallel computation (MPC) model. Currently, a main open question under this topic is whether it is possible to design an algorithm that can process any join with load O(N polylog N/p 1/ρ* ) — measured in the number of words communicated per machine — where N is the total number of tuples in the input relations, ρ * is the join’s fractional edge covering number, and p is the number of machines. We settle the question in the negative for the class of tuple-based algorithms (all the known MPC join algorithms fall in this class) by proving the existence of a join query with ρ * = 2 that requires a load of Ω ( N/p 1/3 ) to evaluate. Our lower bound provides solid evidence that the “AGM bound” alone is not sufficient for characterizing the hardness of join evaluation in MPC (a phenomenon that does not exist in RAM). The hard join instance identified in our argument is cyclic, which leaves the question of whether O(N polylog N/p 1/ρ* ) is still possible for acyclic joins. We answer this question in the affirmative by showing that any acyclic join can be evaluated with load O(N / p 1/ρ* ), which is asymptotically optimal (there are no polylogarithmic factors in our bound). The separation between cyclic and acyclic joins is yet another phenomenon that is absent in RAM. Our algorithm owes to the discovery of a new mathematical structure — we call “canonical edge cover” — of acyclic hypergraphs, which has numerous non-trivial properties and makes an elegant addition to database theory.

Massively parallel and streaming algorithms for balanced clustering

Clustering is a fundamental problem with a wide range of applications. In this paper, we study a balanced version of the well-known k-center clustering problem, where the objective is to select k centers from a given set of points, and assign to each center at most L of the input points, so as to minimize the maximum distance from a point to the center to which it is assigned. We assume soft capacities, where points are allowed to be selected as center more than once. Motivated by applications involving massive datasets, we study the problem in the massively parallel computation (MPC) and streaming models. In particular, we present a two-round MPC algorithm for the balanced k-center problem, achieving an approximation factor of 5α+2, where α is the approximation factor of the corresponding sequential algorithm. This substantially improves the currently best approximation factor of 32α, available for the problem. We show that the approximation factor of our algorithm can be further improved to 5+ε in all spaces with bounded doubling dimension, including the Euclidean space. We also consider the balanced k-center problem in the streaming model, and present a constant-factor streaming algorithm in any metric space using O(knε) memory, and a factor 5+ε streaming algorithm using O(kpolylogn) memory in doubling metrics.

Artificial-Intelligence integrated circuits: Comparison of GPU, FPGA and ASIC

In the recent years, the boom in technology industries has been greatly accelerated by the development of artificial intelligence (AI). AI, which is based on machine learning (ML), can only be developed rapidly because of the continuously increasing computational capacity of AI processors. Compared to general-purpose processors (GPPs), AI processors have specially designed architectures to accelerate the operations of AI applications, such as convolution, matrix, and massive parallel computing. The objectives of this paper are: (1) to illustrate the differences between general-purpose processors and AI processors; (2) to summarise the characteristic three mainstream AI processors: GPU, FPGA and ASIC, and draw a comparison among them. It shows that GPUs provide very competitive performance with high power consumption; FPGAs can offer high efficiency at low cost; and AISCs provide the highest performance with the lowest power consumption, but cost the most.

Flash-based in-memory computing for stochastic computing in image edge detection

The “memory wall” of traditional von Neumann computing systems severely restricts the efficiency of data-intensive task execution, while in-memory computing (IMC) architecture is a promising approach to breaking the bottleneck. Although variations and instability in ultra-scaled memory cells seriously degrade the calculation accuracy in IMC architectures, stochastic computing (SC) can compensate for these shortcomings due to its low sensitivity to cell disturbances. Furthermore, massive parallel computing can be processed to improve the speed and efficiency of the system. In this paper, by designing logic functions in NOR flash arrays, SC in IMC for the image edge detection is realized, demonstrating ultra-low computational complexity and power consumption (25.5 fJ/pixel at 2-bit sequence length). More impressively, the noise immunity is 6 times higher than that of the traditional binary method, showing good tolerances to cell variation and reliability degradation when implementing massive parallel computation in the array.

Gaussian-Approximated Poisson Preconditioner for Iterative Diagonalization in Real-Space Density Functional Theory

Various real-space methods optimized on massive parallel computers have been developed for efficient large-scale density functional theory (DFT) calculations of materials and biomolecules. The iterative diagonalization of the Hamiltonian matrix is a computational bottleneck in real-space DFT calculations. Despite the development of various iterative eigensolvers, the absence of efficient real-space preconditioners has hindered their overall efficiency. An efficient preconditioner must satisfy two conditions: appropriate acceleration of the convergence of the iterative process and inexpensive computation. This study proposed a Gaussian-approximated Poisson preconditioner (GAPP) that satisfied both conditions and was suitable for real-space methods. A low computational cost was realized through the Gaussian approximation of a Poisson Green's function. Fast convergence was achieved through the proper determination of Gaussian coefficients to fit the Coulomb energies. The performance of GAPP was evaluated for several molecular and extended systems, and it showed the highest efficiency among the existing preconditioners adopted in real-space codes.

Static Evaluation of a Midimew Connected Torus Network for Next Generation Supercomputers

Many artificially intelligent systems solve complex health- and agriculture-related problems that require great computational power. Such systems are used for tracking medical records, genome sequence analysis, image-based plant disease detection, food supply chain traceability, and photosynthesis simulation. Massively parallel computers (MPCs) are among those used to solve these computation-intensive problems. MPCs comprise a million nodes; connecting such a large number of nodes is a daunting task. Therefore, hierarchical interconnection networks (HINs) have been introduced to solve this problem. A midimew-connected torus network (MTN) is a HIN that has basic modules (BM) as torus networks that are connected hierarchically by midimew links. This paper presents the performance of MTNs in terms of static topological parameters and cost-effectiveness, as measured through simulations. An MTN was compared with other networks, including mesh, torus, TESH, TTN, MMN, and TFBN. The results showed that our MTN had a low diameter with a high bisection width and arc connectivity. In addition, our MTN had a high cost–performance trade-off factor (CPTF), a high cost-effective factor (CEF), low packing density, and moderate message-traffic density with marginally higher costs, as compared to other networks, due to wire complexity. However, our MTN provided better bandwidth with higher static fault tolerance. Therefore, MTNs are suggested for further evaluation of the effective implementation of MPCs.

MemCA: All-Memristor Design for Deterministic and Probabilistic Cellular Automata Hardware Realization

Inspired by the behavior of natural systems, Cellular Automata (CA) tackle the demanding long-distance information transfer of conventional computers by the massive parallel computation performed by a set of locally-coupled dynamical nodes. Although CA are envisioned as powerful deterministic computers, their intrinsic capabilities are expanded after the memristor’s probabilistic switching is introduced into CA cells, resulting in new hybrid deterministic and probabilistic memristor-based CA (MemCA). In the proposed MemCA hardware realization, memristor devices are incorporated in both the cell and rule modules, composing the very first all-memristor CA hardware, designed with mixed CMOS/Memristor circuits. The proposed implementation accomplishes high operating speed and reduced area requirements, exploiting also memristor as an entropy source in every CA cell. MemCA’s functioning is showcased in deterministic and probabilistic operation, which can be externally modified by the selection of programming voltage amplitude, without changing the design. Also, the proposed MemCA system includes a reconfigurable rule module implementation that allows for spatial and temporal rule inhomogeneity.

An Analogue In-Memory Ridge Regression Circuit With Application to Massive MIMO Acceleration

In-memory computing (IMC) has emerged as one of the most promising candidates for distributed computing frameworks such as edge computing, owing to its unrivalled energy efficiency and high throughput. By leveraging arrays of emerging devices, such as resistive random access memories (RRAM), to implement massive parallel computation, IMC overcomes the main limitations of classic von Neumann architectures. Meanwhile, next generation telecommunication networks are bound to rely ever more intensively on matrix computations to allow simultaneous transmission and reception over multiple spatial channels, an approach known as Massive Multiple-Input Multiple-Output (MIMO). Here, we propose a closed-loop in-memory computing circuit for the acceleration of Ridge Regression, an algebraic prior that finds application in all phases of a typical massive MIMO transaction, namely channel estimation, uplink and downlink. Particularly, we show the circuit’s capability to perform Zero-Forcing (ZF) and Regularized Zero-Forcing (RZF) detection and beamforming, benchmarking its performance in a realistic framework and comparing results with a commercial graphic processing unit (GPU). Our results indicate a 4 orders-of-magnitude increase in energy efficiency and a 3 orders-of-magnitude increase in area efficiency for the same throughput of a digital solution, supporting IMC for energy efficient pre- and post-processing in next-generation B5G and 6G networks.

Programming approaches for scalability, performance, and portability of combustion physics codes

This paper presents the process, strategy, and results associated with porting a typical combustion physics flow solver to current state-of-the-art and future massively-parallel computer architectures. Major focus is placed on the distinct algorithmic structure of these types of codes and how it can be integrated with modern programming paradigms for heterogeneous platforms (i.e., distributed many-core systems with accelerators). An end-to-end case study is presented that exemplifies the process in a generic manner, which then serves as a clear guide with respect to the strategy and best practices leading to a robust and adaptable framework that performs well, is durable over time, is portable, and requires minimal human-effort. This end is accomplished beginning with the use of a mature, validated, structured, multiblock code framework optimized for application of both Large Eddy Simulation (LES) and Direct Numerical Simulation (DNS). This code has been ported to a variety of platforms over the past decade, including most recently the Oak Ridge Leadership Computing Facility’s “Summit” Platform. The experience gained on these multiple platforms provides general insights and thus the results presented are not specific to any one code or platform other than the overarching trend toward distributed many-core systems with accelerators in order to move toward exascale performance. The resultant performance and scalability of the ported code is demonstrated on a real-world application; a state-of-the-art rotating detonation rocket engine simulation that matches the complex geometry and boundary conditions imposed as part of a companion experimental campaign.

Reliability of arrangement networks in terms of the h-restricted edge connectivity

The h-restricted edge connectivity λh is an important parameter to measure the reliability of massive parallel computing system. The λh of G is the minimum cardinality of a set of edges in G, if any, whose deletion disconnects G, and every remaining component has minimum degree at least h. The arrangement graph is a generalization of the star graph, but its order is more flexible than that of the star graph. This paper investigates the h-restricted edge connectivity of the (n,k)-arrangement graph An,k, and determines that λ1(An,k)=2k(n−k)−2, λ2(An,k)=3k(n−k)−6 with k≤n−2.

Artificial Intelligence: New Frontiers in Real-Time Inverse Scattering and Electromagnetic Imaging

In recent years, artificial intelligence (AI) techniques have been developed rapidly. With the help of big data, massive parallel computing, and optimization algorithms, machine learning (ML) and (more recently) deep learning (DL) strategies have been equipped with enhanced learning and generalization capabilities. Besides becoming an essential framework in image and speech signal processing, AI has also been widely applied to solve several electromagnetic (EM) problems with unprecedented computational efficiency, including inverse scattering (IS) and EM imaging. In this article, a review of the most recent progresses in the application of ML and DL for such problems is given. We humbly hope a brief summary could help us better understand the pros and cons of this research topic and foster future research in using AI to address paramount challenges in the field of EM vision.

Guest Editorial Artificial Intelligence: New Frontiers in Real-Time Inverse Scattering and Electromagnetic Imaging

Understanding and solving complex problems in the physical world has been an intelligent endeavor of humankind. Moreover, the study of artificial intelligence (AI) embodies the dream of designing machines like humans. Research in deep-learning (DL) techniques has attracted much attention in many application areas. With the help of big data technology, massive parallel computing, and fast optimization algorithms, DL has greatly improved the performance of many problems in speech and image processing, power transportation networks, and bio-electromagnetics, among others.

Generic and scalable DNA-based logic design methodology for massive parallel computation

Generic and scalable DNA-based logic design methodology for massive parallel computation

Robust massive parallel information processing environments in biology and medicine: case study

The current frontiers in the description and simulation of advanced physical and biological phenomena observed within all scientific disciplines are pointing toward the importance of the development of robust mathematical descriptions that are error resilient. Complexity research is lacking deeper knowledge of the design methodology of processes that are capable to recreate the robustness, which is going to be studied on massive-parallel computations (MPCs) implemented by cellular automata (CA). A simple, two-state cellular automaton having an extremely simple updating rule, which was created by John H. Conway called the ’Game of Life’ (GoL) is applied to simulate and logic gate using emergents. This is followed by simulations of a robust, generalized GoL, which works with nine states instead of two, that is called R-GoL (open-source). extra states enable higher intercellular communication. The logic gate is simulated by the GoL. It is destroyed by injection of random faulty evaluations with even the smallest probability. The simulations of the R-GoL are initiated with random values. several types of emergent structures, which are robust to injection of random errors, are observed for different setups of the R-GoL rule. The GoL is not robust. The R-GoL is capable to create and maintain oscillating, emergent structures that are robust under constant injection of random, faulty evaluations with up to 1% of errors. The R-GoL express long-range synchronization, which is together with robustness facilitated by designed intercellular communication.

Explicit double-phase-field formulation and implementation for bending behavior of UHPC-NC composite beams

The Phase-field model emerges as a promising method since it can automatically capture the initiation, propagation, coalescence, and branching of cracks without any tracking algorithms. Despite its success in simulating brittle and quasi-brittle cracks, tensile (mode I) and shear (mode II) fractures cannot be distinguished effectively. Moreover, the phase-field model is rarely applied to specimens at the structural level (i.e., reinforced concrete beams), due to the demanding computational cost. To address these shortfalls, two sets of phase-field variables are formulated to describe the tensile and shear fractures of concrete, respectively, through crack-direction-based decomposition of strain energy density into tensile, shear and compressive parts. Computational burdens are greatly alleviated by implementing the model with an explicit finite element solver (Abaqus/Explicit), allowing for massive parallel computing. The sensitivity analysis is performed to investigate the effects of critical parameters with a 1D numerical tensile bar. The double-phase-field model is applied to study the bending behavior of composite beams consisting of normal-strength concrete (NC) overlayer and ultra-high performance concrete (UHPC) substrate. To this end, five UHPC-NC beams, with various thickness ratios of UHPC and NC layers, were prepared and loaded to failure. District bending strengths, cracking patterns, deformation characteristics and failure modes of the four beams were revealed and discussed. It shows that the proposed model is necessary to accurately capture the bending behavior of UHPC-NC composite beams. This model is also applied to discuss the effects of critical parameters on the bending behaviors of UHPC-NC composite beams. To benefit potential users, detailed explanations regarding the data structures of input files and user subroutines (VUEL, VUMAT, VUSDFLD in Abaqus/Explicit) are given and released on GitHub repository.

Distributed Computing in a Pandemic

The current COVID-19 global pandemic caused by the SARS-CoV-2 betacoronavirus has resulted in over a million deaths and is having a grave socio-economic impact, hence there is an urgency to find solutions to key research challenges. Much of this COVID-19 research depends on distributed computing. In this article, I review distributed architectures -- various types of clusters, grids and clouds -- that can be leveraged to perform these tasks at scale, at high-throughput, with a high degree of parallelism, and which can also be used to work collaboratively. High-performance computing (HPC) clusters will be used to carry out much of this work. Several bigdata processing tasks used in reducing the spread of SARS-CoV-2 require high-throughput approaches, and a variety of tools, which Hadoop and Spark offer, even using commodity hardware. Extremely large-scale COVID-19 research has also utilised some of the world's fastest supercomputers, such as IBM's SUMMIT -- for ensemble docking high-throughput screening against SARS-CoV-2 targets for drug-repurposing, and high-throughput gene analysis -- and Sentinel, an XPE-Cray based system used to explore natural products. Grid computing has facilitated the formation of the world's first Exascale grid computer. This has accelerated COVID-19 research in molecular dynamics simulations of SARS-CoV-2 spike protein interactions through massively-parallel computation and was performed with over 1 million volunteer computing devices using the Folding@home platform. Grids and clouds both can also be used for international collaboration by enabling access to important datasets and providing services that allow researchers to focus on research rather than on time-consuming data-management tasks.

A Near-Optimal Parallel Algorithm for Joining Binary Relations

We present a constant-round algorithm in the massively parallel computation (MPC) model for evaluating a natural join where every input relation has two attributes. Our algorithm achieves a load of $\tilde{O}(m/p^{1/\rho})$ where $m$ is the total size of the input relations, $p$ is the number of machines, $\rho$ is the join's fractional edge covering number, and $\tilde{O}(.)$ hides a polylogarithmic factor. The load matches a known lower bound up to a polylogarithmic factor. At the core of the proposed algorithm is a new theorem (which we name the "isolated cartesian product theorem") that provides fresh insight into the problem's mathematical structure. Our result implies that the subgraph enumeration problem, where the goal is to report all the occurrences of a constant-sized subgraph pattern, can be settled optimally (up to a polylogarithmic factor) in the MPC model.