Opportunities For Parallelism Research Articles

LUAEMA: A Loop Unrolling Approach Extending Memory Accessing for Vector Very-Long-Instruction-Word Digital Signal Processor with Multiple Register Files

Loop unrolling can provide more instruction-level parallelism opportunities for code and enables a greater range of instruction pipeline scheduling. In high-performance very-long-instruction-word (VLIW) digital signal processors (DSPs), there are special registers to address. To further improve the instruction-level parallelism of code for such DSPs by making full use of these registers, in this paper, we propose a more effective loop unrolling approach through extending memory accessing (LUAEMA). In this approach, the final unrolling factor is computed by a model in which every register kind and every memory accessing operation are considered. For basic digital signal processing algorithms, the unrolling factor under the LUAEMA is larger than that under the conventional loop unrolling approach. We also provide the opportunity to reduce the number of instructions in a loop during the code transformation of loop unrolling. The experimental results show that the loop unrolling approach proposed in this paper can achieve an average speedup ratio ranging from 1.14 to 1.81 compared with the conventional loop unrolling approach. For some algorithms, the peak speedup ratio is up to 2.11.

DECA: Discrete Event inspired Cellular Automata for grain structure prediction in additive manufacturing

Microstructure largely dictates macroscopic material properties and is strongly affected by processing. Therefore, the simulation of microstructure evolution in response to thermal fields during processing is of significant interest within the computational materials science community. Additive manufacturing (AM) has emerged as a technique for producing complex geometries and unique microstructures. Yet, complex and rapid thermal cycles in AM pose computational challenges for existing microstructure models. This work proposes a discrete event inspired cellular automata (CA) approach, titled DECA, to accelerate simulation of grain structure evolution in AM. In contrast to conventional time-stepped CA models, this model directly solves the times capture events would take place allowing for stepping in events rather than time (a technique also found in the field of discrete-event simulation). In comparison to purely serial discrete-event models, DECA allows for temporary violation of the causality constraint, but detects and corrects these violations, leading to an emergent phenomenon dubbed causality rippling, in which previously calculated capture events are overwritten. The amount of repeated calculations, defined by the capture ratio, is taken as a measure of computational inefficiency, and the model parameters that affect this ratio are evaluated. The new DECA approach was found to be more computationally efficient than conventional time-stepped CA models while guaranteeing an accurate solution, which can only be achieved in the conventional models for vanishingly small time steps. Finally, opportunities for parallelization and scaling of the new approach are discussed.

DAHA: Accelerating GNN Training with Data and Hardware Aware Execution Planning

Graph neural networks (GNNs) have been gaining a reputation for effective modeling of graph data. Yet, it is challenging to train GNNs efficiently. Many frameworks have been proposed but most of them suffer from high batch preparation cost and data transfer cost for mini-batch training. In addition, existing works have limitations on the device utilization pattern, which results in fewer opportunities for pipeline parallelism. In this paper, we present DAHA, a GNN training framework with data and hardware aware execution planning to accelerate end-to-end GNN training. We first propose a data and hardware aware cost model that is lightweight and gives accurate estimates on per-operation time cost for arbitrary input and hardware settings. Based on the cost model, we further explore the optimal execution plan for the data and hardware with three optimization strategies with pipeline parallelism: (1) group-based in-turn pipelining of batch preparation neural training to explore more optimization opportunities and prevent batch preparation bottlenecks; (2) data and hardware aware rewriting for intra-batch execution planning to improve computation efficiency and create more opportunities for pipeline parallelism; and (3) inter-batch scheduling to further boost the training efficiency. Extensive experiments demonstrate that DAHA can consistently and significantly accelerate end-to-end GNN training and generalize to different message-passing GNN models.

Fast data-dependence profiling through prior static analysis

Data-dependence profiling is a program-analysis technique for detecting parallelism opportunities in sequential programs. It captures data dependences that actually occur during program execution, filtering parallelism-preventing dependences that purely static methods assume only because they lack critical runtime information, such as the values of pointers and array indices. Profiling, however, suffers from high runtime overhead. In our earlier work, we accelerated data-dependence profiling by excluding polyhedral loops that can be handled statically using certain compilers and eliminating scalar variables that create statically-identifiable data dependences. In this paper, we combine the two methods and integrate them into DiscoPoP, a data-dependence profiler and parallelism discovery tool. Additionally, we detect reduction patterns statically and unify the three static analyses with the DiscoPoP framework to significantly diminish the profiling overhead and for a wider range of programs. We have evaluated our unified approaches with 49 benchmarks from three benchmark suites and two computer simulation applications. The evaluation results show that our approach reports fewer false positive and negative data dependences than the original data-dependence profiler and reduces the profiling time by at least 43%, with a median reduction of 76% across all programs. Also, we identify 40% of reduction cases statically and eliminate the associated profiling overhead for these cases.

Time‐parallel integration and phase averaging for the nonlinear shallow‐water equations on the sphere

We describe a proof‐of‐concept development and application of a phase‐averaging technique to the nonlinear rotating shallow‐water equations on the sphere, discretised using compatible finite‐element methods. Phase averaging consists of averaging the nonlinearity over phase shifts in the exponential of the linear wave operator. Phase averaging aims to capture the slow dynamics in a solution that is smoother in time (in transformed variables), so that larger timesteps may be taken. We overcome the two key technical challenges that stand in the way of studying the phase averaging and advancing its implementation: (1) we have developed a stable matrix exponential specific to finite elements and (2) we have developed a parallel finite averaging procedure. Following recent studies, we consider finite‐width phase‐averaging windows, since the equations have a finite timescale separation. In our numerical implementation, the averaging integral is replaced by a Riemann sum, where each term can be evaluated in parallel. This creates an opportunity for parallelism in the timestepping method, which we use here to compute our solutions. Here, we focus on the stability and accuracy of the numerical solution. We confirm that there is an optimal averaging window, in agreement with theory. Critically, we observe that the combined time discretisation and averaging error is much smaller than the time discretisation error in a semi‐implicit method applied to the same spatial discretisation. An evaluation of the parallel aspects will follow in later work.

Arbitrarily Parallelizable Code: A Model of Computation Evaluated on a Message-Passing Many-Core System

The number of processing elements per solution is growing. From embedded devices now employing (often heterogeneous) multi-core processors, across many-core scientific computing platforms, to distributed systems comprising thousands of interconnected processors, parallel programming of one form or another is now the norm. Understanding how to efficiently parallelize code, however, is still an open problem, and the difficulties are exacerbated across heterogeneous processing, and especially at run time, when it is sometimes desirable to change the parallelization strategy to meet non-functional requirements (e.g., load balancing and power consumption). In this article, we investigate the use of a programming model based on series-parallel partial orders: computations are expressed as directed graphs that expose parallelization opportunities and necessary sequencing by construction. This programming model is suitable as an intermediate representation for higher-level languages. We then describe a model of computation for such a programming model that maps such graphs into a stack-based structure more amenable to hardware processing. We describe the formal small-step semantics for this model of computation and use this formal description to show that the model can be arbitrarily parallelized, at compile and runtime, with correct execution guaranteed by design. We empirically support this claim and evaluate parallelization benefits using a prototype open-source compiler, targeting a message-passing many-core simulation. We empirically verify the correctness of arbitrary parallelization, supporting the validity of our formal semantics, analyze the distribution of operations within cores to understand the implementation impact of the paradigm, and assess execution time improvements when five micro-benchmarks are automatically and randomly parallelized across 2 × 2 and 4 × 4 multi-core configurations, resulting in execution time decrease by up to 95% in the best case.

Parallel block Neo-Hookean XPBD using graph clustering

The eXtended Position Based Dynamics algorithm (XPBD) enables unified simulation of various materials from fluids to both elastic solids and stiff solids. In particular, finite element based neo-Hookean models can simulate near incompressible materials by means of a decoupled compliant constraint formulation. Due to XPBD’s reliance on local constraint projections in the solver loop, its computational nature lends itself to parallelization by means of graph coloring algorithms used to determine partitions of independent constraints which can be solved simultaneously. However, minimal graph coloring is bounded from below by the maximum valence of the finite element mesh, thus hindering parallelization opportunities. In this paper, we propose a novel graph clustering approach on the constraint graph which groups highly dependent constraints into supernodes. By applying graph coloring on the supernodal constraint graph, we are able to significantly reduce the number of partitions, thus enhancing parallelization of the solver. Furthermore, we accelerate convergence of the neo-Hookean XPBD solver by a coupled constraint formulation, resulting in enhanced stability and efficiency compared to previous approaches.

Potential of Dual Drug Delivery Systems: MOF as Hybrid Nanocarrier for Dual Drug Delivery in Cancer Treatment

AbstractSynergistic action of combination therapy can be enhanced by appropriate selection of drugs combination that might reduce the likelihood acquired resistance to chemotherapy in cancer. Nano based formulations offer a parallel opportunity to design advance combination drug delivery systems with specific internal and external stimulus. In recent years, there has been a lot of interest in the study of metal–organic frameworks (MOFs) for biomedical applications. Because of their extraordinary chemical properties such as tunable porous structure with high surface area MOFs are an infancy material used in drug delivery systems as a nacarrier. MOFs are regarded as a promising class of nanocarriers for drug delivery owing to well‐defined structure, ultrahigh surface area and porosity, tunable pore size, and easy chemical functionalization. Herein we summarize the most recent progress in stimuli‐based dual drug‐delivery systems including inorganic and organic nanocarriers. Moreover, we have highlighted the specific stimuli responsive nano‐MOFs as dual drug delivery systems, which offer a fresh perspective for development of hybrid nanocarrier to overcome the drawbacks of pure organic and inorganic carriers, such as toxicity, lower payload capacity and burst effect.

VNF Embedding and Assignment for Network Function Parallelism

The emergence of Network Function Virtualization (NFV) and Service Function Chaining (SFC) together enable flexible and agile network management and traffic engineering. Recently, Network Function Parallelism (NFP) has been proposed to break the need for sequential services and, hence, significantly reduce the latency of SFC. While some studies have investigated how to identify parallel paths to enhance the efficiency of parallelism, less effort has been paid to efficient network function embedding with consideration of parallelism opportunities. Hence, this work aims at solving the instance deployment problem so as to prevent parallel functions from waiting for each other before continuing to the next stage. As it is extremely difficult to optimize VNF (Virtual Network Functions) embedding for uncertain parallelism opportunities, we propose a practical embedding scoring mechanism to enhance the likelihood of low-cost function parallelism. Our scoring design tends to cluster independent functions for efficient parallelism while ensuring load balancing. After instance deployment, we then assign function instances to parallel chains so as to minimize their end-to-end latency while balancing the workload of instances. Our evaluation results show that the proposed scoringbased embedding scheme ensures homogeneous delays of parallel functions and, hence, reduces the end-to-end latency for NFP by up to 22% as compared to the embedding algorithm without considering parallelism opportunities.

Synthesizing fine-grained synchronization protocols for implicit monitors

A monitor is a widely-used concurrent programming abstraction that encapsulates all shared state between threads. Monitors can be classified as being either implicit or explicit depending on the primitives they provide. Implicit monitors are much easier to program but typically not as efficient. To address this gap, there has been recent research on automatically synthesizing explicit-signal monitors from an implicit specification, but prior work does not exploit all paralellization opportunities due to the use of a single lock for the entire monitor. This paper presents a new technique for synthesizing fine-grained explicit-synchronization protocols from implicit monitors. Our method is based on two key innovations: First, we present a new static analysis for inferring safe interleavings that allow violating mutual exclusion of monitor operations without changing its semantics. Second, we use the results of this static analysis to generate a MaxSAT instance whose models correspond to correct-by-construction synchronization protocols. We have implemented our approach in a tool called Cortado and evaluate it on monitors that contain parallelization opportunities. Our evaluation shows that Cortado can synthesize synchronization policies that are competitive with, or even better than, expert-written ones on these benchmarks.

Distributive Justice as a Means of Combating Systemic Racism in Healthcare

Distributive Justice as a Means of Combating Systemic Racism in Healthcare

Fast data series indexing for in-memory data

Data series similarity search is a core operation for several data series analysis applications across many different domains. However, the state-of-the-art techniques fail to deliver the time performance required for interactive exploration, or analysis of large data series collections. In this work, we propose MESSI, the first data series index designed for in-memory operation on modern hardware. Our index takes advantage of the modern hardware parallelization opportunities (i.e., SIMD instructions, multi-socket and multi-core architectures), in order to accelerate both index construction and similarity search processing times. Moreover, it benefits from a careful design in the setup and coordination of the parallel workers and data structures, so that it maximizes its performance for in-memory operations. MESSI supports similarity search using both the Euclidean and dynamic time warping (DTW) distances. Our experiments with synthetic and real datasets demonstrate that overall MESSI is up to 4x faster at index construction and up to 11x faster at query answering than the state-of-the-art parallel approach. MESSI is the first to answer exact similarity search queries on 100GB datasets in \(\sim \)50 ms (30–75 ms across diverse datasets), which enables real-time, interactive data exploration on very large data series collections.

An accurate and fair evaluation methodology for SNN-based inferencing with full-stack hardware design space explorations

Artificial Neural Networks (ANNs) achieve high accuracy in various cognitive tasks (i.e., inferences), but often fail to meet power and latency budgets due to intensive computational overheads. To address the challenge, Spiking Neural Networks (SNNs) have emerged as high-performance and power-efficient alternatives thanks to their theoretically efficient spike-driven computations. The spike-based computations have a high potential of achieving cost-effective inferencing with their low-precision data representations, simple neuron operations, and new parallelization opportunities. To determine which network (i.e., ANN or SNN) is suitable for which purposes, it is essential to accurately evaluate the cost-effectiveness of an SNN and compare it to the corresponding ANN. However, existing studies overestimate or underestimate the cost-effectiveness of SNNs over ANNs as they consider only the limited design points and compare SNNs against naive ANN hardware baselines.In this study, we propose a full-stack SNN evaluation methodology to accurately evaluate the cost-effectiveness of SNNs. Quantifying the potential of SNNs is highly challenging as the evaluations require full-stack knowledge on SNNs’ unique computational features and how each affects the mechanisms of the underlying hardware. For this purpose, we identify five representative SNN-specific design points that affect hardware performance and demonstrate the impact of each design point with the quantified experimental results. Next, we modify the existing ANN accelerator to support the identified SNN-specific design points and implement a cycle-accurate simulator to evaluate how each point affects the overall cost-effectiveness. As a case study, we evaluate SNNs converted from pretrained ANNs and compare them against the ANN counterparts using our simulator. Our study is the first work to accurately evaluate the cost-effectiveness of SNNs and make fair comparisons against ANNs. In addition, our methodology provides important guidelines for the next-generation SNN accelerator designs.

High-Utilization, High-Flexibility Depth-First CNN Coprocessor for Image Pixel Processing on FPGA

Recently, CNNs are increasingly exploited for pixel processing tasks, such as denoising, which opens up new challenges due to the increased activation and operation count. This article presents a CNN coprocessor architecture to solve these challenges on field-programmable gate array (FPGA) through four main contributions. First, the I/O communication between the host processor and the FPGA is reduced to a minimum using a depth-first (DF) principle. Three new DF approaches are presented. Second, to ensure high throughput, the increased parallelization opportunities of the proposed line-based DF operation are analyzed. Third, introducing programmability to the compute array is introduced to enable a broad deployment while maintaining high utilization of the available multipliers digital signal processings (DSPs), independently of the kernel dimensions and without control of the host processor. This is in contrast with many state-of-the-art FPGA implementations, focusing on only one algorithm and/or one kernel topology. Fourth, a model is built to investigate the influence of architecture parameters and show the benefits of DF. The scalable design can be deployed on a wide range of FPGAs, maintaining 78%-93% DSP utilization across all algorithms (denoising, optical flow, depth estimation, segmentation, and super-resolution) and FPGA platforms. Up to 695 GOPS is achieved on a Zynq XCZU9EG board, matching state-of-the-art performance with a more flexible design. The throughput is compared with other pixel processing architectures on FPGA.

Scaling up public mental health care in Sub-Saharan Africa: insights from infectious disease.

Models estimate that the disability burden from mental disorders in Sub-Saharan Africa (SSA) will more than double in the next 40 years. Similar to HIV, mental disorders are stigmatized in many SSA settings and addressing them requires community engagement and long-term treatment. Yet, in contrast to HIV, the public mental healthcare cascade has not been sustained, despite robust data on scalable strategies. We draw on findings from our International AIDS Society (IAS) 2020 virtual workshop and make recommendations for next steps in the scale up of the SSA public mental healthcare continuum. Early HIV surveillance and care cascade targets are discussed as important strategies for HIV response in SSA that should be adopted for mental health. Advocacy, including engagement with civil society, and targeted economic arguments to policymakers, are reviewed in the context of HIV success in SSA. Parallel opportunities for mental disorders are identified. Learning from HIV, communication of strategies that advance mental health care needs in SSA must be prioritized for broad global audiences. The COVID-19 pandemic is setting off a colossal escalation of global mental health care needs, well-publicized across scientific, media, policymaker, and civil society domains. The pandemic highlights disparities in healthcare access and reinvigorates the push for universal coverage. Learning from HIV strategies, we must seize this historical moment to improve the public mental health care cascade in SSA and capitalize on the powerful alliances ready to be forged. As noted by Ambassador Goosby in our AIDS 2020 workshop, 'The time is now'.

Exploiting Parallelism Opportunities with Deep Learning Frameworks

State-of-the-art machine learning frameworks support a wide variety of design features to enable a flexible machine learning programming interface and to ease the programmability burden on machine learning developers. Identifying and using a performance-optimal setting in feature-rich frameworks, however, involves a non-trivial amount of performance profiling efforts and often relies on domain-specific knowledge. This article takes a deep dive into analyzing the performance impact of key design features in a machine learning framework and quantifies the role of parallelism. The observations and insights distill into a simple set of guidelines that one can use to achieve much higher training and inference speedup. Across a diverse set of real-world deep learning models, the evaluation results show that the proposed performance tuning guidelines outperform the Intel and TensorFlow recommended settings by 1.30× and 1.38×, respectively.

Defining mass transfer in a capillary wave micro-bioreactor for dose-response and other cell-based assays

For high-throughput cell culture and associated analytics, droplet-based cultivation systems open up the opportunities for parallelization and rapid data generation. In contrast to microfluidics with continuous flow, sessile droplet approaches enhance the flexibility for fluid manipulation with less operational effort. Generating biologically favorable conditions and promoting cell growth in a droplet, however, is particularly challenging due to mass transfer limitations, which has to be solved by implementing an effective mixing technique.Here, capillary waves induced by vertical oscillation are used to mix inside a sessile droplet micro-bioreactor (MBR) avoiding additional moving parts inside the fluid. Depending on the excitation frequency, different patterns are formed on the oscillating liquid surface, which are described by a model of a vibrated sessile droplet. Analyzing mixing times and oxygen transport into the liquid, a strong dependency of mass transfer on the oscillation parameters, especially the excitation frequency, is demonstrated. Oscillations at distinct capillary wave resonant frequencies lead to rapid homogenization with mixing times of 2 s and volumetric liquid-phase mass transfer coefficients of more than 340 h−1. This shows that the mass transfer in a droplet MBR can be specifically controlled via capillary waves, what is subsequently demonstrated for cultivations of Escherichia coli BL21 cells. Therefore, the presented MBR with vertical oscillation mixing is a promising analytical tool to perform cellular assays and experiments in parallel with intensified and tailored mass transfer.

Pure Functions in C: A Small Keyword for Automatic Parallelization

The need for parallel task execution has been steadily growing in recent years since manufacturers mainly improve processor performance by increasing the number of installed cores instead of scaling the processor’s frequency. To make use of this potential, an essential technique to increase the parallelism of a program is to parallelize loops. Several automatic loop nest parallelizers have been developed in the past such as PluTo. The main restriction of these tools is that the loops must be statically analyzable which, among other things, disallows function calls within the loops. In this article, we present a seemingly simple extension to the C programming language which marks functions without side-effects. These functions can then basically be ignored when the automatic parallelizer checks the parallelizability of loops. We integrated the approach into the GCC compiler toolchain and evaluated it by running several real-world applications. Our experiments show that the C extension helps to identify additional parallelization opportunities and, thus, to significantly increase the performance of applications.

Binary-level data dependence analysis of hot execution regions using abstract interpretation at runtime.

With the widespread of multicore systems, automatic parallelization becomes more pronounced, particularly for legacy programs, where the source code is not generally available. An essential operation in any parallelization system is detecting data dependence among parallelization candidate instructions. Conducting dependence analysis at the binary-level is more challenging than that at the source-level due to the much lower semantics of the binary code. In this paper, we consider using the elaborate 'static' analysis of abstract interpretation, for the first time, at runtime for data dependence detection. Specifically, our system interprets instructions at a hot region, while at the same time, collect programs semantics for seen program points, thereby conducting abstract interpretation analysis dynamically. The analysis is guaranteed to be correct as long as execution does not exit the region prematurely. Moreover, successive hot region re-entries will resume previous analysis, albeit much faster in case no major change in the program semantics. Such approach provides for more rigorous analysis than other simple dynamic analysis which would typically miss parallelization opportunities. The proposed approach also does not require any hardware support, availability of the source code, as well as any code re-compilation. To study the performance and accuracy of our approach, we have extended the Padrone dynamic code modification framework, and conduct an initial study on a set of PolyBench kernels and selected programs from SPEC CPU. Experimental results show accurate dependence detection with low overhead.

LShape Partitioning: Parallel Skyline Query Processing using MapReduce

A skyline query searches the data points that are not dominated by others in the dataset. It is widely adopted for many applications which require multi-criteria decision making. However, skyline query processing is considerably time-consuming for a high-dimensional large scale dataset. Parallel computing techniques are therefore needed to address this challenge, among which MapReduce is one of the most popular frameworks to process big data. A great number of efficient MapReduce skyline algorithms have been proposed in the literature and most of their designs focus on partitioning and pruning the given dataset. However, there are still opportunities for further parallelism. In this study, we propose two parallel skyline processing algorithms using a novel LShape partitioning strategy and an effective Propagation Filtering method. These two algorithms are 2Phase LShape and 1Phase LShape, used for multiple reducers and single reducer, respectively. By extensive experiments, we verify that our algorithms outperformed the state-of-the-art approaches, especially for high-dimensional large scale datasets.