Heterogeneous Parallel Systems Research Articles

Production Systems with Parallel Heterogeneous Servers of Limited Capacity: Accurate Modeling and Performance Analysis

Heterogeneous systems of limited capacity have general applications in manufacturing, but also in logistic or service systems due to the differences in server or workstation performance or work assignment; this is in close relationship with system flexibility, where saturation and blocking are ordinary situations of systems with high demand and limited capacity, and thus, accurate loss quantification is essential for performance evaluation. Multi-class systems of limited capacity have been studied much less than parallel homogeneous systems (Erlang models). In this context, accurate models for parallel heterogeneous ordered-entry systems were developed: without any prior queue, i.e., M/Mi/c/c, and with a k-capacity queue, i.e., M/Mi/c/c + k. These new matrix models gave an exact state formulation, and their accuracy was verified using discrete event simulation and comparison with literature results. Also, the effect of the queue capacity was studied in relationship to the pattern of service rates. Next, the heterogeneous recirculating system model was also developed with good approximation results. Finally, the proposed models were applied to evaluate systems with non-exponential service times using a new hybrid methodology by combining the Markovian model and the Monte Carlo method (MCM) for normal or lognormal service times, which also yielded useful good approximations to the simulated system.

Age replacement policy for heterogeneous parallel systems

The optimization policy on age replacement mostly focuses on systems comprised of identical components. In this paper, both discrete and continuous time age replacement policies are considered by relaxing the assumption of identical components and working with heterogeneous parallel system, i.e. system with not necessarily identical components. In particular, necessary conditions are obtained for the existence and uniqueness of optimal replacement cycle/time for the parallel system with two nonidentical components under the proposed policy. The extension of the results to a system with more than two components is also presented.

A Heterogeneous Parallel Non-von Neumann Architecture System for Accurate and Efficient Machine Learning Molecular Dynamics

This paper proposes a special-purpose system to achieve high-accuracy and high-efficiency machine learning (ML) molecular dynamics (MD) calculations. The system consists of field programmable gate array (FPGA) and application specific integrated circuit (ASIC) working in heterogeneous parallelization. To be specific, a multiplication-less neural network (NN) is deployed on the non-von Neumann (NvN)-based ASIC (SilTerra 180 nm process) to evaluate atomic forces, which is the most computationally expensive part of MD. All other calculations of MD are done using FPGA (Xilinx XC7Z100). It is shown that, to achieve similar-level accuracy, the proposed NvN-based system based on low-end fabrication technologies (180 nm) is 1.6 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> faster and 10 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{2}$</tex-math> </inline-formula> -10 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{3}\times $</tex-math> </inline-formula> more energy efficiency than state-of-the-art vN-based MLMD using graphics processing units (GPUs) based on much more advanced technologies (12 nm), indicating superiority of the proposed NvN-based heterogeneous parallel architecture.

Assessing Application Efficiency and Performance Portability in Single-Source Programming for Heterogeneous Parallel Systems

We analyze the performance portability of the skeleton-based, single-source multi-backend high-level programming framework SkePU across multiple different CPU–GPU heterogeneous systems. Thereby, we provide a systematic application efficiency characterization of SkePU-generated code in comparison to equivalent hand-written code in more low-level parallel programming models such as OpenMP and CUDA. For this purpose, we contribute ports of the STREAM benchmark suite and of a part of the NAS Parallel Benchmark suite to SkePU. We show that for STREAM and the EP benchmark, SkePU regularly scores efficiency values above 80% and in particular for CPU systems, SkePU can outperform hand-written code.

A Novel Heterogeneous Parallel System Architecture Based EtherCAT Hard Real-Time Master in High Performance Control System

EtherCAT is one of the preferred real-time Ethernet technologies. However, EtherCAT is not applicable in high-end control fields due to real-time constraints. Clock synchronization and cycle time are the most representative limitations. In this paper, a novel Heterogeneous Parallel System Architecture (HPSA) with features of parallel computation and hard real-time is presented. An HPSA-based EtherCAT hard real-time master is developed to significantly improve clock synchronization and shorten cycle time. Traditional EtherCAT masters feature serial processing and run on a PC. This HPSA-based master consists of two parts: EtherCAT master stack (EMS) and EtherCAT operating system (EOS). EMS implements the parallel operation of EtherCAT to realize the shorter cycle time, and EOS brings a hard real-time environment to the HPSA-based master to improve clock synchronization. Furthermore, this HPSA-based master operates on a heterogeneous System-on-a-chip (SoC). EMS and EOS form a heterogeneous architecture inside this SoC to achieve low-latency process scheduling. Experimental results show that in our HPSA-based EtherCAT hard real-time master, the cycle time reaches the sub-50 μs range, and the synchronization error reduces to several nanoseconds. Thus, this HPSA-based master has great application value in high-performance control systems.

HPVM: Hardware-Agnostic Programming for Heterogeneous Parallel Systems

We present Heterogeneous Parallel Virtual Machine (HPVM), a compiler framework for hardware-agnostic programming on heterogeneous compute platforms. HPVM introduces a hardware-agnostic parallel intermediate representation with constructs for the hierarchical task, data, and pipeline parallelism, including dataflow parallelism, and supports multiple front-end languages. In addition, HPVM provides optimization passes that navigate performance, energy, and accuracy tradeoffs, and includes retargetable back ends for a wide range of diverse hardware targets, including central processing units, graphics processing units, domain-specific accelerators, and field-programmable gate arrays. Across diverse hardware platforms, HPVM optimizations provide significant performance and energy improvements, while preserving object-code portability. With these capabilities, HPVM facilitates developers, domain experts, and hardware vendors in programming modern heterogeneous systems.

Performance Estimation of High-Level Dataflow Program on Heterogeneous Platforms by Dynamic Network Execution

The performance of programs executed on heterogeneous parallel platforms largely depends on the design choices regarding how to partition the processing on the various different processing units. In other words, it depends on the assumptions and parameters that define the partitioning, mapping, scheduling, and allocation of data exchanges among the various processing elements of the platform executing the program. The advantage of programs written in languages using the dataflow model of computation (MoC) is that executing the program with different configurations and parameter settings does not require rewriting the application software for each configuration setting, but only requires generating a new synthesis of the execution code corresponding to different parameters. The synthesis stage of dataflow programs is usually supported by automatic code generation tools. Another competitive advantage of dataflow software methodologies is that they are well-suited to support designs on heterogeneous parallel systems as they are inherently free of memory access contention issues and naturally expose the available intrinsic parallelism. So as to fully exploit these advantages and to be able to efficiently search the configuration space to find the design points that better satisfy the desired design constraints, it is necessary to develop tools and associated methodologies capable of evaluating the performance of different configurations and to drive the search for good design configurations, according to the desired performance criteria. The number of possible design assumptions and associated parameter settings is usually so large (i.e., the dimensions and size of the design space) that intuition as well as trial and error are clearly unfeasible, inefficient approaches. This paper describes a method for the clock-accurate profiling of software applications developed using the dataflow programming paradigm such as the formal RVL-CAL language. The profiling can be applied when the application program has been compiled and executed on GPU/CPU heterogeneous hardware platforms utilizing two main methodologies, denoted as static and dynamic. This paper also describes how a method for the qualitative evaluation of the performance of such programs as a function of the supplied configuration parameters can be successfully applied to heterogeneous platforms. The technique was illustrated using two different application software examples and several design points.

A Deterministic Portable Parallel Pseudo-Random Number Generator for Pattern-Based Programming of Heterogeneous Parallel Systems

SkePU is a pattern-based high-level programming model for transparent program execution on heterogeneous parallel computing systems. A key feature of SkePU is that, in general, the selection of the execution platform for a skeleton-based function call need not be determined statically. On single-node systems, SkePU can select among CPU, multithreaded CPU, single or multi-GPU execution. Many scientific applications use pseudo-random number generators (PRNGs) as part of the computation. In the interest of correctness and debugging, deterministic parallel execution is a desirable property, which however requires a deterministically parallelized pseudo-random number generator. We present the API and implementation of a deterministic, portable parallel PRNG extension to SkePU that is scalable by design and exhibits the same behavior regardless where and with how many resources it is executed. We evaluate it with four probabilistic applications and show that the PRNG enables scalability on both multi-core CPU and GPU resources, and hence supports the universal portability of SkePU code even in the presence of PRNG calls, while source code complexity is reduced.

Densenet Scheduling for Heterogeneous Parallel Systems

Densenet Scheduling for Heterogeneous Parallel Systems

Preventive maintenance for heterogeneous parallel systems with two failure modes

The parallel structure in the reliability engineering is widely used in many industrial systems such as power generation systems, pump systems, production systems and computing systems. Designing the optimal maintenance policies for these parallel systems can minimize the operation cost while achieving a satisfying system availability. Previous studies have studied various maintenance optimization for parallel systems. However, little research has considered the multiple failure modes that widely exist in many real industrial systems. This research considers a parallel system that consists of heterogeneous components subject to two types of competing failures: a one-stage catastrophic failure and a two-stage delay-time failure. Inspections are conducted periodically to identify the state of each component. Three maintenance policies are proposed in this research. To find the optimal inspection interval, we analyze the renewal process of the system to minimize the long-term expected cost per unit time. A simulation procedure is proposed to evaluate the long-term expected cost per unit time. Numerical experiments are carried out to illustrate the applications of the proposed maintenance policies. The discussion provides useful insights on choosing the best policy under different system performance requirements.

Design of local heterogeneous system control networks of a new generation with the preservation of the optimality of the main topological functionals of the network

The paper discusses On-Board Computing Control Systems (OBCS) in astronautics, avionics, autonomous mobile devices, robotics, weapons control and multi-core microprocessors. This is sort of a “backbone”, which unites many sensors, calculators, control and executive devices. The architecture of these networks was developed some 30-40 years ago. At that time, these systems met the technical conditions in terms of dynamics and reliability. Nowadays, these systems must perform their functions for 10 to 15 years without maintenance. The performance of system networks must be high enough to solve such tasks as monitoring “swarms” that comprise hundreds of objects or work as a “garbage collectors” in space orbits. Nevertheless modern system networks continue to be based on bus or multi-bus architectures. Since these systems are serial for active nodes, a multi-bus solution is a main way to increase the performance of networks by using very high frequencies that amount to 2 ÷ 4 GHz. It’s an extensive path of development, which is problematic. More acceptable would be an intensive path of development, which, in electronics and computer engineering, is associated with the parallelism of task execution. It means that the operating frequencies may not be ultra-high, not exceeding that of modern devices for frequencies of 10 – 600 MHz. However, such devices should work in a parallel mode. The paper proposes a new approach to designing of heterogeneous parallel control system networks, solving parallel tasks, and a conflict-free management of “passive” nodes. To the best of our knowledge, such control system networks are not available as yet.

Optimizing the LINPACK Algorithm for Large-Scale PCIe-Based CPU-GPU Heterogeneous Systems

There is a widening gap between GPU and other components (CPU, PCIe bus and communication network) in heterogeneous parallel system. The gap forces us to orchestrate cooperative execution among these components much more carefully than ever before. By taking the LINPACK benchmark as a case study, this article proposes a fine-grained pipelining algorithm on large-scale CPU-GPU heterogeneous cluster systems. First, we build an algorithmic model that reveals a new approach to GPU-centric and fine-grained pipelining algorithm design. Then, we present four model-driven pipelining algorithms that incrementally squeeze bubbles in the pipeline so that it is occupied by more useful floating-point calculations. The algorithms are implemented on both the AMD and NVIDIA GPU platforms. The finally optimized LINPACK program achieves 107 PFlops on 25, 600 GPUs (70 percent floating-point efficiency). Several insights have been drawn to suggest tradeoff of algorithm design, programming support, and architecture design.

Editor’s Note: Special Issue on High-level Programming for Heterogeneous Parallel Systems (2019)

Editor’s Note: Special Issue on High-level Programming for Heterogeneous Parallel Systems (2019)

STOCHASTIC ORDERING OF LIFETIMES OF PARALLEL AND SERIES SYSTEMS COMPRISING HETEROGENEOUS DEPENDENT COMPONENTS WITH GENERALIZED BIRNBAUM-SAUNDERS DISTRIBUTIONS

In this paper, we discuss stochastic orderings of lifetimes of two heterogeneous parallel and series systems with heterogeneous dependent components having generalized Birnbaum–Saunders distributions. The comparisons presented here are based on the vector majorization of parameters. The ordering results are established in some special cases for the generalized Birnbaum–Saunders distribution based on the multivariate elliptical, normal, t, logistic, and skew-normal kernels. Further, we use these results by considering Archimedean copulas to model the dependence structure among systems with generalized Birnbaum–Saunders components. These results have been used to derive some upper and lower bounds for survival functions of lifetimes of parallel and series systems.

A parallel DEA-based method for evaluating parallel independent subunits with heterogeneous outputs

Data envelopment analysis (DEA) is a methodology for assessing the relative efficiency of a set of homogeneous decision-making units (DMUs), i.e., a set of DMUs belonging to the same technology and having the same input and output indicators. However, in many practical settings, the assumption of homogeneity does not hold. Especially in a multi-function parallel system, each subunit may not have the same function, so it may produce different outputs (e.g., the professors in a university have the responsibilities of teaching and research which consume same inputs while produce different outputs). How to compare a subunit to others in a multi-function heterogeneous parallel system? Therefore, in this study, we extend the DEA model to consider the one-sided heterogeneous problem in a multi-function parallel structure, handling subunit sets that have heterogeneity in outputs. Here, we propose parallel DEA-based methods and the results show that if the non-existent outputs are replaced with zeros or missing values will lead to overestimate the efficiency of the DMU. Then, an application of our proposed approach to the regional innovation systems’ performance evaluation regarding 30 provinces in China from 2012 to 2016 is provided and concludes with several findings.

A Novel, Highly Integrated Simulator for Parallel and Distributed Systems

In an era of complex networked parallel heterogeneous systems, simulating independently only parts, components, or attributes of a system-under-design is a cumbersome, inaccurate, and inefficient approach. Moreover, by considering each part of a system in an isolated manner, and due to the numerous and highly complicated interactions between the different components, the system optimization capabilities are severely limited. The presented fully-distributed simulation framework (called as COSSIM) is the first known open-source, high-performance simulator that can handle holistically system-of-systems including processors, peripherals and networks; such an approach is very appealing to both Cyber Physical Systems (CPS) and Highly Parallel Heterogeneous Systems designers and application developers. Our highly integrated approach is further augmented with accurate power estimation and security sub-tools that can tap on all system components and perform security and robustness analysis of the overall system under design—something that was unfeasible up to now. Additionally, a sophisticated Eclipse-based Graphical User Interface (GUI) has been developed to provide easy simulation setup, execution, and visualization of results. COSSIM has been evaluated when executing the widely used Netperf benchmark suite as well as a number of real-world applications. Final results demonstrate that the presented approach has up to 99% accuracy (when compared with the performance of the real system), while the overall simulation time can be accelerated almost linearly with the number of CPUs utilized by the simulator.

Appropriate Synchronization Time Allocation for Distributed Heterogeneous Parallel Computing Systems

Parallel computing system components should be harmonized, and this harmonization is kept existent using synchronization time. Synchronization time affects the system in two ways. First, if we have too little synchronization time, some tasks face the problem of harmonization, as they need appropriate time to update and synchronize with the system. Second, if we allocate a large amount of time, stall system created. Random allocation of synchronization time for parallel systems slows down not only the booting time of the system but also the execution time of each application involved in the system. This paper presents a simulator used to test and allocate appropriate synchronization time for distributed and parallel heterogeneous systems. The simulator creates the parallel and heterogeneous system to be evaluated, and lets the user vary the synchronization time to optimize the booting time. NS3-cGEM5 simulator in this paper is formed by HLA-RTI federation integration of the two independent architecture and network simulators - NS3 and cGEM5. Therefore, nodes created on these simulators need synchronizations for harmonized system performance. We tested and allocated the appropriate synchronization time for our sample parallel system composed of one x86 server and three ARM clients.

Performance portability study for massively parallel computational fluid dynamics application on scalable heterogeneous architectures

Patient-specific hemodynamic simulations have the potential to greatly improve both the diagnosis and treatment of a variety of vascular diseases. Portability will enable wider adoption of computational fluid dynamics (CFD) applications in the biomedical research community and targeting to platforms ideally suited to different vascular regions. In this work, we present a case study in performance portability that assesses (1) the ease of porting an MPI application optimized for one specific architecture to new platforms using variants of hybrid MPI＋X programming models; (2) performance portability seen when simulating blood flow in three different vascular regions on diverse heterogeneous architectures; (3) model-based performance prediction for future architectures; and (4) performance scaling of the hybrid MPI＋X programming on parallel heterogeneous systems. We discuss the lessons learned in porting HARVEY, a massively parallel CFD application, from traditional multicore CPUs to diverse heterogeneous architectures ranging from NVIDIA/AMD GPUs to Intel MICs and Altera FPGAs.

Автоматизоване проектування OpenCL програм на основі алгебро-алгоритмічного підходу

Further progress in improving the quality of parallel software development is linked to the use of heterogeneous architectures of parallel computing systems. Heterogeneous parallel systems, in particular, include hybrid computing platforms combining the use of central and graphics processing units. One of the facilities for programming such systems is OpenCL. The paper proposes the further development of previously developed algebra-algorithmic tools in the direction of automated design and synthesis of OpenCL programs. The particular feature of the proposed approach consists in using a high-level language based on Glushkov’s system of algorithmic algebra. The approach is illustrated on the development of a parallel interpolation algorithm, which is the part of the numerical weather forecasting program. The results of the experiment consisting in executing of the generated OpenCL program on a graphics processing unit are given. The program is compared with the implementation for CUDA platform. Problems in programming 2019; 1: 27-36

High-level programming for heterogeneous and hierarchical parallel systems

High-Level Heterogeneous and Hierarchical Parallel Systems (HLPGPU) aims to bring together researchers and practitioners to present new results and ongoing work on those aspects of high-level programming relevant, or specific to general-purpose computing on graphics processing units (GPGPUs) and new architectures. The 2016 HLPGPU symposium was an event co-located with the HiPEAC conference in Prague, Czech Republic. HLPGPU is targeted at high-level parallel techniques, including programming models, libraries and languages, algorithmic skeletons, refactoring tools and techniques for parallel patterns, tools and systems to aid parallel programming, heterogeneous computing, timing analysis and statistical performance models.