Execution Of Parallel Applications Research Articles

A neural network framework for optimizing parallel computing in cloud servers

Energy efficiency has become a major focus in optimizing hardware resource usage for Cloud servers. One approach widely employed to enhance the execution of parallel applications is thread-level parallelism (TLP) exploitation. This technique leverages multiple threads to improve computational efficiency and performance. However, the increasing heterogeneity of resources in cloud environments and the complexity of selecting the optimal configuration for each application pose a significant challenge to cloud users due to the massive number of possible configurations and the need to effectively harness TLP in diverse hardware setups to achieve optimal energy efficiency and performance. To address this challenge, we propose TLP-Allocator, an artificial neural network (ANN) optimization strategy that uses hardware and software metrics to build and train an ANN model. It predicts worker node and thread count combinations that provide optimal energy-delay product (EDP) results. In experiments using ten well-known applications on a private cloud with heterogeneous resources, we show that TLP-Allocator predicts combinations that yield EDP values close to the best achieved by an exhaustive search. It also improves the overall EDP by 38.2% compared to state-of-the-art workloads scheduling on cloud environments.

Improving the efficiency of graph algorithm executions on high‐performance computing

SummaryThe growing need for extracting information from large graphs has been pushing the development of parallel graph algorithms. However, the highly irregular structure of the real‐world graphs limits the performance and energy improvements of graph applications. In this paper, we show that, in most cases, using all the available cores of the multiprocessor is not the best option in terms of the aforementioned non‐functional requirements. Based on that, we propose GraphKat, a framework that enables the simultaneous processing of several algorithms/graphs instead of executing them serially (i.e., one after another), increasing efficiency in terms of performance and energy. GraphKat works in two steps: (i) it characterizes the graph applications with a specific number of threads based on their efficiency levels; and (ii) it defines the execution order of all graph applications in the target system. Experimental results on three multicore processors (Intel and AMD) show that GraphKat improves the overall system's efficiency related to performance (up to ) and energy‐saving (up to 245.21), and reduces the graph applications' execution time (up to ) and energy consumption (up to 6.64) compared to the default execution of parallel applications on HPC systems.

Towards scaling community detection on distributed-memory heterogeneous systems

In most real-world networks, nodes/vertices tend to be organized into tightly-knit modules known as communities or clusters such that nodes within a community are more likely to be connected or related to one another than they are to the rest of the network. Community detection in a network (graph) is aimed at finding a partitioning of the vertices into communities. The goodness of the partitioning is commonly measured using modularity. Maximizing modularity is an NP-complete problem. In 2008, Blondel et al. introduced a multi-phase, multi-iteration heuristic for modularity maximization called the Louvain method. Owing to its speed and ability to yield high quality communities, the Louvain method continues to be one of the most widely used tools for serial community detection.Distributed multi-GPU systems pose significant challenges and opportunities for efficient execution of parallel applications. Graph algorithms, in particular, have been known to be harder to parallelize on such platforms, due to irregular memory accesses, low computation to communication ratios, and load balancing problems that are especially hard to address on multi-GPU systems.In this paper, we present our ongoing work on distributed-memory implementation of Louvain method on heterogeneous systems. We build on our prior work parallelizing the Louvain method for community detection on traditional CPU-only distributed systems without GPUs. Corroborated by an extensive set of experiments on multi-GPU systems, we demonstrate competitive performance to existing distributed-memory CPU-based implementation, up to 3.2× speedup using 16 nodes of OLCF Summit relative to two nodes, and up to 19× speedup relative to the NVIDIA RAPIDS® cuGraph® implementation on a single NVIDIA V100 GPU from DGX-2 platform, while achieving high quality solutions comparable to the original Louvain method. To the best of our knowledge, this work represents the first effort for community detection on distributed multi-GPU systems. Our approach and related findings can be extended to numerous other iterative graph algorithms on multi-GPU systems.

Optimizing the EDP of OpenMP applications via concurrency throttling and frequency boosting

The increasing number of cores in multicore architectures has brought the need for better use of hardware resources. Consequently, different solutions have become widely adopted to optimize the execution of parallel applications: dynamic concurrency throttling (DCT) and turbo-boosting. While DCT artificially reduces the number of active threads to improve the performance and energy consumption, turbo-boosting increases the processor’s frequency above the base operating levels to provide maximum performance, considering the Thermal Design Power (TDP). However, as many parallel applications are unbalanced, combining both optimization knobs is not a straightforward task. Therefore, we propose TBFT. It is a transparent and automatic approach that concurrently exploits DCT and turbo-boosting to optimize OpenMP parallel applications. For that, TBFT implements a DCT Engine to find the ideal or near-ideal number of threads in parallel regions and a Boosting Engine to adjust the boosting operating mode w.r.t. the CPU workload. When executing twelve well-known benchmarks on three multicore systems, we show that TBFT outperforms different state-of-the-art strategies without jeopardizing the performance.

Smart resource allocation of concurrent execution of parallel applications

AbstractThread‐level parallelism (TLP) has been widely exploited to optimize computational resource usage in high‐performance systems. However, as many applications do not scale as the number of threads increase, resources will be wasted when the application executes with the maximum possible number of threads (i.e., the default execution) rather than fewer threads (thread throttling) that may use the resources more efficiently. Hence, instead of executing only one application with as many threads as possible, one can run more applications simultaneously by applying thread throttling to each one. The primary outcome of this strategy is a significant reduction in the total execution time and energy consumption when the system needs to execute a list of applications. Given that, we propose a smart resource allocation (SRA) for concurrent parallel application execution. It automatically finds the ideal degree of TLP for each application and guides the simultaneous parallel applications execution. When running 25 well‐known benchmarks on three multicore systems and comparing SRA to state‐of‐the‐art strategies (e.g., Batch, Equal policy, and Scalability), SRA improves the EDP by 87.4% over the Batch strategy; 75.5% over the Equal policy; and 38.8% over the scalability strategy.

A systematic methodology to migrate complex real-time software systems to multi-core platforms

This paper proposes a systematic three-stage methodology for migrating complex real-time industrial software systems from single-core to multi-core computing platforms. Single-core platforms have limited computational capabilities that prevent integration of computationally demanding applications such as image processing within the existing system. Modern multi-core processors offer a promising solution to address these limitations by providing increased computational power and allowing parallel execution of different applications within the system. However, the transition from traditional single-core to contemporary multi-core computing platforms is non-trivial and requires a systematic and well-defined migration process. This paper reviews some of the existing migration methods and provides a systematic multi-phase migration process with emphasis on software architecture recovery and transformation to explicitly address the timing and dependability attributes expected of industrial software systems. The methodology was evaluated using a survey-based approach and the results indicate that the presented methodology is feasible, useable and useful for real-time industrial software systems.

Dynamic Load Balancing in Parallel Execution of Cellular Automata

The allocation of the computational load across different processing elements is an important issue in parallel computing. Indeed, an unbalanced load distribution can strongly affect the performances of a parallel system caused by an excess of synchronization idle times due to less loaded processes waiting for more loaded ones. In this article, we focus on the load balancing issues in the context of the parallel execution of spatial-related applications where the domain space is partitioned in regions that are assigned to different processing elements. In particular, without loss of generality, we consider the well-known spatial-related Cellular Automata computational paradigm for evaluating the proposed dynamic load balancing approach. The main contribution of this article is the derivation of simple closed-form expressions that allow to compute the optimal workload assignment in a dynamic fashion, with the goal of guaranteeing a fully balanced workload distribution during the parallel execution. Based on these expressions, an algorithm for balanced execution of cellular automata is presented and implemented using the MPI technology. Eventually, an experimental section practically shows the behaviour of the proposed dynamic load balancing approach and proves its performance improvement, compared to the not-balanced version, as witnessed by the appreciable reduction of execution times.

Towards Optimal Matrix Partitioning for Data Parallel Computing on a Hybrid Heterogeneous Server

Optimal partitioning of a square computational domain over several heterogeneous processors, balancing the load of the processors and minimizing the inter-processor communication cost, is crucial for data parallel dense linear algebra and other applications having similar communication pattern on modern hybrid servers. Although a solution has been found for two processors, the cases of three and more processors are still open. The state of-the-art solution for three processors uses an approximation communication cost function which fails to accurately account for the total amount of data moved between processors, leaving thus the question of its global optimality unanswered. In this work, we formulate and solve a mathematical problem of optimal partitioning a real-valued square over three heterogeneous processors using a new cost function, which accurately accounts for the total amount of data communicated between processors. We also develop an original method for accurate experimental evaluation of the communication time of data movement between memories of the compute devices in the hybrid platform during the execution of data parallel applications. We successfully use this method in the experimental validation of our mathematical results. Finally, we propose a communication energy model predicting the dynamic energy consumption of data movement between processors and experimentally validate its accuracy. This model predicts, and the experiments confirm, that the performance-optimal partition is not necessarily energy optimal.

Adaptive Task Allocation and Scheduling on NoC-based Multicore Platforms with Multitasking Processors

The application workloads in modern multicore platforms are becoming increasingly dynamic. It becomes challenging when multiple applications need to be executed in parallel in such systems. Mapping and scheduling of these applications are critical for system performance and energy consumption, especially in Network-on-Chip– (NoC) based multicore systems. These systems with multitasking processors offer a better opportunity for parallel application execution. Mapping solutions generated at design time may be inappropriate for dynamic workloads. To improve the utilization of the underlying multicore platform and cope with the dynamism of application workload, often task allocation is carried out dynamically. This article presents a hybrid task allocation and scheduling strategy that exploits the design-time results at runtime. By considering the multitasking capability of the processors, communication energy, and timing characteristics of the tasks, different allocation options are obtained at design time. During runtime, based on the availability of the platform resources and application requirements, the design-time allocations are adapted for mapping and scheduling of tasks, which result in improved runtime performance. Experimental results demonstrate that the proposed approach achieves an on average 11.5%, 22.3%, 28.6%, and 34.6% reduction in communication energy consumption as compared to CAM [18], DEAMS [4], TSMM [38], and CPNN [32], respectively, for NoC-based multicore platforms with multitasking processors. Also, the deadline satisfaction of the tasks of allocated applications improves on an average by 32.8% when compared with the state-of-the-art dynamic resource allocation approaches.

Evaluation of memory performance in NUMA architectures using Stochastic Reward Nets

Understanding memory performance in multi-core platforms is a prerequisite to perform optimizations. To this end, this paper presents analytical models based on Stochastic Reward Nets (SRNs) to model and evaluate the memory performance of Non-Uniform Memory Access (NUMA) multi-core architectures. The approach considers the details of the architecture and first proposes a monolithic SRN model that evaluates the memory performance in terms of the mean memory response time. Since the monolithic model incurs a state space explosion with an increasing number of cores and memory controllers, two approximate models are presented that are able to evaluate large-scale NUMA architectures. The SRNs are validated through measurements on two NUMA multi-core platforms, a 64-core AMD Opteron server and a 72-core Intel system. The results demonstrate the ability of the proposed models to accurately compute the mean memory response time on NUMA architectures. The results also provide valuable information that runtime systems and application designers can use to optimize execution of parallel applications on such architectures.

The main scientific and technical problems of using hybrid HPC clusters in materials science

The article discusses the use of hybrid HPC clusters for the execution of software designed to calculate the electronic structure and atomic scale materials modeling. Modern software systems, which are designed to solve the problems of materials science, use the capabilities of various hardware computing accelerators to increase productivity. The use of such computing technologies requires the adaptation of application program code to hybrid computing architectures, which include classic central processing units (CPUs) and specialized graphics accelerators (GPUs).The use of large computing hybrid systems requires the development of methods for ensuring the workloading of such computing systems that will allow efficient use of computing resources and avoid equipment downtime.First of all, these methods should allow parallel execution of user applications using computational accelerators. However, in practice, software environments designed to solve application problems cannot be deployed in the same computing environment due to software incompatibility. In order to overcome this limitation and ensure the parallel execution of diverse types of materials science tasks, the creation of individual task execution environments based on virtualization technologies and cloud technologies.The continuation of virtualization technologies and the provision of cloud services is the construction of digital platforms. The article proposes the use of a digital platform for hosting scientific materials science services that provide calculations using various application software systems. Digital platforms make it possible to provide a unified user interface to scientific materials science services. The platform provides opportunities for finding the necessary scientific services, transferring source data and results between users, the platform and hybrid high-performance clusters.

Adaptive relaxed synchronization through the use of supervised learning methods

Several authors have proposed the use of relaxed synchronization to speed up the execution of parallel applications that admit tradeoffs between quality and execution time. However, most of these works propose the complete removal of synchronization primitives and do not anticipate the quality of the results to be obtained with different input data. In this paper, we propose a novel strategy for relaxing synchronization, evaluating the feasibility of using supervised learning methods to ensure that the relaxed synchronization technique provides results within acceptable limits of error. We use a varied set of program inputs to create a control base, providing data for the training of supervised learning methods. When the user wishes to execute his/her application with new input data (in the same execution environment), the trained classification algorithm will suggest the relax factor that is best suited for the triple application/input/execution environment. Using this methodology, we obtained a gain of 3.5x for the K-means algorithm applied to videos while maintaining the desired quality rate.

A weighted resource discovery approach in grid computing

Purpose Grid computing is an effective environment for the execution of parallel applications that requires great computing power. This paper aims to present, based on the hierarchical architecture, an improved weighted resource discovery (WRD) algorithm to manage allocation of resources and minimize cost of communications between grid nodes. Design/methodology/approach A behavioral modeling method is addressed to prove the proposed method correctness. The behavioral model of the proposed algorithm is implemented by StarUML tool with two different model-checking mechanisms. Then, the resource discovery correctness is analyzed in terms of reachability condition, fairness condition and deadlock-free using NuSMV model checker. Findings The results show that WRD algorithm has better performance in requiring re-discovery process, the number of examined nodes in each request and discovering the free resources with high-bandwidth links. Originality/value To store information of resources, a new data structure called resource information table is proposed which facilitates resource finding of the algorithm. A behavioral modeling method is addressed to prove the proposed method correctness.

An approach for realistically simulating the performance of scientific applications on high performance computing systems

Scientific applications often contain large, computationally-intensive, and irregular parallel loops or tasks that exhibit stochastic behavior leading to load imbalance. Load imbalance often manifests during the execution of parallel scientific applications on large and complex high performance computing (HPC) systems. The extreme scale of HPC systems on the road to Exascale computing only exacerbates the loss in performance due to load imbalance. Dynamic loop self-scheduling (DLS) techniques are instrumental in improving the performance of scientific applications on HPC systems via load balancing. Selecting a DLS technique that results in the best performance for different problem and system sizes requires a large number of exploratory experiments. Currently, a theoretical model that can be used to predict the scheduling technique that yields the best performance for a given problem and system has not yet been identified. Therefore, simulation is the most appropriate approach for conducting such exploratory experiments in a reasonable amount of time. However, conducting realistic and trustworthy simulations of application performance under different configurations is challenging. This work devises an approach to realistically simulate computationally-intensive scientific applications that employ DLS and execute on HPC systems. The proposed approach minimizes the sources of uncertainty in the simulative experiments results by bridging the native and simulative experimental approaches. A new method is proposed to capture the variation of application performance between different native executions. Several approaches to represent the application tasks (or loop iterations) are compared to establish their influence on the simulative application performance. A novel simulation strategy is introduced that applies the proposed approach, which transforms a native application code into simulative code. The native and simulative performance of two computationally-intensive scientific applications that employ eight task scheduling techniques (static, nonadaptive dynamic, and adaptive dynamic) are compared to evaluate the realism of the proposed simulation approach. The comparison of the performance characteristics extracted from the native and simulative performance shows that the proposed simulation approach fully captured most of the performance characteristics of interest. This work shows and establishes the importance of simulations that realistically predict the performance of DLS techniques for different applications and system configurations.

Upgrading a high performance computing environment for massive data processing

High-performance computing (HPC) and massive data processing (Big Data) are two trends that are beginning to converge. In that process, aspects of hardware architectures, systems support and programming paradigms are being revisited from both perspectives. This paper presents our experience on this path of convergence with the proposal of a framework that addresses some of the programming issues derived from such integration. Our contribution is the development of an integrated environment that integretes (i) COMPSs, a programming framework for the development and execution of parallel applications for distributed infrastructures; (ii) Lemonade, a data mining and analysis tool; and (iii) HDFS, the most widely used distributed file system for Big Data systems. To validate our framework, we used Lemonade to create COMPSs applications that access data through HDFS, and compared them with equivalent applications built with Spark, a popular Big Data framework. The results show that the HDFS integration benefits COMPSs by simplifying data access and by rearranging data transfer, reducing execution time. The integration with Lemonade facilitates COMPSs’s use and may help its popularization in the Data Science community, by providing efficient algorithm implementations for experts from the data domain that want to develop applications with a higher level abstraction.

A green SLA constrained scheduling algorithm for parallel/scientific applications in heterogeneous cluster systems

Cloud services utilize Heterogeneous Clusters as the hardware infrastructure at the backend. Minimal energy consumption in Heterogeneous Cluster System (HCS) is an important but challenging job. It can bring numerous benefits such as drop in operational cost, enhanced system reliability and green environment. Scheduling on HCS, with the objectives of minimizing the energy consumption, is being addressed by the research community at large. This work proposes a scheduling algorithm with green Service Level Agreement (SLA) constrained execution of parallel applications. The green SLA assumes that the tasks are to be scheduled on Dynamic Voltage Frequency Scaling (DVFS) enabled HCS. The proposed algorithm acts in two phases: first, it minimizes the energy consumption for the slacks of the schedule using DVFS technique, and second, a green SLA is made between a customer and service provider to maintain a good energy-performance trade-off. With this strategy, makespan of the schedule is extended a bit but the energy consumption is minimized for the period of extended makespan. It is observed that the decrease in one level of voltage and frequency of high-speed Processing Element (PE) saves more energy compared to that of low-speed PE. To utilize this, jobs running on higher voltage and frequency level of the PE are prioritized first. The performance of the proposed algorithm is studied by simulation on several standard Directed Acyclic Graphs (DAGs) and few other well-known real-world scientific workflows. The comparative study exhibits the effectiveness of the proposed algorithm over state of the art on four real heterogeneous PEs.

A permanent fault tolerant dynamic task allocation approach for Network-on-Chip based multicore systems

Rapid advancement in deep sub-micron regime has made the integration of multiple processing elements possible on a single chip. This has enabled parallel execution of applications on Network-on-Chip (NoC) based multiprocessor platforms. Task mapping and scheduling play crucial roles in timing response and energy consumption of such systems. Tasks present in these applications can be of mixed critical in nature with different importance. However, the on-chip processors executing these tasks of any given application might fail during runtime. Fault tolerance becomes challenging when real-time applications with mixed critical requirements are hosted on such fault prone environment. The complexity of the problem is further magnified in dynamic scenarios when such real-time applications can enter or leave the multicore platform at any time instant. Although several prior works have addressed fault tolerant resource allocation for mixed critical applications, few of these consider permanent processor faults. In this work, an improved fault tolerant resource allocation strategy is presented to mitigate the effect of permanent processor faults on mixed critical applications. The proposed algorithm offers a runtime solution to the unified problem of fault tolerant mapping and scheduling for real-time applications. Both the temporal property of the tasks and the timing information of the faults have been considered while implementing a suitable fault tolerance strategy that reduces the communication energy consumption and provides an improved level of quality of service for the executing applications. A detailed evaluation of the performance of the proposed algorithm has been conducted for different applications. On comparing with other state-of-the-art fault tolerant approaches, the proposed policy shows 28.5% average reduction in communication energy consumption while achieving 34.7% improved quality of service. Additionally, the proposed scheme shows better scalability in comparison to the recent techniques reported in literature.

Using Proactive Fault-Tolerance Approach to Enhance Cloud Service Reliability

The large-scale utilization of cloud computing services for hosting industrial/enterprise applications has led to the emergence of cloud service reliability as an important issue for both cloud service providers and users. To enhance cloud service reliability, two types of fault tolerance schemes, reactive and proactive, have been proposed. Existing schemes rarely consider the problem of coordination among multiple virtual machines (VMs) that jointly complete a parallel application. Without VM coordination, the parallel application execution results will be incorrect. To overcome this problem, we first propose an initial virtual cluster allocation algorithm according to the VM characteristics to reduce the total network resource consumption and total energy consumption in the data center. Then, we model CPU temperature to anticipate a deteriorating physical machine (PM). We migrate VMs from a detected deteriorating PM to some optimal PMs. Finally, the selection of the optimal target PMs is modeled as an optimization problem that is solved using an improved particle swarm optimization algorithm. We evaluate our approach against five related approaches in terms of the overall transmission overhead, overall network resource consumption, and total execution time while executing a set of parallel applications. Experimental results demonstrate the efficiency and effectiveness of our approach.

Parallelization of space-aware applications: Modeling and performance analysis

Many applications in fields like sociology, biology and urban computing, need to cope with an explicit use of a spatial environment, or territory. Such applications, referred to as space-aware applications (SAAs), are based on a set of entities that live and operate in a territory. Parallel execution of space-aware applications is needed to improve the performance when the demand of computational resources increases. Despite the great interest towards SAAs, there is a lack of models and theoretical results for assessing and predicting their execution performance. This paper presents a novel framework, based on Stochastic Time Petri nets, which is able to capture the execution dynamics of parallel SAAs, and model the aspects related to computation, synchronization and communication. The framework has been validated by comparing the predicted performance results for a testbed application, i.e., the ant clustering and sorting algorithm, to those experienced on a real execution platform. An extensive set of experiments have been performed to analyze the impact on the performance of some important parameters, among which the number of parallel nodes and the ratio between computation and communication load.

A Workflow Management System for Scalable Data Mining on Clouds

The extraction of useful information from data is often a complex process that can be conveniently modeled as a data analysis workflow. When very large data sets must be analyzed and/or complex data mining algorithms must be executed, data analysis workflows may take very long times to complete their execution. Therefore, efficient systems are required for the scalable execution of data analysis workflows, by exploiting the computing services of the Cloud platforms where data is increasingly being stored. The objective of the paper is to demonstrate how Cloud software technologies can be integrated to implement an effective environment for designing and executing scalable data analysis workflows. We describe the design and implementation of the Data Mining Cloud Framework (DMCF), a data analysis system that integrates a visual workflow language and a parallel runtime with the Software-as-a-Service (SaaS) model. DMCF was designed taking into account the needs of real data mining applications, with the goal of simplifying the development of data mining applications compared to generic workflow management systems that are not specifically designed for this domain. The result is a high-level environment that, through an integrated visual workflow language, minimizes the programming effort, making easier to domain experts the use of common patterns specifically designed for the development and the parallel execution of data mining applications. The DMCF's visual workflow language, system architecture and runtime mechanisms are presented. We also discuss several data mining workflows developed with DMCF and the scalability obtained executing such workflows on a public Cloud.