Dynamic Load Balancing Scheme Research Articles

On multiprocessor task scheduling using efficient state space search approaches

Obtaining an optimal schedule for a set of precedence-constrained tasks is a well-known NP-complete problem in its general form. In view of the intractability of the problem, most of the previous work relies on heuristics that try to find reasonably high quality solutions in an acceptable amount of time. While optimal polynomial-time algorithms are known only for a few simple cases (and in other cases can only be obtained through an exhaustive search with prohibitively high time complexity), they may be critically important for applications in which performance is the prime objective. Optimal solutions can also serve as a reference to test the performance of various heuristics. Moreover, an optimal schedule for a program at hand needs to be determined only once (and off-line) but the program using that schedule is in general executed several times. In this paper, we propose optimal algorithms for static scheduling of task graphs with arbitrary parameters to multiple homogeneous processors. The first algorithm is based on the A * search technique and uses a computationally efficient cost function for guiding the search with reduced complexity. Additionally, we propose a number of effective state-pruning techniques to reduce the search space. For further lowering the complexity, we propose an efficient parallelization of the search algorithm. We parallelize the algorithm with reduced interprocessor communication as well as with static and dynamic load-balancing schemes to evenly distribute the search states to the processors. We also propose an approximate algorithm that guarantees a bounded deviation from the optimal solution but executes in a considerably shorter time. Based on an extensive experimental evaluation of the algorithms, we conclude that the parallel algorithm with pruning techniques is an efficient scheme for generating optimal solutions of reasonably large problems while the approximate algorithm is effective if slightly degraded solutions are acceptable.

A Feedback Control Mechanism for Balancing I/O- and Memory-Intensive Applications on Clusters

One common assumption of existing models of load balancing is that the weights of resources and I/O buffer size are statically configured and cannot be adjusted based on a dynamic workload. Though the static configuration of these parameters performs well in a cluster where the workload can be modeled and predicted, its performance is poor in dynamic systems in which the workload is unknown. In this paper, a new feedback control mechanism is proposed to improve overall performance of a cluster with a general and practical workload including I/O-intensive and memory-intensive load. This mechanism is also shown to be effective in complementing and enhancing the performance of a number of existing dynamic load-balancing schemes. To capture the current and past workload characteristics, the primary objectives of the feedback mechanism are: (1) dynamically adjusting the resource weights, which indicate the significance of the resources, and (2) minimizing the number of page faults for memory-intensive jobs while increasing the utilization of the I/O buffers for I/O-intensive jobs by manipulating the I/O buffer size. Results from extensive trace-driven simulation experiments show that compared with a number of schemes with fixed resource weights and buffer sizes, the feedback control mechanism delivers a performance improvement in terms of the mean slowdown by up to 282% (with an average of 125%).

Performance evaluation of a dynamic load-balancing library for cluster computing

The performance of scientific applications in heterogeneous environments has been improved with the research advances in dynamic scheduling at application and runtime system levels. This paper presents the performance evaluation of a library as a result of an integrated approach to dynamic load balancing. This approach combines the advantages of optimising data migration via novel dynamic loop-scheduling strategies with the advances in resource management and task migration capabilities offered by a recently developed parallel runtime system. The performance of the library has been investigated by its use in three scientific applications: the N-body simulations, the profiling of automatic quadrature routines and the solution to the 3D heat equation. The investigations focus on the performance degradation owing to the overhead induced by the runtime system software layer. The experimental results obtained indicate only a slight increase in the cost of load balancing owing to this overhead. The results validate the suitability of the runtime system as an implementation platform for dynamic load-balancing schemes and underscore the significance of using the integrated approach, as well as the benefits of using the library especially in cluster applications characterised by irregular and unpredictable behaviour.

A parallel computing application of the genetic algorithm for lubrication optimization

This study investigated the performance of parallel optimization by means of a genetic algorithm (GA) for lubrication analysis. An air-bearing design was used as the illustrated example and the parallel computation was conducted in a single system image (SSI) cluster, a system of loosely network-connected desktop computers. The main advantages of using GAs as optimization tools are for multi-objective optimization, and high probability of achieving global optimum in a complex problem. To prevent a premature convergence in the early stage of evolution for multi-objective optimization, the Pareto optimality was used as an effective criterion in offspring selections. Since the execution of the genetic algorithm (GA) in search of optimum is population-based, the computations can be performed in parallel. In the cases of uneven computational loads a simple dynamic load-balancing scheme is proposed for optimizing the parallel efficiency. It is demonstrated that the huge amount of computing demand of the GA for complex multi-objective optimization problems can be effectively dealt with by parallel computing in an SSI cluster.

Fault tolerant algorithm based on dynamic and active load balancing for redundant services

A new Some-Read-Any-Write (SRAW) fault tolerant algorithm for redundant services is presented that allows a system to adjust failures dynamically in order to keep the availability and improve the performance. SRAW is based upon dynamic and active load balancing. By introducing dynamic and active load balancing scheme into redundant services, not only the processing speed of requests can be greatly improved, but also the load balancing can be simply and efficiently achieved. Integrated with consistency protocol in this paper, SRAW can also be applied to state services. The performance of SRAW algorithm is also analyzed, and comparisons with other fault tolerant algorithms, especially with RAWA, indicate that SRAW efficiently improves the performance of redundant services with guaranteeing system availability.

Performance evaluation of dynamic load balancing scheme with load prediction mechanism using the load growing acceleration for massively parallel computers

AbstractIn this paper, we will propose a new dynamic load balancing scheme in which, based on the load conditions of the whole system of a massively parallel computer, the respective processing elements (PEs) will perform load balancing control in such a way that the loads are balanced as a whole. In the proposed scheme, although each PE exchanges load information with the neighboring PEs, load balancing is performed from a more global viewpoint, not considering only the load conditions of the surroundings of the transfer‐origin PE: instead, each PE takes into consideration the load distribution in the whole system and its increasing and decreasing trends. Furthermore, in order to determine how the load quantity of the whole system changes, prediction of the load quantity is performed by using the load variation acceleration. By doing so, the transfer‐origin PE is made to appropriately select the transfer destination of the load. Evaluation by simulation demonstrated, a performance enhancement of about 60% maximum compared to the scheme in which only the load quantities of the neighboring PEs are collected. © 2004 Wiley Periodicals, Inc. Syst Comp Jpn, 35(11): 69–79, 2004; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10212

Reducing null messages overhead through load balancing in conservative distributed simulation systems

This paper studies the problem of load balancing for conservative distributed simulations for execution on a multicomputer architecture. The synchronization protocol makes use of Chandy–Misra null messages. We propose a dynamic load balancing algorithm which assumes no compile time knowledge about the workload parameters. It is based upon a process migration mechanism, and the notion of CPU-queue length which indicates the workload at each processor. We examine two variations of the algorithm—referred to as the centralized and multi-level hierarchical methods—and experiment with four simulation workload models to study the scalability of our algorithm. The experiments indicate that our dynamic load balancing schemes significantly reduce the run time of an optimized version of Chandy–Misra null message approach, and decreases by 30–40% the synchronization overhead when compared to the use of a static partitioning algorithm. The results also indicate that the hierarchical version outperforms both the centralized version of the dynamic load balancing algorithm and the static partitioning scheme.

THE DEVELOPMENT OF PARALLEL IMPLEMENTATION FOR A CFD BASED FIRE MODEL UTILISING CONVENTIONAL OFFICE BASED PCs

Parallel processing techniques have been used in the past to provide high performance computing resources for activities such as fire-field modelling. This has traditionally been achieved using specialized hardware and software, the expense of which would be difficult to justify for many fire engineering practices. In this article we demonstrate how typical office-based PCs attached to a Local Area Network has the potential to offer the benefits of parallel processing with minimal costs associated with the purchase of additional hardware or software. It was found that good speedups could be achieved on homogeneous networks of PCs, for example a problem composed of ~100,000 cells would run 9.3 times faster on a network of 12 800MHz PCs than on a single 800MHz PC. It was also found that a network of eight 3.2GHz Pentium 4 PCs would run 7.04 times faster than a single 3.2GHz Pentium computer. A dynamic load balancing scheme was also devised to allow the effective use of the software on heterogeneous PC networks. This scheme also ensured that the impact between the parallel processing task and other computer users on the network was minimized.

A simple reduction of non-uniformity in dynamic load balancing of quantized loads on hypercube multiprocessors and hiding balancing overheads

We reconsider the dimension exchange method (DEM), a known dynamic load balancing scheme on hypercube multiprocessors, for its inefficiency when the load is not divisible in arbitrary sizes but divisible only in a fixed size. We show that a direct application of the DEM to this kind of load may result in nonuniformity (the maximum difference in assigned units of load among the processors after balancing) as large as log N for a hypercube of size N. Since the processing time after the balancing depends on the processor with the maximum number of units, it is desirable to reduce the nonuniformity as small as possible. In this paper, we propose a new simple method, odd even method (OEM), which reduces the nonuniformity to no more than ⌈ 1 2 log N⌉ . The claim is proved and confirmed by enumerating all possible combinations of load for hypercubes of limited sizes using a computer. To estimate the accumulated effect of our balancing method under real-world parallel processing environment, a simulation for hypercube multiprocessors using SLAM II tool is performed. The result shows about 30% improvement in speedup in overall processing. In addition, we introduce new techniques for hiding communication overheads involved in balancing. The basic idea is coalescing some phases of balancing to overlap or pipeline the transmission of load whenever possible. They proved effective in making links busy transmitting load as soon as possible, thus reducing the transmission time. It is shown via simulation that pipelining is powerful even in the presence of severe unevenness of initial load distribution. Combined together, proposed techniques reduce (hide) communication overheads by 15–50% depending on initial load distribution.

Parallel programming of gradient-based iterative image reconstruction schemes for optical tomography

Optical tomography (OT) is a fast developing novel imaging modality that uses near-infrared (NIR) light to obtain cross-sectional views of optical properties inside the human body. A major challenge remains the time-consuming, computational-intensive image reconstruction problem that converts NIR transmission measurements into cross-sectional images. To increase the speed of iterative image reconstruction schemes that are commonly applied for OT, we have developed and implemented several parallel algorithms on a cluster of workstations. Static process distribution as well as dynamic load balancing schemes suitable for heterogeneous clusters and varying machine performances are introduced and tested. The resulting algorithms are shown to accelerate the reconstruction process to various degrees, substantially reducing the computation times for clinically relevant problems.

A novel dynamic load balancing scheme for parallel systems

Adaptive mesh refinement (AMR) is a type of multiscale algorithm that achieves high resolution in localized regions of dynamic, multidimensional numerical simulations. One of the key issues related to AMR is dynamic load balancing (DLB), which allows large-scale adaptive applications to run efficiently on parallel systems. In this paper, we present an efficient DLB scheme for structured AMR (SAMR) applications. This scheme interleaves a grid-splitting technique with direct grid movements (e.g., direct movement from an overloaded processor to an underloaded processor), for which the objective is to efficiently redistribute workload among all the processors so as to reduce the parallel execution time. The potential benefits of our DLB scheme are examined by incorporating our techniques into a SAMR cosmology application, the ENZO code. Experiments show that by using our scheme, the parallel execution time can be reduced by up to 57% and the quality of load balancing can be improved by a factor of six, as compared to the original DLB scheme used in ENZO.

Dynamic Load Balancing of Samr Applications on Distributed Systems

Dynamic load balancing(DLB) for parallel systems has been studied extensively; however, DLB for distributed systems is relatively new. To efficiently utilize computing resources provided by distributed systems, an underlying DLB scheme must address both heterogeneous and dynamic features of distributed systems. In this paper, we propose a DLB scheme for Structured Adaptive Mesh Refinement(SAMR) applications on distributed systems. While the proposed scheme can take into consideration (1) the heterogeneity of processors and (2) the heterogeneity and dynamic load of the networks, the focus of this paper is on the latter. The load-balancing processes are divided into two phases: global load balancing and local load balancing. We also provide a heuristic method to evaluate the computational gain and redistribution cost for global redistribution. Experiments show that by using our distributed DLB scheme, the execution time can be reduced by 9%- to using parallel DLB scheme which does not consider the heterogeneous and dynamic features of distributed systems.

A tool for a two-level dynamic load balancing strategy in scientific applications

This paper describes a dynamic load balancing tool intended for computational investigators who have little familiarity with programming for a message-passing environment. Motivated by the PAR DOALL directive available in some compilers for shared-memory systems, the tool is designed to simplify the manual conversion of sequential programs containing computationally intensive one- or two-dimensional loops with independent iterates into parallel programs that execute with high efficiency on general-purpose clusters. The tool implements a dynamic loop scheduling strategy to address load imbalance which may be induced by the non-uniformity of loop iterate times, and by the heterogeneity of processors. The tool is based on the Message Passing Interface library for wide availability. Experimental results of two scientific applications that utilize the tool on a Linux cluster are presented to demonstrate sample achievable performance, and to underscore the effectiveness of the two-level dynamic load balancing strategy.

Load balancing for unstructured mesh applications

The efficient use of distributed memory parallel systems requires the load on each processor to be well balanced. In cases where the load changes unpredictably during the computation, a dynamic load balancing strategy is needed. Load balancing problems have been studied extensively in recent years, particularly in the context of unstructured mesh based applications. Static load balancing can be approximated by a graph partitioning problem and many efficient algorithms have been developed. Significant progress has also been made in the development of dynamic load balancing algorithms. This paper looks at the history and the state of the art of both classes of algorithms, with a particular emphasis on mesh based applications. However the underlying algorithms, including those for graph partitioning and flow calculation, are sufficiently generic to be applicable to other applications.

Analyzing fuzzy surface modeling using load-balanced computation

Models that can code and render uncertainty support users doing spatial analysis, because they provide direct access to and visualization of the imprecision induced by data compression. Interval or fuzzy models provide an important alternative to traditional approaches, especially for scientists and decision makers eager to explore different scenarios, possibly using different parameters. We found parallel computing to be a key for using advanced modeling techniques. In many cases, however, the high computational cost of even simple interrogation processes might prohibit using fuzzy models: In this context, we present a load-balanced solution that efficiently uses parallel computing to analyze surface models based on the fuzzy B-spline model. We obtained efficient portability of the algorithm by introducing a dynamic and flexible load-balancing strategy that provides good results on different architectures, including a Cray-T3D.

Parallel dynamic load balancing strategies for adaptive irregular applications

Dynamic unstructured mesh adaptation is a powerful technique for solving computational problems with evolving physical features; however, an efficient parallel implementation is rather difficult because of the load imbalance that mesh adaptation creates. To address this problem, we have developed two dynamic load balancing strategies for parallel adaptive irregular applications. The first, called PLUM, is an architecture-independent framework particularly geared toward adaptive numerical computations and requires that all data be globally redistributed after each adaptation to achieve load balance. The second is a more general-purpose topology-independent load balancer that utilizes symmetric broadcast networks (SBN) as the underlying communication pattern, with a goal to providing a global view of system loads across processors. Results indicate that both PLUM and the SBN-based approach have their relative merits, and that they achieve excellent load balance at the cost of minimal extra overhead.

An Object-Oriented Parallel Particle-in-Cell Code for Beam Dynamics Simulation in Linear Accelerators

We present an object-oriented three-dimensional parallel particle-in-cell (PIC) code for simulation of beam dynamics in linear accelerators (linacs). An important feature of this code is the use of split-operator methods to integrate single-particle magnetic optics techniques with parallel PIC techniques. By choosing a splitting scheme that separates the self-fields from the complicated externally applied fields, we are able to utilize a large step size and still retain high accuracy. The method employed is symplectic and can be generalized to arbitrarily high order accuracy if desired. A two-dimensional parallel domain decomposition approach is employed within a message-passing programming paradigm along with a dynamic load balancing scheme. Performance tests on an SGI/Cray T3E-900 and an SGI Origin 2000 show good scalability of the object-oriented code. We present, as an example, a simulation of high current beam transport in the accelerator production of tritium (APT) linac design.

History-driven dynamic load balancing for recurring applications on networks of workstations

The existing dynamic approaches to load balancing on parallel or distributed computer systems are primarily based on the current system load. Often, variations in the future exertable load are neglected or assumed as fixed. In this study, a new dynamic load balancing scheme (DYLOBA) is presented, where both the current system load and the load to be exerted by the application are equally important. The target system chosen is a general purpose network of workstations (NOW). The approach utilizes the past execution statistics of the applications. In this sense, information on the run time system load and resource requirement of the applications, averaged over past executions, is integrated. DYLOBA subsumes that most demanding applications are recurring. Thus, in the long run, it will be in possession of the execution histories of most applications. The system load balances both independent and parallel applications. Once an application joins DYLOBA it is permanently given a unique identification, to monitor it during future executions. DYLOBA was subjected to selective artificial parallel applications. The performance results under DYLOBA showed significant improvements over three other, rather conventional, task assignment schemes: random, fixed, and worst.

Locality-Preserving Load-Balancing Mechanisms for Synchronous Simulations on Shared-Memory Multiprocessors

In the past decade, many synchronous algorithms have been proposed for parallel and discrete simulations. However, the actual performance of these algorithms have been far from ideal, especially when event granularity is small. Barring the case of low parallelism in the given simulation models, one of the main reasons of low speedups is in the uneven load distribution among processors. To amend for this, both static and dynamic load balancing approaches have been proposed. Nevertheless, static schemes based on partitioning of LPs are often subject to the dynamic behavior of the specific simulation models and are therefore application dependent; dynamic load balancing schemes, on the other hand, often suffer from loss of localities and hence cache misses, which could severely penalize on fine-grained event processing. In this paper, we present several new locality-preserving load balancing mechanisms for synchronous simulations on shared-memory multiprocessors. We focus on the type of synchronous simulations where the number of LPs to be processed within a cycle decreases monotonically. We show both theoretically and empirically that some of these mechanisms incur very low overhead. The mechanisms have been implemented by using MIT's Cilk and tested with a number of simulation applications. The results confirm that one of the new mechanisms is indeed more efficient and scalable than common existing approaches.

LOCALITY-PRESERVING LOAD-BALANCING MECHANISMS FOR SYNCHRONOUS SIMULATIONS ON SHARED-MEMORY MULTIPROCESSORS

In the past decade, many synchronous algorithms have been proposed for parallel and discrete simulations. However, the actual performance of these algorithms have been far from ideal, especially when event granularity is small. Barring the case of low parallelism in the given simulation models, one of the main reasons of low speedups is in the uneven load distribution among processors. To amend for this, both static and dynamic load balancing approaches have been proposed. Nevertheless, static schemes based on partitioning of LPs are often subject to the dynamic behavior of the specific simulation models and are therefore application dependent; dynamic load balancing schemes, on the other hand, often suffer from loss of localities and hence cache misses, which could severely penalize on fine-grained event processing. In this paper, we present several new locality-preserving load balancing mechanisms for synchronous simulations on shared-memory multiprocessors. We focus on the type of synchronous simulations where the number of LPs to be processed within a cycle decreases monotonically. We show both theoretically and empirically that some of these mechanisms incur very low overhead. The mechanisms have been implemented by using MIT's Cilk and tested with a number of simulation applications. The results confirm that one of the new mechanisms is indeed more efficient and scalable than common existing approaches.