Heterogeneous Grid Resources Research Articles

ATLAS Global Shares implementation in PanDA

PanDA (Production and Distributed Analysis) is the workload management system for ATLAS across the Worldwide LHC Computing Grid. While analysis tasks are submitted to PanDA by over a thousand users following personal schedules (e.g. PhD or conference deadlines), production campaigns are scheduled by a central Physics Coordination group based on the organization’s calendar. The Physics Coordination group needs to allocate the amount of Grid resources dedicated to each activity, in order to manage sharing of CPU resources among various parallel campaigns and to make sure that results can be achieved in time for important deadlines. While dynamic and static shares on batch systems have been around for a long time, we are trying to move away from local resource partitioning and manage shares at a global level in the PanDA system. The global solution is not straightforward, given different requirements of the activities (number of cores, memory, I/O and CPU intensity), the heterogeneity of Grid resources (site/HW capabilities, batch configuration and queue setup) and constraints on data locality. We have therefore started the Global Shares project that follows a requirements-driven multi-step execution plan, starting from definition of nestable shares, implementing share-aware job dispatch, aligning internal processes with global shares and finally implementing a pilot stream control for controlling the batch slots that keeps late binding. This contribution will explain the development work and architectural changes in PanDA to implement Global Shares, and describe how the Global Shares project has enabled the central control of resources and significantly reduced manual operations.

A Dynamic Load Balancing Algorithm for Computational Grid using Ant Colony Optimization

Objective: To design and implement an algorithm for load balancing with convenient utilization of heterogeneous grid resources. Methods: In this paper, we introduce Ant based Dynamic Load Balancing Algorithm (ADLBA), a decentralized dynamic load balancing algorithm using Ant Colony Optimization (ACO), which selects the best resources to be allocated to the tasks considering economic cost, resources' capacity, and local load. Results: We used the Gridsim toolkit to evaluate the efficiency of ADLBA against the Randomized Algorithm (RA) with various number of tasks and resource allocation polices. Our study results show that ADLBA outperforms RA in terms of execution cost and total application execution time (makespan), and they also show that using time-shared allocation policy in the resources leads to better results in both algorithms. Conclusion: We found that ADLBA is suitable for grid users which aim to execute their applications quickly with lower cost.

Guranteed Best Effort QoS Algorithm for Task Schedulingon Heterogeneous Grid Resources

Grid Computing enables the flexible sharing with resources among various dynamically created virtual organizations. Task Scheduling in Grid environment is a very important problem. To utilize the Grid resources efficiently, an efficient task scheduling algorithm is helping to schedule the tasks to the resources with reduced makespan. The problem of mapping metatask to the machines is shown to be NP-Complete. The NP-Complete problem can be solved using heuristic approaches. The proposed heuristic approach is based on QoS requirements. This paper proposes a Guaranteed Best Effort QoS Algorithm (GBEQoS) which achieves user satisfaction with minimum time and cost constraint in completing the task execution. The experimental results reveal that the proposed GBEQoS algorithm achieves reduced makespan with higher user satisfaction.

A Novel Dynamic Resilient Ensemble Physical Proximity Resource Management Scheme Adopting for Distributed Environments

Heterogeneous grid resources in hand by completely different organization spread throughout the internet or wide area networks. In existing Distributed Hash Table (DHT) is the organization used for that mapping key to the nodes of a network based on a standardized hashing performs. to meet high scalable and dynamism resilient, thence we have a tendency to propose a Grid resource management theme called ranked Cycloid Overlay design (HCO) that\'s data locality-preserving. In HCO, we have a tendency to establish program/data locality by cluster resources based on their physical proximity and practical matching with user application. We have a tendency to any develop dynamism-resilient resource management rule, cluster-token forwarding rule, and deadline-driven resource management algorithms square measure planned. On adopting these algorithms consumes less overhead, speedy, dynamism-resilient multi resource discovery. The well-organized resource cluster, reduces communications price, and enhances resource discovery success rate flexibly extant in large scale applications.

A Generic Reference Architecture for Collaboratory Scientific Virtual Laboratory

The paper presents a generic reference architecture framework for collaboratory experiment virtual laboratory. The model presented is open source driven, flexible and based on modern tools and technologies. This in effect will allow geographically remote scientists with limited internal laboratory resources, access to wealth of experimental datasets, computing facilities, and distributed hard-to-duplicate laboratory devices. The key issues discussed are architectural design and choice of technology used for creating virtual laboratory. This architecture offers great levels of flexibility, simplicity, and interoperability that are needed to allow integration between heterogeneous distributed grid resources and its clients and executors. The framework, besides theoretical modelling, will provide a road map for future research and open questions.

Novel Adaptive Job Scheduling Algorithm on Heterogeneous Grid Resources

Grid provides an infrastructure for sharing geographically distributed heterogeneous resources to process many applications and mainly used for solving scientific problems that requires more computation time. Problem statement: Grid is a dynamic environment, where the resources may join or leave the environment at any time and the jobs also arrives at different intervals of time. To meet the demands and requirements of the dynamic environment, to maximize the resource utilization and to minimize the makespan an effective grid scheduling technique is needed. Approach: We propose grid architecture as a collection of clusters with multiple worker nodes in each cluster. We propose a new scheduling algorithm Novel Adaptive Decentralized Job Scheduling Algorithm (NADJSA) that applies both Divisible Load Theory (DLT) and Least Cost Method (LCM) and also considers the user demands. Results: The proposed Novel Adaptive Decentralized Job Scheduling Algorithm is compared with the Decentralized Hybrid Job Scheduling Algorithm. Conclusion: The proposed Novel Adaptive Decentralized Job Scheduling Algorithm minimizes the makespan, improves the resource utilization and satisfies the user demands and well suits for the grid environment.

High level QoS-driven model for Grid applications in a simulated environment

This paper presents a model for high-level Quality of Service (QoS) maintenance within business-context domains and associated simulation results achieved via an expansion of the GridSim toolkit. Grid Computing traditionally has been linked with scientific environments, where heterogeneous resources were networked and employed for carrying out compute-intensive and data-intensive scientific experiments or applications that may have not been possible before. The natural progression is that business-oriented applications will build on this success and utilise the large number of heterogeneous Grid resources, potentially available. The success of introducing these applications into the mainstream is directly related to whether Grid Resource Providers can deliver a suitable level of QoS to the Grid Resource Consumer (GRC) and the ability of the GRC to request high level QoS such as the numbers of CPUs required or the RAM required, on demand. Moreover, we present dynamically calculated metrics for measuring QoS such as reliability, using up-to-date information on resources. We introduce a novel model, Business Grid Quality of Service (BGQoS), for a new generation of commercial and business-oriented Grid applications which may wish to make use of Grid environments. BGQoS allows GRCs to specify varying types of high level QoS requirements which are delivered via querying up-to-date resource information, matchmaking and monitoring operations. In addition, testing is required and this has posed a problem where testing on physical Grid test-beds is either impractical or not viable economically. Simulation is therefore important.

A survey of economic models in grid computing

Grid computing offers the network of large scale computing resources. Economic models are effective in collaborating large scale heterogeneous grid resources that are typically owned by different organizations. Not all the models provide same benefits for users in utilizing the resources. Similarly, the profit earned by resource providers also differs for different economic models. We survey the economic models used in grid computing since its inception until 2010. We discuss their advantages and disadvantages and analyze their suitability for usage in a dynamic grid environment. To the best of our knowledge, no such survey has been conducted in the literature up to now.

Evaluating Heuristics for Scheduling Dependent Jobs in Grid Computing Environments

Job scheduling plays a critical role in the utilisation of grid resources by mapping a number of jobs to grid resources. However, the heterogeneity of grid resources adds some challenges to the work of job scheduling, especially when jobs have dependencies which can be represented as Direct Acyclic Graphs (DAGs). It is widely recognised that scheduling m jobs to n resources with an objective to achieve a minimum makespan has shown to be NP-complete, requiring the development of heuristics. Although a number of heuristics are available for job scheduling optimisation, selecting the best heuristic to use in a given grid environment remains a difficult problem due to the fact that the performance of each original heuristic is usually evaluated under different assumptions. This paper evaluates 12 representative heuristics for dependent job scheduling under one set of common assumptions. The results are presented and analysed, which provides an even basis in comparison of the performance of those heuristics. To facilitate performance evaluation, a DAG simulator is implemented which provides a set of tools for DAG job configuration, execution, and monitoring. The components of the DAG simulator are also presented in this paper.

The User-Level Scheduling of Divisible Load Parallel Applications With Resource Selection and Adaptive Workload Balancing on the Grid

This paper presents a hybrid resource management environment, operating on both application and system levels developed for minimizing the execution time of parallel applications with divisible workload on heterogeneous grid resources. The system is based on the adaptive workload balancing algorithm (AWLB) incorporated into the distributed analysis environment (DIANE) user-level scheduling (ULS) environment. The AWLB ensures optimal workload distribution based on the discovered application requirements and measured resource parameters. The ULS maintains the user-level resource pool, enables resource selection and controls the execution. We present the results of performance comparison of default self-scheduling used in DIANE with AWLB-based scheduling, evaluate dynamic resource pool and resource selection mechanisms, and examine dependencies of application performance on aggregate characteristics of selected resources and application profile.

S-MDS: Semantic Monitoring and Discovery System for the Grid

Due to the dynamic characteristics and heterogeneity of Grid resources, effective and accurate Grid resource monitoring and discovery are a challenging problem. To deal with this, the semantic Grid extends the current Grid by giving information and services well-defined meaning through machine-processable descriptions. In this paper, we present a monitoring and discovery system that is based on the semantic Grid built on the top of the WS-Resource Framework. The system provides an infrastructure to create, aggregate, and maintain semantic metadata of Grid resources so that they can be semantically queried using Resource Description Framework query languages such as SPARQL. We present the system architecture and optimization methods to maintain and store the semantic metadata. Through a system implementation based on the Globus Toolkit software, we show the practicality and simplicity of the framework. We also present evaluation results showing the effectiveness and robustness of the system.

Hierarchy-based resource discovery algorithm in grid environment

In order to solve the problems of conventional resource discovery framework, a hierarchy-based model was discussed, which combined super-peer model and contact node, a membership join protocol was introduced, and a resource discovery algorithm was proposed based on Time To Live (TTL) limit and cache list. The network framework with good scalability can adapt to complexity and heterogeneity of grid resources. The simulation shows that the algorithm works well, query time is minimized and efficiency of resource discovery is improved in grid environment.

Interoperability of GADU in Using Heterogeneous Grid Resources for Bioinformatics Applications

Bioinformatics tools used for efficient and computationally intensive analysis of genetic sequences require large-scale computational resources to accommodate the growing data. Grid computational resources such as the Open Science Grid and TeraGrid have proved useful for scientific discovery. The genome analysis and database update system (GADU) is a high-throughput computational system developed to automate the steps involved in accessing the Grid resources for running bioinformatics applications. This paper describes the requirements for building an automated scalable system such as GADU that can run jobs on different Grids. The paper describes the resource-independent configuration of GADU using the Pegasus-based virtual data system that makes high-throughput computational tools interoperable on heterogeneous Grid resources. The paper also highlights the features implemented to make GADU a gateway to computationally intensive bioinformatics applications on the Grid. The paper will not go into the details of problems involved or the lessons learned in using individual Grid resources as it has already been published in our paper on genome analysis research environment (GNARE) and will focus primarily on the architecture that makes GADU resource independent and interoperable across heterogeneous Grid resources.

A Grid-based Virtual Reactor: Parallel performance and adaptive load balancing

We address the problem of porting parallel distributed applications from static homogeneous cluster environments to dynamic heterogeneous Grid resources. We introduce a generic technique for adaptive load balancing of parallel applications on heterogeneous resources and evaluate it using a case study application: a Virtual Reactor for simulation of plasma chemical vapour deposition. This application has a modular architecture with a number of loosely coupled components suitable for distribution over the Grid. It requires large parameter space exploration that allows using Grid resources for high-throughput computing. The Virtual Reactor contains a number of parallel solvers originally designed for homogeneous computer clusters that needed adaptation to the heterogeneity of the Grid. In this paper we study the performance of one of the parallel solvers, apply the technique developed for adaptive load balancing, evaluate the efficiency of this approach and outline an automated procedure for optimal utilization of heterogeneous Grid resources for high-performance parallel computing.

An architecture to support scalable distributed virtual environment systems on grid

A Distributed Virtual Environment (DVE) system offers a computer-generated virtual world in which individuals located at different places in the physical world can interact with one another. In order to achieve real-time response for a large user base, DVE systems need to have a scalable architecture. In this paper, we present the design of a grid-enabled service oriented framework for facilitating the construction of scalable DVE systems on computing grids. A service component called "gamelet" is proposed, whose distinctive mark is its high mobility for supporting dynamic load sharing. We propose a gamelet migration protocol which can ensure the transparency and efficiency of gamelet migration, and an adaptive gamelet load-balancing (AGLB) algorithm for making gamelet redistribution decisions at runtime. The algorithm considers both the synchronization costs of the DVE system and network latencies inherent in the grid nodes. The activities of the users and the heterogeneity of grid resources are also considered in order to carry out load sharing more effectively. We evaluate the performance of the proposed mechanisms through a multiplayer online game prototype implemented using the Globus toolkit. The results show that our approach can achieve faster response times and higher throughputs than some existing approaches.

GridSim: a toolkit for the modeling and simulation of distributed resource management and scheduling for Grid computing

AbstractClusters, Grids, and peer‐to‐peer (P2P) networks have emerged as popular paradigms for next generation parallel and distributed computing. They enable aggregation of distributed resources for solving large‐scale problems in science, engineering, and commerce. In Grid and P2P computing environments, the resources are usually geographically distributed in multiple administrative domains, managed and owned by different organizations with different policies, and interconnected by wide‐area networks or the Internet. This introduces a number of resource management and application scheduling challenges in the domain of security, resource and policy heterogeneity, fault tolerance, continuously changing resource conditions, and politics. The resource management and scheduling systems for Grid computing need to manage resources and application execution depending on either resource consumers' or owners' requirements, and continuously adapt to changes in resource availability.The management of resources and scheduling of applications in such large‐scale distributed systems is a complex undertaking. In order to prove the effectiveness of resource brokers and associated scheduling algorithms, their performance needs to be evaluated under different scenarios such as varying number of resources and users with different requirements. In a Grid environment, it is hard and even impossible to perform scheduler performance evaluation in a repeatable and controllable manner as resources and users are distributed across multiple organizations with their own policies. To overcome this limitation, we have developed a Java‐based discrete‐event Grid simulation toolkit called GridSim. The toolkit supports modeling and simulation of heterogeneous Grid resources (both time‐ and space‐shared), users and application models. It provides primitives for creation of application tasks, mapping of tasks to resources, and their management. To demonstrate suitability of the GridSim toolkit, we have simulated a Nimrod‐G like Grid resource broker and evaluated the performance of deadline and budget constrained cost‐ and time‐minimization scheduling algorithms. Copyright © 2002 John Wiley & Sons, Ltd.