Heterogeneous Grid Research Articles

A grid computing based approach for the power system dynamic security assessment

This paper addresses the problem of parallel dynamic security assessment applications from static homogeneous cluster environment to dynamic heterogeneous grid environment. Functional parallelism and data parallelism are supported by each of the message passing interface model and TCP/IP model. To consider the differences in heterogeneous computing resources and complexity of large-scale power system communities, a kernel-based multilevel algorithm is proposed for network partitioning. Since the bottleneck in distributed computation is low speed network communication, a bi-level latency exploitation technique is introduced for numerically solving system differential equations. The proposed grid-based implementation includes the core simulation engine, grid computing middleware, a Python interface and Python front-end utilities. Tests for a 39-bus network, a 4000-bus network and a 10,000-bus network are reported, and the results of these experiments demonstrate that the proposed scheme is able to execute the distributed simulations on computational grid infrastructure and provide efficient parallelism.

Grid Independent Task Scheduling Multi-Objective Optimization Model and Genetic Algorithm

The characteristic of heterogeneous grid environment results in that the task scheduling is constrained by a number of factors such as the length of scheduling, the performance of security, and the cost of scheduling , etc. Firstly, under the consideration of the task scheduling demand for secure performance. A security benefit function and the corresponding time cost model are constructed. Secondly , according to the past behavior of grid resource nodes, a simple and efficient node’s credibility dynamic evaluation model is built by using the weighted function. Thirdly, b ased on these, a new constrained multi-objective grid task scheduling mode is proposed. In order to solve this model, the relationship between the characteristic of the task and the different performance among nodes is used to define the individual relationship matrix; then the subjection degree function is used to transform each relationship matrix into a fuzzy matrix . A ccording to the different effects of each objective in the final objective , each weight in the final decision-making is determined. By doing so, the multi-objective model is transformed into a single-objective model. Through the design of new crossover operator and an uniform mutation operator, a genetic algorithm called MUGA for the transformed problem is proposed. Finally, s imulation results show that the proposed algorithm is better than the compared ones in terms of the length of the task scheduling, security efficiency value, reliability and scheduling costs.

Parallel Quasirandom Applications on Heterogeneous Grid

In this paper we present error and performance analysis of quasi-Monte Carlo algorithms for solving multidimensional integrals (up to 100 dimensions) on the grid using MPI. We take into account the fact that the Grid is a potentially heterogeneous computing environment, where the user does not know the specifics of the target architecture. Therefore parallel algorithms should be able to adapt to this heterogeneity, providing automated load-balancing. Monte Carlo algorithms can be tailored to such environments, provided parallel pseudorandom number generators are available. The use of quasi-Monte Carlo algorithms poses more difficulties. In both cases the efficient implementation of the algorithms depends on the functionality of the corresponding packages for generating pseudorandom or quasirandom numbers. We propose efficient parallel implementation of the Sobol sequence for a grid environment and we demonstrate numerical experiments on a heterogeneous grid. To achieve high parallel efficiency we use a newly developed special grid service called Job Track Service which provides efficient management of available computing resources through reservations.

Supporting Fault-Tolerance for Time-Critical Events in Distributed Environments

In this paper, we consider the problem of supporting fault tolerance foradaptiveandtime-criticalapplications in heterogeneous and unreliable grid computing environments. Our goal for this class of applications is to optimize a user-specifiedbenefit functionwhile meeting the time deadline. Our first contribution in this paper is a multi-objective optimization algorithm for scheduling the application onto the most efficient and reliable resources. In this way, the processing can achieve the maximum benefit while also maximizing thesuccess-rate, which is the probability of finishing execution without failures. However, for the cases where failures do occur, we have developed ahybrid failure recoveryscheme to ensure that the application can complete within the pre-specified time interval. Our experimental results show that our scheduling algorithm can achieve better benefit when compared to several heuristics-based greedy scheduling algorithms, while still having a negligible overhead. Benefit is further improved when we apply the hybrid failure recovery scheme, and the success-rate becomes 100%.

New Approach to Dynamic and Seamless Management of Network and Computational Resources in a Heterogeneous Grid Environment

Progress in the field of distributed computing and development of advanced Grid services has a significant influence on the concrete results of cooperation between distributed computing centers in various parts of the world. Advanced applications using remote access to distributed computing resources, data repositories and laboratories impose strong requirements on data networks. Very often these requirements can be met only by technologically advanced optical networks. Guaranteed availability of bandwidth for data transfer between computing nodes, low and stable over time and the minimum packet loss are just some of the parameters that are important for the new generation networks. Moreover, the problem of scalability, particularly in the context of an increase of the size of Grid systems with high requirements to the network, significantly affects the complexity of infrastructure management systems, mainly due to the heterogeneity of the environment and the potential administrative divisions. In order to increase processing efficiency of distributed computing, it is necessary to automate processes related to the management of infrastructure components. This is particularly important in the context of Grid environments connected via optical networks. Depending on the type of computing application, it is a network layer which is responsible for allocating inter-connections between computing sites with particular requirements on duration and the requested bandwidth. The possibility of a dynamic allocation of connections between computing sites is an essential element which allows for the efficient and effective use of optical networks in complex computational processes.

DockFlow: Achieving interoperability of protein docking tools across heterogeneous Grid middleware

Enabling the seamless integration between applications executing on heterogeneous Grid middleware poses a number of challenges to both application scientists and middleware developers. We highlight some of these challenges in the context of designing and implementing DockFlow. DockFlow is a virtual screening environment integrating four Grid-based protein docking tools that execute on different Grid middleware technologies at different locations. We propose extensions that can be applied to any Grid-based workflow system to support the run-time interoperability between the available tools. The extensions are generic, and as an example we describe how they have been implemented in the InforSense workflow system. We also present experimental results that evaluate the tradeoffs between performance and usability of the proposed methods.

The use of scintillometry for validating aggregation schemes over heterogeneous grids

A number of studies have been devoted to derive the diurnal course of regional evapotranspiration (ET) especially in semi-arid areas where the assessment of this term is of crucial importance for water resources management. One approach to derive regional evapotranspiration is based on the use of aggregation schemes in conjunction with energy-balance or land-surface models. However, the effectiveness of this approach cannot be fully assessed without a comparison between the model's flux simulations and the ground truth observations. In the present study, the issue of using scintillometry for validating spatial and temporal aggregation schemes over heterogeneous grids has been investigated. Data collected within the SUDMED project over the oliveyard of Agdal which was located near the city of Marrakech, Morocco, have been used to test the aggregation schemes. The Agdal oliveyard was made up of two contrasted fields, or patches. Even though the two sites appear relatively homogeneous, they differ strongly in terms of soil moisture status and vegetation percent cover. The higher soil moisture in the northern site creates heterogeneity at the scale of the entire olive yard (i.e. at grid-scale). Firstly, the diurnal course of the grid-scale evapotranspiration (〈ET sim〉 SA) estimated from spatial aggregation scheme is compared to that derived from scintillometry (〈ET LAS〉). 〈ET sim〉 SA is obtained as the residual term of the energy balance providing the estimates of the available energy (AE = R n − G), where R n and G are the net radiation and the soil heat flux, respectively, and sensible heat flux. The latter is estimated by using a simple two-layer model developed by Lhomme et al. (1994). The root mean square difference (RMSD) and the correlation coefficient ( R 2) between 〈ET sim〉 SA and 〈ET LAS〉 were about 46 W m −2 and 0.78, respectively. Secondly, we compared the diurnal course of grid-scale evapotranspiration (〈ET sim〉 TA) estimated from the temporal aggregation scheme with the 〈ET LAS〉. 〈ET sim〉 TA is obtained by extrapolating the instantaneous values of the available energy and the evaporative fraction (EF = ET/AE) estimated at the satellite overpass to daily ones. The instantaneous values of AE and EF have been derived using remotely sensed surface temperature measured using a ground-based infrared thermometer combined with ancillary micrometeorological data such as wind speed, incoming and outgoing solar radiation, and temperature and humidity of the air. The RMSD and the R 2 were about 43 W m −2 and 0.7, respectively. Despite the complexity of the site induced by the strong heterogeneity in the soil moisture which is related to the employed irrigation method (flood irrigation), and the consequences in terms of the footprint of the instruments, the obtained statistical results showed that both aggregation schemes performed successfully with regard to estimates of the evapotranspiration over heterogeneous grids. Finally, to further assess the performance of the developed approach, a second dataset collected in northern Mexico has been also used. The result shows that the approach provides acceptable values of aggregated evapotranspiration. Consequently, scintillometry can potentially be used in the development and the validation of aggregation approaches to improve the representation of surface heterogeneity land-surface–atmosphere models operating at large scales.

Exploring Job Migration Technique for P2P Grid Systems

P2P Grids can potentially solve large-scale scientific problems by using geographically distributed heterogeneous resources. However, a number of major technical obstacles must be overcome before this potential can be realized. One problem critical to the effective utilization of P2P Grids is the efficient scheduling of jobs. This study addresses the above-mentioned problem by describing and evaluating a P2P communication model, a P2P resource monitoring system and a job migration mechanism. In this study, the authors propose a P2P communication mechanism, which is built to deliver various information across heterogeneous Grid systems. Based on this P2P communication mechanism, they can develop job migration technology and then improve the usage of distributed computing resources.

An Interoperable Grid Information System for Integrated Resource Monitoring Based on Virtual Organizations

In many Grid infrastructures different kinds of information services are in use, which utilize different incompatible data structures and interfaces to encode and provide their data. Homogeneous monitoring of these infrastructures with the monitoring data being accessible everywhere independently of the middleware which provided it, is the basis for a consistent status reporting on the Grids’ resources and services. Thus, interoperability or interoperation between the different information services in a heterogeneous Grid infrastructure is required. Monitoring data must contain the identity of the affected Virtual Organization (VO) so that it can be related to the resources and services the VO has allocated to enable VO-specific information provision. This paper describes a distributed architecture for an interoperable information service, which combines data unification and categorization with policies for VO membership, VO resource management and data transformations. This service builds the basis for an integrated and interoperating monitoring of Grids, which provide their data to more than one VO and utilize heterogeneous information services.

Communication cost effective scheduling policies of nonclairvoyant jobs with load balancing in a grid

Effective load distribution is of great importance at grids, which are complex heterogeneous distributed systems. In this paper we study site allocation scheduling of nonclairvoyant jobs in a 2-level heterogeneous grid architecture. Three scheduling policies at grid level which utilize site load information are examined. The aim is the reduction of site load information traffic, while at the same time mean response time of jobs and fairness in utilization between the heterogeneous sites are of great interest. A simulation model is used to evaluate performance under various conditions. Simulation results show that considerable decrement in site load information traffic and utilization fairness can be achieved at the expense of a slight increase in response time.

Towards cross-middleware authentication and single sign-on for ARC Grid middleware

When pursuing the task of making access to Grids as simple as possible, security is one of the most important challenges in production Grid infrastructures, especially when the Grid applications span multiple administrative domains as well as heterogeneous Grid middlewares. A typical example is wide scale e-Science applications which need to coordinate resources shared among a number of independent institutions with different Grid middlewares deployed on these resources. In this paper, we describe security implementation and considerations used in the upcoming version of the Advanced Resource Connector (ARC) middleware, where the heterogeneity issue has been addressed. The main goal of ARC implementation in terms of security is to let the middleware be capable of interoperating with other Grid middlewares by leveraging on standard specifications. The key aspect of the work is to enhance the current proxy certificate based authentication and single sign-on by utilizing and enhancing the standardized Web Service specifications such as Security Assertion Markup Languages (SAML), single sign-on (SSO) profile and Web Services Security in order to achieve cross-middleware authentication and single sign-on.

Supporting dynamic access to virtualized data resources via WSRF-based services

Today grid applications require not only lots of computational power but data at a very large scale too. Although grid computing was initially conceptualized as the technology that focuses on solving compute-intensive problems, this focus has gradually shifted to applications where data is distributed over various locations. Access to these data resources stored in heterogeneous grid storage systems located at geographically distributed virtual organizations in an integrated and uniform way is a challenging problem. The Web Services Resource Framework (WSRF) has recently emerged as the standard for the development and integration of grid services. This paper proposes and presents Gravy4WS, a middleware architecture based on WSRF Web services that enables the dynamic access to virtualized grid data resources. A novel scheduling algorithm called DCE (Delegating-Cluster-Execution based Scheduling) is proposed to improve load balancing of the system. The implementation of Gravy4WS using WSRF libraries and services provided by Globus Toolkit 4 is described together with its performance evaluation.

An adaptive multi-policy grid service for biological sequence comparison

In the last decade, we have observed an unprecedented development in molecular biology. An extremely high number of organisms have been sequenced in genome projects and included in genomic databases, for further analysis. These databases present an exponential growth rate and they are intensively accessed daily, all over the world. Once a sequence is obtained, its function and/or structure must be determined. Direct experimentation is considered to be the most reliable method to do that. However, the experiments that must be conducted are very complex and time consuming. For this reason, it is far more productive to use computational methods to infer biological information from a sequence. This is usually done by comparing the new sequence with sequences that already had their characteristics determined. BLAST is the most widely used heuristic tool for sequence comparison. Thousands of BLAST searches are made daily, all over the world. In order to further reduce the BLAST execution time, cluster and grid environments can be effectively used. This paper proposes and evaluates an adaptive task allocation framework to perform BLAST searches in a grid environment. The framework, called PackageBLAST, provides an infrastructure that executes distributed BLAST genomic database comparisons. In addition, it is flexible since the user can choose or incorporate new task allocation strategies. Furthermore, we propose a mechanism to compute grid nodes’ execution weight, adapting the chosen allocation policy to the observed computational power and local load of the nodes. Our results present very good speedups. For instance, in a 16-machine heterogeneous grid testbed, a speedup of 14.59 was achieved, reducing the BLAST execution time from 30.88 min to 2.11 min. Also, we show that the adaptive task allocation strategy was able to handle successfully the complexity of a grid environment.

Grid resource management policies for load-balancing and energy-saving by vacation queuing theory

The resource management is the central component of grid system. The analysis of the workload log file of LCG including the job arrival and the resource utilization daily cycle shows that the idle sites in the Grid are the source of load imbalance and energy waste. Here we focus on these two issues: balancing the workload by transferring jobs to idle sites at prime time to minimize the response time and maximize the resource utilization; power management by switch the idle sites to sleeping mode at non-prime time to minimize the energy consume. We form the M/G/1 queue model with server vacations, startup and closedown to analysis the performance metrics to instruct the design of load-balancing and energy-saving policies. We provide our Adaptive Receiver Initiated (ARI) load-balancing strategy and power-management policy for energy-saving. The simulation experiments prove the accuracy of our analysis and the comparisons results indicate our policies are largely suitable for large-scale heterogeneous grid environment.

The User-Level Scheduling of Divisible Load Parallel Applications With Resource Selection and Adaptive Workload Balancing on the Grid

This paper presents a hybrid resource management environment, operating on both application and system levels developed for minimizing the execution time of parallel applications with divisible workload on heterogeneous grid resources. The system is based on the adaptive workload balancing algorithm (AWLB) incorporated into the distributed analysis environment (DIANE) user-level scheduling (ULS) environment. The AWLB ensures optimal workload distribution based on the discovered application requirements and measured resource parameters. The ULS maintains the user-level resource pool, enables resource selection and controls the execution. We present the results of performance comparison of default self-scheduling used in DIANE with AWLB-based scheduling, evaluate dynamic resource pool and resource selection mechanisms, and examine dependencies of application performance on aggregate characteristics of selected resources and application profile.

Flexible Rollback Recovery in Dynamic Heterogeneous Grid Computing

Large applications executing on Grid or cluster architectures consisting of hundreds or thousands of computational nodes create problems with respect to reliability. The source of the problems are node failures and the need for dynamic configuration over extensive run-time. This paper presents two fault-tolerance mechanisms called theft induced checkpointing and systematic event logging. These are transparent protocols capable of overcoming problems associated with both, benign faults, i.e., crash faults, and node or subnet volatility. Specifically, the protocols base the state of the execution on a dataflow graph, allowing for efficient recovery in dynamic heterogeneous systems as well as multi-threaded applications. By allowing recovery even under different numbers of processors, the approaches are especially suitable for applications with need for adaptive or reactionary configuration control. The low-cost protocols offer the capability of controlling or bounding the overhead. A formal cost model is presented, followed by an experimental evaluation. It is shown that the overhead of the protocol is very small and the maximum work lost by a crashed process is small and bounded.

Resolution of large symmetric eigenproblems on a world-wide grid

We propose a parallel and distributed application for the resolution of the large real symmetric eigenproblem on world-wide heterogeneous grids. It is based on the explicit restarted Lanczos algorithm which is a Krylov subspace method. We take into account the specificities of computational resources and deal with communication constraints over the internet. We propose techniques such as out-of-core and data persistence. We also show that a restarted algorithm and the combination of several paradigms of parallelism are relevant in this context. The results of the experiments stress the impact of the main numerical parameters of the Lanczos method. We also compare many platform configurations built on the top of two complementary experimental environments. We first use a realistic world-wide grid harnessing two geographical sites, in France and in Japan, interconnected by the internet. Then, we use the French national testbed Grid5000. We conclude on the scalability of our application and we forecast the next bounding factor of the implementation. It can be solved by means of a dedicated data management layer such as Omni Storage.

A decision support grid for integrated molecular solvent design and chemical process selection

The process engineering domain makes an interesting application area for grid technologies as problem solving generally involves multiple software and data resources and heavy computations. There are numerous computationally demanding applications that can benefit from grid computing such as molecular simulations and large-scale optimizations. We report on a prototype grid for integrated process and molecular design to demonstrate the potential of grids in the process-engineering arena. The prototype integrates in-house solvent and process design tools with external distributed computation and storage resources as well as web-databases and computer networks. A middleware architecture is proposed in the form of a problem-solving environment as a means of interfacing the heterogeneous grid components. The grid operations and functionalities are exposed to the user through a prototype web-portal that enables the seamless access and utilization of the available resources. The developed prototype is illustrated with an application case.

Deploying HEP applications using Xen and Globus Virtual Workspaces

The deployment of HEP applications in heterogeneous grid environments can be challenging because many of the applications are dependent on specific OS versions and have a large number of complex software dependencies. Virtual machine monitors such as Xen could be used to package HEP applications, complete with their execution environments, to run on resources that do not meet their operating system requirements. Our previous work has shown HEP applications running within Xen suffer little or no performance penalty as a result of virtualization. However, a practical strategy is required for remotely deploying, booting, and controlling virtual machines on a remote cluster. One tool that promises to overcome the deployment hurdles using standard grid technology is the Globus Virtual Workspaces project. We describe strategies for the deployment of Xen virtual machines using Globus Virtual Workspace middleware that simplify the deployment of HEP applications.

Parallel programming on a high‐performance application‐runtime

AbstractHigh‐performance application development remains challenging, particularly for scientists making the transition to a heterogeneous grid environment. In general areas of computing, virtual environments such as Java and .Net have proved to be successful in fostering application development, allowing users to target and compile to a single environment, rather than a range of platforms, instruction sets and libraries. However, existing runtime environments are focused on business and desktop computing and they do not support the necessary high‐performance computing (HPC) abstractions required by e‐Scientists. Our work is focused on developing an application‐runtime that can support these services natively. The result is a new approach to the development of an application‐runtime for HPC: the Motor system has been developed by integrating a high‐performance communication library directly within a virtual machine. The Motor message passing library is integrated alongside and in cooperation with other runtime libraries and services while retaining a strong message passing performance. As a result, the application developer is provided with a common environment for HPC application development. This environment supports both procedural languages, such as C, and modern object‐oriented languages, such as C#. This paper describes the unique Motor architecture, presents its implementation and demonstrates its performance and use. Copyright © 2008 John Wiley & Sons, Ltd.