European DataGrid Project Research Articles

A Two-Level Fuzzy Value-Based Replica Replacement Algorithm in Data Grids

One of the challenges of data grid is to access widely distributed data fast and efficiently and providing maximum data availability with minimum latency. Data replication is an efficient way used to address this challenge by replicating and storing replicas, making it possible to access similar data in different locations of the data grid and can shorten the time of getting the files. However, as the number and storage size of grid sites is limited and restricted, an optimized and effective replacement algorithm is needed to improve the efficiency of replication. In this paper, the authors propose a novel two-level replacement algorithm which uses Fuzzy Replica Preserving Value Evaluator System (FRPVES) for evaluating the value of each replica. The algorithm was tested using a grid simulator, OptorSim developed by European Data Grid projects. Results from simulation procedure show that the authors' proposed algorithm has better performance in comparison with other algorithms in terms of job execution time, total number of replications and effective network usage.

A four-phase data replication algorithm for data grid

<p>Nowadays, scientific applications generate a huge amount of data in terabytes or petabytes. Data grids currently proposed solutions to large scale data management problems including efficient file transfer and replication. Data is typically replicated in a Data Grid to improve the job response time and data availability. A reasonable number and right locations for replicas has become a challenge in the Data Grid. In this paper, a four-phase dynamic data replication algorithm based on Temporal and Geographical locality is proposed. It includes: 1) evaluating and identifying the popular data and triggering a replication operation when the popularity data passes a dynamic threshold; 2) analyzing and modeling the relationship between system availability and the number of replicas, and calculating a suitable number of new replicas; 3) evaluating and identifying the popular data in each site, and placing replicas among them; 4) removing files with least cost of average access time when encountering insufficient space for replication. The algorithm was tested using a grid simulator, OptorSim developed by European Data Grid Projects. The simulation results show that the proposed algorithm has better performance in comparison with other algorithms in terms of job execution time, effective network usage and percentage of storage filled.</p>

A New Architecture for Group Replication in Data Grid

Nowadays, grid systems are vital technology for programs running with high performance and problems solving with large- scale in scientific, engineering and business. In grid systems, heterogeneous computational resources and data should be shared between independent organizations that are scatter geographically. A data grid is a kind of grid types that make relations computational and storage resources. Data replication is an efficient way in data grid to obtain high performance and high availability by saving numerous replicas in different locations e.g. grid sites. In this research, we propose a new architecture for dynamic Group data replication. In our architecture, we added two components to OptorSim architecture: Group Replication Management component (GRM) and Management of Popular Files Group component (MPFG). OptorSim developed by European Data Grid projects for evaluate replication algorithm. By using this architecture, popular files group will be replicated in grid sites at the end of each predefined time interval.

Ten years of European Grids: What have we learnt?

The European DataGrid project started in 2001, and was followed by the three phases of EGEE and the recent transition to EGI. This paper discusses the history of both middleware development and Grid operations in these projects, and in particular the impact on the development of the LHC Computing Grid. It considers to what extent the initial ambitions have been realised, which aspects have been successful and what lessons can be derived from the things which were less so, both in technical and sociological terms. In particular it considers the middleware technologies used for data management, workload management, information systems and security, and the difficulties of operating a highly distributed worldwide production infrastructure, drawing on practical experience with many aspects of the various Grid projects over the last decade.

PDDRA: A new pre-fetching based dynamic data replication algorithm in data grids

In recent years, grid technology has had such a fast growth that it has been used in many scientific experiments and research centers. A large number of storage elements and computational resources are combined to generate a grid which gives us shared access to extra computing power. In particular, data grid deals with data intensive applications and provides intensive resources across widely distributed communities. Data replication is an efficient way for distributing replicas among the data grids, making it possible to access similar data in different locations of the data grid. Replication reduces data access time and improves the performance of the system. In this paper, we propose a new dynamic data replication algorithm named PDDRA that optimizes the traditional algorithms. Our proposed algorithm is based on an assumption: members in a VO (Virtual Organization) have similar interests in files. Based on this assumption and also file access history, PDDRA predicts future needs of grid sites and pre-fetches a sequence of files to the requester grid site, so the next time that this site needs a file, it will be locally available. This will considerably reduce access latency, response time and bandwidth consumption. PDDRA consists of three phases: storing file access patterns, requesting a file and performing replication and pre-fetching and replacement. The algorithm was tested using a grid simulator, OptorSim developed by European Data Grid projects. The simulation results show that our proposed algorithm has better performance in comparison with other algorithms in terms of job execution time, effective network usage, total number of replications, hit ratio and percentage of storage filled.

Dynamic replication in a data grid using a Modified BHR Region Based Algorithm

Grid computing is emerging as a key part of the infrastructure for a wide range of disciplines in science and engineering, including astronomy, high energy physics, molecular biology and earth sciences. These applications handle large data sets that need to be transferred and replicated among different grid sites. A data grid deals with data intensive applications in scientific and enterprise computing. Data grid technology is developed to permit data sharing across many organizations in geographically disperse locations. Replication of data to different sites will help researchers around the world analyse and initiate future experiments. The general idea of replication is to store copies of data in different locations so that data can be easily recovered if a copy at one location is lost or unavailable. In a large-scale data grid, replication provides a suitable solution for managing data files, which enhances data reliability and availability. In this paper, a Modified BHR algorithm is proposed to overcome the limitations of the standard BHR algorithm. The algorithm is simulated using a data grid simulator, OptorSim, developed by European Data Grid projects. The performance of the proposed algorithm is improved by minimizing the data access time and avoiding unnecessary replication.

Quattor: managing (complex) grid sites

Quattor is a tool developed to efficiently manage fabrics with hundreds or thousands of Linux machines, while still being able to manage smaller clusters easily. It was originally developed inside the European Data Grid (EDG) project and is now in use at more than 50 grid sites running gLite middleware, ranging from small LCG T3s to very large sites like CERN.Quattor's ability to factorize and to reuse common parts of service configurations permitted the development of the QWG templates: a complete set of standard templates to configure the OS and gLite middleware. Any site can just import and customize the configuration without editing the bulk of the templates. Collaboration around these templates results in a very efficient sharing of installation and configuration information between those sites using them.

Price‐sensitive resource brokering with the Hybrid Pricing Model and widely overlapping price domains

AbstractThe DGAS‐Sim(ulator) simulates the components of the Grid middleware that are involved in the job scheduling process, reflecting the architecture of the Workload Management System of the European DataGrid project. Its purpose is to study the impact of different resource pricing schemes and resource brokering strategies on workload balancing. In this paper we present further simulation results for a price‐sensitive utility function (the resource broker always chooses the most economic computing element) and the Hybrid Pricing Model (fixed base prices and linear price adjustments within fixed variation limits to reflect the current workload of the computing element). We specifically consider a Grid economy scenario with widely overlapping domains of the prices of the single resources and compare the outcome with our previously presented results. Copyright © 2005 John Wiley & Sons, Ltd.

The GENIUS Web Portal

The development of computational Grids is making huge amounts of computing power and data storage available for a lot of scientific applications. At this stage of development, the use of the Grid is mainly based on Command Line Interface (CLI) tools that are not very friendly and can be considered an obstacle to the use of these powerful tools. The objective of this paper is to present a solution to this problem. To ease the access of new users to the grid the GENIUS (Grid Enabled web eNvironment for site Independent User job Submission) grid portal has been jointly developed by INFN and NICE within the context of both the Italian INFN Grid and the European DataGrid Projects. Here we devote particular care to the description of job creation and submission and the services for transparent access to user's data and applications. Using GENIUS, the obstacle of complicated CLI can be overtaken and simple web interfaces can be built for specific user communities and applications. Here we show examples in the field of bio-medical applications. The use of Grid can be made easy with the use of Grid portals such as GENIUS.

The CanonicalProducer: An Instrument Monitoring Component of the Relational Grid Monitoring Architecture (R-GMA)

We describe how the R-GMA (Relational Grid Monitoring Architecture) can be used to allow for instrument monitoring in a Grid environment. The R-GMA has been developed within the European DataGrid Project (EDG) as a Grid Information and Monitoring System. It is based on the Grid Monitoring Architecture (GMA) from the Global Grid Forum (GGF), which is a simple Consumer-Producer model. The special strength of this implementation comes from the power of the relational model. It offers a global view of the information as if each Virtual Organisation had one large relational database. It provides a number of different Producer types with different characteristics; for example some support streaming of information. We describe the R-GMA component that allows for instrument monitoring, the CanonicalProducer. We also describe an example use of this approach in the European CrossGrid project, SANTA-G, a network monitoring tool.

Replica Management in the European DataGrid Project

Within the European DataGrid project, Work Package 2 has designed and implemented a set of integrated replica management services for use by data intensive scientific applications. These services, based on the web services model, enable movement and replication of data at high speed from one geographical site to another, management of distributed replicated data, optimization of access to data, and the provision of a metadata management tool. In this paper we describe the architecture and implementation of these services and evaluate their performance under demanding Grid conditions.

File-based replica management

Data replication is one of the best known strategies to achieve high levels of availability and fault tolerance, as well as minimal access times for large, distributed user communities using a world-wide Data Grid. In certain scientific application domains, the data volume can reach the order of several petabytes; in these domains, data replication and access optimization play an important role in the manageability and usability of the Grid. In this paper, we present the design and implementation of a replica management Grid middleware that was developed within the EDG project [European DataGrid Project (EDG), http://www.eu-egee.org] and is designed to be extensible so that user communities can adjust its detailed behavior according to their QoS requirements.

Grid Network Monitoring in the European Datagrid Project

A Grid Network monitoring and performance measurement infrastructure is a key component of a large-scale Grid. Requirements for network monitoring with respect to Grid operations as well as performance characteristics and their relevance to Grid context have to be carefully identified before realizing such a system. In this paper we analyze the main issues raised by the design, development, integration in a Grid middleware and deployment of a network monitoring infrastructure in a real Grid environment. Then we present the details of the architecture we proposed in the context of the European DataGRID project and the MapCenter tool we developed to vizualize the Grid status in real time and to give access to our measurement infrastructure.

Replica consistency in a Data Grid

A Data Grid is a wide area computing infrastructure that employs Grid technologies to provide storage capacity and processing power to applications that handle very large quantities of data. Data Grids rely on data replication to achieve better performance and reliability by storing copies of data sets on different Grid nodes. When a data set can be modified by applications, the problem of maintaining consistency among existing copies arises. The consistency problem also concerns metadata, i.e., additional information about application data sets such as indices, directories, or catalogues. This kind of metadata is used both by the applications and by the Grid middleware to manage the data. For instance, the Replica Management Service (the Grid middleware component that controls data replication) uses catalogues to find the replicas of each data set. Such catalogues can also be replicated and their consistency is crucial to the correct operation of the Grid. Therefore, metadata consistency generally poses stricter requirements than data consistency. In this paper we report on the development of a Replica Consistency Service based on the middleware mainly developed by the European Data Grid Project. The paper summarises the main issues in the replica consistency problem, and lays out a high-level architectural design for a Replica Consistency Service. Finally, results from simulations of different consistency models are presented.

LHCb distributed computing and the grid

The current architecture of the LHCb distributed system for Monte Carlo data production is described, together with details of the implementation and performance of a regional site at INFN-CNAF. An overview is given of the current and planned use of Grid technology from the European Datagrid Project (EDG), and of the development of an experiment-specific user interface to Grid services.

Author Index

The current architecture of the LHCb distributed system for Monte Carlo data production is described. An overview is given of the current and planned use of Grid technology from the European Datagrid Project, and of the development of an experiment-specific user interface to Grid services.

European DataGrid project: status and plans

The European DataGrid (EDG) project has reached, after 1.5 years, the middle of its lifetime. In this article we give an overview of the status, components, procedures and plans of the EDG project as of June 2002. The objective of the EDG project is to assist the next generation of scientific exploration, computation and analysis of large-scale data sets—from hundreds of terabytes to petabytes, across widely distributed scientific communities. The primary goal of the first phase of the EDG project was to assemble a distributed testbed to demonstrate that the EDG middleware components could be integrated into a production-quality computational Grid , as well as to gain experience with such a system. The very first version of the EDG testbed was deployed toward the end of 2001. At the first official European Union review of the project on March 1, 2002, it has been found that the project is on the right track to achieve its goals. Since then the EDG middleware, testbed components and procedures have been continuously refined according to the requirements of our user communities and our experience.

LHCb distributed computing and the GRID

Abstract The current architecture of the LHCb distributed system for Monte Carlo data production is described. An overview is given of the current and planned use of Grid technology from the European Datagrid Project, and of the development of an experiment-specific user interface to Grid services.