File Replicas Research Articles

Dynamic and Decentralized Storage Load Balancing with Analogy to Thermal Diffusion for P2P File Sharing

In this paper we propose a file replication scheme inspired by a thermal diffusion phenomenon for storage load balancing in unstructured peer-to-peer (P2P) file sharing networks. The proposed scheme is designed such that the storage utilization ratios of peers will be uniform, in the same way that the temperature in a field becomes uniform in a thermal diffusion phenomenon. The proposed scheme creates replicas of files in peers probabilistically, where the probability is controlled by using parameters that can be used to find the trade-off between storage load balancing and search performance in unstructured P2P file sharing networks. First, we show through theoretical analysis that the statistical behavior of the storage load balancing controlled by the proposed scheme has an analogy with the thermal diffusion phenomenon. We then show through simulation that the proposed scheme not only has superior performance with respect to balancing the storage load among peers (the primary objective of the present proposal) but also allows the performance trade-off to be widely found. Finally, we qualitatively discuss a guideline for setting the parameter values in order to widely find the performance trade-off from the simulation results.

Malugo: A peer-to-peer storage system

We consider the problem of routing locality in peer-to-peer storage systems where peers store and exchange data among themselves. With the global information, peers will take the data locality into consideration when they implement their replication mechanisms to keep a number of file replicas all over the systems. In this paper, we mainly propose a peer-to-peer storage system – Malugo. Algorithms for the implementation of the peers' locating and file operation processes are also presented. Simulation results show that the proposed system successfully constructs an efficient and stable peer-to-peer storage environment with considerations of data and routing locality among peers.

Strategies Toward Optimal Access to File Replicas in Data Grid Environments

Data Grid is one of key technologies to build up large-scale dataset storage system by connecting scattered storage resources dispersedly located in the Grid. One major challenge in data grids is how to provide good and timely access to huge amount of data in distributed locations, given the high latency of interconnection networks. Parallel downloading methods can improve download efficiency and performance, and for such, processes should have started from appropriate locations. In this research paper, we present the design framework of PU-DG Optimizer toolbox (also known as PU-DG Optibox) for data grid environments. The proposed toolbox is a package containing a number of high-end techniques and algorithms running as middleware on top of data grid platforms, in order to optimize file downloads, by improving its efficiency and performance. Moreover, PU-DG Optibox not only provides users and developers possibilities for setting their own priority strategies, as also different downloading modes. Furthermore, workload balancing to avoid low quality computing node to execute highly complex job is also included in this design. Experimental results of techniques packaged in the proposed toolbox demonstrate its potential and effectiveness.

A proactive low-overhead file replication scheme for structured P2P content delivery networks

File replication is a widely used technique for high performance in peer-to-peer content delivery networks. A file replication technique should be efficient and at the same time facilitates efficient file consistency maintenance. However, most traditional methods do not consider nodes’ available capacity and physical location in file replication, leading to high overhead for both file replication and consistency maintenance. This paper presents a proactive low-overhead file replication scheme, namely Plover. By making file replicas among physically close nodes based on nodes’ available capacities, Plover not only achieves high efficiency in file replication but also supports low-cost and timely consistency maintenance. It also includes an efficient file query redirection algorithm for load balancing between replica nodes. Theoretical analysis and simulation results demonstrate the effectiveness of Plover in comparison with other file replication schemes. It dramatically reduces the overhead of both file replication and consistency maintenance compared to other schemes. In addition, it yields significant improvements in reduction of overloaded nodes.

DIRAC: a community grid solution

The DIRAC system was developed in order to provide a complete solution for using the distributed computing resources of the LHCb experiment at CERN for data production and analysis. It allows a concurrent use of over 10K CPUs and 10M file replicas distributed over many tens of sites. The sites can be part of a Computing Grid such as WLCG or standalone computing clusters all integrated in a single management structure. DIRAC is a generic system with the LHCb specific functionality incorporated through a number of plug-in modules. It can be easily adapted to the needs of other communities. Special attention is paid to the resilience of the DIRAC components to allow an efficient use of non-reliable resources. The DIRAC production management components provide a framework for building highly automated data production systems including data distribution and data driven workload scheduling. In this paper we give an overview of the DIRAC system architecture and design choices. We show how different components are put together to compose an integrated data processing system including all the aspects of the LHCb experiment - from the MC production and raw data reconstruction to the final user analysis.

Towards building a conflict‐free mobile distributed file system

AbstractThe rising demand for mobile computing has created a need for an improved file system that supports mobile clients. Current file systems with support for mobility provide availability through file replicas that are cached at the client side. However, mobile clients may experience different obstacles with regards to the local cache, such as the limited network bandwidth, the intermittent connection, and serious conflicts when synchronizing back to the server. In this paper, we propose a novel mobile distributed file system design, which provides high availability and reliable storage for files and guarantees that file operations are executed regardless of concurrency and failure issues. The design is intended to fit mobile clients (e.g. PDAs and cell phones) that have limited storage space and cannot store all of the data they need, and yet require access to these data at all times. We adopt a server‐side caching in order to guarantee sufficient caching space to all mobile clients, and to ensure the availability of files in the case of clients' failures. We present our algorithm, describe its implementation, simulate its high availability functions, and report on its performance evaluation using a cluster of workstations. Our simulation results indicate clearly that our algorithm exhibits a significant degree of automation and conflict‐free mobile file system. Copyright © 2007 John Wiley & Sons, Ltd.

Optimal search performance in unstructured peer-to-peer networks with clustered demands

This paper derives the optimal search time and the optimal search cost that can be achieved in unstructured peer-to-peer networks when the demand pattern exhibits clustering (i.e. file popularities vary across the set of nodes in the network). Clustering in file popularity patterns is evident from measurements on deployed peer-to-peer file sharing networks. In this paper, we provide mechanisms for modeling clustering in file popularity distributions and the consequent non-uniform distribution of file replicas. We derive relations that show the effect of the number of replicas of a file on the search time and on the search cost for a search for that file for the clustered demands case in such networks for both random walk and flooding search mechanisms. The derived relations are used to obtain the optimal search performance for the case of flooding search mechanisms. The potential performance benefit that clustering in demand patterns affords is captured by our results. Interestingly, the performance gains are shown to be independent of whether the search network topology reflects the clustering in file popularity (the optimal file replica distribution to obtain these performance gains, however, does depend on the search network topology).

Analysis of search and replication in unstructured peer-to-peer networks

This paper investigates the effect of the number of file replicas on search performance in unstructured peer-to-peer networks. We observe that for a search network with a random graph topology where file replicas are uniformly distributed, the hop distance to a replica of a file is logarithmic in the number of replicas. Using this observation we show that flooding-based search is optimized when the number of replicas is proportional to the file request rates. This replica distribution is also optimal for download time and since flooding has logarithmically better search time than random walk under its optimal replica distribution, we investigate the query-processing load using this distribution.

A dynamic distributed replica management mechanism based on accessing frequency detecting

This paper presents a novel decentralized dynamic replication management mechanism based on accessing frequency detecting (FDRM).In FDRM, in order to provide better system performance and reduce network traffic, system node scans the local replica of a certain file to detect its access pattern, and makes decision independently to add, delete or migrate a replica for that file. In addition, the scanning interval of a replica is variable according to the accessing frequency to that replica, which makes FDRM more sensitive to the change of system behavior, and we can get better performance with less system overhead. In the end, the paper evaluates the efficiency of FSRM by experimental result and shows the performance enhancement.

Fault-tolerant data management in the gaston peer-to-peer file system

Gaston is a peer-to-peer large-scale file system designed to provide a fault-tolerant and highly available file service for a virtually unlimited number of users. Data management in Gaston disseminates and stores replicas of files on multiple machines to achieve the requested level of data availability and uses a dynamic tree-topology structure to connect replication schema members. We present generic algorithms for replication schema creation and maintenance according to file user requirements and autonomous constraints that are set on individual nodes. We also show specific data object structure as well as mechanisms for secure and efficient update propagation among replicas with data consistency control. Finally, we introduce a scalable and efficient technique improving fault-tolerance of the tree-topology structure connecting replicas.

SAM-GRID: A system utilizing grid middleware and SAM to enable full function grid computing

We present a grid system, which is in development, employing an architecture comprising the primary functional components of job handling, data handling, and monitoring and information services. Each component is built using existing Grid middleware. The Job handling utilizes Condor Match Making Services to broker job submissions, Condor-G to schedule, and GRAM to submit and execute jobs on remote compute resources. The information services provide strategic information of the system including a file replica catalogue, compute availability, and network data-throughput rate predictions, which are made available to the other components. Data handling services are provided by SAM, the data management system built for the Dzero experiment at Fermilab, to optimize data delivery, and cache and replicate data as needed at the processing nodes. The SAM-Grid system is being built to provide experiments in progress at Fermilab the ability to utilize worldwide computing resources to process enormous quantities of data for complex physics analyses.

Taming aggressive replication in the Pangaea wide-area file system

Pangaea is a wide-area file system that supports data sharing among a community of widely distributed users. It is built on a symmetrically decentralized infrastructure that consists of commodity computers provided by the end users. Computers act autonomously to serve data to their local users. When possible, they exchange data with nearby peers to improve the system's overall performance, availability, and network economy. This approach is realized by aggressively creating a replica of a file whenever and wherever it is accessed.This paper presents the design, implementation, and evaluation of the Pangaea file system. Pangaea offers efficient, randomized algorithms to manage highly dynamic and potentially large groups of file replicas. It applies optimistic consistency semantics to replica contents, but it also offers stronger guarantees when required by the users. The evaluation demonstrates that Pangaea outperforms existing distributed file systems in large heterogeneous environments, typical of the Internet and of large corporate intranets.

Grid Data Mirroring Package (GDMP)

The GDMP client‐server software system is a generic file replication tool that replicates files securely and efficiently from one site to another in a Data Grid environment using Globus Grid tools. In addition, it manages replica catalogue entries for file replicas and thus maintains a consistent view on names and locations of replicated files. Files to be transferred can be of any particular file format and GDMP treats them all in the same way. However, for Objectivity database files a particular plug‐in exists. All files are assumed to be read‐only.

Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility

This paper presents and evaluates the storage management and caching in PAST, a large-scale peer-to-peer persistent storage utility. PAST is based on a self-organizing, Internet-based overlay network of storage nodes that cooperatively route file queries, store multiple replicas of files, and cache additional copies of popular files.In the PAST system, storage nodes and files are each assigned uniformly distributed identifiers, and replicas of a file are stored at nodes whose identifier matches most closely the file's identifier. This statistical assignment of files to storage nodes approximately balances the number of files stored on each node. However, non-uniform storage node capacities and file sizes require more explicit storage load balancing to permit graceful behavior under high global storage utilization; likewise, non-uniform popularity of files requires caching to minimize fetch distance and to balance the query load.We present and evaluate PAST, with an emphasis on its storage management and caching system. Extensive trace-driven experiments show that the system minimizes fetch distance, that it balances the query load for popular files, and that it displays graceful degradation of performance as the global storage utilization increases beyond 95%.

Building a large location table to find replicas of physics objects

The problem of building a large location table for physics objects occurs within a number of planned physics data management systems, like those that control reclustering and wide-area replication. To satisfy their efficiency goals, these systems have to make local or remote replicas of individual physics objects, which contain raw or reconstructed data for a single event, rather than replicas of large run or n-tuple files. This replication implies the use of a table to resolve the logical, location independent object descriptor into a physical location where an object replica can be found. For modern physics experiments the table needs to scale to at least some 10 10 objects. We argue that such a table can be efficiently implemented by limiting the freedom of lookup operations, and by exploiting some specific properties of the physics data model. One specific viable implementation is discussed.

CIF applications. II.CIFIO: for CIF input/output in theXtalsystem

CIFIO is the primary CIF input and output program in the Xtal System and provides for global data exchange and machine-readible publication submission. CIFIO also creates and reads an Xtal archive file which is a CIF-like ASCII replica of an Xtal binary file. CIFIO is written in Ratmac and may be implemented, along with the other 94 calculations in the Xtal3.2 package, on any computer.

Cluster-based file replication in large-scale distributed systems

The increasing need for data sharing in large-scale distributed systems may place a heavy burden on critical resources such as file servers and networks. Our examination of the workload in one large commercial engineering environment shows that wide-spread sharing of unstable files among tens to hundreds of users is common. Traditional client-based file cacheing techniques are not scalable in such environments. We propose Frolic, a scheme for cluster-based file replication in large-scale distributed file systems. A cluster is a group of workstations and one or more file servers on a local area network. Large distributed systems may have tens or hundreds of clusters connected by a backbone network. By dynamically creating and maintaining replicas of shared files on the file servers in the clusters using those files, we effectively reduce reliance on central servers supporting such files, as well as reduce the distances between the accessing sites and data. We propose and study algorithms for the two main issues in Frolic, 1) locating a valid file replica, and 2) maintaining consistency among replicas. Our simulation experiments using a statistical workload model based upon measurement data and real workload characteristics show that cluster-based file replication can significantly reduce file access delays and server and backbone network utilizations in large-scale distributed systems over a wide range of workload conditions. The workload characteristics most critical to replication performance are: the size of shared files, the number of clusters that modify a file, and the number of consecutive accesses to files from a particular cluster.