Design of optimal distributed file systems

Abstract

In this paper, the problem of determining an optimal location strategy for an individual program execution is considered. In addition, we propose a heuristic approach for the dynamic file allocation problem. In order to reduce the complexity of the optimization problems, a cluster-based approach is used.To access the data files of a distributed file system, a user initiates a program execution. Based on the current allocation of the program and data files as well as the knowledge about the characteristics of the programs, a first optimization calculates the optimal cluster for each individual program execution. The objective of this optimization is the minimization of the intercluster traffic of an individual program execution. Within the optimal cluster, a simple load-balancing strategy is used to determine the corresponding executing node.A second optimization looks for file allocations where the global intercluster traffic is minimized subject to the following constraints: minimal number of file copies, availability, and storage capacity.Experimental results showing the efficiency of the proposed algorithms are examined, and the implications of the model for the design of very large distributed file systems are discussed.