Federated file systems for clusters with remote memory communication

Abstract

We present the design, prototype implementation and initial evaluation of FedFS a novel cluster file system architecture that provides a global file space by aggregating the local file systems of the cluster nodes into a loose federation. The federated file system (FedFS) is created ad-hoc for a distributed application that runs on the cluster, and its lifetime is limited by the lifetime of the distributed application. FedFS provides location-independent global file naming, load balancing, and file migration and replication. It relies on the local file systems to perform the file I/O operations. The local file systems retain their autonomy, in the sense that their structure and content do not change to support the federated file system. Other applications may run on the local file systems without realizing that the same file system is part of one or multiple FedFS. If the distributed application permits, nodes can dynamically join or leave the federation anytime, with no modifications required to the local file system organization. FedFS is implemented as an I/O library over VIA, which supports remote memory operations. The applicability and performance of the federated file system architecture is evaluated by building a distributed NFS file server.