Scalability of computer clusters

Abstract

For this study, we have chosen to work with Parallel Virtual File System (PVFS) from Clemson University; it is a distributed file system for Linux that implements wide stripping. First, we have programmed and calibrated a PVFS simulator; using this simulator, we demonstrated that PVFS has a very good scalability where the number of nodes in the cluster is smaller than a given threshold. This threshold actually corresponds to the saturation of the network bandwidth. To go beyond this limit, we propose to use wide striping and replication in the same file system. We also propose a new data distribution technique based upon chained declustering, that warranties high availability and scalability. This also allow's the system to be improved with minimal cost, without service interruption and with a minimal degradation of service. In addition, the granularity of the file system is ideal: the size of the cluster can be adjusted to the needed performances with a precision of one node. Finally, we propose a complete architecture, using cluster of clusters, where the performances are not limited by the network performances. In order to validate our file system, we use the PVFS simulator where the improvements have been implemented. The results show that the performances of the system are close to the ideal case. Once the size of the origin cluster is well defined, the total number of nodes in the system is not limited any more, and the performances increase linearly. We have also simulated an upgrade of the system, in order to measure the perturbation caused by the update of the new nodes: it is minimal and can be controlled by priority mechanisms.