Distributed parallel file system for I/O intensive parallel computing on clusters

Abstract

Local area networks are now widely used to run parallel applications. They are particularly suitable to run I/O intensive applications, because each node usually includes disk space. However, programming these applications is rather difficult. The programmer must partition data into disk nudes and, at run time, transfer data from disk space into the memory of each node that uses the data and vice versa. Also, such partitioning gives data a fixed location in disk space, and is usually not adequate for performance because processors do not access mostly data in their local memory or disk, but on disk space in remote nodes. This paper presents a distributed parallel file system that both eases the programming and improves the performance of parallel I/O intensive applications. Our file system eases programming through mapping into memory files of size up to hundreds of Giga bytes. It improves performance through automatically diffusing, or migrating and replicating, data in files to the local memory or local disk of the processors that use the data. Data diffusion occurs under a multiple-readers-single-writer protocol. On applications tested the performance gain can be up to 20 % compared to versions using the MPI file system.