Remote MPI-I/O on a Parallel Virtual File System Using a Circular Buffer for High Throughput

Abstract

A flexible intermediate library named Stampi realizes seamless remote MPI-I/O operations on interconnected computers with the help of its MPI-I/O process which is invoked on a remote computer. The MPI-I/O process carries out I/O operations by using vendor’s MPI-I/O library according to I/O requests from user processes. If the vendor’s one is not available, UNIX I/O functions are used instead of it. A parallel virtual file system (PVFS) was supported in the remote MPI-I/O mechanism for data-intensive applications. Although this mechanism provided parallel I/O operations on a PVFS file system, its performance with UNIX I/O functions was low. Attempts to obtain high throughput have been made for this case with adopting a circular buffer mechanism in the MPI-I/O process to cache a part of or whole data. By optimizing configuration of the buffer, remote MPI-I/O operations with UNIX I/O functions outperformed those with direct calls of PVFS I/O functions on a PVFS file system.