Integration of parallel I/O library and flash native accelerators

Abstract

As illustrated by the IO500 ranking [9] performance and scalability of storage systems have improved dramatically in the recent years. Most of these progresses are due to novel hardware and architectures. The resulting solutions tend to be more complex and harder to manage from an end-user perspective. This work proposes to address simultaneously the user experience and the performance issue. The solution is based on DDN's Infinite Memory Engine (IME) [12], an all Flash distributed cache that sits between the compute nodes and the parallel file system. IME delivers extreme performance for bulk data and complex I/O patterns but does not accelerate metadata traffic. To shield the end-user from the additional complexity of dealing with an extra storage-tier, IME accesses are embedded within SIONlib. SIONlib is a parallel I/O library, which has the ability to provide a task local like access scheme for a single file in the underlying file system that allows to reduce the number of metadata operations. This combined construction with IME and SIONlib allows end-users to obtain the Flash level of performance transparently for their applications and also to receive a performance boost on metadata operations. In respect of traditional file systems, according to the partest benchmark suite, the solution improves performance over 1.8 times on a single node and up to 4 times for multiple nodes. Moreover, experimenting with SIONlib parameters we observe that the resulting performances are more robust towards different data layout patterns and transfer sizes than with traditional file systems.