Overlay striping and optimal parallel I/O for modern applications

Abstract

Disk array systems are rapidly becoming the secondary-storage media of choice for many emerging applications with large storage and high bandwidth requirements. Striping data across the disks of a disk array introduces significant performance benefits mainly because the effective transfer rate of the secondary storage is increased by a factor equal to the stripe width. However, the choice of the optimal stripe width is an open problem: no general formal analysis has been reported and intuition alone fails to provide good guidelines. As a result one may find occasionally contradictory recommendations in the literature. With this work we first contribute an analytical calculation of the optimal stripe width. Second, we recognize that the optimal stripe width is sensitive to the multiprogramming level, which is not known a priori and fluctuates with time. Thus, calculations of the optimal stripe width are, by themselves only, of little practical use. For this reason we propose a novel striping technique, called overlay striping, which allows objects to be retrieved using a number of alternative stripe widths. We provide the detailed algorithms for our overlay striping method and we study the associated storage overhead and performance improvements and we show that we can achieve near optimal performance for very wide ranges of the possible multiprogramming levels, while incurring small storage overheads.