A Novel Stair-Case Replication (SCR) Based Fault Tolerance for MPI Applications

Abstract

When computational clusters increase in size, their mean time to failure reduces drastically. We generally use checkpoint to minimize the loss of computation. Most check pointing techniques, however, require central storage for storing checkpoints. This results in a bottleneck and severely limits the scalability of check pointing, while also proving to be too expensive for dedicated check pointing networks and storage systems. We propose a Stair-Case Replication (SCR) Based MPI check pointing facility. Our reference implementation is based on LAM/MPI; however, it is directly applicable to any MPI implementation. We use the staircase method of fault-tolerant MPI with asynchronous replication, eliminating the need for central or network storage. We evaluate centralized storage, a Sun-X4500-based solution, an EMC storage area network (SAN), and the Ibrix commercial parallel file system and show that they are not scalable, particularly after 64 CPUs. We use the staircase MPI method which allows the access point in a lower complexity level to the higher complexity level which improves the efficiency of the previous method.