On-the-Fly, Robust Translation of MPI Libraries

Abstract

Most parallel scientific applications rely on third-party libraries, some of which may have multiple implementations including open-source and vendor-proprietary. While sharing an application programming interface (API), many of these implementations do not have a shared application binary interface (ABI) and require recompiling applications to change the library implementation used. For many applications, recompiling is a long and complex process and sometimes not even an option when the application is shipped binary only. ABI incompatibility strikes at the heart of portability, productivity, and performance by (1) impeding application execution across different systems; (2) adding developer hours rebuilding an application; and (3) not taking advantage of host-optimized libraries.In this paper, we present a methodology and framework to solve ABI incompatibility across MPI libraries, which follow a well-defined API. The proposed framework called Wi4MPI translates the ABI dynamically from the MPI library used to build the application to a different MPI library available at run time. We show Wi4MPI works robustly on a wide spectrum of architectures, networks, and MPI libraries. Furthermore, we demonstrate its usefulness on several use cases highlighting significant portability, performance, and productivity benefits.