Abstract
Convergence between high-performance computing (HPC) and big data analytics (BDA) is currently an established research area that has spawned new opportunities for unifying the platform layer and data abstractions in these ecosystems. This work presents an architectural model that enables the interoperability of established BDA and HPC execution models, reflecting the key design features that interest both the HPC and BDA communities, and including an abstract data collection and operational model that generates a unified interface for hybrid applications. This architecture can be implemented in different ways depending on the process- and data-centric platforms of choice and the mechanisms put in place to effectively meet the requirements of the architecture. The Spark-DIY platform is introduced in the paper as a prototype implementation of the architecture proposed. It preserves the interfaces and execution environment of the popular BDA platform Apache Spark, making it compatible with any Spark-based application and tool, while providing efficient communication and kernel execution via DIY, a powerful communication pattern library built on top of MPI. Later, Spark-DIY is analyzed in terms of performance by building a representative use case from the hydrogeology domain, EnKF-HGS. This application is a clear example of how current HPC simulations are evolving toward hybrid HPC-BDA applications, integrating HPC simulations within a BDA environment.
Highlights
Convergence between high-performance computing (HPC) and big data analytics (BDA) is an established research area that has spawned new research topics such as data-intensive scientific computing, high-performance data analytics, and hybrid platforms and infrastructures based on virtualization techniques and novel storage hierarchies
We summarize our contributions as follows: 1) A definition of a generic unified distributed data abstraction (UDDA) and its associated unified operational model (UOM), which sets the foundation of a theoretical frame for the analysis and definition of composite HPC-BDA applications
The rest of this paper is organized as follows: Sections II and III introduce the BDA and HPC ecosystems, respectively, and develop on their current state; Section IV presents relevant works related to the HPC-BDA convergence problem; Section V analyzes the challenges and opportunities of the convergence of such paradigms; Section VI details the proposal of an abstract architecture suitable for the interoperation of process- and data-centric platforms, which is later implemented in Section VII, using Apache Spark and a communication library built on Message Passing Interface (MPI), and evaluated in Section VIII on a real use case from the hydrogeology domain; and Section IX summarizes this work, its applications, and directions for future research
Summary
Convergence between high-performance computing (HPC) and big data analytics (BDA) is an established research area that has spawned new research topics such as data-intensive scientific computing, high-performance data analytics, and hybrid platforms and infrastructures based on virtualization techniques and novel storage hierarchies. The rest of this paper is organized as follows: Sections II and III introduce the BDA and HPC ecosystems, respectively, and develop on their current state; Section IV presents relevant works related to the HPC-BDA convergence problem; Section V analyzes the challenges and opportunities of the convergence of such paradigms; Section VI details the proposal of an abstract architecture suitable for the interoperation of process- and data-centric platforms, which is later implemented, using Apache Spark and a communication library built on MPI, and evaluated in Section VIII on a real use case from the hydrogeology domain; and Section IX summarizes this work, its applications, and directions for future research The rest of this paper is organized as follows: Sections II and III introduce the BDA and HPC ecosystems, respectively, and develop on their current state; Section IV presents relevant works related to the HPC-BDA convergence problem; Section V analyzes the challenges and opportunities of the convergence of such paradigms; Section VI details the proposal of an abstract architecture suitable for the interoperation of process- and data-centric platforms, which is later implemented in Section VII, using Apache Spark and a communication library built on MPI, and evaluated in Section VIII on a real use case from the hydrogeology domain; and Section IX summarizes this work, its applications, and directions for future research
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
