Big Data and HPC Convergence for Smart Infrastructures: A Review and Proposed Architecture

Abstract

The world has seen exponential data growth due to social media, mobility, E-commerce, and other factors. The issues related to avalanche of data being produced are immense and cover variety of challenges that need a careful consideration. The use of HPDA (High Performance Data Analytics) is increasing at brisk speed in many industries and has resulted in expansion of HPC market in many new territories. HPC (High Performance Computing) and big data are different systems, not only at the technical level, but also have different ecosystems. HPC systems are mainly developed for computationally intensive applications but recently data intensive applications are also among the major workload in HPC environment. Big data analytics have grown in different perspectives and have separate developer communities. As we head towards the exascale and smart infrastructure era, the necessary integration of big data and HPC is currently a hot topic of research but still at very infant stages. Both systems have different architecture and their integration brings many challenges. The aim of this work is to identify the driving forces, challenges, current and future trends associated with the integration of HPC and big data. This paper is an extension of our earlier work. We have reviewed programming models and frameworks of big data and HPC. The big data and HPC challenges in the exascale-computing era are discussed. Additional elaborations are provided on HPC and big data convergence research efforts and future directions are provided. The HPC-big data convergence architecture proposed in our earlier paper has been enhanced.