PCJ Java library as a solution to integrate HPC, Big Data and Artificial Intelligence workloads

Marek Nowicki,Łukasz Górski,Piotr Bała

doi:10.1186/s40537-021-00454-6

Marek Nowicki, Łukasz Górski + Show 1 more

Open Access

https://doi.org/10.1186/s40537-021-00454-6

Copy DOI

Abstract

With the development of peta- and exascale size computational systems there is growing interest in running Big Data and Artificial Intelligence (AI) applications on them. Big Data and AI applications are implemented in Java, Scala, Python and other languages that are not widely used in High-Performance Computing (HPC) which is still dominated by C and Fortran. Moreover, they are based on dedicated environments such as Hadoop or Spark which are difficult to integrate with the traditional HPC management systems. We have developed the Parallel Computing in Java (PCJ) library, a tool for scalable high-performance computing and Big Data processing in Java. In this paper, we present the basic functionality of the PCJ library with examples of highly scalable applications running on the large resources. The performance results are presented for different classes of applications including traditional computational intensive (HPC) workloads (e.g. stencil), as well as communication-intensive algorithms such as Fast Fourier Transform (FFT). We present implementation details and performance results for Big Data type processing running on petascale size systems. The examples of large scale AI workloads parallelized using PCJ are presented.

Highlights

Artificial Intelligence (AI), known as computational intelligence, is becoming more and more popular in a large number of disciplines
Big Data and AI applications are implemented in Java, Scala, Python and other languages that are not widely used in High-Performance Computing (HPC), which is still dominated by C and Fortran
"HPC workloads" section contains the performance results are presented for a different class of applications including traditional computational intensive (HPC) workloads, as well as communication-intensive algorithms such as Fast Fourier Transform (FFT), in "Data analitycs" section we present implementation details and performance results for Big Data type processing running on petascale size systems

Summary

Introduction

Artificial Intelligence (AI), known as computational intelligence, is becoming more and more popular in a large number of disciplines. It helps to solve problems for which it is impossible or at least very hard to write a traditional algorithm. A deep learning approach is very famous and widely studied. More broadly speaking, machine learning and neural network approaches, parses very large training data, learns from it by fixing its internal state. The bigger the volume and variety of the training data the neural network can better learn the environment and give better answers for the previously not observed data.

Methods

Results

Conclusion