Multi-core multi-node parallelization of the radio interferometric imaging pipeline DDFacet

Abstract

The next generation of radio telescopes, such as the Square Kilometer Array (SKA), will need to process an incredible amount of data in real-time. In addition, the sensitivity of SKA will require a new generation of calibration and imaging software to exploit its full potential. The wide-field direction-dependent spectral deconvolution framework, called DDFacet, has been successfully used in several existing SKA pathfinders and precursors like MeerKAT and LOFAR. This imager allows a multi-core execution based on facets parallelization and a multi-node execution based on observations parallelization. However, because of the amount of data to be processed, the data on a single observation will have to be distributed on several nodes. This paper proposes the first two-level parallelization of DDFacet in the case of a single observation. A multi-core parallelization based on facets and a multi-node parallelization based on frequency distribution grouped in Measurement Sets. We show that this multi-core multi-node parallelization has successfully reduced the total execution time by a factor of 5.7 on a LOFAR dataset.