Abstract
With the construction of the Square Kilometre Array (SKA), the world's largest radio telescope, a new era in the detection of 21 cm neutral hydrogen (HI) spectral line is about to be ushered in, and the SKA spectral line sciences present tremendous opportunities and unprecedented data processing challenges. In this paper, we propose a fast processing pipeline for spectral line data based on the distribution execution framework Data Activated Liu Graph Engine (DALiuGE), and compare it with the Message Passing Interface (MPI) based method. The proposed pipeline is proved to be more stable and efficient on several comparison experiments. The scalability and execution efficiency performance of SKA spectral line data fast processing pipeline using DALiuGE for data volume changes have been verified, and it is believed that the DALiuGE based pipeline can be a solution for more scientific tasks facing massive data processing on SKA Regional Centre (SRC).
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
