Science Data Processor Research Articles

Architecting complex, long-lived scientific software

Software is a critical aspect of large-scale science, providing essential capabilities for making scientific discoveries. Large-scale scientific projects are vast in scope, with lifespans measured in decades and costs exceeding hundreds of millions of dollars. Successfully designing software that can exist for that span of time, at that scale, is challenging for even the most capable software companies. Yet scientific endeavors face challenges with funding, staffing, and operate in complex, poorly understood software settings. In this paper we discuss the practice of early-phase software architecture in the Square Kilometre Array Observatory’s Science Data Processor. The Science Data Processor is a critical software component in this next-generation radio astronomy instrument. We customized an existing set of processes for software architecture analysis and design to this project’s unique circumstances. We report on the series of comprehensive software architecture plans that were the result. The plans were used to obtain construction approval in a critical design review with outside stakeholders. We conclude with implications for other long-lived software architectures in the scientific domain, including potential risks and mitigations.

WS-Snapshot: an effective algorithm for wide-field and large-scale imaging

ABSTRACT The Square Kilometre Array (SKA) is the largest radio interferometer under construction in the world. The high accuracy, wide-field, and large size imaging significantly challenge the construction of the Science Data Processor (SDP) of SKA. We propose a hybrid imaging method based on improved W-Stacking and snapshots. The w-range is reduced by fitting the snapshot uv plane, thus effectively enhancing the performance of the improved W-Stacking algorithm. We present a detailed implementation of WS-Snapshot. With full-scale SKA1-LOW simulations, we present the imaging performance and imaging quality results for different parameter cases. The results show that the WS-Snapshot method enables more efficient distributed processing and significantly reduces the computational time overhead within an acceptable accuracy range, which would be crucial for subsequent SKA science studies.

Performance Evaluation of Baseline-dependent Averaging Based on Full-scale SKA1-LOW Simulation

The Square Kilometre Array (SKA) is the largest radio interferometer under construction in the world. Its immense amount of visibility data poses a considerable challenge to the subsequent processing by the science data processor (SDP). Baseline dependent averaging (BDA), which reduces the amount of visibility data based on the baseline distribution of the radio interferometer, has become a focus of SKA SDP development. This paper developed and implemented a full-featured BDA module based on Radio Astronomy Simulation, Calibration and Imaging Library (RASCIL). Simulated observations were then performed with RASCIL based on a full-scale SKA1-LOW configuration. The performance of the BDA was systematically investigated and evaluated based on the simulated data. The experimental results confirmed that the amount of visibility data is reduced by about 50% to 85% for different time intervals (). In addition, different have a significant effect on the imaging quality. The smaller is, the smaller the degradation of imaging quality.

Radio-astronomical imaging on graphics processors

Abstract Realizing the next generation of radio telescopes such as the Square Kilometre Array (SKA) requires both more efficient hardware and algorithms than today’s technology provides. The image-domain gridding (IDG) algorithm is a novel approach towards solving the most compute-intensive parts of creating sky images: gridding and degridding. It alleviates the performance bottlenecks of traditional AW-projection gridding by applying instrumental and environmental corrections in the image domain instead of in the Fourier domain. In this paper, we present a thorough performance analysis of this algorithm for an Intel Xeon CPU, Intel Xeon Phi, and GPUs from AMD and NVIDIA. We show that, by evaluating trigonometric functions in hardware, GPUs are both much faster and more energy efficient than a CPU or Xeon Phi. Furthermore, on GPUs, IDG is an order of magnitude faster and more energy efficient than traditional AW-projection. IDG on GPUs is the ideal candidate imaging technique for the SKA, as it meets the computational and energy constraints of the SKA Science Data Processor system.

Big telescope, big data: towards exascale with the Square Kilometre Array.

Unlike optical telescopes, radio interferometers do not image the sky directly but require specialized image formation algorithms. For the Square Kilometre Array (SKA), the computational requirements of this image formation are extremely demanding due to the huge data rates produced by the telescope. This processing will be performed by the SKA Science Data Processor facilities and a network of SKA Regional Centres, which must not only deal with SKA-scale data volumes but also with stringent science-driven image fidelity requirements.This article is part of a discussion meeting issue ‘Numerical algorithms for high-performance computational science’.

Preparing for the SKA: The Chinese perspectives

Since 1993, astronomers and engineers from more than 100 institutions in 20 different countries including China, have engaged in the international mega-science project—the Square Kilometre Array (SKA), which will be the world’s largest synthesis radio telescope. The SKA is currently in its pre-construction phase, and China has been playing a very active role. China has participated in 6 of the total of 10 SKA work packages, and carried out R&D in dish array, aperture array, wideband single pixel feeds, phased array feeds, signal and data transport, and science data processor. In the new SKA science book published in 2015, Chinese scholars have made their contributions in 20% of all the chapter proposals. After the SKA1 re-baselining, 20 Chinese astronomers participated in 11 out of the 13 SKA scientific working groups. The “2+1 Strategy” for China’s SKA science priorities has been established, and some important progress has been made in cosmological re-ionization detection as well as in SKA data processing. In June 2015, the Chinese participation in SKA1 passed through the Chinese Academy of Sciences Panel Review. The Chinese Community is now deeply involved into the preparation of the SKA, with activities such as SKA science cases study, researching on high-precision light-weight antennas, developing the Chinese prototype array, as well as some efforts in building an SKA regional data center in China. This article gives a brief historical review of the SKA, overviews some international and domestic progress of the project during its preconstruction phase, and also discusses some future plans of the Chinese SKA.

DALiuGE: A graph execution framework for harnessing the astronomical data deluge

Abstract The Data Activated Liu 1 Graph Engine – DALiuGE 2 – is an execution framework for processing large astronomical datasets at a scale required by the Square Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex data reduction pipelines consisting of both datasets and algorithmic components and an implementation run-time to execute such pipelines on distributed resources. By mapping the logical view of a pipeline to its physical realisation, DALiuGE separates the concerns of multiple stakeholders, allowing them to collectively optimise large-scale data processing solutions in a coherent manner. The execution in DALiuGE is data-activated, where each individual data item autonomously triggers the processing on itself. Such decentralisation also makes the execution framework very scalable and flexible, supporting pipeline sizes ranging from less than ten tasks running on a laptop to tens of millions of concurrent tasks on the second fastest supercomputer in the world. DALiuGE has been used in production for reducing interferometry datasets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide Spectral Radioheliograph; and is being developed as the execution framework prototype for the Science Data Processor (SDP) consortium of the Square Kilometre Array (SKA) telescope. This paper presents a technical overview of DALiuGE and discusses case studies from the CHILES and MUSER projects that use DALiuGE to execute production pipelines. In a companion paper, we provide in-depth analysis of DALiuGE’s scalability to very large numbers of tasks on two supercomputing facilities.

The Square Kilometre Array Science Data Processor. Preliminary compute platform design

The Square Kilometre Array is a next-generation radio-telescope, to be built in South Africa and Western Australia. It is currently in its detailed design phase, with procurement and construction scheduled to start in 2017. The SKA Science Data Processor is the high-performance computing element of the instrument, responsible for producing science-ready data. This is a major IT project, with the Science Data Processor expected to challenge the computing state-of-the art even in 2020. In this paper we introduce the preliminary Science Data Processor design and the principles that guide the design process, as well as the constraints to the design. We introduce a highly scalable and flexible system architecture capable of handling the SDP workload.

Comparisons of MIPAS/ENVISAT ozone profiles with SMR/ODIN and HALOE/UARS observations

Ozone volume mixing ratio (VMR) profiles are measured by the Michelson Interferometer for passive atmospheric sounding (MIPAS) on ENVISAT. The data sets produced by the science data processor at Institut für Meteorologie und Klimaforschung (IMK), Germany are compared with those obtained by halogen occultation experiment (HALOE) on UARS and by sub-millimetre radiometer (SMR) on ODIN. For the stratospheric measurements taken during September/October 2002, the three instruments show reasonable agreement, with global mean differences within 0.1–0.3 ppmv. The typical zonal mean differences are of 0.4 ppmv for HALOE and 0.6 ppmv for SMR (4–6%) in the ozone VMR peak region at 25–30 km near the equator, though larger differences of 0.8–1 ppmv (8–10%) are also observed in a small latitude–altitude region in the tropic. A positive bias of about 0.2–0.4 ppmv in the MIPAS data in the 35–40 km region has also been found. Further studies are under way to explain these differences.