Radio Interferometric Calibration Research Articles

A Bayesian approach to high-fidelity interferometric calibration – I. Mathematical formalism

ABSTRACT High-fidelity radio interferometric data calibration that minimizes spurious spectral structure in the calibrated data is essential in astrophysical applications, such as 21 cm cosmology, which rely on knowledge of the relative spectral smoothness of distinct astrophysical emission components to extract the signal of interest. Existing approaches to radio interferometric calibration have been shown to impart spurious spectral structure to the calibrated data if the sky model used to calibrate the data is incomplete. In this paper, we introduce BayesCal: a novel solution to the sky-model incompleteness problem in interferometric calibration, designed to enable high-fidelity data calibration. The BayesCal data model supplements the a priori known component of the forward model of the sky with a statistical model for the missing and uncertain flux contribution to the data, constrained by a prior on the power in the model. We demonstrate how the parameters of this model can be marginalized out analytically, reducing the dimensionality of the parameter space to be sampled from and allowing one to sample directly from the posterior probability distribution of the calibration parameters. Additionally, we show how physically motivated priors derived from theoretical and measurement-based constraints on the spectral smoothness of the instrumental gains can be used to constrain the calibration solutions. In a companion paper, we apply this algorithm to simulated observations with a HERA-like array and demonstrate that it enables up to four orders of magnitude suppression of power in spurious spectral fluctuations relative to standard calibration approaches.

Tackling the Unique Challenges of Low-frequency Solar Polarimetry with the Square Kilometre Array Low Precursor: The Algorithm

Coronal magnetic fields are well known to be one of the crucial parameters defining coronal physics and space weather. However, measuring the global coronal magnetic fields remains challenging. The polarization properties of coronal radio emissions are sensitive to coronal magnetic fields. While they can prove to be useful probes of coronal and heliospheric magnetic fields, their usage has been limited by technical and algorithmic challenges. We present a robust algorithm for precise polarization calibration and imaging of low-radio frequency solar observations and demonstrate it on data from the Murchison Widefield Array, a Square Kilometre Array (SKA) precursor. This algorithm is based on the Measurement Equation framework, which forms the basis of all modern radio interferometric calibration and imaging. It delivers high-dynamic-range and high-fidelity full-Stokes solar radio images with instrumental polarization leakages <1%, on par with general astronomical radio imaging, and represents the state of the art. Opening up this rewarding, yet unexplored, phase space will enable multiple novel science investigations and offer considerable discovery potential. Examples include detection of low-level circular polarization from thermal coronal emission to estimate large-scale quiescent coronal fields; polarization of faint gyrosynchrotron emissions from coronal mass ejections for robust estimation of plasma parameters; and detection of the first-ever linear polarization at these frequencies. This method has been developed with the SKA in mind and will enable a new era of high-fidelity spectropolarimetric snapshot solar imaging at low radio frequencies.

Solution intervals considered harmful: on the optimality of radio interferometric gain solutions

ABSTRACT Solution intervals are often used to improve the signal-to-noise ratio during radio interferometric gain calibration. This work investigates how factors such as the noise level, intrinsic gain variability, degree of model incompleteness, and the presence of radio frequency interference impact the selection of solution intervals for calibration. We perform different interferometric simulations to demonstrate how these factors, in combination with the choice of solution intervals, affect calibration and imaging outputs and discuss practical guidelines for choosing optimal solution intervals. Furthermore, we present an algorithm capable of automatically selecting suitable solution intervals during calibration. By applying the algorithm to both simulated and real data, we show that it can successfully choose solution intervals that strike a good balance between capturing intrinsic gain variability and not fitting noise as long as the data are not too inhomogeneously flagged. Furthermore, we elaborate on several practical aspects that emphasize the need to develop regularized calibration algorithms that do not require solution intervals.

Unified radio interferometric calibration and imaging with joint uncertainty quantification

The data reduction procedure for radio interferometers can be viewed as a combined calibration and imaging problem. We present an algorithm that unifies cross-calibration, self-calibration, and imaging. Because it is a Bayesian method, this algorithm not only calculates an estimate of the sky brightness distribution, but also provides an estimate of the joint uncertainty which entails both the uncertainty of the calibration and that of the actual observation. The algorithm is formulated in the language of information field theory and uses Metric Gaussian Variational Inference (MGVI) as the underlying statistical method. So far only direction-independent antenna-based calibration is considered. This restriction may be released in future work. An implementation of the algorithm is contributed as well.

Statistical performance of radio interferometric calibration

Calibration is an essential step in radio interferometric data processing that corrects the data for systematic errors and in addition, subtracts bright foreground interference to reveal weak signals hidden in the residual. These weak and unknown signals are much sought after to reach many science goals but the effect of calibration on such signals is an ever present concern. The main reason for this is the incompleteness of the model used in calibration. Distributed calibration based on consensus optimization has been shown to mitigate the effect due to model incompleteness by calibrating data covering a wide bandwidth in a computationally efficient manner. In this paper, we study the statistical performance of direction dependent distributed calibration, i.e., the distortion caused by calibration on the residual statistics. In order to study this, we consider the mapping between the input uncalibrated data and the output residual data. We derive an analytical relationship for the influence of the input on the residual and use this to find the relationship between the input and output probability density functions. Using simulations we show that the smallest eigenvalue of the Jacobian of this mapping is a reliable indicator of the statistical performance of calibration. The analysis developed in this paper can also be applied to other data processing steps in radio interferometry such as imaging and foreground subtraction as well as to many other machine learning problems.

The variance of radio interferometric calibration solutions

This paper investigates the possibility of improving radio interferometric images using an algorithm inspired by an optical method known as “lucky imaging”, which would give more weight to the best-calibrated visibilities used to make a given image. A fundamental relationship between the statistics of interferometric calibration solution residuals and those of the image-plane pixels is derived in this paper. This relationship allows us to understand and describe the statistical properties of the residual image. In this framework, the noise map can be described as the Fourier transform of the covariance between residual visibilities in a new differential Fourier plane. Image-plane artefacts can be seen as one realisation of the pixel covariance distribution, which can be estimated from the antenna gain statistics. Based on this relationship, we propose a means of improving images made with calibrated visibilities using weighting schemes. This improvement would occur after calibration, but before imaging; it is thus ideally used between major iterations of self-calibration loops. Applying the weighting scheme to simulated data improves the noise level in the final image at negligible computational cost.

Cubical – fast radio interferometric calibration suite exploiting complex optimization

It has recently been shown that radio interferometric gain calibration can be expressed succinctly in the language of complex optimisation. In addition to providing an elegant framework for further development, it exposes properties of the calibration problem which can be exploited to accelerate traditional non-linear least squares solvers such as Gauss-Newton and Levenberg-Marquardt. We extend existing derivations to chains of Jones terms: products of several gains which model different aberrant effects. In doing so, we find that the useful properties found in the single term case still hold. We also develop several specialised solvers which deal with complex gains parameterised by real values. The newly developed solvers have been implemented in a Python package called CubiCal, which uses a combination of Cython, multiprocessing and shared memory to leverage the power of modern hardware. We apply CubiCal to both simulated and real data, and perform both direction-independent and direction-dependent self-calibration. Finally, we present the results of some rudimentary profiling to show that CubiCal is competitive with respect to existing calibration tools such as MeqTrees.

Data multiplexing in radio interferometric calibration

New and upcoming radio interferometers will produce unprecedented amounts of data that demand extremely powerful computers for processing. This is a limiting factor due to the large computational power and energy costs involved. Such limitations restrict several key data processing steps in radio interferometry. One such step is calibration where systematic errors in the data are determined and corrected. Accurate calibration is an essential component in reaching many scientific goals in radio astronomy and the use of consensus optimization that exploits the continuity of systematic errors across frequency significantly improves calibration accuracy. In order to reach full consensus, data at all frequencies need to be calibrated simultaneously. In the SKA regime, this can become intractable if the available compute agents do not have the resources to process data from all frequency channels simultaneously. In this paper, we propose a multiplexing scheme that is based on the alternating direction method of multipliers (ADMM) with cyclic updates. With this scheme, it is possible to simultaneously calibrate the full dataset using far fewer compute agents than the number of frequencies at which data are available. We give simulation results to show the feasibility of the proposed multiplexing scheme in simultaneously calibrating a full dataset when a limited number of compute agents are available.

Limits on the validity of the thin-layer model of the ionosphere for radio interferometric calibration

For a ground-based radio interferometer observing at low frequencies, the ionosphere causes propagation delays and refraction of cosmic radio waves which result in phase errors in the received signal. These phase errors can be corrected using a calibration method that assumes a two-dimensional phase screen at a fixed altitude above the surface of the Earth, known as the thin-layer model. Here we investigate the validity of the thin-layer model and provide a simple equation with which users can check when this approximation can be applied to observations for varying time of day, zenith angle, interferometer latitude, baseline length, ionospheric electron content and observing frequency.

Distributed radio interferometric calibration

Increasing data volumes delivered by a new generation of radio interferometers require computationally efficient and robust calibration algorithms. In this paper, we propose distributed calibration as a way of improving both computational cost as well as robustness in calibration. We exploit the data parallelism across frequency that is inherent in radio astronomical observations that are recorded as multiple channels at different frequencies. Moreover, we also exploit the smoothness of the variation of calibration parameters across frequency. Data parallelism enables us to distribute the computing load across a network of compute agents. Smoothness in frequency enables us reformulate calibration as a consensus optimization problem. With this formulation, we enable flow of information between compute agents calibrating data at different frequencies, without actually passing the data, and thereby improving robustness. We present simulation results to show the feasibility as well as the advantages of distributed calibration as opposed to conventional calibration.

Radio interferometric gain calibration as a complex optimization problem

Recent developments in optimization theory have extended some traditional algorithms for least-squares optimization of real-valued functions (Gauss–Newton, Levenberg–Marquardt, etc.) into the domain of complex functions of a complex variable. This employs a formalism called the Wirtinger derivative, and derives a full-complex Jacobian counterpart to the conventional real Jacobian. We apply these developments to the problem of radio interferometric gain calibration, and show how the general complex Jacobian formalism, when combined with conventional optimization approaches, yields a whole new family of calibration algorithms, including those for the polarized and direction-dependent gain regime. We further extend the Wirtinger calculus to an operator-based matrix calculus for describing the polarized calibration regime. Using approximate matrix inversion results in computationally efficient implementations; we show that some recently proposed calibration algorithms such as StefCal and peeling can be understood as special cases of this, and place them in the context of the general formalism. Finally, we present an implementation and some applied results of CohJones, another specialized direction-dependent calibration algorithm derived from the formalism.

Radio interferometric calibration via ordered-subsets algorithms: OS–LS and OS–SAGE calibrations

The main objective of this work is to accelerate the Maximum-Likelihood (ML) estimation procedure in radio interferometric calibration. We introduce the OS-LS and the OS-SAGE radio interferometric calibration methods, as a combination of the Ordered-Subsets (OS) method with the Least-Squares (LS) and Space Alternating Generalized Expectation maximization (SAGE) calibration techniques, respectively. The OS algorithm speeds up the ML estimation and achieves nearly the same level of accuracy of solutions as the one obtained by the non-OS methods. We apply the OS-LS and OS-SAGE calibration methods to simulated observations and show that these methods have a much higher convergence rate relative to the conventional LS and SAGE techniques. Moreover, the obtained results show that the OS-SAGE calibration technique has a superior performance compared to the OS-LS calibration method in the sense of achieving more accurate results while having significantly less computational cost.

Clustered calibration: an improvement to radio interferometric direction-dependent self-calibration

The new generation of radio synthesis arrays, such as LOFAR and SKA, have been designed to surpass existing arrays in terms of sensitivity, angular resolution and frequency coverage. This evolution has led to the development of advanced calibration techniques that ensure the delivery of accurate results at the lowest possible computational cost. However, the performance of such calibration techniques is still limited by the compact, bright sources in the sky, used as calibrators. It is important to have a bright enough source that is well distinguished from the background noise level in order to achieve satisfactory results in calibration. We present "clustered calibration" as a modification to traditional radio interferometric calibration, in order to accommodate faint sources that are almost below the background noise level into the calibration process. The main idea is to employ the information of the bright sources' measured signals as an aid to calibrate fainter sources that are nearby the bright sources. In the case where we do not have bright enough sources, a source cluster could act as a bright source that can be distinguished from background noise. We construct a number of source clusters assuming that the signals of the sources belonging to a single cluster are corrupted by almost the same errors, and each cluster is calibrated as a single source, using the combined coherencies of its sources simultaneously. This upgrades the power of an individual faint source by the effective power of its cluster. We give performance analysis of clustered calibration to show the superiority of this approach compared to the traditional unclustered calibration. We also provide analytical criteria to choose the optimum number of clusters for a given observation in an efficient manner.

On the interpolation of calibration solutions obtained in radio interferometry

Full polarimetric radio interferometric calibration is performed by estimating 2 by 2 Jones matrices representing instrumental and propagation effects. The solutions obtained in this way differ from the true solutions by a 2 by 2 unitary matrix ambiguity. This ambiguity is common to all stations for which a solution is obtained but it is different for solutions obtained at different time and frequency intervals. Therefore, straightforward interpolation of solutions obtained at different time and frequency intervals is not possible. In this paper, we propose to use the theory of quotient manifolds for obtaining correct interpolants that are immune to unitary matrix ambiguities.

Revisiting the radio interferometer measurement equation

Paper I of the series re-derived the radio interferometry measurement equation (RIME) from first principles, and extended the Jones formalism to the full-sky case, incorporating direction-dependent effects (DDEs). This paper aims to describe both classical radio interferometric calibration (selfcal and related methods), and the recent developments in the treatment of DDEs, using the RIME-based mathematical framework developed in Paper I. It also aims to demonstrate the ease with which the various effects can be described and understood. The first section of this paper uses the RIME formalism to describe self-calibration, both with a full RIME, and with the approximate equations of older software packages, and shows how this is affected by DDEs. The second section gives an overview of real-life DDEs and proposed methods of dealing with them. This results in a formal RIME-based description and comparison of existing and proposed approaches to the problem of DDEs.