Calibration Pipeline Research Articles

Exploring cosmic origins with CORE: Mitigation of systematic effects

We present an analysis of the main systematic effects that could impact the measurement of CMB polarization with the proposed CORE space mission. We employ timeline-to-map simulations to verify that the CORE instrumental set-up and scanning strategy allow us to measure sky polarization to a level of accuracy adequate to the mission science goals. We also show how the CORE observations can be processed to mitigate the level of contamination by potentially worrying systematics, including intensity-to-polarization leakage due to bandpass mismatch, asymmetric main beams, pointing errors and correlated noise. We use analysis techniques that are well validated on data from current missions such as Planck to demonstrate how the residual contamination of the measurements by these effects can be brought to a level low enough not to hamper the scientific capability of the mission, nor significantly increase the overall error budget. We also present a prototype of the CORE photometric calibration pipeline, based on that used for Planck, and discuss its robustness to systematics, showing how CORE can achieve its calibration requirements. While a fine-grained assessment of the impact of systematics requires a level of knowledge of the system that can only be achieved in a future study phase, the analysis presented here strongly suggests that the main areas of concern for the CORE mission can be addressed using existing knowledge, techniques and algorithms.

Visible Infrared Imaging Radiometer Suite reflective solar bands on-orbit calibration using solar diffuser illuminated by scattered light through the nadir port.

A new variant to the standard on-orbit calibration of the reflective solar bands (RSBs) using a solar diffuser (SD) is formulated. Instead of direct solar exposure through the SD port in the front of the instrument as originally designed, the variant method uses light reflecting off Earth's surface coming through the nadir port as the light source to illuminate the built-in onboard SD. The methodology is applied to the Visible Infrared Imaging Radiometer Suite on board the Suomi National Polar-orbiting Partnership satellite, and is shown to be viable and useful. This approach effectively preserves the standard calibration pipeline other than using a different set of illumination data, corresponding to a different illumination source, for computing the radiance emanating from the SD. It has the added advantages of not dealing with operational needs for the standard calibration activities and completely bypassing the characterization of the transmission function of the attenuation screen in the front of the SD port. The RSB calibration coefficients are computed from the data of scattered light from the SD sector per each orbit, and a 16-day average is taken. The variant calibration coefficients are shown to well match the standard solar-based RSB calibration coefficients for Bands M5 to M8, but diverging results emerge for Bands M1 to M4, highlighting the known non-ideal behavior in the degradation of SD that contributes to the worsening error in RSB calibration. The result also shows a consistent 2% variation mission-long for all RSBs, showing the overall consistency of this first analysis of the new method but also the level of the uncertainty. The result and the implications of this study are discussed.

Improved Reference Sampling and Subtraction: A Technique for Reducing the Read Noise of Near-infrared Detector Systems

Near-infrared array detectors, like the James Webb Space Telescope (JWST) NIRSpec’s Teledyne’s H2RGs, often provide reference pixels and a reference output. These are used to remove correlated noise. Improved reference sampling and subtraction (IRS2) is a statistical technique for using this reference information optimally in a least-squares sense. Compared with the traditional H2RG readout, IRS2 uses a different clocking pattern to interleave many more reference pixels into the data than is otherwise possible. Compared with standard reference correction techniques, IRS2 subtracts the reference pixels and reference output using a statistically optimized set of frequency-dependent weights. The benefits include somewhat lower noise variance and much less obvious correlated noise. NIRSpec’s IRS2 images are cosmetically clean, with less banding than in traditional data from the same system. This article describes the IRS2 clocking pattern and presents the equations needed to use IRS2 in systems other than NIRSpec. For NIRSpec, applying these equations is already an option in the calibration pipeline. As an aid to instrument builders, we provide our prototype IRS2 calibration software and sample JWST NIRSpec data. The same techniques are applicable to other detector systems, including those based on Teledyne’s H4RG arrays. The H4RG’s interleaved reference pixel readout mode is effectively one IRS2 pattern.

3D visual perception for self-driving cars using a multi-camera system: Calibration, mapping, localization, and obstacle detection

Cameras are a crucial exteroceptive sensor for self-driving cars as they are low-cost and small, provide appearance information about the environment, and work in various weather conditions. They can be used for multiple purposes such as visual navigation and obstacle detection. We can use a surround multi-camera system to cover the full 360-degree field-of-view around the car. In this way, we avoid blind spots which can otherwise lead to accidents. To minimize the number of cameras needed for surround perception, we utilize fisheye cameras. Consequently, standard vision pipelines for 3D mapping, visual localization, obstacle detection, etc. need to be adapted to take full advantage of the availability of multiple cameras rather than treat each camera individually. In addition, processing of fisheye images has to be supported. In this paper, we describe the camera calibration and subsequent processing pipeline for multi-fisheye-camera systems developed as part of the V-Charge project. This project seeks to enable automated valet parking for self-driving cars. Our pipeline is able to precisely calibrate multi-camera systems, build sparse 3D maps for visual navigation, visually localize the car with respect to these maps, generate accurate dense maps, as well as detect obstacles based on real-time depth map extraction.

The MOTH II Doppler-Magnetographs and Data Calibration Pipeline

AbstractThe calibration pipeline of the level zero images obtained from the Magneto-Optical filters at Two Heights (MOTH II) instrument is presented. MOTH II consists of two 20 cm aperture instruments, each using a Magneto-Optical Filter (MOF): one at 5896 Å (Na D2-line), the other one at 7700 Å (K I-line). MOTH II instruments thus provide full disk line-of-sight Doppler velocity and magnetic field measurements at two heights in the solar atmosphere. The developed MOTH II pipeline employs a set of standard calibration corrections, a correction for signal leakage, due to the non-ideal behavior of the polarizers, and the geometrical registration between the eight images acquired by four CMOS cameras, relative to two components of the signal in two circular polarization states, in each of the two channels. MOTH II data are used to investigate atmospheric dynamics (e.g., internal gravity waves and magneto-acoustic portals) and Space Weather phenomena. Particularly, flare forecasting algorithms, based on the detection of magnetic active regions (ARs) and associated flare probability estimation, are currently under development. The possible matching of MOTH II data with SDO/HMI and SDO/AIA images into a flux rope model, developed in collaboration between Harvard-Smithsonian CfA and MIT Laboratory for Nuclear Science, is being tested.

Calibration of LOFAR data on the cloud

Abstract New scientific instruments are starting to generate an unprecedented amount of data. The Low Frequency Array (LOFAR), one of the Square Kilometre Array (SKA) pathfinders, is already producing data on a petabyte scale. The calibration of these data presents a huge challenge for final users: (a) extensive storage and computing resources are required; (b) the installation and maintenance of the software required for the processing is not trivial; and (c) the requirements of calibration pipelines, which are experimental and under development, are quickly evolving. After encountering some limitations in classical infrastructures like dedicated clusters, we investigated the viability of cloud infrastructures as a solution. We found that the installation and operation of LOFAR data calibration pipelines is not only possible, but can also be efficient in cloud infrastructures. The main advantages were: (1) the ease of software installation and maintenance, and the availability of standard APIs and tools, widely used in the industry; this reduces the requirement for significant manual intervention, which can have a highly negative impact in some infrastructures; (2) the flexibility to adapt the infrastructure to the needs of the problem, especially as those demands change over time; (3) the on-demand consumption of (shared) resources. We found that a critical factor (also in other infrastructures) is the availability of scratch storage areas of an appropriate size. We found no significant impediments associated with the speed of data transfer, the use of virtualization, the use of external block storage, or the memory available (provided a minimum threshold is reached). Finally, we considered the cost-effectiveness of a commercial cloud like Amazon Web Services. While a cloud solution is more expensive than the operation of a large, fully-utilized cluster completely dedicated to LOFAR data reduction, we found that its costs are competitive if the number of datasets to be analysed is not high, or if the costs of maintaining a system capable of calibrating LOFAR data become high. Coupled with the advantages discussed above, this suggests that a cloud infrastructure may be favourable for many users.

GPhoton: THE GALEX PHOTON DATA ARCHIVE

ABSTRACT gPhoton is a new database product and software package that enables analysis of GALEX ultraviolet data at the photon level. The project’s stand-alone, pure-Python calibration pipeline reproduces the functionality of the original mission pipeline to reduce raw spacecraft data to lists of time-tagged, sky-projected photons, which are then hosted in a publicly available database by the Mikulski Archive at Space Telescope. This database contains approximately 130 terabytes of data describing approximately 1.1 trillion sky-projected events with a timestamp resolution of five milliseconds. A handful of Python and command-line modules serve as a front end to interact with the database and to generate calibrated light curves and images from the photon-level data at user-defined temporal and spatial scales. The gPhoton software and source code are in active development and publicly available under a permissive license. We describe the motivation, design, and implementation of the calibration pipeline, database, and tools, with emphasis on divergence from prior work, as well as challenges created by the large data volume. We summarize the astrometric and photometric performance of gPhoton relative to the original mission pipeline. For a brief example of short time-domain science capabilities enabled by gPhoton, we show new flares from the known M-dwarf flare star CR Draconis. The gPhoton software has permanent object identifiers with the ASCL (ascl:1603.004) and DOI (doi:10.17909/T9CC7G). This paper describes the software as of version v1.27.2.

GaiaData Release 1

Context. Gaia is an ESA cornerstone mission launched on 19 December 2013 aiming to obtain the most complete and precise 3D map of our Galaxy by observing more than one billion sources. This paper is part of a series of documents explaining the data processing and its results for Gaia Data Release 1, focussing on the G band photometry. Aims. This paper describes the calibration model of the Gaia photometric passband for Gaia Data Release 1. Methods. The overall principle of splitting the process into internal and external calibrations is outlined. In the internal calibration, a self-consistent photometric system is generated. Then, the external calibration provides the link to the absolute photometric flux scales. Results. The Gaia photometric calibration pipeline explained here was applied to the first data release with good results. Details are given of the various calibration elements including the mathematical formulation of the models used and of the extraction and preparation of the required input parameters (e.g. colour terms). The external calibration in this first release provides the absolute zero point and photometric transformations from the Gaia G passband to other common photometric systems. Conclusions. This paper describes the photometric calibration implemented for the first Gaia data release and the instrumental effects taken into account. For this first release no aperture losses, radiation damage, and other second-order effects have not yet been implemented in the calibration.

THE MURCHISON WIDEFIELD ARRAY 21 cm POWER SPECTRUM ANALYSIS METHODOLOGY

ABSTRACT We present the 21 cm power spectrum analysis approach of the Murchison Widefield Array Epoch of Reionization project. In this paper, we compare the outputs of multiple pipelines for the purpose of validating statistical limits cosmological hydrogen at redshifts between 6 and 12. Multiple independent data calibration and reduction pipelines are used to make power spectrum limits on a fiducial night of data. Comparing the outputs of imaging and power spectrum stages highlights differences in calibration, foreground subtraction, and power spectrum calculation. The power spectra found using these different methods span a space defined by the various tradeoffs between speed, accuracy, and systematic control. Lessons learned from comparing the pipelines range from the algorithmic to the prosaically mundane; all demonstrate the many pitfalls of neglecting reproducibility. We briefly discuss the way these different methods attempt to handle the question of evaluating a significant detection in the presence of foregrounds.

The Calibration Reference Data System

Abstract We describe a software architecture and implementation for using rules to determine which calibration files are appropriate for calibrating a given observation. This new system, the Calibration Reference Data System (CRDS), replaces what had been previously used for the Hubble Space Telescope (HST) calibration pipelines, the Calibration Database System (CDBS). CRDS will be used for the James Webb Space Telescope (JWST) calibration pipelines, and is currently being used for HST calibration pipelines. CRDS can be easily generalized for use in similar applications that need a rules-based system for selecting the appropriate item for a given dataset; we give some examples of such generalizations that will likely be used for JWST. The core functionality of the Calibration Reference Data System is available under an Open Source license. CRDS is briefly contrasted with a sampling of other similar systems used at other observatories.

Radiometric calibration of the Visible Infrared Imaging Radiometer Suite reflective solar bands with robust characterizations and hybrid calibration coefficients

The Visible Infrared Imaging Radiometer Suite (VIIRS) is now entering its fourth year of in-orbit global environmental observation and is producing a wide range of scientific output. The ocean color products in particular require a level of accuracy from the reflective solar bands (RSBs) that is a magnitude higher than the specification. In this work, we present an updated and completed core calibration pipeline that achieves the best sensor data records (SDR) to date and helps the ocean color products to reach maturity. We review the core calibration methodology of the RSBs and describe each essential input, including the solar diffuser stability monitor, the solar diffuser (SD), and lunar calibrations. Their associated issues, along with the successful mitigation and improved results, are described and presented. In particular, we illuminate the inaccuracy suffered due to the evolving angular dependence in the degradation of the on-board SD that impacts the heart of the RSB calibration, but also show that lunar-based calibration instead provides the correct long-term baseline for the successful restoration of the core methodology. The new look-up tables, which combine the coefficients from the SD-based and lunar-based calibrations, produce the optimal result, with an estimated accuracy of ∼0.2%. This hybrid approach highlights significant progress in the VIIRS RSB calibration and marks a completion of the core calibration result upon which other physical impacts or scientific issues can then be more accurately examined. We demonstrate the significant improvement and its impact on the ocean color products by comparing the current official output to the newly generated result. Lastly, we point out that this hybrid calibration coefficients scheme is made possible by a VIIRS design and layout change over its predecessor, the Moderate Resolution Imaging Spectroradiometer, that allows both the SD and the moon to be viewed by the RSB at the same angle of incidence. Thus, this design element warrants serious consideration for other satellite sensors utilizing a similar calibration methodology.

Flight Calibration of the LROC Narrow Angle Camera

Characterization and calibration are vital for instrument commanding and image interpretation in remote sensing. The Lunar Reconnaissance Orbiter Camera Narrow Angle Camera (LROC NAC) takes 500 Mpixel greyscale images of lunar scenes at 0.5 meters/pixel. It uses two nominally identical line scan cameras for a larger crosstrack field of view. Stray light, spatial crosstalk, and nonlinearity were characterized using flight images of the Earth and the lunar limb. These are important for imaging shadowed craters, studying ∼1 meter size objects, and photometry respectively. Background, nonlinearity, and flatfield corrections have been implemented in the calibration pipeline. An eight-column pattern in the background is corrected. The detector is linear for \(\mathrm{DN} = 600\mbox{--}2000\) but a signal-dependent additive correction is required and applied for \(\mathrm{DN}<600\). A predictive model of detector temperature and dark level was developed to command dark level offset. This avoids images with a cutoff at \(\mathrm{DN}=0\) and minimizes quantization error in companding. Absolute radiometric calibration is derived from comparison of NAC images with ground-based images taken with the Robotic Lunar Observatory (ROLO) at much lower spatial resolution but with the same photometric angles.

The Detection of Outliers in Nondestructive Integrations with the Generalized Extreme Studentized Deviate Test

The identification of outliers in nondestructive integrations represents an important step in the routine processing of on-orbit or ground observations and in providing data of the highest quality to the observer. We present the Generalized Extreme Studentized Deviate (ESD) Test for multiple outliers and apply it to the identification of cosmic ray hits in simulated integrations and in ground laboratory observations collected with a James Webb Space Telescope (JWST) near-IR detector. The completeness ratio and fraction of false detections of the simulated cosmic-ray impacts are characterized for a range of parameters expected in real-world observations. Random Telegraph Noise (RTN) pixels, which are intrinsically variable, are also successfully identified with this test. The algorithm of this versatile statistical test is straightforward to implement and can serve to flag and mask bad or variable pixels in automated data calibration pipelines.

Real-time calibration of the AARTFAAC array for transient detection

The search for transient phenomena at low radio frequencies is now coming of age with the development of radio sky monitors with a large field of view, which are made feasible by new developments in calibration algorithms and computing. However, accurate calibration of such arrays is challenging, especially within the latency requirements of near real-time transient monitors, and is the main cause of limiting their sensitivities. This paper describes a strategy for real-time, wide-field direction-dependent calibration of the Amsterdam-ASTRON Radio Transients Facility and Analysis Center (AARTFAAC) array, which is a sensitive, continuously available all-sky monitor based on the LOw Frequency ARray (LOFAR). The monitor operates in a zenith pointing, snapshot imaging mode for image plane detection of bright radio transients. We show that a tracking calibration approach with solution propagation satisfies our latency, computing, and calibration accuracy constraints. We characterize the instrument and verify the calibration strategy under a variety of observing conditions. This brings out several phenomena, which can bias the calibration. The real-time nature of the application further imposes strict latency and computational constraints. We find that although ionosphere-induced phase errors present a major impediment to accurate calibration, these can be corrected in the direction of the brightest few sources to significantly improve image quality. Our real-time calibration pipeline implementation processes a single spectral channel of a snapshot observation in ~0.2 s on test hardware, which is well within its latency budget. Autonomously calibrating and imaging one second snapshots, our approach leads to a typical image noise of ~10 Jy for a ~90 kHz channel, reaching dynamic ranges of ~2000:1. We also show that difference imaging allows thermal-noise limited transient detection, despite the instrument being confusion-noise limited.

FisHiCal: an R package for iterative FISH-based calibration of Hi-C data.

The fluorescence in situ hybridization (FISH) method has been providing valuable information on physical distances between loci (via image analysis) for several decades. Recently, high-throughput data on nearby chemical contacts between and within chromosomes became available with the Hi-C method. Here, we present FisHiCal, an R package for an iterative FISH-based Hi-C calibration that exploits in full the information coming from these methods. We describe here our calibration model and present 3D inference methods that we have developed for increasing its usability, namely, 3D reconstruction through local stress minimization and detection of spatial inconsistencies. We next confirm our calibration across three human cell lines and explain how the output of our methods could inform our model, defining an iterative calibration pipeline, with applications for quality assessment and meta-analysis. FisHiCal v1.1 is available from http://cran.r-project.org/.

Herschel SPIRE FTS telescope model correction

Emission from the Herschel telescope is the dominant source of radiation for the majority of SPIRE Fourier transform spectrometer (FTS) observations, despite the exceptionally low emissivity of the primary and secondary mirrors. Accurate modelling and removal of the telescope contribution is, therefore, an important and challenging aspect of FTS calibration and data reduction pipeline. A dust-contaminated telescope model with time invariant mirror emissivity was adopted before the Herschel launch. However, measured FTS spectra show a clear evolution of the telescope contribution over the mission and strong need for a correction to the standard telescope model in order to reduce residual background (of up to 7 Jy) in the final data products. Systematic changes in observations of dark sky, taken over the course of the mission, provide a measure of the evolution between observed telescope emission and the telescope model. These dark sky observations have been used to derive a time dependent correction to the telescope emissivity that reduces the systematic error in the continuum of the final FTS spectra to ~0.35 Jy.

The Hopkins Ultraviolet Telescope: The Final Archive

ABSTRACTThe Hopkins Ultraviolet Telescope (HUT) was a 0.9 m telescope and moderate-resolution (Δλ = 3 Å) far-ultraviolet (820–1850 Å) spectrograph that flew twice on the space shuttle, in 1990 December (Astro-1, STS-35) and 1995 March (Astro-2, STS-67). The resulting spectra were originally archived in a nonstandard format that lacked important descriptive metadata. To increase their utility, we have modified the original data-reduction software to produce a new and more user-friendly data product, a time-tagged photon list similar in format to the Intermediate Data Files (IDFs) produced by the Far Ultraviolet Spectroscopic Explorer calibration pipeline. We have transferred all relevant pointing and instrument-status information from locally-archived science and engineering databases into new FITS header keywords for each data set. Using this new pipeline, we have reprocessed the entire HUT archive from both missions, producing a new set of calibrated spectral products in a modern FITS format that is fully compliant with Virtual Observatory requirements. For each exposure, we have generated quick-look plots of the fully-calibrated spectrum and associated pointing history information. Finally, we have retrieved from our archives HUT TV guider images, which provide information on aperture positioning relative to guide stars, and converted them into FITS-format image files. All of these new data products are available in the new HUT section of the Mikulski Archive for Space Telescopes (MAST), along with historical and reference documents from both missions. In this article, we document the improved data-processing steps applied to the data and show examples of the new data products.

A new experimental setup for making thermal emission measurements in a simulated lunar environment

One of the key problems in determining lunar surface composition for thermal-infrared measurements is the lack of comparable laboratory-measured spectra. As the surface is typically composed of fine-grained particulates, the lunar environment induces a thermal gradient within the near sub-surface, altering the emission spectra: this environment must therefore be simulated in the laboratory, considerably increasing the complexity of the measurement. Previous measurements have created this thermal gradient by either heating the cup in which the sample sits or by illuminating the sample using a solar-like source. This is the first setup able to measure in both configurations, allowing direct comparisons to be made between the two. Also, measurements across a wider spectral range and at a much higher spectral resolution can be acquired using this new setup. These are required to support new measurements made by the Diviner Lunar Radiometer, the first multi-spectral thermal-infrared instrument to orbit the Moon. Results from the two different heating methods are presented, with measurements of a fine-grained quartz sample compared to previous similar measurements, plus measurements of a common lunar highland material, anorthite. The results show that quartz gives the same results for both methods of heating, as predicted by previous studies, though the anorthite spectra are different. The new calibration pipeline required to convert the raw data into emissivity spectra is described also.

LOW-FREQUENCY IMAGING OF FIELDS AT HIGH GALACTIC LATITUDE WITH THE MURCHISON WIDEFIELD ARRAY 32 ELEMENT PROTOTYPE

The Murchison Widefield Array (MWA) is a new low-frequency, wide field-of-view radio interferometer under development at the Murchison Radio-astronomy Observatory (MRO) in Western Australia. We have used a 32-element MWA prototype interferometer (MWA-32T) to observe two 50-degree diameter fields in the southern sky in the 110 MHz to 200 MHz band in order to evaluate the performance of the MWA-32T, to develop techniques for epoch of reionization experiments, and to make measurements of astronomical foregrounds. We developed a calibration and imaging pipeline for the MWA-32T, and used it to produce ~15' angular resolution maps of the two fields. We perform a blind source extraction using these confusion-limited images, and detect 655 sources at high significance with an additional 871 lower significance source candidates. We compare these sources with existing low-frequency radio surveys in order to assess the MWA-32T system performance, wide field analysis algorithms, and catalog quality. Our source catalog is found to agree well with existing low-frequency surveys in these regions of the sky and with statistical distributions of point sources derived from Northern Hemisphere surveys; it represents one of the deepest surveys to date of this sky field in the 110 MHz to 200 MHz band.

On the calibration of full-polarization 86 GHz global VLBI observations

We report the development of a semi-automatic pipeline for the calibration of 86 GHz full-polarization observations performed with the Global Millimeter-VLBI array (GMVA) and describe the calibration strategy followed in the data reduction. Our calibration pipeline involves non-standard procedures, since VLBI polarimetry at frequencies above 43 GHz is not yet well established. We also present, for the first time, a full-polarization global-VLBI image at 86 GHz (source 3C 345), as an example of the final product of our calibration pipeline, and discuss the effect of instrumental limitations on the fidelity of the polarization images. Our calibration strategy is not exclusive for the GMVA, and could be applied on other VLBI arrays at millimeter wavelengths. The use of this pipeline will allow GMVA observers to get fully-calibrated datasets shortly after the data correlation.