The MWA archive infrastructure: archiving terabytes of data over dedicated WAN connections

Abstract

The Murchison Wide Field Array (MWA) is being upgraded from 32 tiles to 128 tiles of 16 dual-polarization dipole antennas. In the course of this project the software and the data infrastructure are also undergoing a major overhaul in order to cope with a more continuous and remote operational model; and the substantial increase in data rate (400 MB/s from the correlator plus 160 MB/s from the real time imaging pipeline). During the course of 2012/13 the data collected by the MWA will be transported via a dedicated 40 Gbit WAN network link between the Murchison Radio Observatory (MRO) and Perth (700 km). However, this network will not be available for some time; and until then, the data will be transported using disk arrays instead. Once in Perth, the data will be ingested into a tape library. The archiving process itself consists of various steps executed either at the MRO site or in Perth, and makes use of a modied version of the archiving system from the Atacama Large Millimeter Array (ALMA). This includes the extraction of meta-data from the original raw data and the ingestion and generation of the appropriate data links for the MWA archive. The MWA correlator generates a collection of small les all belonging to the same observation. In order to optimise the network transfer and the storage of those les they are transparently packed into larger containers at the MRO site and subsequently handled as one big le. This paper describes the setup of the MWA archiving infrastructure.