Efficient Lossless Compression of Integer Astronomical Data

Abstract

Each new generation of telescope produces increasingly larger astronomical data volumes, which are expected to reach the order of exabytes in the next decade. Effective and fast data compression methods are paramount to help the scientific community contain storage costs and improve transmission times. Astronomical data differs significantly from natural and Earth-observation images, asking for specifically tailored compression approaches. This paper presents a novel lossless compression technique that employs the discrete Haar wavelet transform within the JPEG 2000 standard. Its performance is compared to that of a comprehensive selection of compressors, including fpack, the most common technique in astronomical observatories, as well as other algorithms highly competitive for other types of data. Experiments are performed on a large data set of 16 bit integer images, produced by telescopes around the world and representative of a wide variety of astronomical scenarios. The proposed technique has two modes. The first mode outperforms all the other tested techniques in terms of compression performance. It surpasses the most competitive configuration of fpack by, respectively, 5.3% (about 0.3 bits per sample), having also 4.5% lower compression and decompression times. The second mode is the fastest among all tested techniques. Its compression and decompression times are 2.5 and 3.5 times faster than the fastest configuration of fpack, while also yielding a 2.4% better compression performance (0.15 bits per sample).