Data Reduction Using Lossy Compression for Cosmology and Astrophysics Workflows

Abstract

This paper concerns the use of compression methods applied to large scientific data. Specifically the paper addresses the effect of lossy compression on approximation error. Computer simulations, experiments and imaging technologies generate terabyte-scale datasets making necessary new approaches for compression coupled with data analysis. Lossless compression techniques compress data with no loss of information, but they generally do not produce a large-enough reduction when compared to lossy compression methods. Lossy multi-resolution compression techniques make it possible to compress large datasets significantly with small numerical error, preserving coherent features and statistical properties needed for analysis. Lossy data compression reduces I/O data transfer cost and makes it possible to store more data at higher temporal resolution. We present results obtained with lossy multi-resolution compression, with a focus on astrophysics datasets. Our results confirm that lossy data compression is capable of preserving data characteristics very well, even at extremely high degrees of compression.