Abstract
This paper concerns the use of compression methods applied to large scientific data. Specifically the paper addresses the effect of lossy compression on approximation error. Computer simulations, experiments and imaging technologies generate terabyte-scale datasets making necessary new approaches for compression coupled with data analysis. Lossless compression techniques compress data with no loss of information, but they generally do not produce a large-enough reduction when compared to lossy compression methods. Lossy multi-resolution compression techniques make it possible to compress large datasets significantly with small numerical error, preserving coherent features and statistical properties needed for analysis. Lossy data compression reduces I/O data transfer cost and makes it possible to store more data at higher temporal resolution. We present results obtained with lossy multi-resolution compression, with a focus on astrophysics datasets. Our results confirm that lossy data compression is capable of preserving data characteristics very well, even at extremely high degrees of compression.
Highlights
Detecting the structure of the universe and its properties, helps to uncover the great mysteries of how and why the universe as it is today came to be
Los Alamos National Laboratory requests that the publisher identify this article as work performed under the auspices of the U.S Department of Energy
Los Alamos National Laboratory strongly supports academic freedom and a researcher's right to publish; as an institution, the Laboratory does not endorse the viewpoint of a publication or guarantee its technical correctness
Summary
Detecting the structure of the universe and its properties, helps to uncover the great mysteries of how and why the universe as it is today came to be. Title: Data Reduction Using Lossy Compression for Cosmology and Astrophysics Workflows Compression is a solution but lossless methods aren’t good enough! Limited compute and memory budget for in-situ compression
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
