A strategy for satellite data archival. Low noise variable-rate vector quantization with application to AVHRR satellite images: A tutorial review

Abstract

Abstract The volume of satellite data amassed by modern day weather and climate satellites is so enormous that it has become virtually impossible for researchers to access the original resolution data collected by the satellites. Typically, researchers are forced to deal with lower resolution reduced data (e.g. cloud cover or temperatures) and often at a resolution degraded by one to three orders of magnitude when compared with the resolution of the original data. The uncertainties in the reduced data are often unknown. This state of affairs will only get worse, since the data to be collected under the guidance of NASA’s global change program may increase by several orders of magnitude in the coming decades. We need to experiment with mathematically rigorous ways to tame the original data without significantly degrading the information content. Compression of original data by objective mathematical techniques is a promising approach. This study adopts recent compression techniques developed in the field of communications and applies them to weather satellite data. Typically, these techniques compress the raw data by factors ranging from 10 to 100. The compressed data can be decompressed to retrieve the near original data at the site of the user. The mean error in the compression–decompression process varies from a few percent to several percent. As a demonstration, we consider the advanced very high resolution radiometer (AVHRR) radiances with a nadir resolution of 1 km ×1 km. For this demonstration, we adopt a well understood compression technique which is the so-called vector quantization technique. Several vector quantization techniques (full-search, tree-search, pruned-balanced-tree, greedy-tree and pruned-greedy-tree) are compared in performance and ease of implementation. The discussion focuses on the pruned greedy tree-structured vector quantizer because it is highly suited to the compression of AVHRR satellite images. For the case considered here, visual and scientific reproducibility of the original high resolution images are very good. The rms error for roughly 95% of the pixels in a scene is within 2%, even at a 32 : 1 compression ratio. The error in spatially averaged fields is less than 0.1% for averaging scales in excess of 50 km ×50 km. Some important spatial structural information is lost, however. It is found that the same image when compressed using JPEG standard shows significant loss of numerical accuracy at the same compression ratio of 32 : 1. But improvements and developments in compression techniques can minimize these errors and afford researchers the luxury of storing and working with high resolution data at roughly at about 0.03 of the space required by the original data.