Square Root Compression and Noise Effects in Digitally Transformed Images

Abstract

We report on a particular example of noise and data representation interacting to introduce systematic error into scientific measurements. Many instruments collect integer digitized values and apply nonlinear coding, in particular square root coding, to compress the data for transfer or downlink; this can introduce surprising systematic errors when they are decoded for analysis. Square root coding and subsequent decoding typically introduces a variable ±1 count value-dependent systematic bias in the data after reconstitution. This is significant when large numbers of measurements (e.g., image pixels) are averaged together. Using direct modeling of the probability distribution of particular coded values in the presence of instrument noise, one may apply Bayes’ theorem to construct a decoding table that reduces this error source to a very small fraction of a digitizer step; in our example, systematic error from square root coding is reduced by a factor of 20 from 0.23 to 0.012 count rms. The method is suitable both for new experiments such as the upcoming PUNCH mission, and also for post facto application to existing data sets—even if the instrument noise properties are only loosely known. Further, the method does not depend on the specifics of the coding formula, and may be applied to other forms of nonlinear coding or representation of data values.