Iterative algorithms for the post-processing of high-dimensional data

Abstract

Scientific computations or measurements may result in huge volumes of high-dimensional data, for instance 1020 or 100300 elements. Often these can be thought of representing a real-valued function on a high-dimensional domain. In this and also in most other cases the data can be conceptually arranged in the format of a tensor of high degree, and stored in some truncated or lossy compressed format. We look at some common post-processing tasks which are too time and storage consuming in the uncompressed data format and not obvious in the compressed format, as such huge data sets can not be stored in their entirety, and the value of an element is not readily accessible through simple look-up. The tasks we consider are finding the location of maximum or minimum, or finding all elements in some interval — i.e. level sets, or the number of elements with a value in such a level set, the probability of an element being in a particular level set, and the mean and variance of the total collection. The algorithms to be described are fixed point iterations of particular point-wise functions of the data, which will then exhibit the desired result. To formulate these algorithms, the data is considered as an element of a commutative algebra, and in an abstract sense, the algorithms, which only use these algebraic operations, are independent of the representation. We allow the actual computational representation to be a lossy compression, and we allow the algebra operations to be performed in an approximate fashion, so as to maintain a high compression level. One such example format which is addressed explicitly and described in some detail is the representation of the data as a tensor with compression in the form of a low-rank representation.