Denoising Score Matching via Random Fourier Features

Abstract

The density estimation is one of the core problems in statistics. Despite this, existing techniques like maximum likelihood estimation are computationally inefficient in case of complex parametric families due to the intractability of the normalizing constant. For this reason, an interest in score matching has increased, being independent on the normalizing constant. However, such an estimator is consistent only for distributions with the full space support. One of the approaches to make it consistent is to add noise to the input data called Denoising Score Matching. In this work we build computationally efficient algorithm for density estimate using kernel exponential family as a model distribution. The usage of the kernel exponential family is motivated by the richness of this class of densities. To avoid calculating an intractable normalizing constant we use Denoising Score Matching objective. The computational complexity issue is approached by applying Random Fourier Features-based approximation of the kernel function. We derive an exact analytical expression for this case which allows dropping additional regularization terms based on the higher-order derivatives as they are already implicitly included. Moreover, the obtained expression explicitly depends on the noise variance, so that the validation loss can be straightforwardly used to tune the noise level. Along with benchmark experiments, the method was tested on various synthetic distributions to study the behavior of the method in different cases. The empirical study shows comparable quality to the competing approaches, while the proposed method being computationally faster. The latter one enables scaling up to complex high-dimensional data.