Density Distillation for Fast Nonparametric Density Estimation.

Abstract

Nonparametric density estimation has been extensively used in various application scenarios and theoretical models. However, the modeling of these powerful methods is inseparable from the sample data and comes at the cost of repeated and intensive kernel calculations, which makes their efficiency greatly affected by the sample scale, data dimension, and evaluation scale. Inspired by the knowledge distillation method, a student-teacher paradigm model named density convolutional neural network (DCNN) is proposed in this article. The method extracts the density knowledge of the samples based on the density convolution rule and transfers it to a compact and small deep neural network, in order to separate the sample data from the modeling and avoid the cumbersome kernel calculations. Experimental results show the superiority of the proposed method to various nonparametric estimation methods in terms of accuracy, stability, processing efficiency, and low-storage advantage. Especially, for the estimation speed, a univariate density estimation on 1.0E + 08 evaluation points using GPU only takes 1.57 s, and a 10-D multivariate density estimation on 1.0E + 08 evaluation points only takes 10.50 s, which makes our method very suitable for real-time and large-scale repetitive density estimation tasks.