Convolutional kernel networks based on a convex combination of cosine kernels

Abstract

Convolutional Kernel Networks (CKNs) are efficient multilayer kernel machines, which are constructed by approximating a convolution kernel with a mapping based on Gaussian functions. In this paper, we introduce a new approximation of the same convolution kernel based on a convex combination of cosine kernels. CKNs are structurally similar to Convolutional Neural Networks (CNNs), but the convolution operation in CKNs is based on the Euclidean distance, which is not common in convolutional networks. We show that the CKN model obtained by the proposed approximation leads to the ordinary convolution operation, which is based on the inner product. From this point of view, the proposed model is a step forward towards bridging the gap between kernel methods and deep learning. In this paper, we use two methods for learning filters of the proposed CKN: Random Fourier Features, which is a randomized data-independent method for approximating shift-invariant kernels, and a novel method based on the minimization of the sum of squared errors of approximating shift-invariant kernels. Although the RFF method is much faster than ordinary CKN, it requires a high number of random features in order to obtain an acceptable accuracy. To overcome this problem, we proposed the second method, in which the filters are learned in a data-dependent fashion. We evaluate the proposed model on visual recognition datasets MNIST, CIFAR-10, C-Cube, and FERET. Our experiments show that the proposed model surpasses ordinary CKNs in terms of accuracy. Specifically, on CIFAR-10, the accuracy of the proposed method is 1.7% higher than ordinary CKN.