A CNN-RNN Hybrid Model with 2D Wavelet Transform Layer for Image Classification

Abstract

Convolutional neural networks (CNNs) have recently achieved impressive performances in image processing tasks such as image classification and object recognition. However, CNNs only process images in the spatial domain whereas spectral analysis operates in the frequency domain. In this paper, we propose the 2D wavelet transform layer. The learned features from images are viewed as two-directional sequential data, and we use two LSTM layers that sweep both horizontally and vertically across the image to compress feature matrices. Based on this, the 2D wavelet transform decomposes the above feature matrices as a learned mixing of different harmonic functions, and therefore integrating the spectral analysis into CNNs. We also select 3 × 3 convolutional mixing style with Gaussian+LSM filter to mix the output of the 2D wavelet transform to generate new output features. Finally, we combine the sequential and spectral features to build our CNN-RNN architecture with skip layers and apply it to image classification. Our proposed network is evaluated on three widely-used benchmark datasets: CIFAR-10, CIFAR-100 and Tiny ImageNet. Experiments show that our CNN-RNN hybrid model achieves better accuracy in image classification tasks.