Context extraction module for deep convolutional neural networks

Abstract

Convolutional layers convolve the input feature maps to generate valuable output features, and they help deep learning methods significantly in solving complex problems. In order to tackle problems efficiently, deep learning solutions should ensure that the parameters of the model do not increase significantly with the complexity of the problem. Pointwise convolutions are primarily used for parameter reduction in many deep learning architectures. They are convolutional filters of kernel size 1×1. The pointwise convolution, however, ignores the spatial information around the points it is processing. This design is by choice, in order to reduce the overall parameters and computations. However, we hypothesize that this shortcoming of pointwise convolution has a significant impact on network performance. We propose a novel alternative design for pointwise convolution, which uses spatial information from the input efficiently. Our approach extracts spatial context information from the input at two scales and further refines the extracted context based on the channel importance. Finally, we add the refined context to the output of the pointwise convolution. This is the first work that improves pointwise convolution by incorporating context information. Our design significantly improves the performance of the networks without substantially increasing the number of parameters and computations. We perform experiments on coarse/fine-grained image classification, few-shot fine-grained classification, and on object detection. We further perform various ablation experiments to validate the significance of the different components used in our design. Lastly, we show experimentally that our proposed technique can be combined with existing state-of-the-art network performance improvement approaches to further improve the network performance.