Learning-based padding: From connectivity on data borders to data padding

Abstract

Padding is employed vastly in convolutional neural networks (CNNs) to maintain the output size. In current padding schemes, the input feature maps are padded using quite simple strategies, e.g., constant zero values in zero padding, which is the most common padding choice. In this work, we propose to use learning-based paradigms to calculate the padding data from the connectivity of the input images on the data borders. Two different modules, i.e., learning-based padding by convolution (LPC) and learning-based padding by attention (LPA), are designed to obtain the padding data from the interdependencies of the image border data along the channel and spatial dimensions, respectively. The designed LPC or LPA is formulated as a generic, plug-and-play unit, which can be a direct replacement for the conventional padding technique. Extensive experiments on image classification and semantic segmentation tasks show that the proposed padding schemes can consistently obtain higher accuracy than standard padding schemes, in various deep network backbones. The codes and trained modules are available at https://github.com/ICSResearch/LP.