Kernel-convoluted Deep Neural Networks with Data Augmentation

Abstract

The Mixup method, which uses linearly interpolated data, has emerged as an effective data augmentation tool to improve generalization performance and the robustness to adversarial examples. The motivation is to curtail undesirable oscillations by its implicit model constraint to behave linearly at in-between observed data points and promote smoothness. In this work, we formally investigate this premise, propose a way to impose smoothness constraints explicitly, and extend it to incorporate implicit model constraints. First, we derive a new function class composed of kernel-convoluted models (KCM) where the smoothness constraint is directly imposed by locally averaging the original functions with a kernel function. Second, we propose to incorporate the Mixup method into KCM to expand the domains of smoothness. In both cases of the KCM and the KCM adapted with the Mixup, we provide risk analysis, respectively, under mild conditions on kernel functions. As a result, we show that the upper bound of the excess risk over a new function class is not slower than that of the excess risk over the original function class. Using CIFAR-10 and CIFAR-100 datasets, our experiments demonstrate that the KCM with the Mixup outperforms the Mixup method in terms of generalization and robustness to adversarial examples.