Empirical Evaluation of the Shift and Scale Parameters in Batch Normalization

Abstract

Batch Normalization (BatchNorm) is a technique that enables the training of deep neural networks, especially Convolutional Neural Networks (CNN) for computer vision tasks. It has been empirically demonstrated that BatchNorm increases per- formance, stability, and accuracy, although the reasons for these improvements are unclear. BatchNorm consists of a normalization step with trainable shift and scale parameters. In this paper, we examine the role of normalization and the shift and scale parameters in BatchNorm. We implement two new optimizers in PyTorch: a version of BatchNorm that we refer to as AffineLayer, which includes the shift and scale transform without normalization, and a version with just the normalization step, which we call BatchNorm-minus. We compare the performance of our AffineLayer and BatchNorm-minus implementations to standard BatchNorm, and we also compare these to the case where no batch normalization is used. We experiment with the ResNet18 and ResNet50 models over various batch sizes. Among other findings, we provide empirical evidence that the success of BatchNorm may be primarily due to improved weight initialization.