AuxBranch: Binarization residual-aware network design via auxiliary branch search

Abstract

While network binarization is a promising method in memory saving and speedup on hardware, it inevitably leads to binarization residual of intermediate features, resulting in performance capability degradation. To alleviate the above issue, we focus on the network topology design scheme to the more suitable network structure for the extreme-low-bit scenario. In this paper, we propose the baseline-auxiliary expanding network design method to compensate for the binarization residual of features via searching for auxiliary branches, denoted as AuxBranch. The intermediate feature maps are reasonably enhanced by combining baseline and auxiliary features, mimicking the corresponding feature output of the full-precision network. In addition, we devise a hybrid performance estimator (PE) with three elements of preliminary accuracy, feature similarity, and computational complexity. The PE jointly performs an efficient architecture search for binarization baseline and enables automatic computation complexity adjustment under diverse constraints. Extensive experiments show that our approach is superior in terms of accuracy and computational performance, and is plug-and-play for different network backbones and binarization policies. Our code is available at https://github.com/VipaiLab/AuxBranch.