Residual encoding framework to compress DNN parameters for fast transfer

Abstract

Efficient communication is significant for federated learning and DNN model deployment. However, transferring hundreds of millions of DNN parameters over networks with limited bandwidth results in long communication delays or even data losses. To alleviate or even remove the communication bottleneck, efficient methods for parameter compression can be applied. Inspired by video encoding, which exploits inter-frame similarity for compression, we investigate the strong temporal correlations of parameter updates in two near epochs of the DNN model and introduce a model parameter residual encoding framework. By transmitting encoded residual between model parameters in two near epochs, the receiver can reconstruct new model parameters and finish the updates with less communication cost. Furthermore, with respect to our framework, we develop lossless and lossy model parameter compression methods and demonstrate them on popular classification and detection networks. The results show that the lossless method can compress the data size of the parameters to less than 90%, and the lossy method can shrink the parameter size to less than 50% with a fair low loss. Our source code is released at https://github.com/zhouliguo/DNN_param_encode.