C-RSA: Byzantine-robust and communication-efficient distributed learning in the non-convex and non-IID regime

Abstract

The emerging federated learning applications raise challenges of Byzantine-robustness and communication efficiency in distributed non-convex learning over non-IID data. To address these issues, in this paper we propose a compressed Byzantine-robust stochastic model aggregation method, abbreviated as C-RSA. C-RSA utilizes robust stochastic model aggregation to obtain Byzantine-robustness over non-IID data, and compresses the transmitted messages for achieving high communication efficiency. Theoretically, we exploit Moreau envelope and proximal point projection as technical tools to analyze the convergence of C-RSA for distributed non-convex learning. Extensive numerical experiments are conducted on neural network training tasks to demonstrate the superior performance of C-RSA.