Byzantine-Robust and Communication-Efficient Distributed Non-Convex Learning Over Non-IID Data

Abstract

Motivated by the emerging federated learning applications, we jointly consider the problems of Byzantine-robustness and communication efficiency in distributed non-convex learning over non-IID data. We propose a compressed robust stochastic model aggregation (CRSA) method, which applies the idea of robust stochastic model aggregation to achieve Byzantine-robustness over non-IID data, while compresses the transmitted messages so as to achieve communication efficiency. Utilizing the tools of Moreau envelope and proximal point projection, we establish the convergence of C-RSA for distributed non-convex learning problems. Numerical experiments on training a large-scale neural network demonstrate the effectiveness of the proposed C-RSA method.