Efficient statistical estimation for a non-randomly distributed system with application to large-scale data neural network

Abstract

This paper proposes a communication-efficient pilot sample surrogate loss function framework that is able to solve efficient statistical estimation for a non-randomly distributed system. Specifically, we first collect a small size pilot sample from worker machines, then approximate the global loss function by a surrogate one, which only relates to the pilot sample and the gradients of local datasets. Without any restrictive condition about randomness, the established asymptotical properties show that the resulting estimator obtained by minimizing the surrogate loss is equivalent with the global estimator. Since the pilot sample and gradients can easily be communicated between the master and worker machines, the communication cost is significantly reduced. What is more, as a specific application, we apply our new method to the neural network with large-scale data for fast and accurate optimization. Monte Carlo simulations and real-world application are also used to validate our method.