Learning Auction in Coded Distributed Computing with Heterogeneous User Demands

Abstract

Coded distributed computing(CDC) has shown great potentials to solve the unexpected delay caused by stragglers and communication load in distributed computing. We propose a novel learning auction to allocate computing resource efficiently in a CDC scenario. The user demand types are usually het-erogeneous according to different variation trends of the value with finish time and workload, which can be modeled by deep learning. As the goal of social welfare maximizationthe platform would allocate computing resources according to inferred value functions of users. Due to the uncertain finish time and nonlinear structures of deep learning models, the considered optimization problem is non-convex. We then reformulate the non-convex optimization problem into a mixed integer program(MIP). After analyzing the inference error caused by deep learning, a payment rule referred to VCG is designed to achieve incentive alignment and individual rationality. Besides, experiments have been performed to show the superiority of our mechanism.