Serverless computing is becoming a promising paradigm for Distributed Deep Neural Network (DDNN) training in the cloud, as it allows users to decompose complex model training into a number of <i>functions</i> without managing virtual machines or servers. Though provided with a simpler resource interface (i.e., function number and memory size), inadequate function resource provisioning (either under-provisioning or over-provisioning) easily leads to <i>unpredictable</i> DDNN training performance in serverless platforms. Our empirical studies on AWS Lambda indicate that, such <i>unpredictable performance</i> of serverless DDNN training is mainly caused by the resource bottleneck of Parameter Servers (PS) and small local batch size. In this article, we design and implement <i><inline-formula><tex-math notation="LaTeX">$\lambda$</tex-math><alternatives><mml:math><mml:mi>λ</mml:mi></mml:math><inline-graphic xlink:href="xu-ieq1-3054656.gif"/></alternatives></inline-formula>DNN</i>, a cost-efficient function resource provisioning framework to provide predictable performance for serverless DDNN training workloads, while saving the budget of provisioned functions. Leveraging the PS network bandwidth and function CPU utilization, we build a <i>lightweight</i> analytical DDNN training performance model to enable our design of <i><inline-formula><tex-math notation="LaTeX">$\lambda$</tex-math><alternatives><mml:math><mml:mi>λ</mml:mi></mml:math><inline-graphic xlink:href="xu-ieq2-3054656.gif"/></alternatives></inline-formula>DNN</i> resource provisioning strategy, so as to guarantee DDNN training performance with serverless functions. Extensive prototype experiments on AWS Lambda and complementary trace-driven simulations demonstrate that, <i><inline-formula><tex-math notation="LaTeX">$\lambda$</tex-math><alternatives><mml:math><mml:mi>λ</mml:mi></mml:math><inline-graphic xlink:href="xu-ieq3-3054656.gif"/></alternatives></inline-formula>DNN</i> can deliver predictable DDNN training performance and save the monetary cost of function resources by up to 66.7 percent, compared with the state-of-the-art resource provisioning strategies, yet with an acceptable runtime overhead.
Read full abstract