PARS-Push: Personalized, Asynchronous and Robust Decentralized Optimization

Abstract

We study the multi-step Model-Agnostic Meta-Learning (MAML) framework where a group of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> agents seeks to find a common point that enables “few-shot” learning (personalization) via local stochastic gradient steps on their local functions. We formulate the personalized optimization problem under the MAML framework and propose PARS-Push, a decentralized asynchronous algorithm robust to message failures, communication delays, and directed message sharing. We characterize the convergence rate of PARS-Push under arbitrary multi-step personalization for smooth strongly convex, and smooth non-convex functions. Moreover, we provide numerical experiments showing its performance under heterogeneous data setups.