Abstract
Parameter server paradigm has shown great performance superiority for handling deep learning (DL) applications. One crucial issue in this regard is the presence of stragglers, which significantly retards DL training progress. Previous approaches for solving straggler may not consider the resource utilization of a cluster. This motivates us to make an attempt at designing a new scheme that mitigates straggler problem in DL from the perspective of dynamic balance workloads among workers. To optimize the method of mitigating straggler problem in the traditional parameter server, we propose Help-Reassign Synchronization (HRS) mechanism which has high flexibility to adapt to the dynamic cluster without parameter settings. Furthermore, we propose a Deep Reinforcement Learning (DRL)-based algorithm Decentralized Actor-critic-based Experience Replay (DAER) that can automatically identify and determine helper workers (helper) and helpee workers (helpee). The whole idea has been implemented in a scheme called FlexHS which mitigates straggler problem by creating a dynamic balance between the number of helper and backup overhead. Evaluation under various algorithms evidences the superiority of our scheme.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.