Embracing Multiheterogeneity and Privacy Security Simultaneously: A Dynamic Privacy-Aware Federated Reinforcement Learning Approach.

Abstract

With growing demand for privacy-preserving reinforcement learning (RL) applications, federated RL (FRL) has emerged as a potential solution. However, existing FRL methods struggle with multiple sources of heterogeneity, while lacking robust privacy guarantees. In this study, we propose DPA-FedRL, the dynamic privacy-aware FRL framework, to simultaneously mitigate both issues. First, we innovatively put forward the concept of "multiheterogeneity" and embed the environmental heterogeneity into agents' state representations. Next, to ensure privacy during model aggregation, we incorporate a differentially private mechanism in form of Gaussian noise and modify its global sensitivity, tailored to suit FRL's unique characteristics. Encouragingly, our approach dynamically allocates privacy budget based on heterogeneity levels, which strikes a balance between privacy and utility. From the theoretical perspective, we give rigorous convergence, privacy, and sensitivity guarantees for our proposed method. Through extensive experiments on diverse datasets, we demonstrate that DPA-FedRL surpasses state-of-the-art approaches (PPO-DP-SGD, PAvg, and QAvg) in some highly heterogeneous environments. Notably, our novel privacy attack simulations enable quantitative privacy assessment, validating that DPA-FedRL offers over 1.359 × stronger protection than baselines.