Meta Generative Flow Networks with personalization for task-specific adaptation

Abstract

Multi-task reinforcement learning and meta-reinforcement learning have been developed to enhance the learning capabilities on new tasks, but they tend to focus on maximizing rewards, leading to poor performance in exploration. To address this issue, we propose Meta Generative Flow Networks (GFlowMeta): an integration of Generative Flow Networks (GFlowNets) into meta-learning algorithms. By leveraging the unique capabilities of GFlowNets to generate diverse candidate solutions, GFlowMeta exhibits enhanced exploration of tasks. However, GFlowMeta's performance deteriorates when confronted with heterogeneous transitions from distinct tasks, leading to a decrease in its effectiveness. Therefore, we subsequently introduce a personalization approach called personalized Meta Generative Flow Networks (pGFlowMeta), which combines task-specific personalized policies with a meta policy. Each personalized policy balances the loss on its personalized task and the difference from the meta policy, while the meta policy aims to minimize the average loss of all tasks. The theoretical analysis shows that the proposed pGFlowMeta converges at a sublinear rate. Furthermore, extensive experiments demonstrate the superiority of pGFlowMeta over state-of-the-art reinforcement learning algorithms.