Improving Speech Separation with Adversarial Network and Reinforcement Learning

Abstract

In contrast to the conventional deep neural network for single-channel speech separation, we propose a separation framework based on adversarial network and reinforcement learning. The purpose of the adversarial network inspired by the generative adversarial network is to make the separated result and ground-truth with the same data distribution by evaluating the discrepancy between them. Meanwhile, in order to enable the model to bias the generation towards desirable metrics and reduce the discrepancy between training loss (such as mean squared error) and testing metric (such as SDR), we present the future success based on reinforcement learning. We directly optimize the performance metric to accomplish exactly that. With the combination of adversarial network and reinforcement learning, our model is able to improve the performance of single-channel speech separation.