Bridging the cross-modal gap using adversarial training for speech-to-text translation

Abstract

The end-to-end speech translation (ST) model usually adopts the encoder-decoder structure, which takes the speech of the source language as input and directly outputs its translation result in the target language. Since the model performs cross-modal translation, it needs to extract semantically rich representations from speech. Compared with text, speech is fine-grained and contains more noise, which puts a great burden on the encoder of the ST model. The modal gap between speech and text usually causes the ST model to perform inferior to the corresponding machine translation (MT) model. To bridge the cross-modal gap, this paper proposes to use adversarial training to relieve the burden on the ST encoder by providing internal supervision signals. In this approach, the encoder in the ST model can extract representations that contain rich semantics, which greatly improves performance. Experiments on the datasets of Augmented Librispeech English-French and MuST-C English-German show the effectiveness of our method. Further analysis indicates that our proposed method can perform well in low-resource conditions compared to strong baselines.