Transfer knowledge for punctuation prediction via adversarial training

Abstract

Previous studies demonstrate that part-of-speech (POS) tags are helpful for punctuation restoration tasks. However, extra computation cost will be needed during decoding, due to POS tags are provided by an external POS tagger. This paper proposes to transfer knowledge via adversarial training and orthogonality constraints to fill in the gap. Adversarial multi-task learning is introduced to learn task invariant knowledge from the extra POS tagging task for a punctuation prediction task. Furthermore, orthogonality constraints are used to make private and shared features dissimilar. Only the punctuation predicting task is used during decoding. So extra computation is not needed. Experiments are conducted on IWSLT2011 datasets. The results show that the punctuation predicting models trained with adversarial learning obtain performance gains over the baseline models on test sets. The results also demonstrate that the models trained with orthogonality constraints further obtain performance improvement.