Virtual Adversarial Training Applied to Neural Higher-Order Factors for Phone Classification

Abstract

We explore virtual adversarial training (VAT) applied to neu-ral higher-order conditional random fields for sequence label-ing. VAT is a recently introduced regularization method pro-moting local distributional smoothness: It counteracts the prob-lem that predictions of many state-of-the-art classifiers are un-stable to adversarial perturbations. Unlike random noise, ad-versarial perturbations are minimal and bounded perturbationsthat flip the predicted label. We utilize VAT to regularize neuralhigher-order factors in conditional random fields. These fac-tors are for example important for phone classification wherephone representations strongly depend on the context phones.However, without using VAT for regularization, the use of suchfactors was limited as they were prone to overfitting. In exten-sive experiments, we successfully apply VAT to improve per-formance on the TIMIT phone classification task. In particular,we achieve a phone error rate of13.0%, exceeding the state-of-the-art performance by a wide margin.Index Terms: Virtual adversarial training, local distributionalsmoothing, deep higher-order factors, neural higher-order con-ditional random field, phone classificatio.