Multi-View and Multi-Objective Semi-Supervised Learning for HMM-Based Automatic Speech Recognition

Abstract

Current hidden Markov acoustic modeling for large-vocabulary continuous speech recognition (LVCSR) heavily relies on the availability of abundant labeled transcriptions. Given that speech labeling is both expensive and time-consuming while there is a huge amount of unlabeled data easily available nowadays, the semi-supervised learning (SSL) from both labeled and unlabeled data aiming to reduce the development cost for LVCSR becomes more important than ever. In this paper, a new SSL approach is proposed which exploits the cross-view transfer learning for LVCSR through a committee machine consisting of multiple views learned from different acoustic features and randomized decision trees. In addition, a multi-objective learning scheme is developed in each view by maximizing a hybrid information-theoretic criterion which is established by the relative entropy between labeled data and their labels and the entropy of unlabeled data. The multi-objective scheme is then generalized to a unified SSL framework which can be interpreted into a variety of learning strategies under different weighting schemes. Experiments conducted on English Broadcast News using 50 hours of transcribed speech with 50 hours and 150 hours of untranscribed speech show the benefits of proposed approaches.