Performing forced alignment with Wav2vec 2.0

Abstract

Forced alignment, the task of aligning segmentation of audio speech files with an orthographic or phonetic transcript, is fundamental to many types of speech research. Yet the available toolkits for forced alignment are mostly based on the classic HMM/GMM systems, which are outperformed by neural network-based speech recognition models, especially the large-scale speech pre-trained models in recent years. We propose a method of forced alignment utilizing the pre-trained transformer-based model, Wav2vec 2.0. This model has been pre-trained on massive audio datasets, and is subsequently fine-tuned on 360 h of speech data guided by connectionist temporal classification (CTC) loss. The model has learned to jointly recognize phonemes and perform segmentation with or without orthographic transcriptions. During inference, the hidden states are converted to frame-level alignments through post-processing. Our preliminary analysis shows that the model performs competitively when segmenting the TIMIT benchmark, even without orthographic transcription provided.