Subword Latent Semantic Analysis for Texttiling-Based Automatic Story Segmentation of Chinese Broadcast News

Abstract

This paper proposes to perform latent semantic analysis (LSA) on character/syllable n-gram sequences of automatic speech recognition (ASR) transcripts, namely subword LSA, as an extension of our previous work on subword text tiling for automatic story segmentation of Chinese broadcast news. LSA represents the 'meaning' of a lexical term by a feature vector conveying the term's relations with other terms. We apply subword LSA vectors to the measurement of inter-sentence lexical score in text tiling-based story segmentation. Subword n-grams are robust to speech recognition errors, especially out-of-vocabulary (OOV) words, in lexical matching on Chinese ASR transcripts. This work combines the concept matching merit of LSA and the robustness of subwords. Experimental results on the TDT2 Mandarin corpus show that subword-LSA-based text tiling can effectively improve the story segmentation performance. Character-bigram-LSA-based text tiling achieves the best Fl-measure of 0.6598 with relative improvement of 17.4% over the conventional word-based text tiling and 6.5% over our previous syllable-bigram-based text tiling.