Phone inventory optimization for multilingual automatic speech recognition

Abstract

This paper describes a phone inventory optimization procedure for application in multilingual automatic speech recognition (ASR). The optimization procedure is based on three knowledge sources that act collectively to guide phonological reduction and selection processes: (1) abstract (language-independent) phonological universals and tendencies that are used in the construction of a hierarchical structure that specifies phone class reduction paths; (2) language-dependent knowledge that includes information of the targeted languages’ phone inventories and individual phone frequencies in language data resources; (3) acoustic data that provides phone discriminability and similarity metrics. Using the optimization procedure, the phone inventories of six languages, American English, Mandarin Chinese, Egyptian Colloquial Arabic, Japanese, German, and Spanish, were merged to create an inventory consisting of 64 distinct cross-phonological units. This reduced phone set was used in all training and testing procedures and resources for the recognition of the six targeted languages. Preliminary recognition results are very encouraging: while purely data-driven approaches to multilingual ASR fail to reach word-recognition rates comparable to monolingual applications, the use of the optimized phone inventory in our multilingual ASR program yields recognition rates approximating that of monolingual ASR.