The Use of Word, Phrase and Intent Accuracy as Measures of Connected Speech Recognition Performance

Abstract

Eleven subjects participated in a study designed to test the accuracy of a newer-generation connected speech recognition system using a 49 word vocabulary likely to be used in an aircraft cockpit environment. The 49 vocabulary words were used to create 392 test phrases. These phrases were divided into three groups: Complex phrases, which contain more than five words, and two groups of Simple phrases, which contain 5 words or less. The simple phrases were divided into Simple Alternate and Simple No-Alternate phrases, depending on whether or not the phrase was the only one in the entire vocabulary capable of carrying out a particular action once recognition occurred. Performance of the recognition system was measured with three accuracy statistics: word accuracy, the most commonly reported statistic in speech recognition research, phrase accuracy, which is gaining popularity in connected speech recognition research, and intent accuracy, which is probably the most relevant statistic that could be reported in research of this type. Significantly different word, phrase, and intent accuracy results were obtained for the three different phrase types.