Performance prediction of word recognition using the probability of word occurrence

Abstract

AbstractThe words in natural language have different occurrence probabilities. Consequently, the information obtained from the event, i.e., the occurrence of a word, is larger than in the case of the occurrence with uniform probability. In other words, it will be effective to utilize the occurrence probability of the word in the recognition and it is sensible to examine its error‐correcting ability.This paper considers the situation where the word occurrence probability is used in the word recognition process and presents a method to estimate the relation between the phoneme/character recognition score and the word recognition score. In the past derivation of the evaluation formula for the word recognition score, it is assumed that the word occurrence probability is uniform for whole words and the difference is ignored.From such a viewpoint, this paper derives the evaluation formula considering the word occurrence probability. By comparing the value estimated by the derived evaluation formula and the value obtained by the simulation for the word recognition, it is found that there is a considerable error due to the approximation and the word recognition score is estimated as approximately 10 percent lower for the phoneme recognition score of 80 percent. Then the approximation procedure is modified and an evaluation formula containing a correction factor is derived. the difference between the value estimated by the corrected evaluation formula and the value obtained by simulation is less than 5 percent for the phoneme recognition score of 80 percent. In other words, the precision is improved and the word recognition score, when the word occurrence probability is utilized, can be estimated accurately.