発話シーンからのキーフレーム検出とキーフレームに基づく単語読唇

Abstract

In this paper, we propose the new keyframe-based lip reading method which does not need the advanced registration of an utterance scene. To extract keyframe, we apply the frame subtraction method and extract frame which the difference value is the local minimum as the keyframe. We compute thirteen shape features from the five lip regions of the extracted keyframe. Then we apply a discriminant analysis to mouth shape recognition. We generate a code sequence based on a mouth shape recognition result. Moreover, in accordance with several rules, we generate candidate code sequences. Finally, we apply DP matching using two kinds of code sequence of based on keyframe and candidate, and select the similar code sequence as the result word. We set Japanese 19 words as the target. We took four speakers’ utterance scene. We carried out three experiments of the keyframe extraction, the mouth shape recognition, and the word recognition. As a result, we obtained average recognition rate of 53.9%. Although there was individual difference, one speaker obtained 72.1% of the highest recognition rate.