Signal detection of lipreading visemes using two dimensional and three dimensional images

Abstract

The actual process by which the lipreader translates the lip movements they identify into a message is very complex. The lip movements observed represent only fragments of the complete message. The main purpose of this study is to investigate (1) the ability of lipreaders to use visual information alone to identify phonemes in varying contexts including nearby coarticulation effects and vowel neighborhoods; (2) lipreading responses using the effect of improved video presentation through 3D video, providing better and more realistic video presentation; and (3) the use of a novel measurement technique, i.e., a signal detection two-alternative-forced choice method of subject response that should provide measures of discrimination between phonemes including “visemes.” Video recordings were made in both 2D and 3D formats. This 3D image presented more realistically the movements such as lip-rounding and micro-movements of viewable articulators in three dimensions. Subjects with normal hearing were presented these video presentations. A Two-Alternative-Forced-Choice (2AFC) paradigm was used. The consonants were viewed with various vowel contexts. D-prime values were obtained for both the 2D and 3D videos. Particular consonant clusters were more discriminable in 3D.