Measuring visual contributions in phonetic categorization

Abstract

Although most of Joanne Miller's work has explored the mapping from the acoustic signal to phonetic categories, she has also investigated the contributions of visual phonetic information in this process (Green and Miller, 1985). Audiovisual integration is commonly assessed using the McGurk effect, an illusion occurring with audiovisually conflicting stimuli. We (Brancazio and Miller, 2005) suggested that the McGurk effect may underestimate visual contributions in speech perception, in part based on our finding that visual speaking rate influences phonetic judgments when the McGurk effect does not occur (with stimuli that typically produce the effect). Rather, audiovisual integration of incongruent stimuli might result in percepts that fall between phonetic categories—including the one consistent with the acoustic signal—and then are mapped onto one of these categories. Thus, variability in the incidence of the McGurk effect might reflect variability in the process of mapping onto phonetic categories more so than variability in audiovisual integration. In recent work, I have sought to disentangle some factors that might influence the magnitude of the McGurk effect. I will describe findings from recent studies involving variations on the standard McGurk paradigm, and discuss the implications for developing finer-grained methods of assessing audiovisual integration.