Fuzzy Logical Model Of Perception Research Articles

Regularized models of audiovisual integration of speech with predictive power for sparse behavioral data

Audiovisual integration can facilitate speech comprehension by integrating information from lip-reading with auditory speech perception. When incongruent acoustic speech is dubbed onto a video of a talking face, this integration can lead to the McGurk illusion of hearing a different phoneme than that spoken by the voice. Several computational models of the information integration process underlying these phenomena exist. All are based on the assumption that the integration process is, in some sense, optimal. They differ, however, in assuming that it is based on either continuous or categorical internal representations. Here we develop models of audiovisual integration of the phonetic information represented on an internal representation that is continuous and cyclical. We compare these models to the Fuzzy Logical Model of Perception (FLMP), which is based on a categorical internal representation. Using cross-validation, we show that model evaluation criteria based on the goodness-of-fit are poor measures of the models’ generalization error even if they take the number of free parameters into account. We also show that the predictive power of all the models benefit from regularization that limits the precision of the internal representation. Finally, we show that, unlike the FLMP, models based on a continuous internal representation have good predictive power when properly regularized.

Efeitos das relações fonológicas sobre a percepção das fricativas sibilantes não-vozeadas do português brasileiro

Este artigo investiga os efeitos das relações fonológicas sobre a percepção das fricativas sibilantes não-vozeadas [s] e [?] em ataque e coda silábica na variedade de português brasileiro falada em Florianópolis/SC. Em consonância com Boomershine et al., espera-se que as relações de contraste e de não-contraste afetem a percepção das fricativas alveolar e palato-alveolar de modo que as referidas categorias fonéticas sejam percebidas como relativamente mais distintas em ataque silábico (relação fonêmica) do que em coda (relação alofônica). A fim de verificar tal hipótese, testes de classificação por similaridade (processamento linguístico) e de discriminação acelerada (processamento psicoacústico) foram aplicados entre 30 falantes/ouvintes florianopolitanos. Os resultados obtidos pelo teste de processamento linguístico sugerem que as relações fonológicas de fato interferem na percepção das fricativas sibilantes pelos falantes/ouvintes florianopolitanos, pois as fricativas alveolar e palato-alveolar foram percebidas como mais distintas em ataque do que em coda. Por outro lado, os resultados do teste de processamento psicoacústico apontaram resultados opostos, a saber, as fricativas sibilantes se mostraram perceptualmente mais distintas em coda do que em ataque. Os resultados divergentes obtidos pelos dois testes indicam que as relações fonológicas não necessariamente condicionam a percepção das fricativas sibilantes em português. Por fim, argumenta-se que a percepção das fricativas sibilantes se dá em termos de sílabas FV (fricativa–vogal) e VF (vogal-fricativa) e que ocorre em estágios distintos de processamento tal como previsto pela Teoria Lógica-Difusa de percepção da fala.

Modeling the Roles of Category and Feature Information in Inference

The goal of the current research is to use the experimental methods and mathematical models of the information integration framework to precisely determine how category and feature information are combined when making an inference. In three experiments, participants were trained on a probabilistic relationship between a category label and the presence of a property and, separately, the relationship between a visual feature and the presence of the property. Participants were then shown the category label alone, the feature alone, or both in combination, and asked to infer the presence or absence of the property. Two information integration models, the fuzzy logical model of perception and the linear integration model, were fit to the data. The modeling results show that participants were non-Bayesian in their combination of the two sources of information, showed diversity in the relative weight placed on category information, and consistently used each source of information to the extent to which it was known.

Multisensory integration, learning, and the predictive coding hypothesis

The multimodal nature of perception has generated several questions of importance pertaining to the encoding, learning, and retrieval of linguistic representations (e.g., Summerfield, 1987; Altieri et al., 2011; van Wassenhove, 2013). Historically, many theoretical accounts of speech perception have been driven by descriptions of auditory encoding; this makes sense because normal-hearing listeners rely predominantly on the auditory signal. However, from both evolutionary and empirical standpoints, comprehensive neurobiological accounts of speech perception must account for interactions across sensory modalities and the interplay of cross-modal and articulatory representations. These include auditory, visual, and somatosensory modalities. In a recent review, van Wassenhove (2013) discussed key frameworks describing how visual cues interface with the auditory modality to improve auditory recognition (Sumby and Pollack, 1954), or otherwise contribute to an illusory percept for mismatched auditory-visual syllables (McGurk and MacDonald, 1976). These frameworks encompass multiple levels of analysis. Some of these higher cognitive processing models that discuss parallel processing (Altieri and Townsend, 2011) or the independent extraction of features from the auditory and visual modalities (Massaro, 1987, Fuzzy Logical Model of Perception), early feature encoding (van Wassenhove et al., 2005), and encoding/timing at the neural level (Poeppel et al., 2008; Schroeder et al., 2008). This commentary on van Wassenhove (2013) will examine predictive coding hypotheses as one theory for how visemes are matched with auditory cues. Crucially, a hypothesized role shall be emphasized for cross-modal neural plasticity and multisensory learning in reinforcing the sharing of cues across modalities into adulthood.

Speech Perception and Reading: Two Parallel Modes of Understanding Language and Implications for Acquiring Literacy Naturally

I review 2 seminal research reports published in this journal during its second decade more than a century ago. Given psychology's subdisciplines, they would not normally be reviewed together because one involves reading and the other speech perception. The small amount of interaction between these domains might have limited research and theoretical progress. In fact, the 2 early research reports revealed common processes involved in these 2 forms of language processing. Their illustration of the role of Wundt's apperceptive process in reading and speech perception anticipated descriptions of contemporary theories of pattern recognition, such as the fuzzy logical model of perception. Based on the commonalities between reading and listening, one can question why they have been viewed so differently. It is commonly believed that learning to read requires formal instruction and schooling, whereas spoken language is acquired from birth onward through natural interactions with people who talk. Most researchers and educators believe that spoken language is acquired naturally from birth onward and even prenatally. Learning to read, on the other hand, is not possible until the child has acquired spoken language, reaches school age, and receives formal instruction. If an appropriate form of written text is made available early in a child's life, however, the current hypothesis is that reading will also be learned inductively and emerge naturally, with no significant negative consequences. If this proposal is true, it should soon be possible to create an interactive system, Technology Assisted Reading Acquisition, to allow children to acquire literacy naturally.

Predictive power in models of audiovisual integration of speech

Seeing the talking face can influence the phoneme perceived from the voice. This facilitates speech perception in the natural case where the face and voice are congruent and can cause the McGurk illusion when they are not. The classical example of the McGurk illusion is when acoustic /aba/ is perceived as /ada/ when dubbed onto a face articulating /aga/. In order to fully understand the underlying process of integrating information across the senses we need a computational account with predictive power. The Fuzzy Logical Model of Perception is one computational account of audiovisual integration in speech perception. Here we describe alternative accounts in which integration is based on an early continuous internal representation on which the phonetic classes fall. We show that these alternative accounts can provide just as good a fit when corrected for the number of free parameters. We also show, using cross-validation, that they have greater, but not great, predictive power. Finally, we show that introducing a regularization term can amend the lack of predictive power. With regularization, models based on continuous representations have the highest predictive power.

Integration of Facial and Newly Learned Visual Cues in Speech Perception

We are developing technology to translate acoustic characteristics of speech into visual cues that can be used to supplement speechreading when hearing is limited. Research and theory have established that perceivers are influenced by multiple sources of sensory and contextual information in spoken language processing. Previous research has also shown that additional sources of information can be learned and used to supplement those that are normally available but have been degraded by sensory impairment or difficult environments. We tested whether people can combine or integrate information from the face and information from newly learned cues in an optimal manner. Subjects first learned the visual cues and then were tested under three conditions.Words were presented with just the face, just the visual cues, or both together. Performance was much better with both cues than with either one alone. Similar to the description of previous results with audible and visible speech, the present results were well described by the Fuzzy Logical Model of Perception (Massaro, 1998), which predicts optimal or maximally efficient integration.

A reanalysis of McGurk data suggests that audiovisual fusion in speech perception is subject-dependent

Audiovisual perception of conflicting stimuli displays a large level of intersubject variability, generally larger than pure auditory or visual data. However, it is not clear whether this actually reflects differences in integration per se or just the consequence of slight differences in unisensory perception. It is argued that the debate has been blurred by methodological problems in the analysis of experimental data, particularly when using the fuzzy-logical model of perception (FLMP) [Massaro, D. W. (1987). Speech Perception by Ear and Eye: A Paradigm for Psychological Inquiry (Laurence Erlbaum Associates, London)] shown to display overfitting abilities with McGurk stimuli [Schwartz, J. L. (2006). J. Acoust. Soc. Am. 120, 1795-1798]. A large corpus of McGurk data is reanalyzed, using a methodology based on (1) comparison of FLMP and a variant with subject-dependent weights of the auditory and visual inputs in the fusion process, weighted FLMP (WFLMP); (2) use of a Bayesian selection model criterion instead of a root mean square error fit in model assessment; and (3) systematic exploration of the number of useful parameters in the models to compare, attempting to discard poorly explicative parameters. It is shown that WFLMP performs significantly better than FLMP, suggesting that audiovisual fusion is indeed subject-dependent, some subjects being more "auditory," and others more "visual." Intersubject variability has important consequences for theoretical understanding of the fusion process, and re-education of hearing impaired people.

Model evaluation using grouped or individual data

Analyzing the data of individuals has several advantages over analyzing the data combined across the individuals (the latter we term group analysis): Grouping can distort the form of data, and different individuals might perform the task using different processes and parameters. These factors notwithstanding, we demonstrate conditions in which group analysis outperforms individual analysis. Such conditions include those in which there are relatively few trials per subject per condition, a situation that sometimes introduces distortions and biases when models are fit and parameters are estimated. We employed a simulation technique in which data were generated from each of two known models, each with parameter variation across simulated individuals. We examined how well the generating model and its competitor each fared in fitting (both sets of) the data, using both individual and group analysis. We examined the accuracy of model selection (the probability that the correct model would be selected by the analysis method). Trials per condition and individuals per experiment were varied systematically. Three pairs of cognitive models were compared: exponential versus power models of forgetting, generalized context versus prototype models of categorization, and the fuzzy logical model of perception versus the linear integration model of information integration. We show that there are situations in which small numbers of trials per condition cause group analysis to outperform individual analysis. Additional tables and figures may be downloaded from the Psychonomic Society Archive of Norms, Stimuli, and Data, www.psychonomic.org/archive.

The motor theory of speech perception revisited

Galantucci, Fowler, and Turvey (2006) have claimed that perceiving speech is perceiving gestures and that the motor system is recruited for perceiving speech. We make the counter argument that perceiving speech is not perceiving gestures, that the motor system is not recruitedfor perceiving speech, and that speech perception can be adequately described by a prototypical pattern recognition model, the fuzzy logical model of perception (FLMP). Empirical evidence taken as support for gesture and motor theory is reconsidered in more detail and in the framework of the FLMR Additional theoretical and logical arguments are made to challenge gesture and motor theory.

Auditory-visual speech perception: What isolated articulators contribute

The present study investigated the effects of viewing specific articulators on auditory-visual speech perception. We used a jaw-only, lips-only, jaw+lips, and full face visual stimulus paired with clear versus degraded auditory stimuli in an expanded factorial design. The unimodal and bimodal speech syllables were identified as one of five possible response alternatives (/ba/, /da/, /tha/, /bda/ or /bga/). The results indicate that both visual and auditory sources of information influenced syllable identification under the lips alone, lips+jaw alone, and whole face conditions. Performance was better described by the fuzzy logical model of perception than by an auditory dominance model or a weighted-averaging model. Overall, the results are consistent with the idea that although there may be differences in the information provided (which reflect the amount of information each articulator contributes to perception), the way in which the speech information is processed remains invariant. Work funded for the research was provided by a National Institute of Health Training in Speech and Hearing Sciences grant to the University of Washington’s Speech and Hearing Sciences Department and by a National Science Foundation Science of Learning Center grant to the University of Washington’s LIFE Center.

Visual Contribution to Speech Perception: Measuring the Intelligibility of Animated Talking Heads

Animated agents are becoming increasingly frequent in research and applications in speech science. An important challenge is to evaluate the effectiveness of the agent in terms of the intelligibility of its visible speech. In three experiments, we extend and test the Sumby and Pollack (1954) metric to allow the comparison of an agent relative to a standard or reference, and also propose a new metric based on the fuzzy logical model of perception (FLMP) to describe the benefit provided by a synthetic animated face relative to the benefit provided by a natural face. A valid metric would allow direct comparisons accross different experiments and would give measures of the benfit of a synthetic animated face relative to a natural face (or indeed any two conditions) and how this benefit varies as a function of the type of synthetic face, the test items (e.g., syllables versus sentences), different individuals, and applications.

The ∕ problem in the Fuzzy-Logical Model of Perception

The "Fuzzy-Logical Model of Perception" (FLMP) has often been questioned for its presumed ability to fit any data, but no clear-cut evidence has been presented yet. This paper demonstrates the ability of the FLMP to fit random data in the "McGurk region," that is in conditions involving conflicting stimuli. This is due to the so-called "0/0 problem," consisting in the fact that any audio-visual response can be fitted by the FLMP if the audio and visual stimuli provide at least one null probability in each possible category. The consequence is a high instability of the root mean square error in the region of the best fit.

Mandarin speech perception by ear and eye follows a universal principle

In this study, the nature of speech perception of native Mandarin Chinese was compared with that of American English speakers, using synthetic visual and auditory continua (from /ba/ to /da/) in an expanded factorial design. In Experiment 1, speakers identified synthetic unimodal and bimodal speech syllables as either /ba/ or /da/. In Experiment 2, Mandarin speakers were given nine possible response alternatives. Syllable identification was influenced by both visual and auditory sources of information for both Mandarin and English speakers. Performance was better described by the fuzzy logical model of perception than by an auditory dominance model or a weighted-averaging model. Overall, the results are consistent with the idea that although there may be differences in information (which reflect differences in phonemic repertoires, phonetic realizations of the syllables, and the phonotactic constraints of languages), the underlying nature of audiovisual speech processing is similar across languages.

Visual attention modulates audiovisual speech perception

Speech perception is audiovisual, as demonstrated by the McGurk effect in which discrepant visual speech alters the auditory speech percept. We studied the role of visual attention in audiovisual speech perception by measuring the McGurk effect in two conditions. In the baseline condition, attention was focused on the talking face. In the distracted attention condition, subjects ignored the face and attended to a visual distractor, which was a leaf moving across the face. The McGurk effect was weaker in the latter condition, indicating that visual attention modulated audiovisual speech perception. This modulation may occur at an early, unisensory processing stage, or it may be due to changes at the stage where auditory and visual information is integrated. We investigated this issue by conventional statistical testing, and by fitting the Fuzzy Logical Model of Perception (Massaro, 1998) to the results. The two methods suggested different interpretations, revealing a paradox in the current methods of analysis.

Audiovisual speech gating: examining information and information processing

We investigated the perceptual processing of facial information and vocal information to test the nature of the multisensory integration process. Single-syllable words were presented in background noise at one of eight stimulus durations ranging from 45 to 80% of the total word duration. Performance was evaluated relative to a fuzzy logical model of perception (FLMP) and an additive model (ADD). These two models differ in the nature of the integration process: optimal multiplicative integration in the FLMP and additive in the ADD. The FLMP provided a significantly better description of the word identifications relative to the ADD. Consistent with the outcomes of several other tasks in audiovisual speech perception, the FLMP also accurately describes the continuous uptake of information during word perception.

Modeling the effects of choice-set size on the processing of letters and words.

Letters and words are better identified when there are fewer available choices. How do readers use choice-set restrictions? By analyzing new experimental data and previously reported data, the author shows that Bayes theorem-based models overestimate readers' use of choice-set restrictions. This result is discordant with choice-similarity models such as R. D. Luce's (1963a) similarity choice model, G. Keren and S. Baggen's (1981) letter recognition model, and D. W. Massaro and G. C. Oden's (1979) fuzzy logical model of perception. Other models posit that choice restrictions affect accuracy only by improving guessing (e.g., J. L. McClelland & D. E. Rumelhart's, 1981, interactive activation model). It is shown that these models underestimate readers' use of choice-set restrictions. Restriction of choice set does improve perception of letters and words, but not optimally. Decision models that may be able to explain this phenomenon are discussed.

Hierarchical categorization of coarticulated phonemes: a theoretical analysis.

This article is concerned with the question of how listeners recognize coarticulated phonemes. The problem is approached from a pattern classification perspective. First, the potential acoustical effects of coarticulation are defined in terms of the patterns that form the input to a classifier. Next, a categorization model called HICAT is introduced that incorporates hierarchical dependencies to optimally deal with this input. The model allows the position, orientation, and steepness of one phoneme boundary to depend on the perceived value of a neighboring phoneme. It is argued that, if listeners do behave like statistical pattern recognizers, they may use the categorization strategies incorporated in the model. The HICAT model is compared with existing categorization models, among which are the fuzzy-logical model of perception and Nearey's diphone-biased secondary-cue model. Finally, a method is presented by which categorization strategies that are likely to be used by listeners can be predicted from distributions of acoustical cues as they occur in natural speech.

Tribute to an ideal exemplar of scientist and person

Roger Shepard's creativity and scientific contributions have left an indelible mark on Psychology and Cognitive Science. In this tribute, I acknowledge and show how his approach to universal laws helped Oden and me shape and develop our universal law of pattern recognition, as formulated in the Fuzzy Logical Model of Perception (FLMP). [Shepard; Tenenbaum & Griffiths]

Modeling Face Identification Processing in Children and Adults

Two face identification experiments were carried out to study whether and how children (5-year-olds) and adults integrate single facial features to identify faces. Using the paradigm of the Fuzzy Logical Model of Perception each experiment used the same expanded factorial design, with three levels of eyes variations crossed with three levels of mouth variations as well as their corresponding half-face conditions. In Experiment 1, an integration of facial features was observed in adults only. But, in adjusting the salience of the features varied, the results of Experiment 2 indicate that children and adults evaluated and integrated information from both features to identify a face. A weighted Fuzzy Logical Model of Perception fit the judgments significantly better than a Single Channel Model and questions previous claims of holistic face processing. Although no developmental differences in the stage of the integration of facial information were observable, differences between children and adults appeared in the information used for face identification.