Perceptual Simulation Research Articles

Auditory perceptual simulation: Simulating speech rates or accents?

When readers engage in Auditory Perceptual Simulation (APS) during silent reading, they mentally simulate characteristics of voices attributed to a particular speaker or a character depicted in the text. Previous research found that auditory perceptual simulation of a faster native English speaker during silent reading led to shorter reading times that auditory perceptual simulation of a slower non-native English speaker. Yet, it was uncertain whether this difference was triggered by the different speech rates of the speakers, or by the difficulty of simulating an unfamiliar accent. The current study investigates this question by comparing faster Indian-English speech and slower American-English speech in the auditory perceptual simulation paradigm. Analyses of reading times of individual words and the full sentence reveal that the auditory perceptual simulation effect again modulated reading rate, and auditory perceptual simulation of the faster Indian-English speech led to faster reading rates compared to auditory perceptual simulation of the slower American-English speech. The comparison between this experiment and the data from Zhou and Christianson (2016) demonstrate further that the “speakers'” speech rates, rather than the difficulty of simulating a non-native accent, is the primary mechanism underlying auditory perceptual simulation effects.

I "hear" what you're "saying": Auditory perceptual simulation, reading speed, and reading comprehension.

Auditory perceptual simulation (APS) during silent reading refers to situations in which the reader actively simulates the voice of a character or other person depicted in a text. In three eye-tracking experiments, APS effects were investigated as people read utterances attributed to a native English speaker, a non-native English speaker, or no speaker at all. APS effects were measured via online eye movements and offline comprehension probes. Results demonstrated that inducing APS during silent reading resulted in observable differences in reading speed when readers simulated the speech of faster compared to slower speakers and compared to silent reading without APS. Social attitude survey results indicated that readers' attitudes towards the native and non-native speech did not consistently influence APS-related effects. APS of both native speech and non-native speech increased reading speed, facilitated deeper, less good-enough sentence processing, and improved comprehension compared to normal silent reading.

Resonant Dynamics of Grounded Cognition: Explanation of Behavioral and Neuroimaging Data Using the ART Neural Network.

Research on grounded cognition suggests that the processing of a word or concept reactivates the perceptual representations that are associated with the referent object. The objective of this work is to demonstrate how behavioral and functional neuroimaging data on grounded cognition can be understood as different manifestations of the same cortical circuit designed to achieve stable category learning, as proposed by the adaptive resonance theory (ART). We showed that the ART neural network provides a mechanistic explanation of why reaction times in behavioral studies depend on the expectation or attentional priming created by the word meaning (Richter and Zwaan, 2009). A mismatch between top-down expectation and bottom-up sensory data activates an orienting subsystem that slows execution of the current task. Furthermore, we simulated the data from functional neuroimaging studies of color knowledge retrieval that showed anterior shift (Chao and Martin, 1999; Thompson-Schill, 2003) and an overlap effect (Simmons et al., 2007; Hsu et al., 2011) in the left fusiform gyrus. We explain the anterior effect as a result of the partial activation of different components of the same ART circuit in the condition of passive viewing. Conversely, a demanding perceptual task requires activation of the whole ART circuit. This condition is reflected in the fMRI image as an overlap between cortical activation during perceptual and conceptual processing. We conclude that the ART neural network is able to explain how the brain grounds symbols in perception via perceptual simulation.

Do we know what we're simulating? Information loss on transferring unconscious perceptual simulation to conscious imagery.

Perceptual simulations are unconscious and automatic, whereas perceptual imagery is conscious and deliberate, but it is unclear how easily one can transfer perceptual information from unconscious to conscious awareness. We investigated whether it is possible to be aware of what one is mentally representing; that is, whether it is possible to consciously examine the contents of a perceptual simulation without information being lost. Studies 1 and 2 found that people cannot accurately evaluate the perceptual content of a representation unless attention is explicitly drawn to each modality individually. In particular, when asked to consider sensory experience as a whole, modality-specific auditory, gustatory, and haptic information is neglected, and olfactory and visual information distorted. Moreover, information loss is greatest for perceptually complex, multimodal simulations. Study 3 examined if such information loss leads to behavioral consequences by examining performance during lexical decision, a task whose semantic effects emerge from automatic access to the full potential of unconscious perceptual simulation. Results showed that modality-specific perceptual strength consistently outperformed modality-general sensory experience ratings in predicting latency and accuracy, which confirms that the effects of Studies 1 and 2 are indeed due to information being lost in the transfer to conscious awareness. These findings suggest that people indeed have difficulty in transferring perceptual information from unconscious simulation to conscious imagery. People cannot be aware of the full contents of a perceptual simulation because the act of bringing it to awareness leads to systematic loss of information. (PsycINFO Database Record

Perceptual Uniform Simulation for Tang Tomb Mural Inpainting by Camera Array

Perceptual Uniform Simulation for Tang Tomb Mural Inpainting by Camera Array

Supplemental Material for Do We Know What We’re Simulating? Information Loss on Transferring Unconscious Perceptual Simulation to Conscious Imagery

Supplemental Material for Do We Know What We’re Simulating? Information Loss on Transferring Unconscious Perceptual Simulation to Conscious Imagery

The modality-switch effect: visually and aurally presented prime sentences activate our senses.

Verifying different sensory modality properties for concepts results in a processing cost known as the modality-switch effect. It has been argued that this cognitive cost is the result of a perceptual simulation. This paper extends this argument and reports an experiment investigating whether the effect is the result of an activation of sensory information which can also be triggered by perceptual linguistically described stimuli. Participants were first exposed to a prime sentence describing a light or a sound’s perceptual property (e.g., “The light is flickering”, “The sound is echoing”), then required to perform a property-verification task on a target sentence (e.g., “Butter is yellowish”, “Leaves rustle”). The content modalities of the prime and target sentences could be compatible (i.e., in the same modality: e.g., visual–visual) or not (i.e., in different modalities). Crucially, we manipulated the stimuli’s presentation modality such that half of the participants was faced with written sentences while the other half was faced with aurally presented sentences. Results show a cost when two different modalities alternate, compared to when the same modality is repeated with both visual and aural stimuli presentations. This result supports the embodied and grounded cognition view which claims that conceptual knowledge is grounded into the perceptual system. Specifically, this evidence suggests that sensory modalities can be pre-activated through the simulation of either read or listened linguistic stimuli describing visual or acoustic perceptual properties.

Environmental design studies on perception and simulation: an urban design approach

Perceptual simulation represents an attempt to anticipate physical reality, whereby people can experience and interpret future environments from a subjective perspective. Working on experiential simulation for urban and landscape design requires an understanding of the relationship between man and the environment from a perceptual and cognitive standpoint. In fact, only by investigating the sensing and cognitive processes behind perception can we establish an informed approach to simulation of places and their ambiances. In particular, we propose a parallelism between man/environment and man/simulation relationships, aiming at giving back a framework for replicating in simulation the multisensory aspects that occur in the perception of the physical world. Hence, the objective of this article is to present how we approach the dimension of perceptual simulation within our research and professional work as urban designers. From a methodological point of view, we explored the topic through two main tasks, namely the selection and reconstruction of the research context and the key issues of perceptual simulation finalized in the second task, i.e. the construction of a set of simulation tools for urban design, intended as a matrix of possible practical applications. In particular, the theoretical framework presented in this work consists of a selection and overview of references relevant to urban design, comprehension of the research context and delivery of the set of tools implemented within our research unit. This matrix of tools represents the novelty of this work and is intended as a practical reference for orienting the choice among different simulation tools within the urban design practice. For instance, it is important to highlight the efficacy of each type of simulation in mimicking the man/environment relationship.

Individual differences in spatial cognition influence mental simulation of language

The factors that contribute to perceptual simulation during sentence comprehension remain underexplored. Extant research on perspective taking in language has largely focused on linguistic constraints, such as the role of pronouns in guiding perspective adoption. In the present study, we identify preferential usage of egocentric and allocentric reference frames in individuals, and test the two groups on a standard sentence-picture verification task. Across three experiments, we show that individual biases in spatial reference frame adoption observed in non-linguistic tasks influence visual simulation of perspective in language. Our findings suggest that typically reported grand-averaged effects may obscure important between-subject differences, and support proposals arguing for representational pluralism, where perceptual information is integrated dynamically and in a way that is sensitive to contextual and especially individual constraints.

Reconsidering “Image Metaphor” in the Light of Perceptual Simulation Theory

“Image metaphor” is defined in Conceptual Metaphor Theory as a mapping of visual structure from one entity onto another based on the mental images they evoke. It is considered an exceptional, one-off phenomenon that can be distinguished clearly from prototypical conceptual metaphors. However, according to Perceptual Simulation Theory, all language, both literal and nonliteral, is understood partially by simulating in our minds what it would be like to actually perceive the things that are being described, which suggests that visualization is not unique to image metaphor. As I argue in this article, the mental imagery invited by image metaphors is nevertheless likely to be particularly vivid, detailed, and complex, as well as involving more conscious and deliberate thought processes. Since it is hard to consider visualization in abstract terms, pictorial equivalents of image metaphors are used to support my claims.

Spatial configuration of vertically related word pairs modulates the N400 component.

This study examined the effect of spatial iconicity on the N400 component. Spatial iconicity is defined as the spatial arrangement of words on a screen relative to the spatial arrangement of their referents (e.g. attic-basement). In two experiments, electroencephalograms were recorded in 32 participants while performing a semantic relatedness judgment task on pairs of words that were either related or unrelated. All of the related word pairs were parts of objects that shared a vertical spatial relationship. In Experiment 1, the words of each pair were presented simultaneously on top of one another. Results showed that related word pairs presented in a spatial arrangement that mismatched the spatial relationship of their referents were associated with increased error rates as well as larger N400 components known to index semantic/conceptual processing cost. These findings thus suggest that the words automatically activated visuospatial simulations of their referents and that semantic/conceptual processing difficulty arose when the vertical arrangement of the word pairs was inconsistent with those simulations. In line with this interpretation, these effects were not present in Experiment 2 when the words of each pair were presented in succession in the middle of the screen. Overall, these results provide evidence that perceptual simulations contribute to some of the underlying processes of the N400 component (see video abstract, Supplemental digital content 1, http://links.lww.com/WNR/A304).

Effect size matters: the role of language statistics and perceptual simulation in conceptual processing

The cognitive science literature increasingly demonstrates that perceptual representations are activated during conceptual processing. Such findings suggest that the debate on whether conceptual processing is predominantly symbolic or perceptual has been resolved. However, studies too frequently provide evidence for perceptual simulations without addressing whether other factors explain dependent variables as well, and if so, to what extent. The current paper examines effect sizes computed from 126 experiments in 51 published embodied cognition studies to clarify the conditions under which perceptual simulations are most important. Results showed that effects of language statistics tend to be as large or larger than those of perceptual stimulation. Moreover, factors that can be associated with immediate processing (button press, word processing) tend to reduce the effect size of perceptual simulation. These findings are considered in respect to the Symbol Interdependency Hypothesis, which argues that language encodes perceptual information, with language statistics explaining quick, good-enough representations and perceptual simulation explaining more effortful, detailed representations.

Moving stimuli guide retrieval and (in)validation of coordination simulations

According to theories of embodied cognition, visual stimuli can either facilitate or impede the retrieval of language meaning as multimodal perceptual simulations. Here, we introduced a novel experimental paradigm to test the hypothesis that moving stimuli (i.e., motion-defined objects) facilitate coordination comprehension. Participants read coordination descriptions and saw two colored lines that matched the descriptions. Two figures then selected the lines either by moving jointly along them or by standing each on a different line. Moving selections yielded high validation scores in conjunction trials and low validation scores in disjunction trials, whereas stationary selections yielded mitigated scores. The results demonstrate that jointly moving stimuli, which are effective cues to visual grouping, help retrieve and validate conjunction simulations composed of dependent stimuli as well as retrieve and invalidate disjunction simulations composed of independent stimuli. These findings challenge accounts based on truth-condition satisfaction that stimuli properties cannot affect language comprehension and thereby reasoning.

Why is the sunny side always up? Explaining the spatial mapping of concepts by language use

Humans appear to rely on spatial mappings to represent and describe concepts. The conceptual cuing effect describes the tendency for participants to orient attention to a spatial location following the presentation of an unrelated cue word (e.g., orienting attention upward after reading the word sky). To date, such effects have predominately been explained within the embodied cognition framework, according to which people's attention is oriented on the basis of prior experience (e.g., sky → up via perceptual simulation). However, this does not provide a compelling explanation for how abstract words have the same ability to orient attention. Why, for example, does dream also orient attention upward? We report on an experiment that investigated the role of language use (specifically, collocation between concept words and spatial words for up and down dimensions) and found that it predicted the cuing effect. The results suggest that language usage patterns may be instrumental in explaining conceptual cuing.

Perceptual Simulation in Gender Categorization: Associations between Gender, Vertical Height, and Spatial Size

The current studies extend perceptual symbol systems theory to the processing of gender categorization by revealing that gender categorization recruits perceptual simulations of spatial height and size dimensions. In study 1, categorization of male faces were faster when the faces were in the “up” (i.e., higher on the vertical axis) rather than the “down” (i.e., lower on the vertical axis) position and vice versa for female face categorization. Study 2 found that responses to male names depicted in larger font were faster than male names depicted in smaller font, whereas opposite response patterns were given for female names. Study 3 confirmed that the effect in Study 2 was not due to metaphoric relationships between gender and social power. Together, these findings suggest that representation of gender (social categorization) also involves processes of perceptual simulation.

Temporal-Spatial Metaphor in Conceptual Representation: Can Spatial Information be Activated When Processing the Temporal Reference Which Implied in the Changing Shape of Objects?

Although time is abstract, unbodied and intangible, human also can feel it in some way. The changing shapes of objects show the existence of time. As time goes on, the shape of objects will change or be changed. The objects in a forward position of time series refer to "earlier", while the objects in a afterward position of time series refer to "later". It has been demonstrated that English participants were faster responding to "earlier" objects presented on the left of the screen and to "later" objects presented on the right of the screen. Both the perceptual symbol theory and the propositional symbol theory assure that this finding can be explained by the opinion of the reading/writing directionality affecting the representation of temporal sequences. Therefore, the former studies haven't solved the disputes of the above two theories. The substantial difference between two theories is whether the information, activated during the concept processing, is perceptual. In present research, two experiments were conducted to investigate whether the perceptual traits(the degree of shape change) affected congruency effects between temporal reference and spatial relation.In each experiment, pairs of words or pictures of objects, which were displayed on the left and right of the screen respectively, were used as targets. It is called metaphor-consistent condition, if the "earlier" objects in this condition were on the left of screen and the "later" ones were on the right; by contrast, it is called metaphor-inconsistent condition. Pairs of objects were divided into two groups. In the one group, the shapes of objects in pairs have changed gradually as time goes on. There are still a lot of same traits between two objects; in the other group, the shapes of objects in pairs have changed abruptly. There is little resemblance between two objects.In Experiment 1, Participants were instructed to carry out semantic relatedness judgment to pairs of object names with go/no-go task. The results showed semantic processing was inhibited in metaphor-inconsistent condition rather than metaphor-consistent condition. Participants were faster responding when the "earlier" objects in the left of the screen and the "later" objects in the right of the screen. It indicated that the temporal reference which implied in the changing shape of objects was activated. Furthermore, there were more significant temporal-spatial congruency effects in the shapes-changing-abruptly group. This result showed that perceptual traits were activated when processing the concept. Experiment 2, which was same as Experiment 1 except changing words into pictures, got the same temporal-spatial congruency effects as Experiment 1. Furthermore, Experiment 2 got a very important and different result. Perceptual traits affected the semantic relatedness judgment to pairs of object pictures which did not observe in Experiment 1. Participants were faster responding to the objects of gradually changing shapes than the ones of changing abruptly shapes. In short, it is claimed that spatial information can be activated when processing temporal reference which implied in the changing shape of objects. Combining the results of two experiments, it suggested that there were both semantic processing and perceptual simulation in the conceptual processing, which further proved the perspectives of dual-processing theory.

Embodied language comprehension: Encoding-based and goal-driven processes.

Theories of embodied language comprehension have proposed that language is understood through perceptual simulation of the sensorimotor characteristics of its meaning. Strong support for this claim requires demonstration of encoding-based activation of sensorimotor representations that is distinct from task-related or goal-driven processes. Participants in 3 eye-tracking experiments were presented with triplets of either numbers or object and animal names. In Experiment 1, participants indicated whether the size of the referent of the middle object or animal name was in between the size of the 2 outer items. In Experiment 2, the object and animal names were encoded for an immediate recognition memory task. In Experiment 3, participants completed the same comparison task of Experiment 1 for both words and numbers. During the comparison tasks, word and number decision times showed a symbolic distance effect, such that response time was inversely related to the size difference between the items. A symbolic distance effect was also observed for animal and object encoding times in cases where encoding time likely reflected some goal-driven processes as well. When semantic size was irrelevant to the task (Experiment 2), it had no effect on word encoding times. Number encoding times showed a numerical distance priming effect: Encoding time increased with numerical difference between items. Together these results suggest that while activation of numerical magnitude representations is encoding-based as well as goal-driven, activation of size information associated with words is goal-driven and does not occur automatically during encoding. This conclusion challenges strong theories of embodied cognition which claim that language comprehension consists of activation of analog sensorimotor representations irrespective of higher level processes related to context or task-specific goals.

Automatic perceptual simulation of first language meanings during second language sentence processing in bilinguals

Research supports the claim that, when understanding language, people perform mental simulation using those parts of the brain which support sensation, action, and emotion. A major criticism of the findings quoted as evidence for embodied simulation, however, is that they could be a result of conscious image generation strategies. Here we exploit the well-known fact that bilinguals routinely and automatically activate both their languages during comprehension to test whether this automatic process is, in turn, modulated by embodied simulatory processes. Dutch participants heard English sentences containing interlingual homophones and implying specific distance relations, and had to subsequently respond to pictures of objects matching or mismatching this implied distance. Participants were significantly slower to reject critical items when their perceptual features matched said distance relationship. These results suggest that bilinguals not only activate task-irrelevant meanings of interlingual homophones, but also automatically simulate these meanings in a detailed perceptual fashion. Our study supports the claim that embodied simulation is not due to participants' conscious strategies, but is an automatic component of meaning construction.

Characterization of reasoning in terms of perceptual simulation

Although characterizing reasoning and natural language semantics in traditional logic captures their complexity and productivity, accounting for the grounding of logical reasoning in perception raises several challenges. These include difficulties in explaining the integration of reasoning and perceptual processing, and in accounting for the evolution of human reasoning from sensorimotor origins. Central to these problems is the fact that traditional logic includes elements such as quantifiers and negation that do not obviously occur in perceptual representations. We propose a formal framework in terms of perceptual simulation that bridges this gap. We demonstrate that perceptual simulations have the power to explain crucial elements of logical human reasoning and also allow us to provide the first unified linguistic analysis of noun phrases, negative polarity items and branching quantifiers within a single cognitively motivated formal framework.

Language statistics and individual differences in processing primary metaphors

AbstractResearch in cognitive linguistics has emphasized the role of embodiment in metaphor comprehension, with experimental research showing activation of perceptual simulations when processing metaphors. Recent research in conceptual processing has demonstrated that findings attributed to embodied cognition can be explained through language statistics. The current study investigates whether language statistics explain processing of primary metaphors and whether this effect is modified by the gender of the participant. Participants saw word pairs with valence (Experiment 1: good–bad), authority (Experiment 2: doctor–patient), temperature (Experiment 3: hot–cold), or gender (Experiment 4: male–female) connotations. The pairs were presented in either a vertical configuration (X above Y or Y above X) matching the primary metaphors (e.g., HAPPY IS UP, CONTROL IS UP) or a horizontal configuration (X left of Y or Y left of X) not matching the primary metaphors. Even though previous research has argued that primary metaphor processing can best be explained by an embodied cognition account, results demonstrate that statistical linguistic frequencies also explain the response times of the stimulus pairs both in vertical and horizontal configurations, because language has encoded embodied relations. In addition, the effect of the statistical linguistic frequencies was modified by participant gender, with female participants being more sensitive to statistical linguistic context than male participants.