Perceptual Simulation Research Articles

An intelligent glove interface with integrated perceptual simulation and motion tracking for enhanced virtual interactions

An intelligent glove interface with integrated perceptual simulation and motion tracking for enhanced virtual interactions

Bridging phenomenology and neural mechanisms of inner speech: ALE meta-analysis on egocentricity and spontaneity in a dual-mechanistic framework

The neural mechanisms of inner speech remain unclear despite its importance in a variety of cognitive processes and its implication in aberrant perceptions such as auditory verbal hallucinations. Previous research has proposed a corollary discharge model in which inner speech is a truncated form of overt speech, relying on speech production-related regions (e.g. left inferior frontal gyrus). This model does not fully capture the diverse phenomenology of inner speech and recent research suggesting alternative perception-related mechanisms of generation. Therefore, we present and test a framework in which inner speech can be generated by two separate mechanisms, depending on its phenomenological qualities: a corollary discharge mechanism relying on speech production regions and a perceptual simulation mechanism within speech perceptual regions. The results of the activation likelihood estimation meta-analysis examining inner speech studies support the idea that varieties of inner speech recruit different neural mechanisms.

Mental simulation and language comprehension: The case of copredication

Empirical evidence suggests that perceptual‐motor simulations are often constitutively involved in language comprehension. Call this “the simulation view of language comprehension”. This article applies the simulation view to illuminate the much‐discussed phenomenon of copredication, where a noun permits multiple predications which seem to select different senses of the noun simultaneously. On the proposed account, the (in)felicitousness of a copredicational sentence is closely associated with the perceptual simulations that the language user deploys in comprehending the sentence.

Different kinds of simulation during literary reading: Insights from a combined fMRI and eye-tracking study

Mental simulation is an important aspect of narrative reading. In a previous study, we found that gaze durations are differentially impacted by different kinds of mental simulation. Motor simulation, perceptual simulation, and mentalizing as elicited by literary short stories influenced eye movements in distinguishable ways (Mak & Willems, 2019). In the current study, we investigated the existence of a common neural locus for these different kinds of simulation. We additionally investigated whether individual differences during reading, as indexed by the eye movements, are reflected in domain-specific activations in the brain. We found a variety of brain areas activated by simulation-eliciting content, both modality-specific brain areas and a general simulation area. Individual variation in percent signal change in activated areas was related to measures of story appreciation as well as personal characteristics (i.e., transportability, perspective taking). Taken together, these findings suggest that mental simulation is supported by both domain-specific processes grounded in previous experiences, and by the neural mechanisms that underlie higher-order language processing (e.g., situation model building, event indexing, integration).

Spike‐Based Spatiotemporal Processing Enabled by Oscillation Neuron for Energy‐Efficient Artificial Sensory Systems

Developing an energy‐efficient artificial sensory system is of great significance for neuroprosthesis, neurorobotics, and intelligent human–machine interfaces. Inspired by the biological perception, achieving this goal through spatiotemporal processing is viable. But some challenges, such as continuous signal coding resulting in high‐energy consumption, are yet to be solved, hindering the realization of human perception emulation. Herein, a perceptual simulation enabled by spike‐based spatiotemporal processing is demonstrated, which is analogous to the biological behavior, through an NbOx‐based oscillation neuron. The time difference between distinct inputs has a notable impact on the output spiking activity of oscillation neuron. On the basis of these features, the temporal‐related perceptions, for example, direction selectivity and sound localization are closely imitated. Unambiguous differentiation of direction or azimuth is enabled according to the output spiking numbers of oscillation neuron. Furthermore, by combining the oscillation neuron with a spiking neural network, azimuth recognition is conceptually established to mimic the human's response to auditory stimuli. Herein, the feasibility of employing spike‐based spatiotemporal processing of oscillation neurons to emulate sensory functionality, paving a highly potential way for realizing energy‐efficient artificial sensory systems, is proved.

Kinesic intelligence, medieval illuminated psalters, and the poetics of the psalms

ABSTRACT Kinesic intelligence enables humans to understand physical movements and produce complex meanings out of them and by means of them. Grounded in embodied cognition, it elicits perceptual simulations of sensorimotor events. An attention to perceptual simulations in literature and art helps not only address readers’ and audiences’ cognitive participation in their reception of artworks, it also helps account for historical traces of cognitive acts, perceptual simulations, and kinesic intelligence in the production of artworks. This essay tests this claim in two ways. Its first part studies the text-image relationships in four medieval manuscripts containing illustrations of the psalms (the Utrecht Psalter and its English derivatives, i.e., the Harley Psalter, the Eadwine Psalter, and the Paris Psalter), while its second part focuses on a challenging line in Psalm 16: adipem suum concluserunt, comparing various translations of it over centuries and showing that its core meaning is conveyed by a sensorimotor metaphor that calls for a sustained attention to the perceptual simulations it triggers.

The concreteness advantage in lexical decision does not depend on perceptual simulations.

words are typically more difficult to identify than concrete words in lexical-decision, word-naming, and recall tasks. This behavioral advantage, known as the concreteness effect, is often considered as evidence for embodied semantics, which emphasizes the role of sensorimotor experience in the comprehension of word meaning. In this view, online sensorimotor simulations triggered by concrete words, but not by abstract words, facilitate access to word meaning and speed up word identification. To test whether perceptual simulation is the driving force underlying the concreteness effect, we compared data from early-blind and sighted individuals performing an auditory lexical-decision task. Subjects were presented with property words referring to abstract (e.g., "logic"), concrete multimodal (e.g., "spherical"), and concrete unimodal visual concepts (e.g., "blue"). According to the embodied account, the processing advantage for concrete unimodal visual words should disappear in the early blind because they cannot rely on visual experience and simulation during semantics processing (i.e., purely visual words should be abstract for early-blind people). On the contrary, we found that both sighted and blind individuals are faster when processing multimodal and unimodal visual words compared with abstract words. This result suggests that the concreteness effect does not depend on perceptual simulations but might be driven by modality-independent properties of word meaning. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Is implicit Level-2 visual perspective-taking embodied? Spontaneous perceptual simulation of others’ perspectives is not impaired by motor restriction

Visual perspective taking may rely on the ability to mentally rotate one’s own body into that of another. Here, we test whether participants’ ability to make active body movements plays a causal role in visual perspective taking. We utilised our recent task that measures whether participants spontaneously represent another’s visual perspective in a (quasi-)perceptual format that can drive own perceptual decision making. Participants reported whether alphanumeric characters, presented in different orientations, are shown in their normal or mirror-inverted form (e.g., “R” vs. “Я”). Between trials, we manipulated whether another person was sitting either left or right of the character and whether participants’ movement was restricted with a chinrest or whether they could move freely. As in our previous research, participants spontaneously took the visual perspective of the other person, recognising rotated letters more rapidly when they appeared upright to the other person in the scene, compared with when they faced away from that person, and these effects increased with age but were (weakly) negatively related to schizotypy and not to autistic traits or social skills. Restricting participants’ ability to make active body movements did not influence these effects. The results, therefore, rule out that active physical movement plays a causal role in computing another’s visual perspective, either to create alignment between own and other’s perspective or to trigger perspective taking processes. The postural adjustments people sometimes make when making judgements from another’s perspective may instead be a bodily consequence of mentally transforming one’s actual to an imagined position in space.

Perceptual simulation in language comprehension and Chinese character reading among third-grade Hong Kong children

The current study investigated perceptual simulation and its relationship with literacy ability in Chinese children. Ninety-three third-grade Hong Kong Chinese children completed a sentence-picture verification task for perceptual simulation. In this task, a sentence mentioning an object was presented first, followed by a picture involving the object. The picture was either a perceptual match with the preceding sentence or a mismatch. The participants were asked to judge whether the object in the picture was mentioned in the preceding sentence. Literacy ability was measured by a Chinese character reading task. Model analysis revealed a significant mismatch effect: the children gave faster correct responses to the perceptually matched pictures than to the mismatched ones. Furthermore, the children with lower literacy abilities displayed larger mismatch effects than those with higher literacy abilities. The results suggest that children with lower literacy abilities may rely more heavily on reactivated perceptual representations in language comprehension.

Effect of perceptual simulation on sentence production using word sets

Many recent studies, primarily about comprehension, have shown that perceptual simulation occurs in language processing. We conducted two studies to investigate how such perceptual simulation influences sentence production. In both studies, the participants were first repeatedly presented with pictures of an object with a specific shape (e.g., an unbroken egg or a broken egg). In Study 1, they produced a sentence using two words (e.g., egg, refrigerator) and the onset times for their speech were recorded. In Study 2, they were presented with three words and asked to produce a sentence from the target word (e.g., egg) and a word they freely chose from the others (e.g., refrigerator or skillet). Results indicated that visual information matching the sentences did not influence speech onset times (Study 1). However, more sentences implied that the shape matched the presented picture compared to the other picture (Study 2). This suggests perceptual simulation plays an important role in sentence production when spontaneously formulating the message to be conveyed.

Energeia as Defamiliarization: Reading Aristotle with Shklovsky’s Eyes

ABSTRACT In their work on Aristotle’s Rhetoric, Monica Westin and George Kennedy resort to defamiliarization, a device coined by one of the leading figures of Russian Formalism, Viktor Shklovsky, to comment on rhetorical energeia. This connection is examined taking into account the recent trends in the scholarship on energeia to determine whether it is methodologically and metatheoretically productive. Drawing on the insights afforded by cognitive approaches to literature and embodiment, the gap between defamiliarization and energeia is bridged by how both devices tap into perception by triggering perceptual simulations.

Evidence for visual simulation during sign language processing.

What are the mental processes that allow us to understand the meaning of words? A large body of evidence suggests that when we process speech, we engage a process of perceptual simulation whereby sensorimotor states are activated as a source of semantic information. But does the same process take place when words are expressed with the hands and perceived through the eyes? To date, it is not known whether perceptual simulation is also observed in sign languages, the manual-visual languages of deaf communities. Continuous flash suppression is a method that addresses this question by measuring the effect of language on detection sensitivity to images that are suppressed from awareness. In spoken languages, it has been reported that listening to a word (e.g., "bottle") activates visual features of an object (e.g., the shape of a bottle), and this in turn facilitates image detection. An interesting but untested question is whether the same process takes place when deaf signers see signs. We found that processing signs boosted the detection of congruent images, making otherwise invisible pictures visible. A boost of visual processing was observed only for signers but not for hearing nonsigners, suggesting that the penetration of the visual system through signs requires a fully fledged manual language. Iconicity did not modulate the effect of signs on detection, neither in signers nor in hearing nonsigners. This suggests that visual simulation during language processing occurs regardless of language modality (sign vs. speech) or iconicity, pointing to a foundational role of simulation for language comprehension. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Supplemental Material for The Concreteness Advantage in Lexical Decision Does Not Depend on Perceptual Simulations

Supplemental Material for The Concreteness Advantage in Lexical Decision Does Not Depend on Perceptual Simulations

Energeia as Defamiliarization: Reading Aristotle with Shklovsky’s Eyes

ABSTRACT In their work on Aristotle’s Rhetoric, Monica Westin and George Kennedy resort to defamiliarization, a device coined by one of the leading figures of Russian Formalism, Viktor Shklovsky, to comment on rhetorical energeia. This connection is examined taking into account the recent trends in the scholarship on energeia to determine whether it is methodologically and metatheoretically productive. Drawing on the insights afforded by cognitive approaches to literature and embodiment, the gap between defamiliarization and energeia is bridged by how both devices tap into perception by triggering perceptual simulations.

Visual perspective-taking and image-like representations: We don't see it

The ability to represent another agent's visual perspective has recently been attributed to a process called “perceptual simulation”, whereby we generate an image-like or “quasi-perceptual” representation of another agent's vision. In an extensive series of experiments we tested this notion. Adult observers were presented with pictures of an agent looking at two horizontal lines, one of which was closer to the agent and hence appeared longer from his/her visual perspective. In each case approximately as many participants judged the closer line to appear shorter as longer (to the agent), i.e., failures to take the agent's perspective. This occurred when clear depth cues were added to emphasise the agent's location relative to the stimuli, when the agent was moved closer to the lines, when the lines were oriented vertically, when judgments could be made while viewing the image, and when participants imagined themselves in the agent's place. It also persisted when we asked participants to imagine what a photo taken from the same location as the agent would show, ruling out a misinterpretation of the instructions. Overall, our data suggest that adults attempt to solve visual perspective-taking problems by drawing upon naïve and often erroneous ideas about how vision works.

Vertical position can affect categorization of the rich and the poor in Chinese culture

This study explored whether vertical position affects social categorization of the rich and the poor. Experiment 1 used high- and low-income occupations as stimuli, and found participants categorized high-income occupations faster when they were presented in the top vertical position compared to the bottom vertical position. In Experiment 2, participants responded using either the “up” or “down” key to categorize high- and low-income occupations, and responded faster to high-income occupations with the “up” key and low-income occupations with the “down” key. In Experiment 3, names identified as belonging to either rich or poor individuals were presented at the top or bottom of a screen, and the results were the same as in Experiments 1 and 2. These findings suggest that social categorization based on wealth involved perceptual simulations of vertical position, and that vertical position affects the social categorization of the rich and the poor.

Data-driven computational models reveal perceptual simulation in word processing

In their strongest formulation, theories of grounded cognition claim that concepts are made up of sensorimotor information. Following such equivalence, perceptual properties of objects should consistently influence processing, even in purely linguistic tasks, where perceptual information is neither solicited nor required. Previous studies have tested this prediction in semantic priming tasks, but they have not observed perceptual influences on participants’ performances. However, those findings suffer from critical shortcomings, which may have prevented potential visually grounded/perceptual effects from being detected. Here, we investigate this topic by applying an innovative method expected to increase the sensitivity in detecting such perceptual effects. Specifically, we adopt an objective, data-driven, computational approach to independently quantify vision-based and language-based similarities for prime-target pairs on a continuous scale. We test whether these measures predict behavioural performance in a semantic priming mega-study with various experimental settings. Vision-based similarity is found to facilitate performance, but a dissociation between vision-based and language-based effects was also observed. Thus, in line with theories of grounded cognition, perceptual properties can facilitate word processing even in purely linguistic tasks, but the behavioural dissociation at the same time challenges strong claims of sensorimotor and conceptual equivalence.

Spatial Size Can Affect Social Categorization of the Rich and the Poor

Previous research has shown that representation of certain social-category knowledge, such as that regarding gender, involves the process of perceptual simulation. The present research extended these findings and explored whether social categorization based on wealth, which is an important dimension of social categorization, involved perceptual simulation of spatial size. In Experiment 1, we used high- and low-income occupations as stimuli; categorization of high-income occupations presented in larger font was faster relative to that of those presented in small font, and vice versa for low-income occupations. In Experiments 2, 3, and 4, we used high-income occupations without social power and low-income occupations, names designated as those of rich and poor people, and idioms describing wealth and poverty as stimuli, respectively. All three experiments showed that responses to wealth-related stimuli in larger font were faster relative to those to the same stimuli in small font, and vice versa for poverty-related stimuli. These results suggest that social categorization based on wealth is grounded in perceptual simulation of spatial size.

The Role of Perceptual Simulation in L2 Vocabulary Acquisition

Research in embodied cognition suggests that perceptual simulation may play a role in language comprehension. In this study we use a combination of experimental and qualitative research to explore the potential of simulation exercises to improve acquisition of esoteric literary English vocabulary by English majors at a Chinese University. Through quantitative analysis of the data from one pre-test and two post-tests of the experimental and control groups, and qualitative examination of the student feedback collected from an open-ended survey and face-to-face interviews, we find that most of the students in both conditions have simulations of the learned words whether based on self-determined Chinese trigger words or given English language prompts to them, and skilled language learners intuitively adopt perceptual simulations. The research findings give teachers some enlightenment on the pedagogical strategies that might encourage less proficient learners to incorporate perceptual simulations in their study habits.

Perceptual Simulation of Vertical Object Movement during Comprehension of Auditory and Audiovisual Text in Children and Adults

ABSTRACT It has been argued that people construct situation models during text reception and that these are analogous, multimodal representations of text grounded in perception and action. On the one hand, abundant evidence has been generated that recipients perceptually simulate features of the situation described in the text. On the other hand, findings indicating that pictures facilitate situation model construction have been explained by assuming a causal link between picture processing, perceptual simulation, and situation model construction. Using a picture verification task, we tested whether 5- to 11 year-old children and adults perceptually simulate vertical object movements during reception of narrative text and whether the presence or absence of pictures during text presentation makes a difference. Our results suggest that both children and adults perceptually simulate vertical object movements. In our work, perceptual simulation was not influenced by pictures, so there is no evidence that it mediates the facilitating effect of pictures on situation model construction.