Action Semantics Research Articles

Neural bases of imitation and pantomime in acute stroke patients: distinct streams for praxis.

Apraxia is a cognitive disorder of skilled movements that characteristically affects the ability to imitate meaningless gestures, or to pantomime the use of tools. Despite substantial research, the neural underpinnings of imitation and pantomime have remained debated. An influential model states that higher motor functions are supported by different processing streams. A dorso-dorsal stream may mediate movements based on physical object properties, like reaching or grasping, whereas skilled tool use or pantomime rely on action representations stored within a ventro-dorsal stream. However, given variable results of past studies, the role of the two streams for imitation of meaningless gestures has remained uncertain, and the importance of the ventro-dorsal stream for pantomime of tool use has been questioned. To clarify the involvement of ventral and dorsal streams in imitation and pantomime, we performed voxel-based lesion-symptom mapping in a sample of 96 consecutive left-hemisphere stroke patients (mean age ± SD, 63.4 ± 14.8 years, 56 male). Patients were examined in the acute phase after ischaemic stroke (after a mean of 5.3, maximum 10 days) to avoid interference of brain reorganization with a reliable lesion-symptom mapping as best as possible. Patients were asked to imitate 20 meaningless hand and finger postures, and to pantomime the use of 14 common tools depicted as line drawings. Following the distinction between movement engrams and action semantics, pantomime errors were characterized as either movement or content errors, respectively. Whereas movement errors referred to incorrect spatio-temporal features of overall recognizable movements, content errors reflected an inability to associate tools with their prototypical actions. Both imitation and pantomime deficits were associated with lesions within the lateral occipitotemporal cortex, posterior inferior parietal lobule, posterior intraparietal sulcus and superior parietal lobule. However, the areas specifically related to the dorso-dorsal stream, i.e. posterior intraparietal sulcus and superior parietal lobule, were more strongly associated with imitation. Conversely, in contrast to imitation, pantomime deficits were associated with ventro-dorsal regions such as the supramarginal gyrus, as well as brain structures counted to the ventral stream, such as the extreme capsule. Ventral stream involvement was especially clear for content errors which were related to anterior temporal damage. However, movement errors were not consistently associated with a specific lesion location. In summary, our results indicate that imitation mainly relies on the dorso-dorsal stream for visuo-motor conversion and on-line movement control. Conversely, pantomime additionally requires ventro-dorsal and ventral streams for access to stored action engrams and retrieval of tool-action relationships.

The left inferior parietal lobe represents stored hand-postures for object use and action prediction.

Action semantics enables us to plan actions with objects and to predict others' object-directed actions as well. Previous studies have suggested that action semantics are represented in a fronto-parietal action network that has also been implicated to play a role in action observation. In the present fMRI study it was investigated how activity within this network changes as a function of the predictability of an action involving multiple objects and requiring the use of action semantics. Participants performed an action prediction task in which they were required to anticipate the use of a centrally presented object that could be moved to an associated target object (e.g., hammer—nail). The availability of actor information (i.e., presenting a hand grasping the central object) and the number of possible target objects (i.e., 0, 1, or 2 target objects) were independently manipulated, resulting in different levels of predictability. It was found that making an action prediction based on actor information resulted in an increased activation in the extrastriate body area (EBA) and the fronto-parietal action observation network (AON). Predicting actions involving a target object resulted in increased activation in the bilateral IPL and frontal motor areas. Within the AON, activity in the left inferior parietal lobe (IPL) and the left premotor cortex (PMC) increased as a function of the level of action predictability. Together these findings suggest that the left IPL represents stored hand-postures that can be used for planning object-directed actions and for predicting other's actions as well.

The scope and limits of action semantics: Reply to comments on ‘Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge’

In our target paper [1], we have provided an integrative review of the current state of affairs regarding research on the use of conceptual knowledge for action. In short, we argue that humans have developed declarative and procedural knowledge about objects, i.e. action semantics that enable them to use objects in a purposeful and effective manner. We were impressed by both the quantity and the quality of the commentaries on our position paper. We would like to thank the commentators for their valuable contributions, pointing out supportive evidence for our framework, indicating additional and complementary strands of research in this field and raising important conceptual and theoretical challenges for our proposed model. Here we take the opportunity to briefly indicate how additional lines of research could be integrated in our framework and how potentially conflicting findings could be reconciled. Joachim Hermsdorfer suggests that the overlap between the kinematics of pantomimed and object-directed actions points towards the existence of a common motor program, while at the same time task context also modulates the low-level features of action execution [2]. The suggestion that the context may exert both an indirect effect (i.e. through a top-down selection of motor programs) and a direct effect on movement execution (e.g. by providing participants with stronger task constraints and sensory feedback through a direct interaction with an object) provides an excellent illustration of the role of context, as proposed in our model of action semantics. A similar role of context on the activation of motor programs is proposed by Anna Borghi [3], who showed that the context (i.e. provided by the task, the inclusion of other objects, semantic information or the presence of other persons) can have a strong influence on the activation of object affordances.

The costs of action: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by Michiel van Elk, Hein van Schie and Harold Bekkering

The costs of action: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by Michiel van Elk, Hein van Schie and Harold Bekkering

Laterality, evolution and the IPL in the semantic–action system: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by Michiel van Elk, Hein van Schie, Harold Bekkering

Laterality, evolution and the IPL in the semantic–action system: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by Michiel van Elk, Hein van Schie, Harold Bekkering

What about mechanical knowledge?: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie, and Bekkering

What about mechanical knowledge?: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie, and Bekkering

SEMANTIC CHARACTERISTICS OF DESTRUCTIVE ACTION VERBS IN THE KHANTY LANGUAGE (BASED ON THE TEXTS IN SHURYSHKARSKY DIALECT)

The article is devoted to the semantic and grammatical features of Khanty verbs with the meaning of destruction. These verbs reflect the semantics of destructive action and its following characteristics also: 1) nature of the subject of the action; 2) nature of the object of the action (quality and quantity features); 3) various aspects of instruments: a) mode of action, b) instrument, c) method; 4) purposeful nature of action, 5) result of action. The aim of our work is the study and description of the lexical-semantic and semantic-syntactic features of the verbs of destructive action and their classification. In accordance with the intended purpose there are the following objectives: 1. Revealing of the verbs of destructive action. 2. Determination of matter and scope of the concept of destruction and specification of semantic classification of verbs with the meaning of destruction. 3. Determination of component structure of meaning of LSV (Lexical-semantic variants) included in each subgroup. Scientific novelty and timeliness of the attempted research are determined by the fact that a comprehensive study of the destructive action verbs in the Khanty language in lexical-semantic aspect has not yet been the subject of a special research, the lexical-semantic analysis of the verbs of the lexical-semantic group is performed for the first time. The object of the research is based on news stories and texts documented by us in Shuryshkarsky dialect of the Khanty language. Language material is presented in Shuryshkarsky dialect of the Khanty language, widely spoken on the territory of Shuryshkarsky district of Yamalo-Nenets Autonomous Okrug.

Insights from kinematic analysis of pantomime and actual use for the hierarchical organization of tool use actions: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie & Bekkering

Insights from kinematic analysis of pantomime and actual use for the hierarchical organization of tool use actions: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie & Bekkering

Hemispheric (a)symmetries in action semantics: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie, and Bekkering

Hemispheric (a)symmetries in action semantics: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie, and Bekkering

The role of action semantics in action control: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by Michiel van Elk, Hein van Schie and Harold Bekkering

In this issue, van Elk, van Schie and Bekkering [4] succeed in presenting a well-organized, highly structured, and impressively complete summary of the available research on how knowledge may affect action control. However, their theoretical efforts remain unconvincing for three reasons. First, even though the suggested framework is presented as a general “model” of how people acquire, store, and make use of action knowledge, the authors restrict their theoretical reasoning to unimanual reaching for/grasping of objects—a surprisingly small subset of possible actions. Other actions often do not fit into the authors’ schema: walking, talking, singing, gesturing, typing, looking, driving, dancing, playing music, doing sports, and reading do not refer to particular objects and/or have goals other than being directed towards particular spatial locations—and yet they all rely on “what” and “how” knowledge. This suggests that the “multimodal object representation” box of the model is too specific to apply to action in a broader sense; it should rather refer to all sorts of events that people can aim at, generate, and imply by means of voluntary actions—as suggested by the Theory of Event Codes [2]. Event representations may also refer to the action itself, such as with a ballet figure, suggesting that the “object representation” box might sometimes merely consist of representations of the sensory consequences the action is intended to produce. It thus makes little sense to distinguish between “multimodal object representations”, “proprioceptive consequences” (which, strangely enough, are distinguished from other sensory consequences), and “sensory consequences”—this may just be different examples of “event representations” [2]. Second, the remaining concepts of the framework are ill-defined: Distinguishing between functional and manipulation knowledge makes sense for object grasping but not for many other actions, including gesturing, dancing, and singing. Worse, the authors’ definition that the action-related knowledge comprises of all “the procedural or manipulation knowledge” that enables us to carry out meaningful actions renders the model tautological: if humans can carry

Affordances and contextual flexibility: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by Michiel van Elk, Hein van Schie and Harold Bekkering

Affordances and contextual flexibility: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by Michiel van Elk, Hein van Schie and Harold Bekkering

Is a single ‘hub’, with lots of spokes, an accurate description of the neural architecture of action semantics?: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie and Bekkering

How is knowledge about tools and their associated actions represented in the brain? In their otherwise excellentreview of the behavioral, developmental, and neuropsychological literatures, Van Elk, van Schie, and Bekkering [16]argue that the so-called ‘hub and spoke’ architecture [12] provides a good answer. Semantics, in this model, is “rep-resented across modality-specific ‘spokes’ that converge in a multimodal association area (i.e. the ‘semantic hub’)that provides an amodal representation of semantic information”. Exactly what type of information is contained inan ‘amodal’ object representation, and how this representation is created from the convergence of information frommultiple modalities, is left unsaid.Leaving aside these thorny theory-laden concerns, in this comment we highlight several problems with the ‘huband spoke’ architecture. The semantic ‘hub’ is claimed to reside in a specific region of the brain, the anterior temporallobes, bilaterally (ATL). This claim is largely based on the notion that the clinical disorder of semantic dementia iscaused solely by damage limited to the ATL. This is not the case. Semantic dementia is a progressive disorder thattypically involves a broad expanse of temporal, and often frontal, cortex, as well as subcortical regions – especiallyduring the later stages when conceptual knowledge is most degraded (e.g., [5]). More to the point, even if the dam-age associated with semantic dementia was limited to the ATL, the fact that this damage produces semantic deficitsinvolving multiple object categories and concepts does not constitute evidence for a semantic ‘hub’ any more so thanwould finding that damage to posterior temporal cortex – a brain region known to represent multiple object categoriesin spatially distinct regions – constitute evidence for an object perception ‘hub’.

Foundational roles for action representations: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie, and Bekkering

Embodiment of conceptual knowledge is one of the most influential ideas to sweep the landscape of cognitive psychology in recent decades [1,2] and as the review by van Elk, van Schie, and Bekkering [6] makes clear, it has deeply affected related subfields as well. We have substantial evidence to support the proposal that sensory and motor representations can affect perceptual and cognitive experiences and that these representations are likely to be integrated in some way with conceptual knowledge. The hard work of distinguishing the functional and epiphenomenal contributions that sensorimotor representations make to processes such as object identification and language comprehension must now begin in earnest [3]. Toward that end, van Elk et al. have provided a framework for understanding some of the contextual influences that govern the involvement of action representations in conceptual processing. They suggested that this framework generates testable predictions regarding the distinction between multimodal and modality-specific representations and the hierarchical control of action. Genuine predictions are very hard to come by when working with frameworks rather than computational models, and what van Elk et al. deliver actually turns out to be suggestions for research questions that might be addressed rather than clear predictions about the results that experiments should produce. Such caution is wise because it is becoming clear that what might be seen as clear and straightforward predictions about the nature and function of action representations often fail when put to empirical test (see [4] for examples involving mental simulation and language comprehension). A striking case of failed expectations regarding action representations comes from a single-cell recording study of hand movements in monkeys [5]. Animals were trained to move their hand forward and to the right or left to contact an illuminated target. They did so while viewing a video image of their hand and the indicated target, rather than viewing the hand and target directly. Distinct populations of neurons in premotor cortex fired when the hand moved to the left versus the right. Rather than coding hand movements directly, however, these populations were actually coding a higher level action goal. This fact was cleverly revealed when the experimenters displayed on the monkey’s monitor not a literal image of their moving hand, but rather a mirror image. Thus, a rightward movement of the hand effected a leftward movement of the image of the hand on the monitor. To capture a target appearing on the left side of the monitor, therefore the monkey was required to make a physical movement of the hand to the right. Under

Interpersonal action semantics: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie, & Bekkering

Interpersonal action semantics: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie, & Bekkering

What does the forward model of an expert hand-tool motor program code?: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie, and Bekkering

What does the forward model of an expert hand-tool motor program code?: Comment on “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge” by van Elk, van Schie, and Bekkering

A STUDY OF ACTOR AND ACTION SEMANTIC RETENTION IN VIDEO SUPERVOXEL SEGMENTATION

Existing methods in the semantic computer vision community seem unable to deal with the explosion and richness of modern, open-source and social video content. Although sophisticated methods such as object detection or bag-of-words models have been well studied, they typically operate on low level features and ultimately suffer from either scalability issues or a lack of semantic meaning. On the other hand, video supervoxel segmentation has recently been established and applied to large scale data processing, which potentially serves as an intermediate representation to high level video semantic extraction. The supervoxels are rich decompositions of the video content: they capture object shape and motion well. However, it is not yet known if the supervoxel segmentation retains the semantics of the underlying video content. In this paper, we conduct a systematic study of how well the actor and action semantics are retained in video supervoxel segmentation. Our study has human observers watching supervoxel segmentation videos and trying to discriminate both actor (human or animal) and action (one of eight everyday actions). We gather and analyze a large set of 640 human perceptions over 96 videos in 3 different supervoxel scales. Furthermore, we design a feature defined on supervoxel segmentation, called supervoxel shape context, which is inspired by the higher order processes in human perception. We conduct actor and action classification experiments with this new feature and compare to various traditional video features. Our ultimate findings suggest that a significant amount of semantics have been well retained in the video supervoxel segmentation and can be used for further video analysis.

The neuroscience of action semantics in neurodegenerative brain diseases

Within the last decade, the neurobiology of action processing has moved from an obscure topic of specialist interest to one of the most popular themes in modern neuroscience. However, the wealth of literature and the diversity of approaches and theoretical models can make the field complex and, at times, bewildering. This review presents the main currents of research, examining their theoretical underpinnings in an interdisciplinary context. The presence of specific deficits in verb and action processing has been documented in a wide range of neurodegenerative diseases, including parkinsonian syndromes and motor neuron disease. Interestingly, most of these disorders affect the motor system, suggesting a systematic relationship between motor functions and their cognitive and linguistic representations. Action processing has been explored with a whole spectrum of methodologies, from neuroimaging to transcranial and intracranial stimulation. The findings have been integrated with other influential concepts and theories, including mirror neurons and embodied cognition. Converging evidence from patient and imaging studies links the concepts of actions and their processing with the execution of actions through the motor system. The theory of embodied cognition remains influential as well as controversial. However, the points of criticism have changed, reflecting recent paradigm shifts.

Action representation: Crosstalk between semantics and pragmatics

Marc Jeannerod pioneered a representational approach to movement and action. In his approach, motor representations provide both, declarative knowledge about action and procedural knowledge for action (action semantics and action pragmatics, respectively). Recent evidence from language comprehension and action simulation supports the claim that action pragmatics and action semantics draw on common representational resources, thus challenging the traditional divide between declarative and procedural action knowledge. To account for these observations, three kinds of theoretical frameworks are discussed: (i) semantics is grounded in pragmatics, (ii) pragmatics is anchored in semantics, and (iii) pragmatics is part and parcel of semantics.

Semantic embodiment, disembodiment or misembodiment? In search of meaning in modules and neuron circuits

“Embodied” proposals claim that the meaning of at least some words, concepts and constructions is grounded in knowledge about actions and objects. An alternative “disembodied” position locates semantics in a symbolic system functionally detached from sensorimotor modules. This latter view is not tenable theoretically and has been empirically falsified by neuroscience research. A minimally-embodied approach now claims that action–perception systems may “color”, but not represent, meaning; however, such minimal embodiment (misembodiment?) still fails to explain why action and perception systems exert causal effects on the processing of symbols from specific semantic classes. Action perception theory (APT) offers neurobiological mechanisms for “embodied” referential, affective and action semantics along with “disembodied” mechanisms of semantic abstraction, generalization and symbol combination, which draw upon multimodal brain systems. In this sense, APT suggests integrative-neuromechanistic explanations of why both sensorimotor and multimodal areas of the human brain differentially contribute to specific facets of meaning and concepts.

Action semantics and movement characteristics engage distinct processing streams during the observation of tool use

The cortical motor system follows a modular organization in which different features of executed movements are supported by distinct streams. Accordingly, different levels of action recognition, such as movement characteristics or action semantics may be processed within distinct networks. The present study aimed to differentiate areas related to the analysis of action features involving semantic knowledge from regions concerned with the evaluation of movement characteristics determined by structural object properties. To this end, the assessment of (i) tool-associated actions in relation to semantically, but not functionally inappropriate recipients (factor "Semantics"), and the evaluation of (ii) tool-associated movements performed with awkward versus correct hand postures (factor "Hand") were experimentally manipulated in an fMRI study with an event-related 2 × 2 factorial design. The videos used as stimuli displayed actions performed with the right hand in third-person perspective. Conjunction analysis of all four experimental conditions showed that observing videos depicting tool-related actions compared to rest was associated with widespread bilateral activity within the frontal lobes, inferior and superior parietal lobules, parts of the temporal lobes, as well as the occipital lobes. Viewing actions executed with incorrect compared to correct hand postures (factor "Hand") elicited significantly more activity within right primary sensory cortex (Brodmann area 2) and superior parietal lobule. Conversely, tool-associated actions displayed after semantically incorrect compared to correct recipients elicited higher activation within a left-lateralized network comprising the ventro-lateral prefrontal cortex (VLPFC), parts of the intraparietal sulcus and the angular gyrus (AG), as well as the supplementary motor area (SMA) and pre-SMA. Probabilistic diffusion tensor imaging-based tractography revealed two distinct fiber connections between AG and the frontal activation: A dorsal pathway via the superior longitudinal fascicle to the caudal part of VLPFC and a ventral pathway reaching the more rostral parts of VLPFC via the extreme capsule. The task-dependent relative modulation of activity within these brain networks composed of activated cortical areas connected by specific white matter tracts may indicate that the assessment of semantic action features relies on both dorso-ventral and ventral processing streams, whereas the analysis of hand postures with respect to objects depends on areas within the dorso-dorsal stream.