Action Sounds Research Articles

Brain activation during processing of mouth actions in patients with disorders of consciousness.

In the past 2 decades, several attempts have been made to promote a correct diagnosis and possible restorative interventions in patients suffering from disorders of consciousness. Sensory stimulation has been proved to be useful in sustaining the level of arousal/awareness and to improve behavioural responsiveness with a significant effect on oro-motor functions. Recently, action observation has been proposed as a stimulation strategy in patients with disorders of consciousness, based on neurophysiological evidence that the motor cortex can be activated not only during action execution but also when actions are merely observed in the absence of motor output, or during listening to action sounds and speech. This mechanism is provided by the activity of mirror neurons. In the present study, a group of patients with disorders of consciousness (11 males, 4 females; median age: 55 years; age range: 19-74 years) underwent task-based functional MRI in which they had, in one condition, to observe and listen to the sound of mouth actions, and in another condition, to listen to verbs with motor or abstract content. In order to verify the presence of residual activation of the mirror neuron system, the brain activations of patients were compared with that of a group of healthy individuals (seven males, eight females; median age: 33.4 years; age range: 24-65 years) performing the same tasks. The results show that brain activations were lower in patients with disorders of consciousness compared with controls, except for primary auditory areas. During the audiovisual task, 5 out of 15 patients with disorders of consciousness showed only residual activation of low-level visual and auditory areas. Activation of high-level parieto-premotor areas was present in six patients. During the listening task, three patients showed only low-level activations, and six patients activated also high-level areas. Interestingly, in both tasks, one patient with a clinical diagnosis of vegetative state showed activations of high-level areas. Region of interest analysis on blood oxygen level dependent signal change in temporal, parietal and premotor cortex revealed a significant linear relation with the level of clinical functioning, assessed with coma recovery scale-revised. We propose a classification of the patient's response based on the presence of low-level and high-level activations, combined with the patient's functional level. These findings support the use of action observation and listening as possible stimulation strategies in patients with disorders of consciousness and highlight the relevance of combined methods based on functional assessment and brain imaging to provide more detailed neuroanatomical specificity about residual activated areas at both cortical and subcortical levels.

Predicting and coding sound into action translation in spinal cord injured people.

Motor activation in response to perception of action-related stimuli may depend on a resonance mechanism subserving action understanding. The extent to which this mechanism is innate or learned from sensorimotor experience is still unclear. Here, we recorded EEG while people with paraplegia or tetraplegia consequent to spinal cord injury (SCI) and healthy control participants were presented with action sounds produced by body parts (mouth, hands or feet) that were or were not affected by SCI. Non-action sounds were used as further control. We observed reduced brain activation in subjects affected by SCI at both pre- and post-stimulus latencies specifically for those actions whose effector was disconnected by the spinal lesion (i.e., hand sound for tetraplegia and leg sound for both paraplegia and tetraplegia). Correlation analyses showed that these modulations were functionally linked with the chronicity of the lesion, indicating that the longer the time the lesion- EEG data acquisition interval and/or the more the lesion occurred at a young age, the weaker was the cortical activity in response to these action sounds. Tellingly, source estimations confirmed that these modulations originated from a deficit in the motor resonance mechanism, by showing diminished activity in premotor (during prediction and perception) and near the primary motor (during perception) areas. Such dissociation along the cortical hierarchy is consistent with both previous reports in healthy subjects and with hierarchical predictive coding accounts. Overall, these data expand on the notion that sensorimotor experience maintains the cortical representations relevant to anticipate and perceive action-related stimuli.

Mimicry in misophonia: A large-scale survey of prevalence and relationship with trigger sounds.

Misophonia is often referred to as a disorder that is characterized by excessive negative emotional responses, including anger and anxiety, to "trigger sounds" which are typically day-to-day sounds, such as those generated from people eating, chewing, and breathing. Misophonia (literally "hatred of sounds") has commonly been understood within an auditory processing framework where sounds cause distress due to aberrant processing in the auditory and emotional systems of the brain. However, a recent proposal suggests that it is the perceived action (e.g., mouth movement in eating/chewing sounds as triggers) of the trigger person, and not the sounds per se, that drives the distress in misophonia. Since observation or listening to sounds of actions of others are known to prompt mimicry in perceivers, we hypothesized that mimicking the action of the trigger person may be prevalent in misophonia. Apart from a few case studies and anecdotal information, a relation between mimicking and misophonia has not been systematically evaluated. In this work, we addressed this limitation by collecting data on misophonia symptoms and mimicry behavior using online questionnaires from 676 participants. Analysis of these data shows that (i) more than 45% of individuals with misophonia reported mimicry, indicating its wide prevalence, (ii) the tendency to mimic varies in direct proportion to misophonia severity, (iii) compared to other human and environmental sounds, trigger sounds of eating and chewing are more likely to trigger mimicking, and (iv) the act of mimicking provides some degree of relief from distress to people with misophonia. This study shows prevalence of mimicry and its relation to misophonia severity and trigger types. The theoretical framework of misophonia needs to incorporate the phenomenon of mimicry and its effect on management of misophonia distress.

Flee: Sounds of Fright and Flight

Animation and live-action are two closely related media, which are foremost distinguished by the ideas and conventions surrounding them. The diverging discourses around animation and live action have tended to focus on animation as something constructed to represent characters and settings and on live action as something capturing actors and sets representing characters and settings. This difference between constructing and capturing, along with the perceived indexicality of the photo, is what seems to suggest live action as the preferred medium for documenting real events. Sound effects, in the form of recorded and edited sounds of objects, actions and environments, are of particular interest here, as they can be considered to balance somewhere between these poles of construction and capture, between the non-indexical and indexical, and ultimately between representation and reproduction. In this article, I will focus on aspects of ‘truth’ (understood as corresponding to some external reality) and ‘realism’ (understood as a representation of external reality) and how something comes to be perceived as truthful or realistic in animated documentaries in relation to the role played by sound effects. By discussing the Danish film Flugt [Flee], I will show how sound effects can aid in creating representations of truth.

Chinese-English bilinguals show linguistic-perceptual links in the brain associating short spoken phrases with corresponding real-world natural action sounds by semantic category

ABSTRACT Higher cognitive functions such as linguistic comprehension must ultimately relate to perceptual systems in the brain, though how and why this forms remains unclear. Different brain networks that mediate perception when hearing have recently been proposed to respect a taxonomic neurobiological model for the processing of different acoustic-semantic categories of real-world natural sounds. Using functional magnetic resonance imaging (fMRI) with Chinese/English bilingual listeners, the present study explored whether reception of short spoken phrases that described corresponding natural sounds, in both Chinese (Mandarin) and English, would engage overlapping brain regions at a semantic category level. The results revealed a double-dissociation of cortical regions that were preferential for representing knowledge of human versus environmental action events, whether conveyed through natural sounds or the corresponding spoken phrases depicted by either comprehended language. These findings of cortical hubs exhibiting linguistic-perceptual knowledge links at a semantic category level should help to advance neurocomputational models of the neurodevelopment of language systems.

Do infants represent human actions cross-modally? An ERP visual-auditory priming study

Recent findings indicate that 7-months-old infants perceive and represent the sounds inherent to moving human bodies. However, it is not known whether infants integrate auditory and visual information in representations of specific human actions. To address this issue, we used ERPs to investigate infants’ neural sensitivity to the correspondence between sounds and images of human actions. In a cross-modal priming paradigm, 7-months-olds were presented with the sounds generated by two types of human body movement, walking and handclapping, after watching the kinematics of those actions in either a congruent or incongruent manner. ERPs recorded from frontal, central and parietal electrodes in response to action sounds indicate that 7-months-old infants perceptually link the visual and auditory cues of human actions. However, at this age these percepts do not seem to be integrated in cognitive multimodal representations of human actions.

Action sound\u2013shape congruencies explain sound symbolism

Sound symbolism, the surprising semantic relationship between meaningless pseudowords (e.g., ‘maluma’, ‘takete’) and abstract (round vs. sharp) shapes, is a hitherto unexplained human-specific knowledge domain. Here we explore whether abstract sound symbolic links can be explained by those between the sounds and shapes of bodily actions. To this end, we asked human subjects to match pseudowords with abstract shapes and, in a different experimental block, the sounds of actions with the shapes of the trajectories of the actions causing these same sounds. Crucially, both conditions were also crossed. Our findings reveal concordant matching in the sound symbolic and action domains, and, importantly, significant correlations between them. We conclude that the sound symbolic knowledge interlinking speech sounds and abstract shapes is explained by audiovisual information immanent to action experience along with acoustic similarities between speech and action sounds. These results demonstrate a fundamental role of action knowledge for abstract sound symbolism, which may have been key to human symbol-manipulation ability.

Incidental or Intentional? Different Brain Responses to One's Own Action Sounds in Hurdling vs. Tap Dancing.

Most human actions produce concomitant sounds. Action sounds can be either part of the action goal (GAS, goal-related action sounds), as for instance in tap dancing, or a mere by-product of the action (BAS, by-product action sounds), as for instance in hurdling. It is currently unclear whether these two types of action sounds—incidental or intentional—differ in their neural representation and whether the impact on the performance evaluation of an action diverges between the two. We here examined whether during the observation of tap dancing compared to hurdling, auditory information is a more important factor for positive action quality ratings. Moreover, we tested whether observation of tap dancing vs. hurdling led to stronger attenuation in primary auditory cortex, and a stronger mismatch signal when sounds do not match our expectations. We recorded individual point-light videos of newly trained participants performing tap dancing and hurdling. In the subsequent functional magnetic resonance imaging (fMRI) session, participants were presented with the videos that displayed their own actions, including corresponding action sounds, and were asked to rate the quality of their performance. Videos were either in their original form or scrambled regarding the visual modality, the auditory modality, or both. As hypothesized, behavioral results showed significantly lower rating scores in the GAS condition compared to the BAS condition when the auditory modality was scrambled. Functional MRI contrasts between BAS and GAS actions revealed higher activation of primary auditory cortex in the BAS condition, speaking in favor of stronger attenuation in GAS, as well as stronger activation of posterior superior temporal gyri and the supplementary motor area in GAS. Results suggest that the processing of self-generated action sounds depends on whether we have the intention to produce a sound with our action or not, and action sounds may be more prone to be used as sensory feedback when they are part of the explicit action goal. Our findings contribute to a better understanding of the function of action sounds for learning and controlling sound-producing actions.

SOUND SYMBOLISM OF FRONT VOWELS IN ENGLISH ONOMATOPOEIC WORDS

This is a qualitative research that focuses on the meaning represented in the phoneme contained in English onomatopoeic words. Onomatopoeia is word that imitates the sounds of human, animal, things, actions, and nature in the world. Onomatopoeia exists in many reading materials such as comics, fables, tales and poetry. This research focuses on the onomatopoeic words which are contained in Oxford English Dictionary for the dictionary is regularly updated. Out of two kinds of phoneme, which are consonants and vowels, this research limits the investigation for only English front vowels. Based on the manner of articulation, English front vowels are divided into front high tense unrounded vowel /i/, front high lax unrounded vowel /ɪ/, front mid tense unrounded vowel /e/, front mid lax unrounded vowel /ɛ/, and front low lax unrounded vowel /ӕ/. This approach used in this research is called sound symbolism which is a study of relation between sound and meaning. This research applies low-level properties, a mechanism in sound symbolism that is associating the sound to the meaning based on the shared perceptual feature in both phoneme and associated stimuli. The mechanism is used as the method of the research for the researcher explores the characteristics of front vowels contained in English onomatopoeic words that are used to represent the sounds produced by human, animals, natures, machines, and other things. Based on the investigation, the result indicates that the higher vowels the more diminutive meaning it indicates, while the lower vowels the more augmentative meaning it indicates.

Understanding the attitude of others by hearing action sounds: the role of the insula

During social interactions, actions and words can be expressed in different ways, for example gently, vigorously or rudely communicating the positive or negative attitude of the agent. These forms of communication are called vitality forms and play a crucial role in social relations. While the neural bases of speech and actions vitality forms have been investigated, there is no information on how we recognize others’ mood/attitude by hearing the sound of their actions. In the present fMRI study we investigated the neural basis of vitality forms while participants heard action sounds in two different conditions: sounds resulting from gentle and rude actions, sounds communicating the same actions without vitality forms (control stimuli). Results showed that hearing action sounds conveying rude and gentle vitality forms respect to the control stimuli produced a specific activation of the dorso-central insula. In addition, hearing both vitality forms action sounds and control stimuli produced the activation of the parieto-frontal circuit typically involved in the observation and the execution of arm actions. In conclusion, our data indicate that, the dorso-central insula is a key region involved in the processing of vitality forms regardless of the modality by which they are conveyed.

Object-Audio Capture System for Sports Broadcasting

Object-audio workflows for traditional flat broadcasts have recently appeared after the introduction of new audio formats, such as MPEG-H and Atmos. These formats enable the creation of object-based mixes that can be dynamically rendered by the end user depending on their reproduction hardware. Until very recently, only post-produced content was being created for these formats, but new broadcast standards in the U.S. and Asia, as well as new hardware encoding engines, have made live sports production in these formats more feasible. These formats allow for a fuller, more immersive sound design and allow for some possibilities of personalization. The issue then arises on how to capture live action from the field that would provide these object-audio workflows with the desired isolated sounds and accompanying metadata. Current capture systems provide a suboptimal amount of isolation from the crowd to highlight individual action sounds and dialog from the field. Also, in most cases, placing traditional microphones near the action is not possible. In this paper, we present new microphone techniques and systems enabling better performance for sound capture that fulfill the needs of future object-audio broadcast formats. This includes beamforming techniques, automatic steering, and systems management of microphone arrays.

Prediction mechanisms in motor and auditory areas and their role in sound perception and language understanding

Is the meaning of an expected stimulus manifest in brain activity even before it appears? Although theories of predictive coding see anticipatory activity as crucial for the understanding of brain function, few studies have explored neurophysiologically manifest semantic predictions. Here, we report predictive negative-going potentials before the onset of action (i.e. whistle and hand clap) and non-action (i.e. pure tone, water drop) sounds. These prediction potentials (PP) indexed the meaning of action-related sounds. Dependent on the body-part-relationship of sound stimuli, neural sources were relatively more prominent in dorsal or ventral motor areas. In contrast, meaningless sounds (pure tones) activated predictive sources in temporal areas close to the auditory cortex; complex environmental sounds induced an anticipatory positivity broadly distributed over the scalp. We also found a systematic relationship between predictive activity and a Mismatch Negativity (MMN) like response to unexpected meaningful words which were presented as rare deviant stimuli amongst frequently repeated sounds. This deviant-elicited potential indexed semantic priming between action sounds and action-related words and semantic mismatch (prediction error). These results suggest a systematic link between perceptual/semantic prediction and matching mechanisms in the processing of sounds and words.

Human action sounds elicit sensorimotor activation early in life

In human adults the auditory representation of others’ actions is capable to activate specific areas of the motor and premotor cortices. Here, we examined the early origins of the neural processing of action sounds to investigate whether and how infants rely on auditory information to understand their close social environment. Sensorimotor activity, as indexed by μ rhythm suppression, was measured using electroencephalography in 14-month-old infants who listened to hand- and foot-produced action sounds (i.e., footsteps and clapping) and to mechanical sounds (i.e., blender). Footstep sounds elicited activation at midline electrodes over the foot area (Cz), and not in correspondence of lateralized clusters over the hand areas (C3 and C4). Greater activation in response to clapping compared to blender and footstep sounds was recorded at electrodes in the left central cluster, over the hand sensorimotor cortex (i.e., C3), but extended to some extent over the midline electrode cluster. Furthermore, our results underscore the role of natural locomotor experience in shaping sensorimotor activation, since infants who gained more walking experience exhibited stronger sensorimotor activation for footstep sounds over left central electrodes. Taken together, current results provide the first evidence that action sounds produced by another person are capable to elicit sensorimotor activation during infancy.

Embodying functionally relevant action sounds in patients with spinal cord injury

Growing evidence indicates that perceptual-motor codes may be associated with and influenced by actual bodily states. Following a spinal cord injury (SCI), for example, individuals exhibit reduced visual sensitivity to biological motion. However, a dearth of direct evidence exists about whether profound alterations in sensorimotor traffic between the body and brain influence audio-motor representations. We tested 20 wheelchair-bound individuals with lower skeletal-level SCI who were unable to feel and move their lower limbs, but have retained upper limb function. In a two-choice, matching-to-sample auditory discrimination task, the participants were asked to determine which of two action sounds matched a sample action sound presented previously. We tested aural discrimination ability using sounds that arose from wheelchair, upper limb, lower limb, and animal actions. Our results indicate that an inability to move the lower limbs did not lead to impairment in the discrimination of lower limb-related action sounds in SCI patients. Importantly, patients with SCI discriminated wheelchair sounds more quickly than individuals with comparable auditory experience (i.e. physical therapists) and inexperienced, able-bodied subjects. Audio-motor associations appear to be modified and enhanced to incorporate external salient tools that now represent extensions of their body schemas.

Hearing and orally mimicking different acoustic-semantic categories of natural sound engage distinct left hemisphere cortical regions

Oral mimicry is thought to represent an essential process for the neurodevelopment of spoken language systems in infants, the evolution of language in hominins, and a process that could possibly aid recovery in stroke patients. Using functional magnetic resonance imaging (fMRI), we previously reported a divergence of auditory cortical pathways mediating perception of specific categories of natural sounds. However, it remained unclear if or how this fundamental sensory organization by the brain might relate to motor output, such as sound mimicry. Here, using fMRI, we revealed a dissociation of activated brain regions preferential for hearing with the intent to imitate and the oral mimicry of animal action sounds versus animal vocalizations as distinct acoustic-semantic categories. This functional dissociation may reflect components of a rudimentary cortical architecture that links systems for processing acoustic-semantic universals of natural sound with motor-related systems mediating oral mimicry at a category level. The observation of different brain regions involved in different aspects of oral mimicry may inform targeted therapies for rehabilitation of functional abilities after stroke.

Mirror and (absence of) counter-mirror responses to action sounds measured with TMS.

To what extent is the mirror neuron mechanism malleable to experience? The answer to this question can help characterising its ontogeny and its role in social cognition. Some suggest that it develops through sensorimotor associations congruent with our own actions. Others argue for its extreme volatility that will encode any sensorimotor association in the environment. Here, we added to this debate by exploring the effects of short goal-directed ‘mirror’ and ‘counter-mirror’ trainings (a ‘mirror’ training is defined as the first type of training encountered by the participants) on human auditory mirror motor-evoked potentials (MEPs). We recorded MEPs in response to two tones void of previous motor meaning, before and after mirror and counter-mirror trainings in which participants generated two tones of different pitch by performing free-choice button presses. The results showed that mirror MEPs, once established, were protected against an equivalent counter-mirror experience: they became manifest very rapidly and the same number of training trials that lead to the initial association did not suffice to reverse the MEP pattern. This steadiness of the association argues that, by serving direct-matching purposes, the mirror mechanism is a good solution for social cognition.

Auditory object perception: A neurobiological model and prospective review

Interaction with the world is a multisensory experience, but most of what is known about the neural correlates of perception comes from studying vision. Auditory inputs enter cortex with its own set of unique qualities, and leads to use in oral communication, speech, music, and the understanding of emotional and intentional states of others, all of which are central to the human experience. To better understand how the auditory system develops, recovers after injury, and how it may have transitioned in its functions over the course of hominin evolution, advances are needed in models of how the human brain is organized to process real-world natural sounds and “auditory objects”. This review presents a simple fundamental neurobiological model of hearing perception at a category level that incorporates principles of bottom-up signal processing together with top-down constraints of grounded cognition theories of knowledge representation. Though mostly derived from human neuroimaging literature, this theoretical framework highlights rudimentary principles of real-world sound processing that may apply to most if not all mammalian species with hearing and acoustic communication abilities. The model encompasses three basic categories of sound-source: (1) action sounds (non-vocalizations) produced by ‘living things’, with human (conspecific) and non-human animal sources representing two subcategories; (2) action sounds produced by ‘non-living things’, including environmental sources and human-made machinery; and (3) vocalizations (‘living things’), with human versus non-human animals as two subcategories therein. The model is presented in the context of cognitive architectures relating to multisensory, sensory-motor, and spoken language organizations. The models’ predictive values are further discussed in the context of anthropological theories of oral communication evolution and the neurodevelopment of spoken language proto-networks in infants/toddlers. These phylogenetic and ontogenetic frameworks both entail cortical network maturations that are proposed to at least in part be organized around a number of universal acoustic-semantic signal attributes of natural sounds, which are addressed herein.

When a hit sounds like a kiss: An electrophysiological exploration of semantic processing in visual narrative

Researchers have long questioned whether information presented through different sensory modalities involves distinct or shared semantic systems. We investigated uni-sensory cross-modal processing by recording event-related brain potentials to words replacing the climactic event in a visual narrative sequence (comics). We compared Onomatopoeic words, which phonetically imitate action sounds (Pow!), with Descriptive words, which describe an action (Punch!), that were (in)congruent within their sequence contexts. Across two experiments, larger N400s appeared to Anomalous Onomatopoeic or Descriptive critical panels than to their congruent counterparts, reflecting a difficulty in semantic access/retrieval. Also, Descriptive words evinced a greater late frontal positivity compared to Onomatopoetic words, suggesting that, though plausible, they may be less predictable/expected in visual narratives. Our results indicate that uni-sensory cross-model integration of word/letter-symbol strings within visual narratives elicit ERP patterns typically observed for written sentence processing, thereby suggesting the engagement of similar domain-independent integration/interpretation mechanisms.

Divergent Human Cortical Regions for Processing Distinct Acoustic-Semantic Categories of Natural Sounds: Animal Action Sounds vs. Vocalizations.

A major gap in our understanding of natural sound processing is knowledge of where or how in a cortical hierarchy differential processing leads to categorical perception at a semantic level. Here, using functional magnetic resonance imaging (fMRI) we sought to determine if and where cortical pathways in humans might diverge for processing action sounds vs. vocalizations as distinct acoustic-semantic categories of real-world sound when matched for duration and intensity. This was tested by using relatively less semantically complex natural sounds produced by non-conspecific animals rather than humans. Our results revealed a striking double-dissociation of activated networks bilaterally. This included a previously well described pathway preferential for processing vocalization signals directed laterally from functionally defined primary auditory cortices to the anterior superior temporal gyri, and a less well-described pathway preferential for processing animal action sounds directed medially to the posterior insulae. We additionally found that some of these regions and associated cortical networks showed parametric sensitivity to high-order quantifiable acoustic signal attributes and/or to perceptual features of the natural stimuli, such as the degree of perceived recognition or intentional understanding. Overall, these results supported a neurobiological theoretical framework for how the mammalian brain may be fundamentally organized to process acoustically and acoustic-semantically distinct categories of ethologically valid, real-world sounds.

Vocal Imitations of Non-Vocal Sounds.

Imitative behaviors are widespread in humans, in particular whenever two persons communicate and interact. Several tokens of spoken languages (onomatopoeias, ideophones, and phonesthemes) also display different degrees of iconicity between the sound of a word and what it refers to. Thus, it probably comes at no surprise that human speakers use a lot of imitative vocalizations and gestures when they communicate about sounds, as sounds are notably difficult to describe. What is more surprising is that vocal imitations of non-vocal everyday sounds (e.g. the sound of a car passing by) are in practice very effective: listeners identify sounds better with vocal imitations than with verbal descriptions, despite the fact that vocal imitations are inaccurate reproductions of a sound created by a particular mechanical system (e.g. a car driving by) through a different system (the voice apparatus). The present study investigated the semantic representations evoked by vocal imitations of sounds by experimentally quantifying how well listeners could match sounds to category labels. The experiment used three different types of sounds: recordings of easily identifiable sounds (sounds of human actions and manufactured products), human vocal imitations, and computational “auditory sketches” (created by algorithmic computations). The results show that performance with the best vocal imitations was similar to the best auditory sketches for most categories of sounds, and even to the referent sounds themselves in some cases. More detailed analyses showed that the acoustic distance between a vocal imitation and a referent sound is not sufficient to account for such performance. Analyses suggested that instead of trying to reproduce the referent sound as accurately as vocally possible, vocal imitations focus on a few important features, which depend on each particular sound category. These results offer perspectives for understanding how human listeners store and access long-term sound representations, and sets the stage for the development of human-computer interfaces based on vocalizations.