Musical Timbre Research Articles

Large-scale brain networks emerge from dynamic processing of musical timbre, key and rhythm

We investigated the neural underpinnings of timbral, tonal, and rhythmic features of a naturalistic musical stimulus. Participants were scanned with functional Magnetic Resonance Imaging (fMRI) while listening to a stimulus with a rich musical structure, a modern tango. We correlated temporal evolutions of timbral, tonal, and rhythmic features of the stimulus, extracted using acoustic feature extraction procedures, with the fMRI time series. Results corroborate those obtained with controlled stimuli in previous studies and highlight additional areas recruited during musical feature processing. While timbral feature processing was associated with activations in cognitive areas of the cerebellum, and sensory and default mode network cerebrocortical areas, musical pulse and tonality processing recruited cortical and subcortical cognitive, motor and emotion-related circuits. In sum, by combining neuroimaging, acoustic feature extraction and behavioral methods, we revealed the large-scale cognitive, motor and limbic brain circuitry dedicated to acoustic feature processing during listening to a naturalistic stimulus. In addition to these novel findings, our study has practical relevance as it provides a powerful means to localize neural processing of individual acoustical features, be it those of music, speech, or soundscapes, in ecological settings.

Impaired perception of temporal fine structure and musical timbre in cochlear implant users

Cochlear implant (CI) users demonstrate severe limitations in perceiving musical timbre, a psychoacoustic feature of sound responsible for ‘tone color’ and one’s ability to identify a musical instrument. The reasons for this limitation remain poorly understood. In this study, we sought to examine the relative contributions of temporal envelope and fine structure for timbre judgments, in light of the fact that speech processing strategies employed by CI systems typically employ envelope extraction algorithms. We synthesized “instrumental chimeras” that systematically combined variable amounts of envelope and fine structure in 25% increments from two different source instruments with either sustained or percussive envelopes. CI users and normal hearing (NH) subjects were presented with 150 chimeras and asked to determine which instrument the chimera more closely resembled in a single-interval two-alternative forced choice task. By combining instruments with similar and dissimilar envelopes, we controlled the valence of envelope for timbre identification and compensated for envelope reconstruction from fine structure information. Our results show that NH subjects utilize envelope and fine structure interchangeably, whereas CI subjects demonstrate overwhelming reliance on temporal envelope. When chimeras were created from dissimilar envelope instrument pairs, NH subjects utilized a combination of envelope (p = 0.008) and fine structure information (p = 0.009) to make timbre judgments. In contrast, CI users utilized envelope information almost exclusively to make timbre judgments (p < 0.001) and ignored fine structure information (p = 0.908). Interestingly, when the value of envelope as a cue was reduced, both NH subjects and CI users utilized fine structure information to make timbre judgments (p < 0.001), although the effect was quite weak in CI users. Our findings confirm that impairments in fine structure processing underlie poor perception of musical timbre in CI users.

Harmonic-temporal model with multiple function for sound synthesis.

The most popular method for synthesizing musical timbre sound is additive synthesis, where the timbre sound is obtained by adding multiple harmonic partials with different time envelopes. Additive synthesis has many advantages such as high flexibility to control each partial individually and easy way to reconstruct musical tones from the spectrum. However, it is difficult to control the harmonic partials as a whole because time envelopes of each harmonic are independent of one another. Furthermore, the model has many parameters which are difficult to control. For example, many instruments require dozens of harmonics, where each envelope consists of about 1000 time points, and the parameters are hard to handle. In this paper, additive synthesis is extended to easily control a whole sound with only a few parameters. In this approach, all the time envelopes of each harmonic are parametrically modeled by a 2-D function, called multiple function. Two types of multiple functions are introduced with the update rules of their parameters based on an EM algorithm. The harmonic-temporal shape of an instrument sound can be directly controlled by changing the parameters of the proposed function slightly.

Cortical Plasticity in 4-Month-Old Infants: Specific Effects of Experience with Musical Timbres

Animal models suggest that the brain is particularly neuroplastic early in development, but previous studies have not systematically controlled the auditory environment in human infants and observed the effects on auditory cortical representations. We exposed 4-month-old infants to melodies in either guitar or marimba timbre (infants were randomly assigned to exposure group) for a total of~160min over the course of a week, after which we measured electroencephalogram (EEG) responses to guitar and marimba tones at pitches not previously heard during the exposure phase. A frontally negative response with a topography consistent with generation in auditory areas, peaking around 450ms, was significantly larger for guitar than marimba tones in the guitar-exposed group but significantly larger for marimba than guitar tones in the marimba-exposed group. This indicates that experience with tones in a particular timbre affects representations for that timbre, and that this effect generalizes to tones not previously experienced during exposure. Furthermore, mismatch responses to occasional small 3% changes in pitch were larger for tones in guitar than marimba timbre only for infants exposed to guitar tones. Together these results indicate that a relatively small amount of passive exposure to a particular timbre in infancy enhances representations of that timbre and leads to more precise pitch processing for that timbre.

Sound Source Mechanics and Musical Timbre Perception: Evidence From Previous Studies

Timbre has been conceived of as a multidimensional sensory attribute and as a carrier of perceptually useful information about the mechanics of the sound source. To date, research on musical timbre has focused on defining its acoustical correlates, whereas fragmentary evidence is available on the influence of mechanical parameters. We quantified the extent to which mechanical properties of the sound source are associated with structures in the data from published identification and dissimilarity-rating studies. We focus on two macroscopic mechanical properties: the musical instrument family and excitation type. Identification confusions are significantly more frequent for same-family instruments. With dissimilarity ratings, same-family or same-excitation tones are judged more similar and tend to occupy the same region of multidimensional-scaling spaces. As such, significant associations between the perception of musical timbre and the mechanics of the sound source emerge even when not explicitly demanded by the task.

Temporal and Spectral Cues for Musical Timbre Perception in Electric Hearing

The purpose of this study was to investigate musical timbre perception in cochlear-implant (CI) listeners using a multidimensional scaling technique to derive a timbre space. Methods Sixteen stimuli that synthesized western musical instruments were used (McAdams, Winsberg, Donnadieu, De Soete, & Krimphoff, 1995). Eight CI listeners and 15 normal-hearing (NH) listeners participated. Each listener made judgments of dissimilarity between stimulus pairs. Acoustical analyses that characterized the temporal and spectral characteristics of each stimulus were performed to examine the psychophysical nature of each perceptual dimension. For NH listeners, the timbre space was best represented in three dimensions, one correlated with the temporal envelope (log-attack time) of the stimuli, one correlated with the spectral envelope (spectral centroid), and one correlated with the spectral fine structure (spectral irregularity) of the stimuli. The timbre space from CI listeners, however, was best represented by two dimensions, one correlated with temporal envelope features and the other weakly correlated with spectral envelope features of the stimuli. Temporal envelope was a dominant cue for timbre perception in CI listeners. Compared to NH listeners, CI listeners showed reduced reliance on both spectral envelope and spectral fine structure cues for timbre perception.

Perception and acoustical analyzes of traditionally orchestrated musical structures versus non‐traditional counterparts.

Many studies use single‐tone or dyadic‐unison stimuli in the exploration of musical timbre. Musical contexts, however, often employ larger numbers of simultaneous voices. Recently, we have conducted perceptual/acoustical studies of consonance, dissonance, and roughness in multitimbral triads often finding that chord quality (major, minor, diminished, augmented) had perceptual primacy over timbre. The present research extends this work to include major and minor triads in similarity scaling, blend, and identification experiments. In addition, four‐tone chords were orchestrated according to procedures articulated in orchestration texts: juxtaposition, interlocking, enclosure, and overlapping. Sampled instrument tones (Sibelius/Kontakt) of oboe, flute, and clarinet were used, allowing comparison to previous research, and non‐traditional combinations with saxophone and trumpet were included. Triads with each tone assigned a different timbre provided nontraditional orchestrations to be compared and contrasted with traditional orchestrations that duplicated timbres within a chord. Perceptual data were correlated to acoustical analyzes of spectral distribution and time‐variancy.

A biomimetic multi‐resolution spectrotemporal model for musical timbre recognition.

Timbre is usually defined as that property of sound that is neither pitch nor loudness that helps in identifying sounds. Such a definition of what it is not is in itself problematic and unsatisfactory. The complexity of characterizing a timbre space stems from the intricate interactions between spectrotemporal dynamics of sound, which overcasts the simple description of individual acoustic dimensions as usually captured by techniques such as multidimensional scaling. By contrast, sound encoding in the mammalian auditory system, and particularly its sensitivity to spectrotemporal modulations, offers a rich feature space to explore perceptual representations of timbre. In the present work, the problem of musical timbre modeling was casted as a generative model based on a reduced set of multi‐resolution spectrotemporal features inspired from encoding of complex sound in the auditory cortex. A probabilistic system based on Gaussian mixtures was built to perform a timbre recognition task on a data set consisti...

Psychoacoustics and organology.

Organology, a branch of ethnomusicology, is the study of musical instruments and their classification. The first Western system–which classified instruments primarily as idiophones, membranophones, chordophones, or aerophones—was proposed by Sachs and von Hornbostel in 1914. This system has been the progenitor and inspiration for numerous other classification systems. While the van Hornbostel–Sachs system has proven to be useful in many contexts, it does not offer a consistent approach in terms of the transfer of energy from driver to generator to resonator. In addition, it does not take into account the common practice in modern popular and art music by which digitized samples of sounds are disembodied and appropriated from musical instruments from around the world. In other words, rather than encountering musical instruments as visual, tactile, and sound-producing, musicians and audiences encounter only pure sound. This paper, therefore, addresses two issues: How would a classification system of musical instrument {\sound} take into account the vibrational characteristics of acoustical musical instruments as well as the results of perceptual research on musical timbre from the past 40 years? The resulting system effectively turns Hornbostel–Sachs “on its ear” by reinterpreting primary classes as subclasses and vice versa.

Recent research in musical timbre perception.

Various methods have been used in an attempt to reveal salient parameters for musical timbre perception and to estimate perceptual distances between timbres based on their spectra. The multidimensional scaling method compares exemplars of musical sounds using listener estimation of timbral dissimilarity between sound pairs. Data are coalesced into a space whose dimensions are easily interpretable in terms of acoustic parameters such as spectral centroid, spectral flux, spectral irregularity, and attack time. Sounds can be normalized with respect to some parameters in order to reveal others. The direct comparison method attempts to discover how discrimination, recognition, and degree of perceptual dissimilarity depend on the degree of modification of the time-varying spectrum or of particular features of the spectrum. Morphing between spectrum pairs is one method of producing spectral modifications. Dissimilarity can be predicted by computing a “distance” between two time-varying spectra. For musical sounds, spectral distance has been computed by taking the average Cartesian distance between spectral measures treated as vectors. For constant-F0 sounds, the most obvious vector is the set of harmonic amplitudes, either in linear or decibel units. Other vectors are based on critical bands and cepstral coefficients. [Work supported by Research Grants Council Grant No. 613508.]

Congenital amusia: A short-term memory deficit for non-verbal, but not verbal sounds

Congenital amusia refers to a lifelong disorder of music processing and is linked to pitch-processing deficits. The present study investigated congenital amusics’ short-term memory for tones, musical timbres and words. Sequences of five events (tones, timbres or words) were presented in pairs and participants had to indicate whether the sequences were the same or different. The performance of congenital amusics confirmed a memory deficit for tone sequences, but showed normal performance for word sequences. For timbre sequences, amusics’ memory performance was impaired in comparison to matched controls. Overall timbre performance was found to be correlated with melodic contour processing (as assessed by the Montreal Battery of Evaluation of Amusia). The present findings show that amusics’ deficits extend to non-verbal sound material other than pitch, in this case timbre, while not affecting memory for verbal material. This is in line with previous suggestions about the domain-specificity of congenital amusia.

Multidimensional perceptual scaling of musical timbre by hearing‐impaired listeners.

We examine the impact of hearing loss and hearing aid processing on the perception of musical timbre. Our objective is to identify significant timbre cues for hearing‐impaired listeners, and to assess the impact of hearing aid signal processing on timbre perception. Hearing aids perform dynamic, level‐dependent spectrum shaping that may influence listeners’ perception of musical instrument timbres and their ability to discriminate among them. Grey [“Multidimensional perceptual scaling of musical timbres,” J. Acoust. Soc. Am. 61, 1270 (1977)] showed that sustaining instrument tones equalized for level, loudness, and duration are distinguished primarily along three perceptual dimensions that are strongly correlated with the acoustical dimensions of: (1) spectral energy distribution, (2) spectral fluctuation, and (3) precedent high‐frequency, low‐amplitude energy. Following the work of Grey, we ask listeners having mild to moderately severe sensorineural hearing loss to rate pairs of synthetic musical instrument tones according to dissimilarity in aided and unaided conditions. We analyze the dissimilarity judgments to identify acoustical correlates for the significant dimensions of timbre discrimination. This analysis helps us to objectively estimate the impact of hearing aid signal processing on the ability of hearing‐impaired listeners to discriminate among musical instruments, an important component of musical listening.

Cortical analysis of auditory scenes and speech

Event Abstract Back to Event Cortical analysis of auditory scenes and speech Spectrotemporal modulations are critical in understanding the perception of complex sounds, especially in conveying intelligibility in speech and quality in music. Neurons in the primary auditory cortex extract and explicitly represent these modulations in their responses. In this talk, I shall review experimental methods to study the representation of these modulations in the cortex and the way they reveal the acoustic features of complex sounds such as phonemes. I shall also discuss various abstractions of this analysis to develop algorithms for the analysis of complex auditory scenes. One example is the question of how spectrotemporal modulation content of speech can be used to assess its intelligibility, to discriminate it from non-speech signals, and to enhance it by performing Weiner-like filtering in the space of the cortical representation. In music, these modulations underlie the perception of sound quality, and hence can be effectively employed to construct a perceptually-meaningful metric of musical timbre. Finally, I shall review recent extensions of these studies that explore the role of feedback and top-down attentional effects on the representation of modulations, measurements that are carried out while animals are engaged in a variety of auditory tasks. The resulting understanding of these rapid adaptive mechanisms has been formulated as algorithms for the segregation of complex sound mixtures to resolve difficult examples of the "cocktail-party problem". Conference: Computational and systems neuroscience 2009, Salt Lake City, UT, United States, 26 Feb - 3 Mar, 2009. Presentation Type: Oral Presentation Topic: Oral Presentations Citation: (2009). Cortical analysis of auditory scenes and speech. Front. Syst. Neurosci. Conference Abstract: Computational and systems neuroscience 2009. doi: 10.3389/conf.neuro.06.2009.03.305 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 04 Feb 2009; Published Online: 04 Feb 2009. Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Google Google Scholar PubMed Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Spectral and temporal modulations essential to spoken word, gender and timbre identification

Human speech and musical sounds contain complex spectral and temporal modulations. Speech intelligibility, perception of melody, and identification of source characteristics (e.g., speaker gender or musical timbre) depend on spectrotemporal modulations but can be surprisingly robust to drastic spectral and temporal degradations. We systematically explored which restricted spectral and temporal modulations are essential to the perception of complex sounds. Degraded sentences and musical sounds were obtained by a novel modulation filtering procedure performed on the sound spectrogram. Temporal modulation filtering smeared the amplitude envelope by removing changes above particular Hz. Spectral modulation filtering smeared the spectral energy across frequency bands by removing changes above particular cyc/kHz. We further complemented this low-pass filtering with more specific notch-filtering. Speech intelligibility, gender recognition and musical instrument identification were assessed in psychophysical experiments. We determined that spectral modulations below ∼3.75 cyc/kHz, and temporal modulations between 1 and 7 Hz are essential for speech comprehension. Gender identification however required the presence of higher spectral modulations. Similarly the timbre and pitch of instruments was affected differentially by notch filters in these two regions of the modulation spectrum. Our research could be used to guide the design of optimal signal processing in hearing aids and cochlear implants.

Cortical analysis of auditory scenes and speech

Event Abstract Back to Event Cortical analysis of auditory scenes and speech Shihab Shamma1* 1 University of Maryland, Maryland, United States Spectrotemporal modulations are critical in understanding the perception of complex sounds, especially in conveying intelligibility in speech and quality in music. Neurons in the primary auditory cortex extract and explicitly represent these modulations in their responses. In this talk, I shall review experimental methods to study the representation of these modulations in the cortex and the way they reveal the acoustic features of complex sounds such as phonemes. I shall also discuss various abstractions of this analysis to develop algorithms for the analysis of complex auditory scenes. One example is the question of how spectrotemporal modulation content of speech can be used to assess its intelligibility, to discriminate it from non-speech signals, and to enhance it by performing Weiner-like filtering in the space of the cortical representation. In music, these modulations underlie the perception of sound quality, and hence can be effectively employed to construct a perceptually-meaningful metric of musical timbre. Finally, I shall review recent extensions of these studies that explore the role of feedback and top-down attentional effects on the representation of modulations, measurements that are carried out while animals are engaged in a variety of auditory tasks. The resulting understanding of these rapid adaptive mechanisms has been formulated as algorithms for the segregation of complex sound mixtures to resolve difficult examples of the "cocktail-party problem". Conference: Bernstein Symposium 2008, Munich, Germany, 8 Oct - 10 Oct, 2008. Presentation Type: Oral Presentation Topic: All Abstracts Citation: Shamma S (2008). Cortical analysis of auditory scenes and speech. Front. Comput. Neurosci. Conference Abstract: Bernstein Symposium 2008. doi: 10.3389/conf.neuro.10.2008.01.002 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 11 Nov 2008; Published Online: 11 Nov 2008. * Correspondence: Shihab Shamma, University of Maryland, Maryland, Maryland, United States, sas@eng.umd.edu Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Shihab Shamma Google Shihab Shamma Google Scholar Shihab Shamma PubMed Shihab Shamma Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Cerebral Processing of Timbre and Loudness: fMRI Evidence for a Contribution of Broca’s Area to Basic Auditory Discrimination

Sound timbre and sound volume processing are basic auditory discrimination processes relevant for human language abilities. Regarding lateralization effects, the prevailing hypotheses ascribe timbre processing to the right hemisphere (RH). Recent experiments also point to a role of the RH for volume discrimination. We investigated the relevance of the RH for timbre and volume processing, aiming at finding possible differences in cerebral representation of these acoustic parameters. Seventeen healthy subjects performed two auditory discrimination tasks on tone pairs, differing either in timbre or volume. FMRI was performed using an EPI-sequence on a 1.5 T scanner. Hemodynamic responses emerged in both tasks within a bilateral network of areas, including cingulate and cerebellum, peaking in primary and secondary auditory cortices (core and belt areas). Laterality analyses revealed a significant leftward dominance at the temporal cortex. Task comparison revealed significant activation within Broca’s area during the timbre task and a trend for an increase of right parietal responses during volume processing. These results contribute to a more differentiated view of timbre processing. Additionally to the engagement of the right temporal cortex during processing of musical timbre, there seem to be language related aspects of timbre that are preferentially processed in the left hemisphere. These findings are discussed within the framework of a model of timbre perception comprising two differentially lateralized subprocesses. Processing of spectral cues (harmonic structure) linked to the right hemisphere and processing of temporal cues (i.e. attack-decay dynamics) linked to the left hemisphere. Moreover, activation of Broca’s area linked to the timbre task indicates a participation of this area in discriminating phonetic changes of the vowel-like non-speech signals, encouraging the argument that basic acoustic cue processing at a pre- or non-speech level is represented within this “classical language area.”

Persistent Responsiveness of Long-Latency Auditory Cortical Activities in Response to Repeated Stimuli of Musical Timbre and Vowel Sounds

Long-latency auditory-evoked magnetic field and potential show strong attenuation of N1m/N1 responses when an identical stimulus is presented repeatedly due to adaptation of auditory cortical neurons. This adaptation is weak in subsequently occurring P2m/P2 responses, being weaker for piano chords than single piano notes. The adaptation of P2m is more suppressed in musicians having long-term musical training than in nonmusicians, whereas the amplitude of P2 is enhanced preferentially in musicians as the spectral complexity of musical tones increases. To address the key issues of whether such high responsiveness of P2m/P2 responses to complex sounds is intrinsic and common to nonmusical sounds, we conducted a magnetoencephalographic study on participants who had no experience of musical training, using consecutive trains of piano and vowel sounds. The dipole moment of the P2m sources located in the auditory cortex indicated significantly suppressed adaptation in the right hemisphere both to piano and vowel sounds. Thus, the persistent responsiveness of the P2m activity may be inherent, not induced by intensive training, and common to spectrally complex sounds. The right hemisphere dominance of the responsiveness to musical and speech sounds suggests analysis of acoustic features of object sounds to be a significant function of P2m activity.

Commentary on "Timbre as an Elusive Component of Imagery for Music," by Freya Bailes

The study of musical timbre by Bailes (2007) raises important questions concerning the relative ease of imaging complex perceptual attributes such as timbre, compared to more unidimensional attributes. I also raise the issue of individual differences in auditory imagery ability, especially for timbre. Submitted 2006 November 21; Accepted 2006 November 25

A Response to Andrea R. Halpern's Commentary

The author responds to points raised in Andrea Halpern's commentary, which appeared in Vol. 2, No. 1 of Empirical Musicology Review. Discussion focuses on the apparent contradiction between self-reports of veridical mental imagery of musical timbre, and cognitive constraints on temporal memory for multidimensional sound.

The influence of polyphony on the dynamical modelling of musical timbre

This letter addresses the problem of pattern recognition of polyphonic musical timbre. Frame-level dynamics of audio features are particularly difficult to model, although they have been identified as crucial perceptive dimensions of timbre perception. Recent studies seem to indicate that traditional means to model data dynamics, such as delta-coefficients, texture windows or Markov modelling, do not provide any improvement over the best static models for real-world, complex polyphonic textures of several seconds’ length. This contradicts experimental data on the perception of individual instrument notes. This letter describes an experiment to identify the cause of this contradiction. We propose that the difficulty of modelling the dynamics of full songs results either from the complex structure of the temporal succession of notes, or from the vertical polyphonic nature of individual notes. We discriminate between both hypothesis by comparing the performance of static and dynamical algorithms on several specially designed datasets, namely monophonic individual notes, polyphonic individual notes, and polyphonic multiple-note textures. We conclude that the main cause of the difficulty of modelling dynamics of real-world polyphonic musical textures is the polyphonic nature of the data.