Naturalistic Listening Research Articles

Correspondence between consonant inventories obtained using modified naturalistic listening in real time (NLRT) and parent report

ObjectiveThe current study examined the correspondence between consonant inventories obtained using the modified NLRT approach and parent report. DesignProspective comparative study. SettingMultisite institutional. ParticipantsParticipants included 70 children with repaired CP + L (mean age = 16 months) who were participating in the multicenter study. ProceduresParents of participants were asked to record approximately two hours of their child's vocalizations/words at home using a Language ENvironmental Analysis (LENATM) recorder. Four ten-minute audio-recorded samples of vocalizations were extracted from the original recording for each participant and analyzed for size of consonant inventory. Parent reported consonant inventory was compared to coder identified consonant inventory. Coders identified an in-inventory consonant using two different criteria: 2+ tokens of each consonant were required in the first analysis and 10+ tokens of each consonant were required in the second analysis. ResultsCoder identified consonant inventory was larger (mean = 7.90) than that reported by parents (mean = 6.06) when a minimum of two tokens per consonant was required for inclusion of a consonant in inventory, while the inventory transcribed by coders was smaller (mean = 4.46) than that reported by parents when inclusion criteria required a minimum of ten tokens per consonant. ConclusionsAlthough the mean number of consonants in inventory between coders and parents was slightly closer using the 10+ versus 2+ criterion for consonant inclusion, the difference was not significant enough to recommend one protocol over the other.

Semantic composition in experimental and naturalistic paradigms

Abstract Naturalistic paradigms using movies or audiobooks have become increasingly popular in cognitive neuroscience, but connecting them to findings from controlled experiments remains rare. Here, we aim to bridge this gap in the context of semantic composition in language processing, which is typically examined using a “minimal” two-word paradigm. Using magnetoencephalography (MEG), we investigated whether the neural signatures of semantic composition observed in an auditory two-word paradigm can extend to naturalistic story listening, and vice versa. Our results demonstrate consistent differentiation between phrases and single nouns in the left anterior and middle temporal lobe, regardless of the context. Notably, this distinction emerged later during naturalistic listening. Yet this latency difference disappeared when accounting for various factors in the naturalistic data, such as prosody, word rate, word frequency, surprisal, and emotional content. These findings suggest the presence of a unified compositional process underlying both isolated and connected speech comprehension.

Making sense of music: Insights from neurophysiology and connectivity analyses in naturalistic listening conditions

According to the latest frameworks, auditory perception and memory involve the constant prediction of future sound events by the brain, based on the continuous extraction of feature regularities from the environment. The neural hierarchical mechanisms for predictive processes in perception and memory for sounds are typically studied in relation to simple acoustic features in isolated sounds or sound patterns inserted in highly certain contexts. Such studies have identified reliable prediction formation and error signals, e.g., the N100 or the mismatch negativity (MMN) evoked responses. In real life, though, individuals often face situations in which uncertainty prevails and where making sense of sounds becomes a hard challenge. In music, not only deviations from predictions are masterly set up by composers to induce emotions but sometimes the sheer uncertainty of sound scenes is exploited for aesthetic purposes, especially in compositional styles such as Western atonal classical music. In very recent magnetoencephalography (MEG) and electroencephalography (EEG) studies, experimental and technical advances in stimulation paradigms and analysis approaches have permitted the identification of prediction-error responses from highly uncertain, atonal contexts and the extraction of prediction-related responses from real, continuous music. Moreover, functional connectivity analyses revealed the emergence of cortico-hippocampal interactions during the formation of auditory memories for more predictable vs. less predictable patterns. These findings contribute to understanding the general brain mechanisms that enable us to predict even highly uncertain sound environments and to possibly make sense of and appreciate even atonal music.

Auditory and reward structures reflect the pleasure of musical expectancies during naturalistic listening

Enjoying music consistently engages key structures of the neural auditory and reward systems such as the right superior temporal gyrus (R STG) and ventral striatum (VS). Expectations seem to play a central role in this effect, as preferences reliably vary according to listeners’ uncertainty about the musical future and surprise about the musical past. Accordingly, VS activity reflects the pleasure of musical surprise, and exhibits stronger correlations with R STG activity as pleasure grows. Yet the reward value of musical surprise – and thus the reason for these surprises engaging the reward system – remains an open question. Recent models of predictive neural processing and learning suggest that forming, testing, and updating hypotheses about one’s environment may be intrinsically rewarding, and that the constantly evolving structure of musical patterns could provide ample opportunity for this procedure. Consistent with these accounts, our group previously found that listeners tend to prefer melodic excerpts taken from real music when it either validates their uncertain melodic predictions (i.e., is high in uncertainty and low in surprise) or when it challenges their highly confident ones (i.e., is low in uncertainty and high in surprise). An independent research group (Cheung et al., 2019) replicated these results with musical chord sequences, and identified their fMRI correlates in the STG, amygdala, and hippocampus but not the VS, raising new questions about the neural mechanisms of musical pleasure that the present study seeks to address. Here, we assessed concurrent liking ratings and hemodynamic fMRI signals as 24 participants listened to 50 naturalistic, real-world musical excerpts that varied across wide spectra of computationally modeled uncertainty and surprise. As in previous studies, liking ratings exhibited an interaction between uncertainty and surprise, with the strongest preferences for high uncertainty/low surprise and low uncertainty/high surprise. FMRI results also replicated previous findings, with music liking effects in the R STG and VS. Furthermore, we identify interactions between uncertainty and surprise on the one hand, and liking and surprise on the other, in VS activity. Altogether, these results provide important support for the hypothesized role of the VS in deriving pleasure from learning about musical structure.

The effects of propofol anaesthesia on molecular-enriched networks during resting-state and naturalistic listening

Placing a patient in a state of anaesthesia is crucial for modern surgical practice. However, the mechanisms by which anaesthetic drugs, such as propofol, impart their effects on consciousness remain poorly understood. Propofol potentiates GABAergic transmission, which purportedly has direct actions on cortex as well as indirect actions via ascending neuromodulatory systems. Functional imaging studies to date have been limited in their ability to unravel how these effects on neurotransmission impact the system-level dynamics of the brain. Here, we leveraged advances in multi-modal imaging, Receptor-Enriched Analysis of functional Connectivity by Targets (REACT), to investigate how different levels of propofol-induced sedation alter neurotransmission-related functional connectivity (FC), both at rest and when individuals are exposed to naturalistic auditory stimulation. Propofol increased GABA-A- and noradrenaline transporter-enriched FC within occipital and somatosensory regions respectively. Additionally, during auditory stimulation, the network related to the dopamine transporter showed reduced FC within bilateral regions of temporal and mid/posterior cingulate cortices, with the right temporal cluster showing an interaction between auditory stimulation and level of consciousness. In bringing together these micro- and macro-scale systems, we provide support for both direct GABAergic and indirect noradrenergic and dopaminergic-related network changes under propofol sedation. Further, we delineate a cognition-related reconfiguration of the dopaminergic network, highlighting the utility of REACT to explore the molecular substrates of consciousness and cognition.

Persistence and generalization of adaptive changes in auditory localization behavior following unilateral conductive hearing loss.

Sound localization relies on the neural processing of binaural and monaural spatial cues generated by the physical properties of the head and body. Hearing loss in one ear compromises binaural computations, impairing the ability to localize sounds in the horizontal plane. With appropriate training, adult individuals can adapt to this binaural imbalance and largely recover their localization accuracy. However, it remains unclear how long this learning is retained or whether it generalizes to other stimuli. We trained ferrets to localize broadband noise bursts in quiet conditions and measured their initial head orienting responses and approach-to-target behavior. To evaluate the persistence of auditory spatial learning, we tested the sound localization performance of the animals over repeated periods of monaural earplugging that were interleaved with short or long periods of normal binaural hearing. To explore learning generalization to other stimulus types, we measured the localization accuracy before and after adaptation using different bandwidth stimuli presented against constant or amplitude-modulated background noise. Retention of learning resulted in a smaller initial deficit when the same ear was occluded on subsequent occasions. Each time, the animals' performance recovered with training to near pre-plug levels of localization accuracy. By contrast, switching the earplug to the contralateral ear resulted in less adaptation, indicating that the capacity to learn a new strategy for localizing sound is more limited if the animals have previously adapted to conductive hearing loss in the opposite ear. Moreover, the degree of adaptation to the training stimulus for individual animals was significantly correlated with the extent to which learning extended to untrained octave band target sounds presented in silence and to broadband targets presented in background noise, suggesting that adaptation and generalization go hand in hand. Together, these findings provide further evidence for plasticity in the weighting of monaural and binaural cues during adaptation to unilateral conductive hearing loss, and show that the training-dependent recovery in spatial hearing can generalize to more naturalistic listening conditions, so long as the target sounds provide sufficient spatial information.

Le Petit Prince multilingual naturalistic fMRI corpus

Neuroimaging using more ecologically valid stimuli such as audiobooks has advanced our understanding of natural language comprehension in the brain. However, prior naturalistic stimuli have typically been restricted to a single language, which limited generalizability beyond small typological domains. Here we present the Le Petit Prince fMRI Corpus (LPPC–fMRI), a multilingual resource for research in the cognitive neuroscience of speech and language during naturalistic listening (OpenNeuro: ds003643). 49 English speakers, 35 Chinese speakers and 28 French speakers listened to the same audiobook The Little Prince in their native language while multi-echo functional magnetic resonance imaging was acquired. We also provide time-aligned speech annotation and word-by-word predictors obtained using natural language processing tools. The resulting timeseries data are shown to be of high quality with good temporal signal-to-noise ratio and high inter-subject correlation. Data-driven functional analyses provide further evidence of data quality. This annotated, multilingual fMRI dataset facilitates future re-analysis that addresses cross-linguistic commonalities and differences in the neural substrate of language processing on multiple perceptual and linguistic levels.

Transcribing toddler vocalizations: comparison of a modified naturalistic listening approach with phonetic transcription

ABSTRACT Previous investigations have demonstrated that naturalistic listening in real time (NLRT) can be used reliability to assess prelinguistic vocalizations and is less time-consuming than phonetic transcription. The current investigation was performed to examine the correspondence of syllable and consonant information obtained using a modified naturalistic listening in real time (NLRT) methodology compared to broad phonetic transcription for 20 toddlers with repaired cleft palate. A forty-minute vocalization sample was obtained for each toddler. Two listeners listened to each recording independently and coded the number of syllables and consonant types produced by each toddler using NLRT. Each recording was phonetically transcribed by each listener 2 to 16 months following the NLRT analysis. High reliability was evident between the modified NSRT method and phonetic transcription for identification of both syllables and consonants. Differences were evident in the actual number of syllables and consonants identified between the two types of assessment across the 20 participants. Possible explanations for those differences are addressed. The results of this investigation indicate that the modified NLRT procedure is a reliable method for determining consonant/inventories of young children. It is less labor intensive than traditional phonetic transcription and may be useful clinically when documentation of early sound development is needed to support early intervention decisions.

Decoding of Envelope vs. Fundamental Frequency During Complex Auditory Stream Segregation.

Hearing-in-noise perception is a challenging task that is critical to human function, but how the brain accomplishes it is not well understood. A candidate mechanism proposes that the neural representation of an attended auditory stream is enhanced relative to background sound via a combination of bottom-up and top-down mechanisms. To date, few studies have compared neural representation and its task-related enhancement across frequency bands that carry different auditory information, such as a sound's amplitude envelope (i.e., syllabic rate or rhythm; 1-9 Hz), and the fundamental frequency of periodic stimuli (i.e., pitch; >40 Hz). Furthermore, hearing-in-noise in the real world is frequently both messier and richer than the majority of tasks used in its study. In the present study, we use continuous sound excerpts that simultaneously offer predictive, visual, and spatial cues to help listeners separate the target from four acoustically similar simultaneously presented sound streams. We show that while both lower and higher frequency information about the entire sound stream is represented in the brain's response, the to-be-attended sound stream is strongly enhanced only in the slower, lower frequency sound representations. These results are consistent with the hypothesis that attended sound representations are strengthened progressively at higher level, later processing stages, and that the interaction of multiple brain systems can aid in this process. Our findings contribute to our understanding of auditory stream separation in difficult, naturalistic listening conditions and demonstrate that pitch and envelope information can be decoded from single-channel EEG data.

Natural music context biases musical instrument categorization.

Perception of sounds occurs in the context of surrounding sounds. When spectral properties differ between earlier (context) and later (target) sounds, categorization of later sounds becomes biased through spectral contrast effects (SCEs). Past research has shown SCEs to bias categorization of speech and music alike. Recent studies have extended SCEs to naturalistic listening conditions when the inherent spectral composition of (unfiltered) sentences biased speech categorization. Here, we tested whether natural (unfiltered) music would similarly bias categorization of French horn and tenor saxophone targets. Preceding contexts were either solo performances of the French horn or tenor saxophone (unfiltered; 1 secondduration in Experiment 1, or 3 seconds durationin Experiment 2) or a string quintet processed to emphasize frequencies in the horn or saxophone (filtered; 1 secondduration). Both approaches produced SCEs, producing more "saxophone" responses following horn/ horn-like contexts and vice versa. One-second filtered contexts produced SCEs as in previous studies, but 1-second unfiltered contexts did not. Three-second unfiltered contexts biased perception, but to a lesser degree than filtered contexts did. These results extend SCEs in musical instrument categorization to everyday listening conditions.

Assessment of prelinguistic vocalizations in real time: a comparison with phonetic transcription and assessment of inter-coder-reliability

ABSTRACT This study investigated reliability of naturalistic listening in real time (NLRT) compared to phonetic transcription. Speech pathology students with brief training in NLRT assessed prelinguistic syllable inventory size and specific syllable types in typically developing infants. A second study also examined inter-coder reliability for canonical babbling, canonical babbling ratio and presence of oral stops in syllable inventory of infants with cleft palate, by means of NLRT. In study 1, ten students independently assessed prelinguistic samples of five 12-month-old typically developing infants using NLRT and phonetic transcription. Coders assessed syllable inventory size as more than twice as large using phonetic transcription as NLRT. Results showed a strong correlation between NLRT and phonetic transcription (syllables with more than five occurrences) for syllable inventory size (r = .60; p < .001). The methods showed similar results for inter-coder reliability of specific syllable types. In study 2, three other students assessed prelinguistic samples of twenty-eight 12-month-old infants with cleft palate by means of NLRT. Results revealed perfect inter-coder agreement for presence/absence of canonical babbling, strong correlations between the three coders’ assessment of syllable inventory size (average r = .83; p < .001), but more inter-coder variability for agreement of specific syllable types. In conclusion, NLRT is a reliable method for assessing prelinguistic measures in infants with and without cleft palate with inter-coder agreement levels comparable to phonetic transcription for specific syllable types.

Natural speech statistics shift phoneme categorization.

All perception takes place in context. Recognition of a given speech sound is influenced by the acoustic properties of surrounding sounds. When the spectral composition of earlier (context) sounds (e.g., more energy at lower first formant [F1] frequencies) differs from that of a later (target) sound (e.g., vowel with intermediate F1), the auditory system magnifies this difference, biasing target categorization (e.g., towards higher-F1 /ɛ/). Historically, these studies used filters to force context sounds to possess desired spectral compositions. This approach is agnostic to the natural signal statistics of speech (inherent spectral compositions without any additional manipulations). The auditory system is thought to be attuned to such stimulus statistics, but this has gone untested. Here, vowel categorization was measured following unfiltered (already possessing the desired spectral composition) or filtered sentences (to match spectral characteristics of unfiltered sentences). Vowel categorization was biased in both cases, with larger biases as the spectral prominences in context sentences increased. This confirms sensitivity to natural signal statistics, extending spectral context effects in speech perception to more naturalistic listening conditions. Importantly, categorization biases were smaller and more variable following unfiltered sentences, raising important questions about how faithfully experiments using filtered contexts model everyday speech perception.

Hierarchical structure guides rapid linguistic predictions during naturalistic listening.

The grammar, or syntax, of human language is typically understood in terms of abstract hierarchical structures. However, theories of language processing that emphasize sequential information, not hierarchy, successfully model diverse phenomena. Recent work probing brain signals has shown mixed evidence for hierarchical information in some tasks. We ask whether sequential or hierarchical information guides the expectations that a human listener forms about a word’s part-of-speech when simply listening to every-day language. We compare the predictions of three computational models against electroencephalography signals recorded from human participants who listen passively to an audiobook story. We find that predictions based on hierarchical structure correlate with the human brain response above-and-beyond predictions based only on sequential information. This establishes a link between hierarchical linguistic structure and neural signals that generalizes across the range of syntactic structures found in every-day language.

A software program to assist coding of prelinguistic vocalizations in real time

ABSTRACTNumerous studies have confirmed that prelinguistic utterances are precursors to speech, and there is ample evidence that, for example, frequency of canonical syllables and syllable inventory size correlate with speech and language measures at older ages.Traditionally, prelinguistic utterances have been assessed by phonetic transcription which is difficult and time-consuming in infants. Recently, a more time-efficient methodology to assess prelinguistic utterances in real time, called naturalistic listening, was developed (Ramsdell et al., 2012). In a large international NIDCR-funded randomized controlled trial, Timing of Primary Surgery for with Cleft Palate (TOPS), including many coders, a software program (TimeStamper) was developed to assist in coding of prelinguistic vocalizations in real time, to ensure consistency of the coding procedures. Coders upload a video (or audio) file and watch and listen to the recording in real time without any possibility of pausing or taking notes. In real time, the coder registers each speech-like syllable as canonical or non-canonical. TimeStamper automatically calculates the percentage of canonical syllables of all syllables registered (canonical babbling ratio). At the end of a recording, TimeStamper assists in assessing presence/absence of canonical babbling and syllable inventory size. The software is presented and instructions for free access are provided.

RETRACTED ARTICLE: Neural auditory intensity discrimination during naturalistic listening: a brain decoding study.

Neural discrimination of auditory intensity is one of the fundamental questions in human auditory perception. Human neuroimaging studies have demonstrated specific neural activations during intensity discrimination tasks. The detailed functional anatomy, however, remains elusive. Most of the existing studies examined the entire auditory cortex as a whole, neglecting the potential functional differentiation within the auditory cortex. Moreover, these previous results based on controlled auditory stimuli might not necessarily extend to the neural mechanism of natural auditory processing. In this study, we propose a novel brain decoding method to examine neural encoding of intensity discrimination during naturalistic auditory experience. Here, we used publically available dataset (http://studyforrest.org) in which high resolution functional magnetic resonance imaging (fMRI) was acquired when the participants freely listened to the audio-version of the movie “Forrest Gump”. We found the sensitivity and accuracy of intensity discrimination in the auditory cortex critically depend on the divergences in intensity levels, yielding a sigmoidal relationship. Furthermore, anatomical subdivisions of auditory cortex revealed great functional diversity and lateralization in intensity discrimination. While bilateral activations were observed in primary auditory cortex, enhanced left-lateralization effects were shown in inferior frontal gyrus, anterior middle temporal gyrus and temporal-occipital junction. Our findings underscore the value of naturalistic paradigms in mapping auditory processing in the brain: naturalistic paradigm contains enriched and complex natural sounds, which not only improve the ecological validity but also alleviate the discrepancies caused by the differences in stimulus types and task designs in previous studies. The multidisciplinary methodology we developed here provides an analytical framework for studying neural encoding of complex auditory stimuli.

Editorial for the Special Issue on Neurosciences and Music: Part 2.

It is a great privilege to have been the guest editor for this second special issue on the Neurosciences of Music, following up on the 2012 issue on the same theme. It hardly seems like three years have passed, but even in this relatively short time period, we have witnessed conceptual and methodological developments that represent a significant maturation of our field. The contributions of the present issue affirm the breadth and quality of research being undertaken in the music and neuroscience community. The present issue comprises nine empirical papers from contributors in seven different countries. The contributions span perceptual, cognitive, affective, neurological, and therapeutic aspects of music, using an array of neuroimaging approaches, statistical approaches, and paradigms. Topics include distinguishing between how the brain responds to the cognitive versus affective aspects of music (Ellison, Moisseinen, Fachner, & Brattico, in press), how the reward system of musicians versus nonmusicians differs during continuous naturalistic listening (Alluri et al., in press), the role of the amygdala and orbitofrontal cortex in expectation (Mikutta et al., in press), the distinction between different forms of pathological and nonpathological musical imagery (Hemming & Merrill, in press) and the neural signatures of beat-based versus duration-based timing (De Pretto & James, in press). Sturm's paper focuses on the neural responses evoked during naturalistic ensemble listening, while Nakata looks at how tapping with an adaptive partner impacts pitch processing. Pfeiffer and Sabe (in press) provides a novel music-based assessment for the assessment of cognition after brain injury and, with a timely reflection on the current state of our community, Albusac-Jorge and Gimenez-Rodriguez (in press) provides a bibliometric study of the Neuroscience of Music field. We include a report on a high profile public lecture given by Daniel Levitin. Finally, we include a report on a lecture given by Daniel Levitin, and an announcement about a major new research center, Music in the Brain, in Aarhus, Denmark.As was also the case for the first special issue on this topic, we must acknowledge the critical role of the Mariani Foundation, in cooperation with the New York Academy of Sciences, who continue to provide an international nexus for dissemination and collaboration via their excellent research conferences. Several of the papers in the present issue were first presented in poster form at The Neurosciences and Music V conference, in Dijon, France, following which we announced a call for articles. …

Identification of Prelinguistic Phonological Categories

The prelinguistic infant's babbling repertoire of syllables--the phonological categories that form the basis for early word learning--is noticed by caregivers who interact with infants around them. Prior research on babbling has not explored the caregiver's role in recognition of early vocal categories as foundations for word learning. In the present work, the authors begin to address this gap. The authors explored vocalizations produced by 8 infants at 3 ages (8, 10, and 12 months) in studies illustrating identification of phonological categories through caregiver report, laboratory procedures simulating the caregiver's natural mode of listening, and the more traditional laboratory approach (phonetic transcription). Caregivers reported small repertoires of syllables for their infants. Repertoires of similar size and phonetic content were discerned in the laboratory by judges who simulated the caregiver's natural mode of listening. However, phonetic transcription with repeated listening to infant recordings yielded repertoire sizes that vastly exceeded those reported by caregivers and naturalistic listeners. The results suggest that caregiver report and naturalistic listening by laboratory staff can provide a new way to explore key characteristics of early infant vocal categories, a way that may provide insight into later speech and language development.

Infants Exposed to Fluent Natural Speech Succeed at Cross-Gender Word Recognition

To examine the possibility that early signal-to-word form mapping capabilities are robust enough to handle substantial indexical variation in the realization of words. Two groups of 7.5-month-olds were tested with the Headturn Preference Procedure. Half of the infants were exposed to words embedded in passages spoken by their mothers and tested on lists of trained and novel isolated words spoken by their fathers. The other half of the infants were yoked pairs listening to unfamiliar speakers. In the test phase, infants listened longer to trained than to novel words, indicating that they successfully segmented the words from the passages. This result was not modulated by infants' familiarity with the speaker. Under more naturalistic listening conditions, 7.5-month-olds exhibit the ability to recognize words in the face of substantial indexical variation regardless of whether speakers are familiar. This suggests that early word representations are, at least to some extent, independent of the speaker's gender and may reflect sophisticated abstraction capabilities on the part of the infants, which would render extreme episodic models of early speech perception untenable. Additional research using similarly ecologically valid testing methods is called for to elucidate the precise nature of early word representations.