Auditory Representation Research Articles

The Role of Auditory Perceptual Simulation in L2 Syntactic Processing

Auditory perceptual simulation (APS) reading is a virtual auditory representation of other speakers reading out a text. A previous L1 study showed that native speakers of English using APS reading read faster and more accurately. This study investigated whether using a similar paradigm can enhance L2 learners’ weak syntactic processing with more salient prosodic information. In eye-tracking experiments, L2 Korean learners of English, with or without APS-manipulation, read experimental sentences that employed the plausibility effect, such as The cat that the mouse chased was fast. The plausibility effect would occur when the participants properly established the structural relation and disappear when they overcame the difficulty with accurate syntactic interpretations. The results demonstrated that the APS-reading group showed larger plausibility effect at the critical region during the initial processing, but smaller plausibility effect during the late processing. The APS group detected the implausibility of the sentence earlier than the control group and had less difficulty accurately parsing the structure. This finding suggests that APS reading can improve L2 learners’ syntactic processing, as it does in L1 processing, by facilitating their prosodic representations.

A comparison of neural responses in the primary auditory cortex, amygdala, and medial prefrontal cortex of cats during auditory discrimination tasks.

Discriminating biologically relevant sounds is crucial for survival. The neurophysiological mechanisms that mediate this process must register both the reward significance and the physical parameters of acoustic stimuli. Previous experiments have revealed that the primary function of the auditory cortex (AC) is to provide a neural representation of the acoustic parameters of sound stimuli. However, how the brain associates acoustic signals with reward remains unresolved. The amygdala (AMY) and medial prefrontal cortex (mPFC) play keys role in emotion and learning, but it is unknown whether AMY and mPFC neurons are involved in sound discrimination or how the roles of AMY and mPFC neurons differ from those of AC neurons. To examine this, we recorded neural activity in the primary auditory cortex (A1), AMY, and mPFC of cats while they performed a Go/No-go task to discriminate sounds with different temporal patterns. We found that the activity of A1 neurons faithfully coded the temporal patterns of sound stimuli; this activity was not affected by the cats' behavioral choices. The neural representation of stimulus patterns decreased in the AMY, but the neural activity increased when the cats were preparing to discriminate the sound stimuli and waiting for reward. Neural activity in the mPFC did not represent sound patterns, but it showed a clear association with reward and was modulated by the cats' behavioral choices. Our results indicate that the initial auditory representation in A1 is gradually transformed into a stimulus-reward association in the AMY and mPFC to ultimately generate a behavioral choice. NEW & NOTEWORTHY We compared the characteristics of neural activities of primary auditory cortex (A1), amygdala (AMY), and medial prefrontal cortex (mPFC) while cats were performing the same auditory discrimination task. Our results show that there is a gradual transformation of the neural code from a faithful temporal representation of the stimulus in A1, which is insensitive to behavioral choices, to an association with the predictive reward in AMY and mPFC, which, to some extent, is correlated with the animal's behavioral choice.

The familiar-melody advantage in auditory perceptual development: Parallels between spoken language acquisition and general auditory perception.

How do learners build up auditory pattern knowledge? Findings from children's spoken word learning suggest more robust auditory representations for highly familiar words than for newly learned words. This argues against spoken language learning as a process of simply acquiring a fixed set of speech sound categories, suggesting instead that specific words may be the relevant units. More generally, one might state this as the specific-learning hypothesis-that acquiring sound pattern knowledge involves learning specific patterns, rather than abstract pattern components. To understand the nature of human language knowledge, it is important to determine whether this specific learning reflects processes unique to spoken language learning or instead reflects more general auditory-learning processes. To investigate whether the specific-learning hypothesis extends to auditory pattern learning more generally, the present study tested the perceptual processing of familiar melodies versus carefully matched unfamiliar melodies. Children performed better at both audiovisual mapping (Exp. 1) and same-different auditory discrimination (Exp. 2) when hearing familiar melodies than when hearing unfamiliar melodies. This is consistent with the specific-learning hypothesis and with exemplar-style general-auditory accounts of pattern learning, although alternative explanations are possible.

Multimodal Object Classification Models Inspired by Multisensory Integration in the Brain.

Two multimodal classification models aimed at enhancing object classification through the integration of semantically congruent unimodal stimuli are introduced. The feature-integrating model, inspired by multisensory integration in the subcortical superior colliculus, combines unimodal features which are subsequently classified by a multimodal classifier. The decision-integrating model, inspired by integration in primary cortical areas, classifies unimodal stimuli independently using unimodal classifiers and classifies the combined decisions using a multimodal classifier. The multimodal classifier models are implemented using multilayer perceptrons and multivariate statistical classifiers. Experiments involving the classification of noisy and attenuated auditory and visual representations of ten digits are designed to demonstrate the properties of the multimodal classifiers and to compare the performances of multimodal and unimodal classifiers. The experimental results show that the multimodal classification systems exhibit an important aspect of the “inverse effectiveness principle” by yielding significantly higher classification accuracies when compared with those of the unimodal classifiers. Furthermore, the flexibility offered by the generalized models enables the simulations and evaluations of various combinations of multimodal stimuli and classifiers under varying uncertainty conditions.

SU82AN ALGORITHM TO IDENTIFY PATIENTS WITH OBSESSIVE-COMPULSIVE DISORDERS IN BIOVU USING NATURAL LANGUAGE PROCESSING, ICD- CODE AND MEDICATION ADMINISTRATION RECORDS IN THE SYNTHETIC DERIVATIVE

Recognizing the identity of a person is fundamental to guide social interactions. We can recognize the identity of a person looking at her face, but also listening to her voice. An important question concerns how visual and auditory information come together, enabling us to recognize identity independently of the modality of the stimulus. This study reports converging evidence across univariate contrasts and multivariate classification showing that the posterior superior temporal sulcus (pSTS), previously known to encode polymodal visual and auditory representations, encodes information about person identity with invariance within and across modality. In particular, pSTS shows selectivity for faces, selectivity for voices, classification of face identity across image transformations within the visual modality, and classification of person identity across modality.

Nonlinear auditory models yield new insights into representations of vowels.

Studies of vowel systems regularly appeal to the need to understand how the auditory system encodes and processes the information in the acoustic signal. The goal of this study is to present computational models to address this need, and to use the models to illustrate responses to vowels at two levels of the auditory pathway. Many of the models previously used to study auditory representations of speech are based on linear filter banks simulating the tuning of the inner ear. These models do not incorporate key nonlinear response properties of the inner ear that influence responses at conversational-speech sound levels. These nonlinear properties shape neural representations in ways that are important for understanding responses in the central nervous system. The model for auditory-nerve (AN) fibers used here incorporates realistic nonlinear properties associated with the basilar membrane, inner hair cells (IHCs), and the IHC-AN synapse. These nonlinearities set up profiles of f0-related fluctuations that vary in amplitude across the population of frequency-tuned AN fibers. Amplitude fluctuations in AN responses are smallest near formant peaks and largest at frequencies between formants. These f0-related fluctuations strongly excite or suppress neurons in the auditory midbrain, the first level of the auditory pathway where tuning for low-frequency fluctuations in sounds occurs. Formant-related amplitude fluctuations provide representations of the vowel spectrum in discharge rates of midbrain neurons. These representations in the midbrain are robust across a wide range of sound levels, including the entire range of conversational-speech levels, and in the presence of realistic background noise levels.

Understanding auditory representations of emotional expressions with neural networks

In contrast to many established emotion recognition systems, convolutional neural networks do not rely on handcrafted features to categorize emotions. Although achieving state-of-the-art performances, it is still not fully understood what these networks learn and how the learned representations correlate with the emotional characteristics of speech. The aim of this work is to contribute to a deeper understanding of the acoustic and prosodic features that are relevant for the perception of emotional states. Firstly, an artificial deep neural network architecture is proposed that learns the auditory features directly from the raw and unprocessed speech signal. Secondly, we introduce two novel methods for the analysis of the implicitly learned representations based on data-driven and network-driven visualization techniques. Using these methods, we identify how the network categorizes an audio signal as a two-dimensional representation of emotions, namely valence and arousal. The proposed approach is a general method to enable a deeper analysis and understanding of the most relevant representations to perceive emotional expressions in speech.

The relationship between the temporal structure of magnetoencephalography recorded brain activity and capacity to form discrete auditory representations.

A function of oscillatory brain activity may be to align activity relative to threshold potentials and in doing so provide limited opportunities for representational neuronal assemblies to form. This low‐level function could apply across frequency bands and potentially affect the temporal dynamics of experience. To test these possibilities, a magnetoencephalography protocol was developed where capacity to form discrete auditory representations over time was assessed relative to oscillatory brain activity. Three sets of preregistered analyses were conducted. First, the capacity to form representations correlated with the prevalence and durations of activity localised to the auditory cortex. Second, brain oscillations became entrained to stimuli over a broad range of frequencies. Finally, a sequence of gamma (γ) band events predicted successful discrete representation, where previous research had indicated similar individuation‐related differences within the alpha (α) range. Together, these findings indicate that a low‐level function of cortical oscillations, which may apply across a range of frequency bands, is periodically to set conditions in which representational neuronal assemblies can manifest, limiting and so affecting the flow of experience.

Auditory Brainstem Representation of the Voice Pitch Contours in the Resolved and Unresolved Components of Mandarin Tones.

Accurate perception of voice pitch plays a vital role in speech understanding, especially for tonal languages such as Mandarin. Lexical tones are primarily distinguished by the fundamental frequency (F0) contour of the acoustic waveform. It has been shown that the auditory system could extract the F0 from the resolved and unresolved harmonics, and the tone identification performance of resolved harmonics was better than unresolved harmonics. To evaluate the neural response to the resolved and unresolved components of Mandarin tones in quiet and in speech-shaped noise, we recorded the frequency-following response. In this study, four types of stimuli were used: speech with either only-resolved harmonics or only-unresolved harmonics, both in quiet and in speech-shaped noise. Frequency-following responses (FFRs) were recorded to alternating-polarity stimuli and were added or subtracted to enhance the neural response to the envelope (FFRENV) or fine structure (FFRTFS), respectively. The neural representation of the F0 strength reflected by the FFRENV was evaluated by the peak autocorrelation value in the temporal domain and the peak phase-locking value (PLV) at F0 in the spectral domain. Both evaluation methods showed that the FFRENV F0 strength in quiet was significantly stronger than in noise for speech including unresolved harmonics, but not for speech including resolved harmonics. The neural representation of the temporal fine structure reflected by the FFRTFS was assessed by the PLV at the harmonic near to F1 (4th of F0). The PLV at harmonic near to F1 (4th of F0) of FFRTFS to resolved harmonics was significantly larger than to unresolved harmonics. Spearman's correlation showed that the FFRENV F0 strength to unresolved harmonics was correlated with tone identification performance in noise (0 dB SNR). These results showed that the FFRENV F0 strength to speech sounds with resolved harmonics was not affected by noise. In contrast, the response to speech sounds with unresolved harmonics, which were significantly smaller in noise compared to quiet. Our results suggest that coding resolved harmonics was more important than coding envelope for tone identification performance in noise.

Multimedia-based Presentation: Exploring Interactivity Emergence in Academic Speaking Classroom

<p>The understanding of knowledge transfer and information delivery is recently in the broader scope due to the development of educational technology. The information delivery is not merely done using verbal message; however, multiple modes of presentation such as verbal and auditory representation can also be the alternative of material delivery. The studies featuring the use of multimedia-based presentation are mostly administered to determine its effectiveness to be implemented in the learning process. In contrast, the exploration of the use of multimedia-based presentation toward the way how it can be a means of interaction seems underexplored. Therefore, to make it be more precise, the present study attempts to explore the practice of multimedia-based presentation in academic speaking classroom and to investigate the interactivity emerged during the presentation process. This study deployed qualitative case study design due to the purpose of gaining the in-depth investigation of the use of multimedia-based presentation and its interactivity emergence in academic speaking classroom. The study was conducted in one of the universities in Surakarta majoring English Education where academic speaking becomes one of the subjects. The presentation document along with the presentation process were analyzed in this study. The results of the analysis show that multimedia-based presentation is designed to visualize the material being conveyed through the icons, pictures, and illustrations that are able to represent the information or knowledge in a more concrete way. Moreover, the interactivity is also emerged through the use of multimedia-based presentation as it simplifies the presenter to do the following: gesturing, dialoguing, and describing. The results implies the opportunity for both teachers and academic speaking presenters to innovate how they present the material by using multimedia-based presentation. In practice, multimedia-based presentation along with its interactivity can clarify the materials, grab the audience attention, and stimulate the audience responses.</p>

Sonification of network traffic flow for monitoring and situational awareness.

Maintaining situational awareness of what is happening within a computer network is challenging, not only because the behaviour happens within machines, but also because data traffic speeds and volumes are beyond human ability to process. Visualisation techniques are widely used to present information about network traffic dynamics. Although they provide operators with an overall view and specific information about particular traffic or attacks on the network, they often still fail to represent the events in an understandable way. Also, because they require visual attention they are not well suited to continuous monitoring scenarios in which network administrators must carry out other tasks. Here we present SoNSTAR (Sonification of Networks for SiTuational AwaReness), a real-time sonification system for monitoring computer networks to support network administrators’ situational awareness. SoNSTAR provides an auditory representation of all the TCP/IP traffic within a network based on the different traffic flows between between network hosts. A user study showed that SoNSTAR raises situational awareness levels by enabling operators to understand network behaviour and with the benefit of lower workload demands (as measured by the NASA TLX method) than visual techniques. SoNSTAR identifies network traffic features by inspecting the status flags of TCP/IP packet headers. Combinations of these features define particular traffic events which are mapped to recorded sounds to generate a soundscape that represents the real-time status of the network traffic environment. The sequence, timing, and loudness of the different sounds allow the network to be monitored and anomalous behaviour to be detected without the need to continuously watch a monitor screen.

The Formation and Cultivation of the Inner Hearing of Music

Music is the art of hearing, and inner hearing is the core of music hearing. Inner hearing is formed, developed and perfected gradually with all kinds of musical practical skills, and its implicit operation process is achieved in the cooperative activities of external sound stimulation and internal auditory experience. The cultivation of inner hearing is one of the primary parts of music skill learning. Developing the acoustic perception image and perfecting music auditory representation should be the first principle to cultivate the inner hearing.

Current Use and Future Perspectives of Spatial Audio Technologies in Electronic Travel Aids

Electronic travel aids (ETAs) have been in focus since technology allowed designing relatively small, light, and mobile devices for assisting the visually impaired. Since visually impaired persons rely on spatial audio cues as their primary sense of orientation, providing an accurate virtual auditory representation of the environment is essential. This paper gives an overview of the current state of spatial audio technologies that can be incorporated in ETAs, with a focus on user requirements. Most currently available ETAs either fail to address user requirements or underestimate the potential of spatial sound itself, which may explain, among other reasons, why no single ETA has gained a widespread acceptance in the blind community. We believe there is ample space for applying the technologies presented in this paper, with the aim of progressively bridging the gap between accessibility and accuracy of spatial audio in ETAs.

Hearing Shapes: Event-related Potentials Reveal the Time Course of Auditory–Visual Sensory Substitution

In auditory-visual sensory substitution, visual information (e.g., shape) can be extracted through strictly auditory input (e.g., soundscapes). Previous studies have shown that image-to-sound conversions that follow simple rules [such as the Meijer algorithm; Meijer, P. B. L. An experimental system for auditory image representation. Transactions on Biomedical Engineering, 39, 111-121, 1992] are highly intuitive and rapidly learned by both blind and sighted individuals. A number of recent fMRI studies have begun to explore the neuroplastic changes that result from sensory substitution training. However, the time course of cross-sensory information transfer in sensory substitution is largely unexplored and may offer insights into the underlying neural mechanisms. In this study, we recorded ERPs to soundscapes before and after sighted participants were trained with the Meijer algorithm. We compared these posttraining versus pretraining ERP differences with those of a control group who received the same set of 80 auditory/visual stimuli but with arbitrary pairings during training. Our behavioral results confirmed the rapid acquisition of cross-sensory mappings, and the group trained with the Meijer algorithm was able to generalize their learning to novel soundscapes at impressive levels of accuracy. The ERP results revealed an early cross-sensory learning effect (150-210 msec) that was significantly enhanced in the algorithm-trained group compared with the control group as well as a later difference (420-480 msec) that was unique to the algorithm-trained group. These ERP modulations are consistent with previous fMRI results and provide additional insight into the time course of cross-sensory information transfer in sensory substitution.

Musical Expectations Enhance Auditory Cortical Processing in Musicians: A Magnetoencephalography Study

The present study investigated the influence of musical expectations on auditory representations in musicians and non-musicians using magnetoencephalography (MEG). Neuroscientific studies have demonstrated that musical syntax is processed in the inferior frontal gyri, eliciting an early right anterior negativity (ERAN), and anatomical evidence has shown that interconnections occur between the frontal cortex and the belt and parabelt regions in the auditory cortex (AC). Therefore, we anticipated that musical expectations would mediate neural activities in the AC via an efferent pathway. To test this hypothesis, we measured the auditory-evoked fields (AEFs) of seven musicians and seven non-musicians while they were listening to a five-chord progression in which the expectancy of the third chord was manipulated (highly expected, less expected, and unexpected). The results revealed that highly expected chords elicited shorter N1m (negative AEF at approximately 100 ms) and P2m (positive AEF at approximately 200 ms) latencies and larger P2m amplitudes in the AC than less-expected and unexpected chords. The relations between P2m amplitudes/latencies and harmonic expectations were similar between the groups; however, musicians’ results were more remarkable than those of non-musicians. These findings suggest that auditory cortical processing is enhanced by musical knowledge and long-term training in a top-down manner, which is reflected in shortened N1m and P2m latencies and enhanced P2m amplitudes in the AC.

Extensive Tonotopic Mapping across Auditory Cortex Is Recapitulated by Spectrally Directed Attention and Systematically Related to Cortical Myeloarchitecture.

Auditory selective attention is vital in natural soundscapes. But it is unclear how attentional focus on the primary dimension of auditory representation—acoustic frequency—might modulate basic auditory functional topography during active listening. In contrast to visual selective attention, which is supported by motor-mediated optimization of input across saccades and pupil dilation, the primate auditory system has fewer means of differentially sampling the world. This makes spectrally-directed endogenous attention a particularly crucial aspect of auditory attention. Using a novel functional paradigm combined with quantitative MRI, we establish in male and female listeners that human frequency-band-selective attention drives activation in both myeloarchitectonically estimated auditory core, and across the majority of tonotopically mapped nonprimary auditory cortex. The attentionally driven best-frequency maps show strong concordance with sensory-driven maps in the same subjects across much of the temporal plane, with poor concordance in areas outside traditional auditory cortex. There is significantly greater activation across most of auditory cortex when best frequency is attended, versus ignored; the same regions do not show this enhancement when attending to the least-preferred frequency band. Finally, the results demonstrate that there is spatial correspondence between the degree of myelination and the strength of the tonotopic signal across a number of regions in auditory cortex. Strong frequency preferences across tonotopically mapped auditory cortex spatially correlate with R1-estimated myeloarchitecture, indicating shared functional and anatomical organization that may underlie intrinsic auditory regionalization.SIGNIFICANCE STATEMENT Perception is an active process, especially sensitive to attentional state. Listeners direct auditory attention to track a violin's melody within an ensemble performance, or to follow a voice in a crowded cafe. Although diverse pathologies reduce quality of life by impacting such spectrally directed auditory attention, its neurobiological bases are unclear. We demonstrate that human primary and nonprimary auditory cortical activation is modulated by spectrally directed attention in a manner that recapitulates its tonotopic sensory organization. Further, the graded activation profiles evoked by single-frequency bands are correlated with attentionally driven activation when these bands are presented in complex soundscapes. Finally, we observe a strong concordance in the degree of cortical myelination and the strength of tonotopic activation across several auditory cortical regions.

Early visual deprivation prompts the use of body-centered frames of reference for auditory localization

The effects of early visual deprivation on auditory spatial processing are controversial. Results from recent psychophysical studies show that people who were born blind have a spatial impairment in localizing sound sources within specific auditory settings, while previous psychophysical studies revealed enhanced auditory spatial abilities in early blind compared to sighted individuals. An explanation of why an auditory spatial deficit is sometimes observed within blind populations and its task-dependency remains to be clarified. We investigated auditory spatial perception in early blind adults and demonstrated that the deficit derives from blind individual’s reduced ability to remap sound locations using an external frame of reference. We found that performance in blind population was severely impaired when they were required to localize brief auditory stimuli with respect to external acoustic landmarks (external reference frame) or when they had to reproduce the spatial distance between two sounds. However, they performed similarly to sighted controls when had to localize sounds with respect to their own hand (body-centered reference frame), or to judge the distances of sounds from their finger. These results suggest that early visual deprivation and the lack of visual contextual cues during the critical period induce a preference for body-centered over external spatial auditory representations.

On the Perceptual Subprocess of Absolute Pitch.

Absolute pitch (AP) is the rare ability of musicians to identify the pitch of tonal sound without external reference. While there have been behavioral and neuroimaging studies on the characteristics of AP, how the AP is implemented in human brains remains largely unknown. AP can be viewed as comprising of two subprocesses: perceptual (processing auditory input to extract a pitch chroma) and associative (linking an auditory representation of pitch chroma with a verbal/non-verbal label). In this review, we focus on the nature of the perceptual subprocess of AP. Two different models on how the perceptual subprocess works have been proposed: either via absolute pitch categorization (APC) or based on absolute pitch memory (APM). A major distinction between the two views is that whether the AP uses unique auditory processing (i.e., APC) that exists only in musicians with AP or it is rooted in a common phenomenon (i.e., APM), only with heightened efficiency. We review relevant behavioral and neuroimaging evidence that supports each notion. Lastly, we list open questions and potential ideas to address them.

Cascaded Amplitude Modulations in Sound Texture Perception.

Sound textures, such as crackling fire or chirping crickets, represent a broad class of sounds defined by their homogeneous temporal structure. It has been suggested that the perception of texture is mediated by time-averaged summary statistics measured from early auditory representations. In this study, we investigated the perception of sound textures that contain rhythmic structure, specifically second-order amplitude modulations that arise from the interaction of different modulation rates, previously described as “beating” in the envelope-frequency domain. We developed an auditory texture model that utilizes a cascade of modulation filterbanks that capture the structure of simple rhythmic patterns. The model was examined in a series of psychophysical listening experiments using synthetic sound textures—stimuli generated using time-averaged statistics measured from real-world textures. In a texture identification task, our results indicated that second-order amplitude modulation sensitivity enhanced recognition. Next, we examined the contribution of the second-order modulation analysis in a preference task, where the proposed auditory texture model was preferred over a range of model deviants that lacked second-order modulation rate sensitivity. Lastly, the discriminability of textures that included second-order amplitude modulations appeared to be perceived using a time-averaging process. Overall, our results demonstrate that the inclusion of second-order modulation analysis generates improvements in the perceived quality of synthetic textures compared to the first-order modulation analysis considered in previous approaches.

Pitch-trained neural networks replicate properties of human pitch perception

Pitch perception exhibits well-established dependencies on stimulus characteristics. In human listeners, pitch is predominantly determined by low-numbered harmonics that are believed to be resolved by the cochlea, but is also conveyed to a lesser extent by high-numbered harmonics that are not individually resolved. Moreover, the relative phases of harmonics influence pitch perception only when the stimulus consists of high-numbered (unresolved) harmonics. To assess whether these dependencies reflect information constraints on the problem of extracting pitch from peripheral auditory representations, we trained a convolutional neural network to estimate the fundamental frequency (F0) of tones in noise. The tones replicated basic spectral properties of natural sounds. The resulting neural network model was tested on complex tones that varied in harmonic composition and phase (as in classic psychoacoustic experiments). The network qualitatively replicated features of human pitch perception, including better performance for tones containing low-numbered harmonics and worse performance when the phases of high-numbered (presumably unresolved) harmonics were randomized. The results are consistent with the notion that human pitch perception exhibits near-optimal performance characteristics for the task of estimating F0 from peripheral auditory representations, in that optimizing for performance of this task is sufficient to reproduce human performance characteristics.