Auditory Representation Research Articles

A multi-second adaptive integration process underlies sound texture perception

Temporally homogenous sound textures—as produced by rain, fire or insect swarms—are thought to be represented with time-average statistics measured from early auditory representations. We explored the averaging process involved in sound texture perception using “texture steps”—stimuli whose statistics changed at some point in time. We reasoned that judgments should be biased by the stimulus history included in the averaging process. Listeners were presented with two sounds, a texture step and a probe texture (with constant statistics). Listeners were asked to compare the endpoint of the texture step to the probe and to select the sound that was most similar to a familiar reference texture. Judgments were biased by the presence of a step 1 second prior to the stimulus endpoint, but the bias was reduced when the step occurred 2.5 seconds from the endpoint. In addition, the bias was substantially larger for textures whose statistics were less homogenous (ocean waves) than textures whose statistics were more homogenous (rain). The results suggest a texture integration process operating over several seconds, but whose integration window is adapted to the homogeneity of the sound signal, averaging over longer periods of time for more variable textures.

Melodic Heating as a Basic Ability in the formation of Musical Intonation Thinking

The article highlights the problem of the significance of such basic skills as melodic hearing in the musical intonation thinking. Analyzed various theoretical and methodological approaches to the development of melodic hearing. It showed how this sense is important in music teaching and creative activities.Analyzing the formation of musical intonation thinking most researchers focusing primarily on its individual components (perception, reflection, creativity). They prove their priority in the process of thinking. However, the complex mechanism of musical thinking could not exist without some basic music skills, such as melodic hearing (intonation hearing), sense of rhythm and memory. In the process of musical intonation thought advisable to rely on the methodicalprovisions on melodic development of intonation hearing. Among them: education system and sense modal connections sounds; learning and auditory representations fixing elements of musical language; formation good singers skills; ability to operate hearing ideas for using them in singing; ability to listen and assess the quality of singing. It is proved that developed intonation hearing depends on the availability of internal hearing, the ability to anticipate sound, music and operate auditory representations. Due to the internal hearing musicians represent mentally sound before performance, and during performance, which allows to control the actual sound.

Multimodal representations of person identity individuated with fMRI

Recognizing the identity of a person is fundamental to guide social interactions. We can recognize the identity of a person looking at her face, but also listening to her voice. An important question concerns how visual and auditory information come together, enabling us to recognize identity independently of the modality of the stimulus. This study reports converging evidence across univariate contrasts and multivariate classification showing that the posterior superior temporal sulcus (pSTS), previously known to encode polymodal visual and auditory representations, encodes information about person identity with invariance within and across modality. In particular, pSTS shows selectivity for faces, selectivity for voices, classification of face identity across image transformations within the visual modality, and classification of person identity across modality.

A deafening flash! Visual interference of auditory signal detection.

In some people, visual stimulation evokes auditory sensations. How prevalent and how perceptually real is this? 22% of our neurotypical adult participants responded 'Yes' when asked whether they heard faint sounds accompanying flash stimuli, and showed significantly better ability to discriminate visual 'Morse-code' sequences. This benefit might arise from an ability to recode visual signals as sounds, thus taking advantage of superior temporal acuity of audition. In support of this, those who showed better visual relative to auditory sequence discrimination also had poorer auditory detection in the presence of uninformative visual flashes, though this was independent of awareness of visually-evoked sounds. Thus a visually-evoked auditory representation may occur subliminally and disrupt detection of real auditory signals. The frequent natural correlation between visual and auditory stimuli might explain the surprising prevalence of this phenomenon. Overall, our results suggest that learned correspondences between strongly correlated modalities may provide a precursor for some synaesthetic abilities.

Compensating Level-Dependent Frequency Representation in Auditory Cortex by Synaptic Integration of Corticocortical Input.

Robust perception of auditory objects over a large range of sound intensities is a fundamental feature of the auditory system. However, firing characteristics of single neurons across the entire auditory system, like the frequency tuning, can change significantly with stimulus intensity. Physiological correlates of level-constancy of auditory representations hence should be manifested on the level of larger neuronal assemblies or population patterns. In this study we have investigated how information of frequency and sound level is integrated on the circuit-level in the primary auditory cortex (AI) of the Mongolian gerbil. We used a combination of pharmacological silencing of corticocortically relayed activity and laminar current source density (CSD) analysis. Our data demonstrate that with increasing stimulus intensities progressively lower frequencies lead to the maximal impulse response within cortical input layers at a given cortical site inherited from thalamocortical synaptic inputs. We further identified a temporally precise intercolumnar synaptic convergence of early thalamocortical and horizontal corticocortical inputs. Later tone-evoked activity in upper layers showed a preservation of broad tonotopic tuning across sound levels without shifts towards lower frequencies. Synaptic integration within corticocortical circuits may hence contribute to a level-robust representation of auditory information on a neuronal population level in the auditory cortex.

Response Behavior in a Video-Web Survey: A Mode Comparison Study

Intrigued by the possibilities of improving the data quality of web surveys by incorporating human-like features, we developed a video-web survey for this study. This paper describes an experiment that compares response behavior in the video-web mode to traditional web and interviewer-administered surveys. The disclosure of sensitive information and respondents’ engagement were examined. Overall, despite the visual and auditory representation of a human interviewer in the video-web mode, video-web seems to have been experienced by respondents much like a traditional web survey. Based on these results, we argue that for human-like features to fully increase the level of engagement it would require the inclusion of responsiveness. However, researchers should be aware of possible social presence effects that may arise when creating web surveys with responsive human-like features.

Use of a sonification system for science learning by people who are blind

Purpose The purpose of this paper is to examine a central need of students who are blind: the ability to access science curriculum content. Design/methodology/approach Agent-based modeling is a relatively new computational modeling paradigm that models complex dynamic systems. NetLogo is a widely used agent-based modeling language that enables exploration and construction of models of complex systems by programming and running the rules and behaviors. Sonification of variables and events in an agent-based NetLogo computer model of gas in a container is used to convey phenomena information. This study examined mainly two research topics: the scientific conceptual knowledge and systems reasoning that were learned as a result of interaction with the listen-to-complexity (L2C) environment as appeared in answers to the pre- and post-tests and the learning topics of kinetic molecular theory of gas in chemistry that was learned as a result of interaction with the L2C environment. The case study research focused on A., a woman who is adventitiously blind, for eight sessions. Findings The participant successfully completed all curricular assignments; her scientific conceptual knowledge and systems reasoning became more specific and aligned with scientific knowledge. Practical implications A practical implication of further studies is that they are likely to have an impact on the accessibility of learning materials, especially in science education for students who are blind, as equal access to low-cost learning environments that are equivalent to those used by sighted users would support their inclusion in the K-12 academic curriculum. Originality/value The innovative and low-cost learning system that is used in this research is based on transmittal of visual information of dynamic and complex systems, providing perceptual compensation by harnessing auditory feedback. For the first time the L2C system is based on sound that represents a dynamic rather than a static array. In this study, the authors explore how a combination of several auditory representations may affect cognitive learning ability.

Crossmodal interference between language and flavour

This work describes crossmodal Stroop interference in flavoural, visual, and auditory representations. A mixed design was used with two randomized groups. As a between-subjects factor, words were presented in visual (group 1) or auditory (group 2) forms. Stimulus congruency (congruent, incongruent, and control) was defined as a within-subjects factor. Reaction times and the number of correct answers were recorded. The results showed a minor crossmodal Stroop interference in conditions of congruency. In contrast, with incongruent and control stimuli, reaction times increased and accuracy rates diminished in both experimental groups. Data from the two groups were compared, and it was concluded that the interference was greater when the distractor was written than when it was spoken. These results are discussed in terms of the difficulty of visual linguistic representation and in relation to previous studies.

Auditory spatial representations of the world are compressed in blind humans

Compared to sighted listeners, blind listeners often display enhanced auditory spatial abilities such as localization in azimuth. However, less is known about whether blind humans can accurately judge distance in extrapersonal space using auditory cues alone. Using virtualization techniques, we show that auditory spatial representations of the world beyond the peripersonal space of blind listeners are compressed compared to those for normally sighted controls. Blind participants overestimated the distance to nearby sources and underestimated the distance to remote sound sources, in both reverberant and anechoic environments, and for speech, music, and noise signals. Functions relating judged and actual virtual distance were well fitted by compressive power functions, indicating that the absence of visual information regarding the distance of sound sources may prevent accurate calibration of the distance information provided by auditory signals.

Neurodynamics of visual and auditory scene size representations

Neurodynamics of visual and auditory scene size representations

Brainwaves in concert: the 20th century sonification of the electroencephalogram.

Over the past decade, sonification, the transformation of data into sound, has received considerable attention from the scientific community. While the sense of vision continues to predominate the hierarchy of senses, auditory data representation has been increasingly recognized as a legitimate technique to complement existing modes of data display (Supper, 2012). Sonification is, however, not an exclusively scientific endeavour; the technique has been commonly applied within the domain of experimental music. This poses a challenge to the field, as some argue for a sharper distinction between scientific and artistic sonification, whereas others proclaim openness to both sides of the science–art spectrum (Supper, 2012). The interplay between science and art is beautifully demonstrated by the sonification of the EEG, a practice that was first described in the early 1930s and subsequently gave rise to a variety of medical and artistic applications. In neurophysiology, sonification was used to complement visual EEG analysis, which had become increasingly complex by the middle of the 20th century. In experimental music, the encounter between physicist Edmond Dewan (1931–2009) and composer Alvin Lucier (b. 1931) inspired the first brainwave composition Music for the solo performer (1965). By the end of the century, advances in EEG and sound technology ultimately gave rise to brain–computer music interfaces (BCMIs), a multidisciplinary achievement that has enhanced expressive abilities of both patients and artists (Miranda, 2014). In this article, we aim to place the sonification of the EEG in its historical context, thereby seeking to draw attention to the previously underexposed scientific technique of sonification, but also to illustrate the way in which the domains of science and art may, perhaps more than occasionally, enhance one another. In 1929, the German psychiatrist Hans Berger (1873–1941) published his first report on the human EEG, in which he described a method for recording …

Mapping the cortical representation of speech sounds in a syllable repetition task

Speech repetition relies on a series of distributed cortical representations and functional pathways. A speaker must map auditory representations of incoming sounds onto learned speech items, maintain an accurate representation of those items in short-term memory, interface that representation with the motor output system, and fluently articulate the target sequence. A “dorsal stream” consisting of posterior temporal, inferior parietal and premotor regions is thought to mediate auditory-motor representations and transformations, but the nature and activation of these representations for different portions of speech repetition tasks remains unclear. Here we mapped the correlates of phonetic and/or phonological information related to the specific phonemes and syllables that were heard, remembered, and produced using a series of cortical searchlight multi-voxel pattern analyses trained on estimates of BOLD responses from individual trials. Based on responses linked to input events (auditory syllable presentation), predictive vowel-level information was found in the left inferior frontal sulcus, while syllable prediction revealed significant clusters in the left ventral premotor cortex and central sulcus and the left mid superior temporal sulcus. Responses linked to output events (the GO signal cueing overt production) revealed strong clusters of vowel-related information bilaterally in the mid to posterior superior temporal sulcus. For the prediction of onset and coda consonants, input-linked responses yielded distributed clusters in the superior temporal cortices, which were further informative for classifiers trained on output-linked responses. Output-linked responses in the Rolandic cortex made strong predictions for the syllables and consonants produced, but their predictive power was reduced for vowels. The results of this study provide a systematic survey of how cortical response patterns covary with the identity of speech sounds, which will help to constrain and guide theoretical models of speech perception, speech production, and phonological working memory.

The Sound of Vision Project: On the Feasibility of an Audio-Haptic Representation of the Environment, for the Visually Impaired.

The Sound of Vision project involves developing a sensory substitution device that is aimed at creating and conveying a rich auditory representation of the surrounding environment to the visually impaired. However, the feasibility of such an approach is strongly constrained by neural flexibility, possibilities of sensory substitution and adaptation to changed sensory input. We review evidence for such flexibility from various perspectives. We discuss neuroplasticity of the adult brain with an emphasis on functional changes in the visually impaired compared to sighted people. We discuss effects of adaptation on brain activity, in particular short-term and long-term effects of repeated exposure to particular stimuli. We then discuss evidence for sensory substitution such as Sound of Vision involves, while finally discussing evidence for adaptation to changes in the auditory environment. We conclude that sensory substitution enterprises such as Sound of Vision are quite feasible in light of the available evidence, which is encouraging regarding such projects.

A Phoneme Perception Test Method for High-Frequency Hearing Aid Fitting.

Outcomes with hearing aids (HAs) can be assessed using various speech tests, but many tests are not sensitive to changes in high-frequency audibility. A Phoneme Perception Test (PPT), designed for the phonemes /s/ and /ʃ/, has been developed to investigate whether detection and recognition tasks are able to measure individual differences in phoneme audibility and recognition for various hearing instrument settings. These capabilities were studied using two different fricative stimulus materials. The first set of materials preserves natural low-level sound components in the low- and mid-frequency ranges (LF set); the second set of materials attempts to limit the audibility to high-frequency fricative noise (nLF set). To study the effect on phoneme detection and recognition when auditory representations of /s/ and /ʃ/ are modified, a too strong nonlinear frequency compression (NLFC) setting was applied. Repeated measure design was used under several different conditions. A total of 31 hearing-impaired individuals participated in this study. Of the 31 participants, 10 individuals did not own HAs but were provided with them during the study and 21 individuals owned HAs and were experienced users. All participants had a symmetrical sensorineural hearing loss. The present study applied a phoneme detection test and a recognition test with two different stimulus sets under different amplification conditions. The statistical analysis focused on the capability of the PPT to measure the effect on audibility and perception of high-frequency information with and without HAs, and between HAs with two different NLFC settings ("default" and "too strong"). Detection thresholds (DTs) and recognition thresholds (RTs) were compared with respective audiometric thresholds in the free field for all available conditions. Significant differences in thresholds between LF and nLF stimuli were observed. The thresholds for nLF stimuli showed higher correlation to the corresponding audiometric thresholds than the thresholds for LF stimuli. The difference in thresholds for unaided and aided conditions was larger for the stimulus set nLF than for the stimulus set LF. Also, thresholds were similar in both aided conditions for stimulus set LF, whereas a large difference between amplifications was observed for the stimulus set nLF. When NLFC was set "too strong," DTs and RTs differed significantly for /s/. The findings from this study strongly suggest that measuring DTs and RTs with the stimulus set nLF is beneficial and useful to quantify the effects of HAs and NLFC on high-frequency speech cues for detection and recognition tasks. The findings also suggest that both tests are necessary because they assess audibility as well as recognition abilities, particularly as they relate to speech modification algorithms. The experiments conducted in this study did not allow for any acclimatization of the participants to increased high-frequency gain or NLFC. Further investigations should therefore examine the impact on DTs and RTs in the PPT as well as the contrasting effects of strong setting of NLFC to DTs and RTs because of (re)learning of modified auditory representations of /s/ and /ʃ/ as caused by NLFC.

Visual and Auditory Representation of Sentiment Classified Data Using SVM

Visual and Auditory Representation of Sentiment Classified Data Using SVM

Analysis of available and listener-received phonetic information in babble-masked consonant identification

Speech communication commonly occurs in the presence of multiple, non-target talkers. Previous work has shown that the amount of glimpsed target speech is a good predictor of overall intelligibility of babble-masked speech (Brungart et al., 2006, JASA) and that an automatic speech recognition system trained on glimpses closely approximates listener accuracy as a function of the number of babble talkers (Cooke, 2006, JASA). The present work uses a number of machine learning models to analyze the auditory information available in babble-masked speech and in available glimpses of babble-masked speech. Regularized Linear Discriminant, Support Vector Machine, and Naive Bayes classifiers were fit to modeled auditory representations of babble-masked CV syllables (with C = t, d, s, z, and V = a). The machine learning models outperformed human listeners substantially (>70% versus 54% accuracy, respectively). Analysis of predicted confusion patterns indicates that the Naive Bayes model most closely approximates human error patterns. The effects of variation in the local and absolute thresholds for glimpse calculation are explored with respect to overall accuracy and error pattern prediction in these machine learning models.

Auditory lateralisation deficits in neglect patients

Although visual deficits due to unilateral spatial neglect (USN) have been frequently described in the literature, fewer studies have been interested in directional hearing impairment in USN. The aim of this study was to explore sound lateralisation deficits in USN. Using a paradigm inspired by Tanaka et al. (1999), interaural time differences (ITD) were presented over headphones to give the illusion of a leftward or a rightward movement of sound. Participants were asked to respond “right” and “left” as soon as possible to indicate whether they heard the sound moving to the right or to the left side of the auditory space. We additionally adopted a single-case method to analyse the performance of 15 patients with right-hemisphere (RH) stroke and added two additional measures to underline sound lateralisation on the left side and on the right side. We included 15 patients with RH stoke (5 with a severe USN, 5 with a mild USN and 5 without USN) and 11 healthy age-matched participants. We expected to replicate findings of abnormal sound lateralisation in USN. However, although a sound lateralisation deficit was observed in USN, two different deficit profiles were identified. Namely, patients with a severe USN seemed to have left sound lateralisation impairment whereas patients with a mild USN seemed to be more influenced by a systematic bias in auditory representation with respect to body meridian axis (egocentric deviation). This latter profile was unexpected as sounds were manipulated with ITD and, thus, would not be perceived as coming from an external source of the head. Future studies should use this paradigm in order to better understand these two distinct profiles.

The auditory representation of speech sounds in human motor cortex

In humans, listening to speech evokes neural responses in the motor cortex. This has been controversially interpreted as evidence that speech sounds are processed as articulatory gestures. However, it is unclear what information is actually encoded by such neural activity. We used high-density direct human cortical recordings while participants spoke and listened to speech sounds. Motor cortex neural patterns during listening were substantially different than during articulation of the same sounds. During listening, we observed neural activity in the superior and inferior regions of ventral motor cortex. During speaking, responses were distributed throughout somatotopic representations of speech articulators in motor cortex. The structure of responses in motor cortex during listening was organized along acoustic features similar to auditory cortex, rather than along articulatory features as during speaking. Motor cortex does not contain articulatory representations of perceived actions in speech, but rather, represents auditory vocal information.

Auditory scene analysis and sonified visual images. Does consonance negatively impact on object formation when using complex sonified stimuli?

A critical task for the brain is the sensory representation and identification of perceptual objects in the world. When the visual sense is impaired, hearing and touch must take primary roles and in recent times compensatory techniques have been developed that employ the tactile or auditory system as a substitute for the visual system. Visual-to-auditory sonifications provide a complex, feature-based auditory representation that must be decoded and integrated into an object-based representation by the listener. However, we don’t yet know what role the auditory system plays in the object integration stage and whether the principles of auditory scene analysis apply. Here we used coarse sonified images in a two-tone discrimination task to test whether auditory feature-based representations of visual objects would be confounded when their features conflicted with the principles of auditory consonance. We found that listeners (N = 36) performed worse in an object recognition task when the auditory feature-based representation was harmonically consonant. We also found that this conflict was not negated with the provision of congruent audio–visual information. The findings suggest that early auditory processes of harmonic grouping dominate the object formation process and that the complexity of the signal, and additional sensory information have limited effect on this.

Response properties of neurons in the cat’s putamen during auditory discrimination

The striatum integrates diverse convergent input and plays a critical role in the goal-directed behaviors. To date, the auditory functions of striatum are less studied. Recently, it was demonstrated that auditory cortico-striatal projections influence behavioral performance during a frequency discrimination task. To reveal the functions of striatal neurons in auditory discrimination, we recorded the single-unit spike activities in the putamen (dorsal striatum) of free-moving cats while performing a Go/No-go task to discriminate the sounds with different modulation rates (12.5Hz vs. 50Hz) or envelopes (damped vs. ramped). We found that the putamen neurons can be broadly divided into four groups according to their contributions to sound discrimination. First, 40% of neurons showed vigorous responses synchronized to the sound envelope, and could precisely discriminate different sounds. Second, 18% of neurons showed a high preference of ramped to damped sounds, but no preference for modulation rate. They could only discriminate the change of sound envelope. Third, 27% of neurons rapidly adapted to the sound stimuli, had no ability of sound discrimination. Fourth, 15% of neurons discriminated the sounds dependent on the reward-prediction. Comparing to passively listening condition, the activities of putamen neurons were significantly enhanced by the engagement of the auditory tasks, but not modulated by the cat’s behavioral choice. The coexistence of multiple types of neurons suggests that the putamen is involved in the transformation from auditory representation to stimulus-reward association.