Auditory Modality Research Articles

Music Valence Can Affect Dichotic Listening Performance

Background and Aim: Music can regulate the activity of brain structures that play a significant role in emotions. The perceived emotion techniques such as dichotic listening clarify the relationships between auditory emotional stimuli and hemispheric asymmetries in the auditory modality. We examined the impact of pleasantness/unpleasantness of music by Acceptable Noise Level (ANL) as a subjective measure of listeners’ willingness to accept background noise. Methods: In this study, 32 participants rated their pleasantness with about ten songs; then, we considered the effect of preferred pleasant and unpleasant music on dichotic music listening and acceptable background noise. There were six forced attention conditions to calculate ANL, followed by measuring the most comfortable level and background noise level for each condition. Results: The pairwise comparison analyses revealed significantly higher ANL in forced attention to pleasant music than to speech (p<0.004) and unpleasant music to the left ear (p≤0.05). The mean ANLs difference in 2 groups of right ear advantage and left ear advantage showed significant intra-hemispheric differences in the forced pleasant music attention than the forced unpleasant music attention conditions (p<0.007), and forced speech conditions (p=0.001), only in the left ear advantage group. In addition, the interaction between conditions and groups showed interhemispheric asymmetry. Conclusion: Music valence and intra- and interhemispheric differences can affect the ANL dichotic processing and, consequently, lower noise tolerance (higher ANL) in forced pleasant music attention conditions. Keywords: Acceptable noise level; dichotic listening test; music valence; functional laterality

Travelogue as Supplementary Learning Tool Towards Students’ Historical Thinking Skills

The study utilized experimental-developmental quantitative research design through the use of teacher-made and expert-validated survey questionnaire, pretest and posttest to find the effects of researcher-designed travelogue as a supplementary learning tool in improving the historical thinking skills of fifty (50) Grade 7 students in a public high school in San Pablo City. Results showed that as to the learners’ attitude toward learning styles, visual, auditory, and kinesthetic modalities sometimes apply to them. As to the level of interest, the students were generally interested in the topics for Araling Panlipunan 7. Scores of the respondents as to historical thinking skills were generally also shifted from the satisfactory, very satisfactory ranges to the outstanding range. This is true for sourcing, corroborating, contextualizing, close reading and explicit instruction. Statistical tests for relationships and differences also reflected significant difference on the historical thinking skills of the learners implying that the use of travelogue has improved the skills of the learners and only visual related to explicit instruction and auditory to corroborating. This concludes that the attitudes of the learners toward learning styles partially relate to some of the historical thinking skills.

Modality, presentation, domain and training effects in statistical learning

While several studies suggest that the nature and properties of the input have significant effects on statistical learning, they have rarely been investigated systematically. In order to understand how input characteristics and their interactions impact statistical learning, we explored the effects of modality (auditory vs. visual), presentation type (serial vs. simultaneous), domain (linguistic vs. non-linguistic), and training type (random, starting small, starting big) on artificial grammar learning in young adults (N = 360). With serial presentation of stimuli, learning was more effective in the auditory than in the visual modality. However, with simultaneous presentation of visual and serial presentation of auditory stimuli, the modality effect was not present. We found a significant domain effect as well: a linguistic advantage over nonlinguistic material, which was driven by the domain effect in the auditory modality. Overall, the auditory linguistic condition had an advantage over other modality-domain types. Training types did not have any overall effect on learning; starting big enhanced performance only in the case of serial visual presentation. These results show that input characteristics such as modality, presentation type, domain and training type influence statistical learning, and suggest that their effects are also dependent on the specific stimuli and structure to be learned.

Neuroanatomical correlates of macrostructural receptive abilities in narrative discourse in unilateral left hemisphere stroke: A behavioural and voxel-based morphometry study.

Little is known about story retelling and comprehension abilities in groups with lower levels of education and socio-economic status (SES). A growing body of evidence suggests the role of an extended network supporting narrative comprehension, but few studies have been conducted in clinical populations, even less in developing countries. To extend our knowledge of the impact of a stroke on macrostructural aspects of discourse processes, namely main and complementary information, in individuals with middle-low to low SES and low levels of education. Relationships were tested between the performance in story retell and comprehension and reading and writing habits (RWH). Also, the associations between retelling and comprehension measures and their structural grey matter (GM) correlates were explored. A total of 17 adults with unilateral left hemisphere (LH) chronic ischaemic stroke without the presence of significant aphasia and 10 matched (age, education and SES) healthy controls (HC) participated in the study. Retell and comprehension tasks were performed after listening or reading narrative stories. Voxel-based morphometry (VBM) analysis was conducted on a subgroup of nine individuals with LH stroke and the 10 matched controls using structural magnetic resonance imaging (MRI). Retelling and comprehension abilities were not significantly different between LH and HC, nonetheless quantitively lower in LH. Exploratory correlations showed that retelling and comprehension abilities in both written and auditory modalities were correlated with naming abilities. At the neural level, written comprehension positively correlated with GM density of the LH, including areas in the temporal pole, superior and middle temporal gyrus as well as the orbitofrontal cortex, precentral and postcentral gyri. Auditory narrative comprehension was associated with GM density of the lingual gyrus in the right hemisphere. The present results suggest that retelling and comprehension of auditory and written narratives are relatively well-preserved in individuals with a LH stroke without significant aphasia, but poorer than in HC. The findings replicate previous studies conducted in groups with higher levels of education and SES both at the behavioural and neural levels. Considering that naming seems to be associated with narrative retell and comprehension in individuals with lower SES and education, this research provides evidence on the importance of pursuing further studies including larger samples with and without aphasia as well as with various SES and education levels. What is already known on this subject Story retell and comprehension of auditory and written discourse have been shown to be affected after stroke, but most studies have been conducted on individuals with middle to high SES and high educational levels. What this paper adds to existing knowledge The study reports on narrative retell and comprehension in both auditory and written modalities in groups of HC and individuals with LH brain damage, with low-to-middle SES and lower levels of education. What are the potential or actual clinical implications of this work? This study highlights the importance of taking into consideration the sociodemographic and RWH of patients when assessing discourse retell and comprehension in both auditory and written modalities. It also underlines the importance of including patients without significant aphasia following LH stroke to look at the effect of both stroke and aphasia on narrative comprehension and story retelling.

National aeronautics and space administration on IGTV: Multimodal discourses of space

This study employed critical analysis of multimodal texts to examine how NASA utilizes multiple modes like text, visuals, font, and sound to communicate space on a modern multimodal platform—Instagram. Four main themes emerged in the analysis of 34 NASA IGTV videos: space as a scientific aim, space as a scientific means, space as a U.S. territorialization effort, and space within the general public’s reach. Multimodal discursive practices enabled NASA to produce institutional discourses and disseminate meanings of space that enacted and reinforced gendered and racial exceptionalism, technological and economic space elitism, and hegemonic territorialization operationalized through space appropriation and commodification. While visuals, text, and auditory mode in many cases worked together to create meanings, the use of visuals communicated that space-related accomplishments are still achieved in white and male environments. Thus, the unpacking of multimodal discursive practices calls for more holistic approaches that address multimodal arrangements and an inclusive approach to the roles of individual modes in meaning-making.

Does emotion recognition change across phases of the ovulatory cycle?

Recognizing emotions is an essential ability for successful interpersonal interaction. Prior research indicates some links between the endocrine system and emotion recognition ability, but only a few studies focused on within-subject differences across distinct ovulatory cycle phases and this ability. These studies have demonstrated mixed results that might be potentially due to heterogeneity in experimental tasks, methodologies, and lacking ecological validity. In the current study, we investigated associations between within-subject differences in ovarian hormones levels and emotion recognition from auditory, visual, and audiovisual modalities in N = 131 naturally cycling participants across the late follicular and mid-luteal phase of the ovulatory cycle. We applied a within-subject design with sessions in the late follicular and mid-luteal cycle phase, and also assessed salivary progesterone and estradiol in these sessions. Our findings did not reveal any significant difference in emotion recognition ability across two cycle phases. Thus, they emphasize the necessity of employing large-scale replication studies with well-established study designs along with proper statistical analyses. Moreover, our findings indicate that the potential link between ovulatory cycle phases (late follicular and mid-luteal) and emotion recognition ability might have been overestimated in previous studies, and may contribute to theoretical and practical implications of socio-cognitive neuroendocrinology.

Perception Accuracy of a Multi-Channel Tactile Feedback System for Assistive Technology.

Assistive technology uses multi-modal feedback devices, focusing on the visual, auditory, and haptic modalities. Tactile devices provide additional information via touch sense. Perception accuracy of vibrations depends on the spectral and temporal attributes of the signal, as well as on the body parts they are attached to. The widespread use of AR/VR devices, wearables, and gaming interfaces requires information about the usability of feedback devices. This paper presents results of an experiment using an 8-channel tactile feedback system with vibrators placed on the wrists, arms, ankles, and forehead. Different vibration patterns were designed and presented using sinusoidal frequency bursts on 2, 4, and 8 channels. In total, 27 subjects reported their sensation formally and informally on questionnaires. Results indicate that 2 and 4 channels could be used simultaneously with high accuracy, and the transducers' optimal placement (best sensitivity) is on the wrists, followed by the ankles. Arm and head positions were inferior and generally inadequate for signal presentation. For optimal performance, signal length should exceed 500 ms. Furthermore, the amplitude level and temporal pattern of the presented signals have to be used for carrying information rather than the frequency of the vibration.

Action Postponing and Restraint Varies among Sensory Modalities.

Proactive inhibition is divided into two components: action postponing (AP), which refers to slowing the onset of response, and action restraint (AR), which refers to preventing the response. To date, several studies have reported alterations in proactive inhibition and its associated neural processing among sensory modalities; however, this remains inconclusive owing to several methodological issues. This study aimed to clarify the differences in AP and AR and their neural processing among visual, auditory, and somatosensory modalities using an appropriate experimental paradigm that can assess AP and AR separately. The postponing time calculated by subtracting simple reaction time from Go signal reaction time was shorter in the visual modality than in the other modalities. This was explained by faster neural processing for conflict monitoring induced by anticipating the presence of the No-go signal, supported by the shorter latency of AP-related N2. Furthermore, the percentage of false alarms, which is the reaction to No-go signals, was lower in the visual modality than in the auditory modality. This was attributed to higher neural resources for conflict monitoring induced by the presence of No-go signals, supported by the larger amplitudes of AR-related N2. Our findings revealed the differences in AP and AR and their neural processing among sensory modalities.

Temporal visual representation elicits early auditory-like responses in hearing but not in deaf individuals

It is evident that the brain is capable of large-scale reorganization following sensory deprivation, but the extent of such reorganization is to date, not clear. The auditory modality is the most accurate to represent temporal information, and deafness is an ideal clinical condition to study the reorganization of temporal representation when the audio signal is not available. Here we show that hearing, but not deaf individuals, show a strong ERP response to visual stimuli in temporal areas during a time-bisection task. This ERP response appears 50–90 ms after the flash and recalls some aspects of the N1 ERP component usually elicited by auditory stimuli. The same ERP is not evident for a visual space-bisection task, suggesting that the early recruitment of temporal cortex is specific for building a highly resolved temporal representation within the visual modality. These findings provide evidence that the lack of auditory input can interfere with typical development of complex visual temporal representations.

A systematic investigation reveals that Ishihara et al.’s (2008) STEARC effect only emerges when time is directly assessed

The Spatial–TEmporal Association of Response Codes (STEARC) effect (Ishihara et al. in Cortex 44:454–461, 2008) is evidence that time is spatially coded along the horizontal axis. It consists in faster left-hand responses to early onset timing and faster right-hand responses to late onset timing. This effect has only been established using tasks that directly required to assess onset timing, while no studies investigated whether this association occurs automatically in the auditory modality. The current study investigated the occurrence of the STEARC effect by using a procedure similar to Ishihara and colleagues. Experiment 1 was a conceptual replication of the original study, in which participants directly discriminated the onset timing (early vs. late) of a target sound after listening to a sequence of auditory clicks. This experiment successfully replicated the STEARC effect and revealed that the onset timing is mapped categorically. In Experiments 2, 3a and 3b participants were asked to discriminate the timbre of the stimuli instead of directly assessing the onset timing. In these experiments, no STEARC effect was observed. This suggests that the auditory STEARC effect is only elicited when time is explicitly processed, thus questioning the automaticity of this phenomenon.

Typological differences influence the bilingual advantage in metacognitive processing.

Previous studies showed a bilingual advantage in metacognitive processing (tracking one's own cognitive performance) in linguistic tasks. However, bilinguals do not constitute a homogeneous population, and it was unclear which aspects of bilingualism affect metacognition. In this project, we tested the hypothesis that simultaneous acquisition and use of typologically different languages leads to development of diverse processing strategies and enhances metacognition. The hypothesis was tested in the visual and auditory modalities in language and nonlanguage domains, in an artificial language learning task. In the auditory modality, the hypothesis was confirmed for linguistic stimuli, with no between-domain transfer of metacognitive abilities was observed at the individual level. In the visual modality, no differences in metacognitive efficiency were observed. Moreover, we found that bilingualism per se and the use of typologically different languages modulated separate metacognitive processes engaged in monitoring cognitive performance in statistical learning task. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Development of statistical learning ability across modalities, domains, and languages

Statistical learning (SL) is defined as our ability to use statistics (e.g., frequencies or transitional probabilities) to detect implicit regularities in the environment. Limited research has examined the developmental trajectory of SL across domains and modalities, and no previous research has made systematic comparisons across domains, modalities, and languages using comparable tasks. The current study investigated the development of SL ability across 9-, 11-, and 13-year-old native Chinese-speaking children in non-linguistic visual and auditory SL, first-language Chinese visual and auditory SL, and second-language English visual and auditory SL. Results showed that children across the three age groups achieved all types of SL, and they performed better in visual modality than in auditory modality. Furthermore, while visual SL constantly improved from 9- to 11- to 13-year-olds, auditory SL improved only from 11- to 13-year-olds but not from 9- to 11-year-olds, which could be explained by the discrepancy in developmental trajectory between auditory language and working memory. This pattern of age and modality interaction was similar across non-linguistic Chinese and English SL. A significant interaction between modality and language type also showed that better learning was achieved in visual SL as compared with auditory SL in both non-linguistic and English stimuli. However, children performed similarly across the two modalities in Chinese, possibly due to the contribution of tonal information. Together, our findings point to the joint function of age, modality, and language type in SL development.

Overselectivity during reading-related tasks for children at risk for reading failure

Stimulus overselectivity is defined as a learner’s performance coming under the control of a subset of elements in a compound stimulus. Research in this area has frequently included individuals with autism spectrum disorder, although overselectivity has also been shown in children and adults without developmental disabilities. In educational settings, textual stimuli serve as academically relevant compound stimuli made up of several elements (i.e., letters). The current study sought to identify potential overselective performances by five children at risk for reading failure when presented with 1-, 2-, or 3-element compound stimuli in a delayed matching-to-sample task. Stimuli were academically relevant targets presented in auditory (i.e., letter sounds or names) or visual (i.e., printed letters) modalities. Overall, stimulus overselectivity was more common during visual tasks and tasks that included compounds with three elements. These findings may inform interventions intended to remediate or prevent overselective performances in academic settings.

The Effects of Dance Movement Therapy in the Treatment of Depression

Two essential components make up the semantic information that was used to pinpoint the video events. They consist of: (a) A spatial component of a video frame, such as the scenery, visible people, and visible objects. (b) A temporal component that is represented by a series of video frames across time, such as the actions of a character or the movements of an object. The video's audio, video, and text components are examined in order to provide the higher-level semantic data. The semantic structure of the movie is discovered using a variety of cues from many modalities, such as auditory and visual elements, bridging the gap between high level semantic notions and low level qualities. A video contains three distinct modalities. The three are as follows: (1) Visual modality, which refers to the scene seen in the video; (2) Auditory modality, which refers to spoken words, background music, and other noises; and (3) Textual modality, which refers to written materials that describe the subject matter of the movie.
 There are numerous video databases and a sizable amount of video data in general. Even while there are tools for maintaining and exploring these collections, many decision-making applications increasingly require tools to extract the critical information hidden in the video data.

The Effect of Auditory and Visual Recommendations on Choice

We explore the effect of recommendation modality on recommendation adherence. Results from five experiments run on various online platforms (N = 6,103 adults from TurkPrime and Prolific) show that people are more likely to adhere to recommendations that they hear (auditory) than recommendations that they read (visual). This effect persists regardless of whether the auditory recommendation is spoken by a human voice or an automated voice and holds for hypothetical and consequential choices. We show that the effect is in part driven by the relative need for closure—manifested in a sense of urgency—that is evoked by the ephemerality of auditory messages. This work suggests that differences in the physical properties of auditory and visual modalities can lead to meaningful psychological and behavioral consequences.

Using visual speech at the cocktail-party: CNV evidence for early speech extraction in younger and older adults

The comprehension of spoken language benefits from visual speech information. One reason for this is the temporal lead of mouth and lip movements over the onset of acoustic speech utterance. Here, we investigated EEG event-related potentials preceding acoustic speech, focusing on a fronto-central contingent negative variation (CNV) prior to the onset of acoustic speech. We explored influences of expectation and visual speech content as well as age-related differences. In a multi-talker two-alternative speech discrimination task, younger and older subjects responded to short words presented simultaneously to competing speech under free-field conditions. Subjects were always presented with audiovisual speech stimuli, while the modality containing the task-relevant information was modulated in a block-wise fashion. Thus, task-relevant speech information was either available as audio-visually congruent stimuli or only in the visual (visual-valid) or the auditory (auditory-valid) modality. Subjects were instructed to fixate a pre-specified position in the left or right hemispace. In each task block, task-relevant stimuli appeared either at the pre-specified position (standard trials, 80%) or at a rare deviant position (20%). Target words were recognized faster and more accurately when visual speech information was available. The CNV prior to the acoustic speech onset was more pronounced with visual-informative than with visually non-informative speech. Especially in the younger group, a less pronounced CNV occurred with purely visual speech in deviant trials, that is, when a task-irrelevant speech stimulus appeared instead of the expected target stimulus. The results indicate that processes preceding the onset of acoustic speech are modulated by expectations and visual speech content, while age differences are rather small.

Haptic Technology in Digital Music Learning Context: A State-of-the-Art Analysis

Abstract: Digital media have become increasingly established in learning contexts in recent decades, and it seems impossible to imagine education without them, especially in recent years. Various technological advances can be observed, such as developments in virtual reality and augmented reality. To give learners a realistic impression of the virtual world, as many sensory impressions as possible should be addressed. However, current developments have mainly addressed the visual and auditory modalities, which make up two of the five human senses. Research and developments for the use of the other senses are being made but at this stage they are not yet ready for mass use. Especially the sense of touch based on skin as the largest human sensory organ or tactile and haptic perception seem to be of interest. Particularly in manual or medical areas where motor skills are required, haptic technologies are declared to be supportive and beneficial. One area that has hardly focused on digital learning so far is the music sector. Learning a musical instrument in this context seems to be an interesting field of research, as it not only promotes motor skills, but also cognitive development in both children and adults.
 To give an update on the technical developments in the field of digital teaching and learning in music, and especially to highlight the use of haptic technologies, we will briefly review the state of the art in this paper. It begins with a brief overview of the basics of digital learning and haptics, as well as previous work in this field. Using the method of a scoping review, the topic of haptic technologies in the field of music education will be researched, analysed, and summarised according to defined criteria to give a condensed overview of it. The selected database and appropriate search strings will be used to achieve the aim of the paper. The results help to shed light on current research gaps and give indications for future developments of haptic technology in the music learning context.

Sensory Target Detection at Local and Global Timescales Reveals a Hierarchy of Supramodal Dynamics in the Human Cortex.

To ensure survival in a dynamic environment, the human neocortex monitors input streams from different sensory organs for important sensory events. Which principles govern whether different senses share common or modality-specific brain networks for sensory target detection? We examined whether complex targets evoke sustained supramodal activity while simple targets rely on modality-specific networks with short-lived supramodal contributions. In a series of hierarchical multisensory target detection studies (n = 77, of either sex) using EEG, we applied a temporal cross-decoding approach to dissociate supramodal and modality-specific cortical dynamics elicited by rule-based global and feature-based local sensory deviations within and between the visual, somatosensory, and auditory modality. Our data show that each sense implements a cortical hierarchy orchestrating supramodal target detection responses, which operate at local and global timescales in successive processing stages. Across different sensory modalities, simple feature-based sensory deviations presented in temporal vicinity to a monotonous input stream triggered a mismatch negativity-like local signal which decayed quickly and early, whereas complex rule-based targets tracked across time evoked a P3b-like global neural response which generalized across a late time window. Converging results from temporal cross-modality decoding analyses across different datasets, we reveal that global neural responses are sustained in a supramodal higher-order network, whereas local neural responses canonically thought to rely on modality-specific regions evolve into short-lived supramodal activity. Together, our findings demonstrate that cortical organization largely follows a gradient in which short-lived modality-specific as well as supramodal processes dominate local responses, whereas higher-order processes encode temporally extended abstract supramodal information fed forward from modality-specific cortices.SIGNIFICANCE STATEMENT Each sense supports a cortical hierarchy of processes tracking deviant sensory events at multiple timescales. Conflicting evidence produced a lively debate around which of these processes are supramodal. Here, we manipulated the temporal complexity of auditory, tactile, and visual targets to determine whether cortical local and global ERP responses to sensory targets share cortical dynamics between the senses. Using temporal cross-decoding, we found that temporally complex targets elicit a supramodal sustained response. Conversely, local responses to temporally confined targets typically considered modality-specific rely on early short-lived supramodal activation. Our finding provides evidence for a supramodal gradient supporting sensory target detection in the cortex, with implications for multiple fields in which these responses are studied (e.g., predictive coding, consciousness, and attention).

Investigating the individual difference in distance calls of Bengalese finches using acoustic analysis

Most songbirds are monogamous and exhibit biparental care. They socially interact with other individuals, including those outside the family, through auditory modality. The capacity to explicitly identify conspecific individuals via acoustic cues enables fast turn-taking communication in a crowd, especially when visual signals are not available. Bengalese finches (Lonchura striata var. domestica) are domesticated strains of the white-rumped munia imported from China to Japan about 260 years ago. Previous studies have reported distance calls, which are emitted when a bird is visually separated from its conspecifics, are sexually dimorphic (both acoustically and perceptually) in Bengalese finches. However, the individuality in distance calls has not been thoroughly discussed yet. In this research, we recorded vocalizations from four pairs of Bengalese finches in kinship. To extract distance calls from those recordings, we use the Harmonics-to-Noise Ratio (HNR) to detect and remove noise, which mainly consists of aperiodic components. By analyzing acoustic features such as length, bandwidth, and peak frequency, we found that calls from the same individuals form clear clusters, and the difference between other individuals was more pronounced in contrast to kins. [Work supported by MEXT/JSPS grant #4903, 17H06380.]

The different brain areas occupied for integrating information of hierarchical linguistic units: a study based on EEG and TMS.

Human language units are hierarchical, and reading acquisition involves integrating multisensory information (typically from auditory and visual modalities) to access meaning. However, it is unclear how the brain processes and integrates language information at different linguistic units (words, phrases, and sentences) provided simultaneously in auditory and visual modalities. To address the issue, we presented participants with sequences of short Chinese sentences through auditory, visual, or combined audio-visual modalities while electroencephalographic responses were recorded. With a frequency tagging approach, we analyzed the neural representations of basic linguistic units (i.e. characters/monosyllabic words) and higher-level linguistic structures (i.e. phrases and sentences) across the 3 modalities separately. We found that audio-visual integration occurs in all linguistic units, and the brain areas involved in the integration varied across different linguistic levels. In particular, the integration of sentences activated the local left prefrontal area. Therefore, we used continuous theta-burst stimulation to verify that the left prefrontal cortex plays a vital role in the audio-visual integration of sentence information. Our findings suggest the advantage of bimodal language comprehension at hierarchical stages in language-related information processing and provide evidence for the causal role of the left prefrontal regions in processing information of audio-visual sentences.