Tonal Information Research Articles

Predicting the intelligibility of Mandarin Chinese with manipulated and intact tonal information for normal-hearing listeners.

Objective indices for predicting speech intelligibility offer a quick and convenient alternative to behavioral measures of speech intelligibility. However, most such indices are designed for a specific language, such as English, and they do not take adequate account of tonal information in speech when applied to languages like Mandarin Chinese (hereafter called Mandarin) for which the patterns of fundamental frequency (F0) variation play an important role in distinguishing speech sounds with similar phonetic content. To address this, two experiments with normal-hearing listeners were conducted examining: (1) The impact of manipulations of tonal information on the intelligibility of Mandarin sentences presented in speech-shaped noise (SSN) at several signal-to-noise ratios (SNRs); (2) The intelligibility of Mandarin sentences with intact tonal information presented in SSN, pink noise, and babble at several SNRs. The outcomes were not correctly predicted by the Hearing Aid Speech Perception Index (HASPI-V1). A new intelligibility metric was developed that used one acoustic feature from HASPI-V1 plus Hilbert time envelope and temporal fine structure information from multiple frequency bands. For the new metric, the Pearson correlation between obtained and predicted intelligibility was 0.923 and the root mean square error was 0.119. The new metric provides a potential tool for evaluating Mandarin intelligibility.

Event-related potentials of familiar monosyllabic words with unexpected lexical tones: A picture-word study of Mandarin-speaking preschoolers with and without a history of late talking.

This study examined how Mandarin-speaking preschoolers with and without a history of late talking (LT) process familiar monosyllabic words with unexpected lexical tones, focusing on both phonological and semantic violations. This study initially enrolled 64 Mandarin-speaking toddlers: 31 with a history of LT (mean age: 27.67 months) and 33 without a history of LT (non-LT) (mean age: 27.85 months). Event-related potentials were recorded at the age of 4 years during a picture-word mismatch task (LT mean age: 51.36 months; non-LT mean age: 51.20 months); in this task, the participants were presented with auditory words either matching (Tone 3) or mismatching with images in terms of their lexical tones; the mismatches encompassed acoustically dissimilar (Tone 1) and similar (Tone 2) mismatches. A significant difference in the phonological mapping negativity (PMN) responses to Tones 1 and 3 was observed only in the non-LT group. However, differences in the N400 responses to Tones 1 and 3 remained consistent across both groups. In addition, greater differences in the PMN responses between Tones 1 and 3 were associated with higher language proficiency during the preschool period. The PMN response serves as an indicator of neural correlates in lexical tone processing, reflecting challenges encountered by preschoolers with a history of LT when processing the lexical tones of familiar words. Furthermore, the PMN response was correlated with concurrent language abilities. These findings indicate the importance of early tonal perception development for Mandarin speakers with a history of LT. RESEARCH HIGHLIGHTS: Preschoolers with a history of late talking (LT), similar to preschoolers without such a history, can establish word expectations and detect the lexical tone violation in real time. However, those with a history of LT require additional time to process acoustic cues and differentiate between word semantics based on lexical tone information. The phonological mapping negativity response serves as an indicator of neural correlates in lexical tone processing, reflecting challenges encountered by preschoolers with a history of LT when processing the lexical tones of familiar words. The present findings indicate the importance of early intervention for Mandarin speakers with a history of LT, with an emphasis on lexical tone processing from toddlerhood.

The prediction of segmental and tonal information in Mandarin Chinese: an eye-tracking investigation

ABSTRACT There is controversy about the extent to which people predict phonology during comprehension. In three visual-world experiments, we ask whether it occurs in Mandarin, a tonal language. Participants heard sentences containing a target word that was highly predictable (Cloze 80.2%, Experiment 1) or very highly predictable (Cloze 93.9%, Experiments 2–3) and saw an array of objects containing one whose name matched the target word (Experiments 1–2), was unrelated to the target word (Experiments 1–3), or matched the target word in segment and tone (Experiments 1–3), in segment only (Experiments 1–3), or tone only (Experiment 3). In comparison to the unrelated object, participants looked more at the segment + tone object (Experiments 1–3), and sometimes at the segment object (Experiments 1 and 3), but not at the tone object. We conclude that participants predict segmental information independently of tone.

The role of coarticulatory tonal information in Cantonese spoken word recognition: an eye-tracking study

Abstract Previous eye-tracking studies suggest listeners closely track coarticulatory segmental information to facilitate spoken word recognition. In tonal languages such as Cantonese, the F0 value of high-level and rising tones is higher when preceding a low tone than preceding a high tone. Given the tonal coarticulation effect in Cantonese, this study used a printed-word version of the visual-world eye-tracking paradigm to investigate Macau Cantonese listeners’ real-time processing of coarticulatory information through two types of word pairings, in which target words and competitor words are either matched or mismatched in coarticulatory pattern (i.e., preceding a low or high tone). The results of the growth curve analysis on the difference between target and competitor fixations show that a fixation advantage of target words increases more quickly over time in the mismatch condition than in the match condition. The overall findings suggest that native listeners closely attend to the coarticulatory tonal information to facilitate lexical activation during moment-by-moment processing. Since the mechanisms of the coarticulatory effect may differ between segments and lexical tones, this research calls for further eye-tracking studies that manipulate coarticulation cues in segmental and suprasegmental domains.

Enhanced noise resilience in passive tone detection via broad-receptive field complex-valued convolutional neural networks.

Tone detection is crucial for passive sonar systems. Numerous algorithms have been developed for passive tone detection, but their effectiveness in detecting weak tones is still limited. To enhance noise resilience in passive tone detection, a broad-receptive field complex-valued structure named attention-driven complex-valued U-Net is proposed. Concretely, two attention mechanisms, namely, temporal attention and harmonic attention, are proposed to broaden the receptive field with high computational efficiency. Complex-valued operators are then introduced to mine both amplitude and phase information of tones. Additionally, a symmetric downsampling and upsampling strategy is proposed to improve the reconstruction accuracy of detailed time-frequency information. Overall, the proposed method demonstrates a strong robustness to noise and a strong ability to generalize. Experimental results on both simulated data and real-world data validate the superiority of the proposed attention-driven complex-valued U-Net against conventional U-shaped structures.

Phonological recovery during Chinese sentence reading: effects of rime and tone

ABSTRACT The present study tested the activation of different phonological units of Chinese characters during silent sentence reading. Fifty-five participants were tested in an eye-tracking experiment. A highly predictable target character in each experimental sentence was replaced by four types of substitutes (i.e. no-violation, tone-violation, rime-violation, and double-violation). The participants exhibited a shorter total reading time in the no-violation and tone-violation conditions than in the double-violation baseline condition, whereas the rime-violation condition did not differ from the baseline. Moreover, the participants did not benefit from tonal information in addition to syllable-level phonological overlap. Our findings are consistent with a notion of late phonological activation in Chinese, and therefore suggest a direct route of lexical activation bypassing phonological mediation during visual word recognition.

Disambiguation of Isolated Manipuri Tonal Contrast Word Pairs Using Acoustic Features

Manipuri is a low-resource, Tibeto-Burman tonal language spoken mainly in Manipur, a northeastern state of India. Tone identification is crucial to speech comprehension for tonal languages, where tone defines the word’s meaning. Automatic Speech Recognition for those languages can perform better by including tonal information from a powerful tone detection system. While significant research has been conducted on tonal languages like Mandarin, Thai, Cantonese, and Vietnamese, a notable gap exists in exploring Manipuri within this context. To address this gap, this work expands our previously developed handcrafted speech corpus, ManiTo, which comprises isolated Manipuri tonal contrast word pairs to study the tones of Manipuri. This extension includes contributions from 20 native speakers. Preliminary findings have confirmed that Manipuri has two unique tones, Falling and Level. The study then conducts a comprehensive acoustic feature analysis. Two sets of features based on Pitch contours, Jitter, and Shimmer measurements are investigated to distinguish the two tones of Manipuri. Support Vector Machine, Long Short-term Memory, Random Forest, and k-Nearest Neighbors are the classifiers adopted to validate the selected feature sets. The results indicate that the second set of features consistently outperformed the first set, demonstrating higher accuracy, particularly when utilizing the Random Forest classifier, which provides valuable insights for further advancements in speech recognition technology for low-resource tonal language Manipuri.

Voices of the Himalayas: Benchmarking Speech Recognition Systems for the Tibetan Language

The Lhasa dialect, the most widely spoken Tibetan dialect in Tibet, is also renowned for its rich historical archive of written scripts. Exploring speech recognition methodologies specific to the Lhasa dialect is paramount in safeguarding Tibet’s distinct linguistic heritage. Previous studies in Tibetan speech recognition have been largely confined to academic research using nonpublic datasets, focusing on elements such as the selection of phone-level acoustic modeling units and the integration of tonal information. However, these studies have not significantly benefited the community due to the scarcity of available data. To mitigate the challenge posed by limited data resources, we present the NICT-Tib1 (phase 1) dataset, a new open-source dataset collected from native speakers dedicated to investigating speech recognition for the Lhasa dialect. Speech recognition with deep neural networks (DNNs) evolved three generations from systems hybrid with hidden Markov model (HMM) (e.g., DNN-HMM) to End-to-End systems (e.g., Transformer), and finally to self-supervised learning (SSL) systems (e.g., Wav2Vec2.0), each generation improving accuracy and simplifying the training process, with the latest generation achieving state-of-the-art performance, especially for low-resource languages. Besides the early DNN-HMM-based system using Kaldi, we further update benchmark systems with the Conformer and Wav2Vec2.0 trained by ESPnet and Huggingface on this dataset, respectively. Experimental results show that these state-of-the-art models outperformed the models in previous work.

Quantitative assessment of English accent variation: Levenshtein distance and perceptual identification

This is a perceptual study with a basis in acoustic phonetics. It investigates the extent to which accents of English can be quantified by the Levenshtein distance (LD), a string metric. LD defines string similarity by the minimum number of character edits required to transform one string into another. Narrowly transcribed speech samples of English provide the phonetic strings required for LD comparison. For the purposes of the current experiment, 24 speakers representing four English accent varieties are included: Midland American (control), British/Australian, Hindi-influenced, and Mandarin-influenced. High- and low-pass filters are used to augment or attenuate perceptual contribution of high- versus low-frequency information in the speech signal. Predictions stem from published correlations between LD and listeners’ perceptional ratings of intelligibility and native-likeness. For a group of monolingual American English listeners, confusion is expected between Midland American and British/Australian accents due to similarly low LDs, further intensified by low-pass filtering. However, fewer confusions are predicted between Hindi- and Mandarin-influenced English, due to varying L1 influences and the absence/presence of tonal information. This research seeks to explore accent perception, investigating the relationship between manipulated aspects of the speech signal and listener identification.

Tone selection for cochlear implants using peak-finding and frequency bin overlap

The Advanced Bionics cochlear implant devices employ a method of allocating static frequency bins to electrodes based on the natural tonotopic organization of the cochlea. Many studies show that there is a significant frequency-to-place mismatch in many subjects, but that over time the brain adjusts to these mismatches. Also, most of the tonal information in human speech is present in the lower end of the frequency spectrum, where there are few frequency bins and electrodes available to represent this information. Here, adjustments to the frequency bin allocation algorithm used in the crowdsourced CI Hackathon code are made to allow a more accurate representation of the original signal in the lower registers. A peak-finding and selecting method is used to find significant peaks regardless of frequency bin, then bins are overlapped to allow more accurate representation of tones near the borders between channels and peaks which may be near to each other.

An fMRI Study of How Deaf Children Process the Two Tones (Second Tone and Third Tone) in Mandarin Chinese

The objective of this study is to investigate the brain activity patterns of deaf children and hearing children during the processing of two different tones (second tone and third tone) using resting-state functional Magnetic Resonance Imaging (fMRI). Furthermore, the study aims to identify the differences in brain activation regions between deaf children and hearing children during the tone processing task. Five deaf children and two hearing children were selected as participants. Resting-state functional Magnetic Resonance Imaging (fMRI) scans were conducted on the subjects using an fMRI scanner. The acquired fMRI data were then preprocessed and analyzed to examine the patterns of brain activity. Deaf children and hearing children exhibit differences in brain activation regions during the execution of tone recognition tasks. These differences can be observed in various areas such as the pre-central gyrus, superior temporal gyrus, middle occipital gyrus, supplementary motor area, superior parietal lobe, and interior frontal gyrus, among others. Through comparisons, the brains of deaf and mute children, although they exhibit relatively reduced activation in certain areas of the auditory cortex, may show enhanced activation in additional regions such as the middle occipital gyrus. This suggests that the brain of deaf children may undergo a reorganization of its functional networks, allowing for improved spatial perception, visual abilities, and other skills. This adaptation enables them to better process tonal information despite the absence of auditory input. These findings contribute to the understanding of the neural basis of tone processing and may help in refining intervention strategies.

The Contribution of Aspirated Feature Information of Initial in Mandarin-Chinese Monosyllable Word Processing

The passage mainly focuses on aspirated feature of stop consonants priming effect in Mandarin-Chinese monosyllable real spoken words. The experiment contrasted 4 prime conditions of prime-target pairs: homogeneous (pa1-pa1), aspirated feature (pha1-pa1), post-homogeneous (ma1-pa1) and unrelated (lu1-pa1). In the experiment, all tonal information is matched and the aspirated feature of initial in Mandarin-Chinese monosyllable real spoken words is the variant. The latter two pairs are control groups. The data were from ten Mandarin native listeners, then they were calculated and got the mean reaction time (RT), the standard deviation (std) and standard error (std. err). By comparing with the unrelated group, the other three shows that if some segmental information are matched, there will be significant negative priming effect. The first two pairs test whether aspirated feature activate priming effect at word recognition in Mandarin-Chinese monosyllable real spoken words by auditory priming, by comparing the homogeneous (H) group with the aspirated group and post-homogeneous group, the result shows that changing the aspirated feature do have a positive priming effect.

Tracking the time-course of spoken word recognition of Cantonese Chinese in sentence context: Evidence from eye movements.

In this study, we conducted an eye-tracking experiment to investigate the effects of sentence context and tonal information on spoken word recognition processes in Cantonese Chinese. We recruited 60 native Cantonese listeners to participate in the eye-tracking experiment. The target words (phonologically similar words) were manipulated to either (1) a congruent context or (2) an incongruent context in the experiment. The resulting eye-movement patterns in the incongruent context condition clearly revealed that (1) sentence context produced a garden-path effect in the initial stage of the spoken word recognition processes and then (2) the lexical tone of the word (bottom-up information) overrode the contextual effects to help listeners to discriminate between different similar-sounding words during lexical access. In conclusion, the patterns of eye-tracking data show the interactive processes between the lexical tone (an acoustic cue within a Cantonese word) and sentence context played in different phases to the spoken word recognition of Cantonese Chinese.

The Effect of Color Saturation of Travel Pictures on Consumer Appeal

In the Internet era, online channels have contributed significantly to tourism marketing and promotion. Consumers will receive tourism information online to reduce information asymmetry. Moreover, with overwhelming levels of information, consumers may only get a limited amount of information. Therefore, the primary concern for marketers lies in capturing consumers’ attention during this stage. Visual design is essential among the various factors researchers have thoroughly investigated. Studies have recognized the importance of the color characteristics of travel photos on consumer decisions. However, these studies often focus their analysis on tonal information while neglecting the intuitive emotional impact of color saturation on consumers. Consequently, this study aims to distinguish and categorize different tourist destinations to explore the impact of saturation characteristics of attraction photos on consumer appeal in online scenarios. This study contributes to existing knowledge of color context theory, enhancing its contextual application. The findings presented in this paper have implications for developing more effective visual marketing strategies.

Processing of acoustic and phonological information of lexical tones at pre-attentive and attentive stages

ABSTRACT While Mismatch Negativity (MMN) and P300 have been found to correlate with the processing of acoustic and phonological information involved in speech perception, there is controversy surrounding how these two components index acoustic and/or phonological processing at pre-attentive and attentive stages. The current study employed both passive and active oddball paradigms to examine neural responses to lexical tones at the two stages in Cantonese speakers, using the paradigm of categorical perception (CP) where the between- and within-category deviants share the same acoustic distance from the standard but differ in the involvement of phonological information. We failed to observe a CP effect in P300, which might indicate that this component doesn’t necessarily index phonological processing, while MMN does, as reflected by the finding of a greater MMN amplitude elicited from the between-category than within-category deviant. Nevertheless, phonological processing might be overridden by acoustic processing among participants who were sensitive to pitch.

An fMRI study of the neural mechanisms of third and fourth tone recognition in deaf children

Vocal tone is an important component of language and it plays a key role in language comprehension and communication. However, children with hearing loss face challenges in vocal tone recognition due to hearing impairment. In this study, five deaf children and two children with normal hearing were recruited to compare the differences in third and fourth tone recognition tasks between deaf and normal children. The results revealed that (1) some of the deaf children's brain regions that process vocal tones did not work properly due to hearing loss; (2) deaf children may rely on different neural networks when processing vocal tone information. (3) Deaf children process vocal tone information with hemispheric characteristics.

An fMRI study of how deaf children process the two tones (first tone and third tone) in Mandarin Chinese

The objective of this study is to investigate the brain activity patterns of deaf children and hearing children during the processing of two different tones (first tone and third tone) using resting-state functional magnetic resonance imaging (fMRI). Furthermore, the study aims to identify the differences in brain activation regions between deaf children and hearing children during the tone processing task. Five deaf children and two hearing children were selected as participants. Resting-state functional magnetic resonance imaging (fMRI) scans were conducted on the subjects using an fMRI scanner. The acquired fMRI data were then preprocessed and analyzed to examine the patterns of brain activity. Deaf children and hearing children exhibit differences in brain activation regions during the execution of tone recognition tasks. These differences can be observed in various areas such as the pre-central gyrus, superior temporal gyrus, middle occipital gyrus, supplementary motor area and interior frontal gyrus, among others. The results suggest that deaf children may rely on different neural networks to process tone information compared to hearing children, possibly exhibiting stronger neural plasticity and compensatory mechanisms. These findings contribute to the understanding of the neural basis of tone processing and may help in refining intervention strategies.

An fMRI study of the neural mechanisms of second and fourth tone recognition in deaf children

Tone is an important component of language and it plays a key role in language comprehension and communication. However, children with hearing loss face challenges in vocal tone recognition due to hearing impairment. In this study, five deaf children and two children with normal hearing were recruited to compare the differences in second and fourth tone recognition tasks between deaf and normal children. The results found that (1) some of the deaf children's brain regions for processing vocal tones did not work properly due to hearing loss; (2) deaf children may rely on different neural networks when processing vocal tone information. (3) Deaf children process vocal tone information with hemispheric characteristics.

Contextual effects on spoken word processing: An eye-tracking study of the time course of tone and vowel activation in Mandarin.

Lexical access is highly contextual. For example, vowel (rime) information is prioritized over tone in the lexical access of isolated words in Mandarin Chinese, but these roles are flipped in constraining contexts. The time course of these contextual effects remains unclear, and so here we tracked the real-time eye gaze of native Mandarin speakers in a visual-world paradigm. While listening to a noun classifier, before the target noun was even uttered, gaze to the target noun was already greater than looking to phonologically unrelated distractors. Critically, there was also more distraction from a cohort competitor (tone information) than a segmental competitor (vowel information) in more semantically constraining contexts. Results confirm that phonological activation in Mandarin lexical access is highly sensitive to context, with tone taking priority over vowel information even before a target word is heard. Results suggest that phonological activation in real-time lexical access may be highly context-specific across languages. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Perceptual integrality of foreign segmental and tonal information: Dimensional transfer hypothesis

AbstractThis study investigates whether (a) Cantonese and (b) English listeners integrally or independently perceive Thai tone and segmental information. Listeners completed a modified AX discrimination task that contained a control block (without segmental variation) and an orthogonal block (with segmental variation). Relative to their own performance in the control block, the Cantonese listeners showed increased response time, decreased proportion of accuracy, and decreased sensitivity index in the orthogonal block. By contrast, the English listeners showed similar response time, proportion of accuracy, and sensitivity index across the two blocks. These reflect integral processing among the Cantonese but not the English listeners. This finding motivates the dimensional transfer hypothesis. The hypothesis posits that L1 perceptual experience shapes the perceptual integrality (or nonintegrality) of foreign suprasegmental and segmental information.