Confusion Patterns Research Articles

An Alternative Explanation for Difficulties with Speech in Background Talkers: Abnormal Fusion of Vowels Across Fundamental Frequency and Ears.

Normal-hearing (NH) listeners use frequency cues, such as fundamental frequency (voice pitch), to segregate sounds into discrete auditory streams. However, many hearing-impaired (HI) individuals have abnormally broad binaural pitch fusion which leads to fusion and averaging of the original monaural pitches into the same stream instead of segregating the two streams (Oh and Reiss, 2017) and may similarly lead to fusion and averaging of speech streams across ears. In this study, using dichotic speech stimuli, we examined the relationship between speech fusion and vowel identification. Dichotic vowel perception was measured in NH and HI listeners, with across-ear fundamental frequency differences varied. Synthetic vowels /i/, /u/, /a/, and /ae/ were generated with three fundamental frequencies (F0) of 106.9, 151.2, and 201.8Hz and presented dichotically through headphones. For HI listeners, stimuli were shaped according to NAL-NL2 prescriptive targets. Although the dichotic vowels presented were always different across ears, listeners were not informed that there were no single vowel trials and could identify one vowel or two different vowels on each trial. When there was no F0 difference between the ears, both NH and HI listeners were more likely to fuse the vowels and identify only one vowel. As ΔF0 increased, NH listeners increased the percentage of two-vowel responses, but HI listeners were more likely to continue to fuse vowels even with large ΔF0. Binaural tone fusion range was significantly correlated with vowel fusion rates in both NH and HI listeners. Confusion patterns with dichotic vowels differed from those seen with concurrent monaural vowels, suggesting different mechanisms behind the errors. Together, the findings suggest that broad fusion leads to spectral blending across ears, even for different ΔF0, and may hinder the stream segregation and understanding of speech in the presence of competing talkers.

Vowel Production in Prelingually Deafened Mandarin-Speaking Children With Cochlear Implants

Purpose The purpose of this study was to characterize the acoustic profile and to evaluate the intelligibility of vowel productions in prelingually deafened, Mandarin-speaking children with cochlear implants (CIs). Method Twenty-five children with CIs and 20 age-matched children with normal hearing (NH) were recorded producing a list of Mandarin disyllabic and trisyllabic words containing 20 Mandarin vowels [a, i, u, y, ɤ, ɿ, ʅ, ai, ei, ia, ie, ye, ua, uo, au, ou, iau, iou, uai, uei] located in the first consonant-vowel syllable. The children with CIs were all prelingually deafened and received unilateral implantation before 7 years of age with an average length of CI use of 4.54 years. In the acoustic analysis, the first two formants (F1 and F2) were extracted at seven equidistant time locations for the tested vowels. The durational and spectral features were compared between the CI and NH groups. In the vowel intelligibility task, the extracted vowel portions in both NH and CI children were presented to six Mandarin-speaking, NH adult listeners for identification. Results The acoustic analysis revealed that the children with CIs deviated from the NH controls in the acoustic features for both single vowels and compound vowels. The acoustic deviations were reflected in longer duration, more scattered vowel categories, smaller vowel space area, and distinct formant trajectories in the children with CIs in comparison to NH controls. The vowel intelligibility results showed that the recognition accuracy of the vowels produced by the children with CIs was significantly lower than that of the NH children. The confusion pattern of vowel recognition in the children with CIs generally followed that in the NH children. Conclusion Our data suggested that the prelingually deafened children with CIs, with a relatively long duration of CI experience, still showed measurable acoustic deviations and lower intelligibility in vowel productions in comparison to the NH children.

Raised Dot Enumeration Via Haptic Exploration.

In two experiments we investigated blindfolded, sighted participants' capacity to extract the number of raised dots from arrays of braille cells that they scanned once via active touch. The arrays could contain between one and 12 raised dots and estimates were based on scanning with one or more fingers on one or both hands (Experiment 1), or when the dots were as maximally or minimally spaced as the braille code permits (Experiment 2). We sought evidence of discontinuities in performance that reflect more than one mode of enumeration. We found that participants' estimates of numerosity increased in a linear fashion with actual numerosity, but were increasingly underestimated beyond numerosity of six, and confidence in the judgment declined linearly with increasing numerosity. Finger combinations made no difference to accuracy, errors, or confidence. Increasing dot density had the effect of diminishing perceptual accuracy, exaggerating underestimation and reducing confidence. While perceptual accuracy was generally high up to six raised dots, patterns of confusions and scaling analyses suggest that numerosities of four or less are perceptually unique. In this article, we discuss these data in terms of enumeration in touch and other modalities, and consider whether this discontinuity in enumeration signifies a subitize-to-count or a count-to-estimate transition.

Cross-modal transfer of talker-identity learning.

A speech signal carries information about meaning and about the talker conveying that meaning. It is now known that these two dimensions are related. There is evidence that gaining experience with a particular talker in one modality not only facilitates better phonetic perception in that modality, but also transfers across modalities to allow better phonetic perception in the other. This finding suggests that experience with a talker provides familiarity with some amodal properties of their articulation such that the experience can be shared across modalities. The present study investigates if experience with talker-specific articulatory information can also support cross-modal talker learning. In Experiment 1 we show that participants can learn to identify ten novel talkers from point-light and sinewave speech, expanding on prior work. Point-light and sinewave speech also supported similar talker identification accuracies, and similar patterns of talker confusions were found across stimulus types. Experiment 2 showed these stimuli could also support cross-modal talker matching, further expanding on prior work. Finally, in Experiment 3 we show that learning to identify talkers in one modality (visual-only point-light speech) facilitates learning of those same talkers in another modality (auditory-only sinewave speech). These results suggest that some of the information for talker identity takes a modality-independent form.

Wearing Face Masks Strongly Confuses Counterparts in Reading Emotions.

Wearing face masks is one of the essential means to prevent the transmission of certain respiratory diseases such as coronavirus disease 2019 (COVID-19). Although acceptance of such masks is increasing in the Western hemisphere, many people feel that social interaction is affected by wearing a mask. In the present experiment, we tested the impact of face masks on the readability of emotions. The participants (N = 41, calculated by an a priori power test; random sample; healthy persons of different ages, 18–87 years) assessed the emotional expressions displayed by 12 different faces. Each face was randomly presented with six different expressions (angry, disgusted, fearful, happy, neutral, and sad) while being fully visible or partly covered by a face mask. Lower accuracy and lower confidence in one’s own assessment of the displayed emotions indicate that emotional reading was strongly irritated by the presence of a mask. We further detected specific confusion patterns, mostly pronounced in the case of misinterpreting disgusted faces as being angry plus assessing many other emotions (e.g., happy, sad, and angry) as neutral. We discuss compensatory actions that can keep social interaction effective (e.g., body language, gesture, and verbal communication), even when relevant visual information is crucially reduced.

Perception and Acoustic Studies of Vowel Intelligibility in Dysphonic Speech

This study aimed to (1) examine the effect of dysphonia and background noise on the identifiability of vowels, and (2) evaluate the relationship between the degree of vowel intelligibility and vowel-related acoustic measurements. A speech perception experiment was conducted with speech samples collected from 10 adult females with healthy voices and 10 adult females with dysphonia. The speech material was 11 vowels of American English in /h/-vowel-/d/ format. Cafeteria noise was added to these samples at a signal-to-noise ratio of -6 dB. A total of 10 adults with normal hearing participated in a speech perception experiment, in which the vowels were presented with and without the noise. F1 and F2 frequencies of the vowels were measured, and their relationships with the vowel intelligibility were statistically evaluated. The group-level analysis showed that vowel intelligibility was lower in dysphonic speech than normal speech, both in quiet and at signal-to-noise ratio of -6 dB. The intelligibility was higher for the high vowels than the low vowels. In general, the vowel confusion pattern was similar between normal and dysphonic speech. However, data from a speaker with severe diplophonia showed a distinct confusion pattern. Voice quality ratings did not significantly correlate with the vowel intelligibility. There was a significant correlation between F2 and the vowel intelligibility in quiet. A post-hoc acoustic analysis revealed that vowels of the speakers with lower vowel intelligibility had lower acoustic energy above 1 kHz. Dysphonia negatively affects vowel intelligibility. Low vowels were more vulnerable to the effect of dysphonia than high vowels. Among different types of dysphonic voice qualities, diplophonia appears to be particularly detrimental to vowel intelligibility. F2 significantly correlated with vowel intelligibility; however, this result requires a careful interpretation. Given that the acoustic energy above 1 kHz resulted in better intelligibility in noise, a treatment strategy that targets this frequency region may help improve intelligibility in noise. Future studies should examine the generalizability of this finding to different age and gender groups, and intelligibility as a whole.

Effects of speech-shaped noise on consonant identification by native and non-native Korean and English listeners

The global proliferation of machine-human interface via voice and audio suggests a growing need for including language-specificity in the design and implementation of acoustic signal processing devices for all individuals, especially those in educational settings. Second language learners are often more susceptible to acoustic distortions in their second language (i.e., L2). Speech understanding challenges that L2 learners face in noisy environments are often compounded by other normal cross-language influences that are expected to occur during L2 speech perception. This study evaluated consonant confusion patterns using consonant-vowel (CV) English tokens spoken by native Korean and English speakers. The speech stimuli were embedded in speech-shaped noise with varying signal-to-noise ratios. Speech scores were measured using an alternative forced-choice paradigm. Overall scores were evaluated for each listener group to first assess whether L2 learners were most robust to acoustic-variances in English when spoken by Korean compared to English native-speakers. Second, their speech scores and acoustic recordings were evaluated for constructive as well as destructive effects of these acoustic variance on their masking and consonant confusion patterns. Results will be presented on the similarity and differences in acoustics and perception between English and Korean native-speakers. The clinical implications of our findings will be discussed.

A common representation of fingers and toes

There are many similarities and differences between the human hands and feet. On a psychological level, there is some evidence from clinical disorders and studies of tactile localisation in healthy adults for deep functional connections between the hands and feet. One form these connections may take is in common high-level mental representations of the hands and feet. Previous studies have shown that there are systematic, but distinct patterns of confusion found between both the fingers and toes. Further, there are clear individual differences between people in the exact patterns of mislocalisations. Here, we investigated whether these idiosyncratic differences in tactile localisation are shared between the fingers and toes, which may indicate a shared high-level representation. We obtained confusion matrices showing the pattern of mislocalisation on the hairy skin surfaces of both the fingers and toes. Using a decoding approach, we show that idiosyncratic differences in individuals' pattern of confusions are shared across the fingers and toes, despite different overall patterns of confusions. These results suggest that there is a common representation of the fingers and toes.

Time-resolved discrimination of audio-visual emotion expressions

Humans seamlessly extract and integrate the emotional content delivered by the face and the voice of others. It is however poorly understood how perceptual decisions unfold in time when people discriminate the expression of emotions transmitted using dynamic facial and vocal signals, as in natural social context. In this study, we relied on a gating paradigm to track how the recognition of emotion expressions across the senses unfold over exposure time. We first demonstrate that across all emotions tested, a discriminatory decision is reached earlier with faces than with voices. Importantly, multisensory stimulation consistently reduced the required accumulation of perceptual evidences needed to reach correct discrimination (Isolation Point). We also observed that expressions with different emotional content provide cumulative evidence at different speeds, with "fear" being the expression with the fastest isolation point across the senses. Finally, the lack of correlation between the confusion patterns in response to facial and vocal signals across time suggest distinct relations between the discriminative features extracted from the two signals. Altogether, these results provide a comprehensive view on how auditory, visual and audiovisual information related to different emotion expressions accumulate in time, highlighting how multisensory context can fasten the discrimination process when minimal information is available.

The role of cue enhancement and frequency fine-tuning in hearing impaired phone recognition

A speech-based hearing test is designed to identify the susceptible error-prone phones for individual hearing impaired (HI) ear. Only robust tokens in the experiment noise levels had been chosen for the test. The noise-robustness of tokens is measured as SNR90 of the token, which is the signal to the speech-weighted noise ratio where a normal hearing (NH) listener would recognize the token with an accuracy of 90% on average. Two sets of tokens T1 and T2 having the same consonant-vowels but different talkers with distinct SNR90 had been presented with flat gain at listeners' most comfortable level. We studied the effects of frequency fine-tuning of the primary cue by presenting tokens of the same consonant but different vowels with similar SNR90. Additionally, we investigated the role of changing the intensity of primary cue in HI phone recognition, by presenting tokens from both sets T1 and T2. On average, 92% of tokens are improved when we replaced the CV with the same CV but with a more robust talker. Additionally, using CVs with similar SNR90, on average, tokens are improved by 75%, 71%, 63%, and 72%, when we replaced vowels /A, ae, I, E/, respectively. The confusion pattern in each case provides insight into how these changes affect the phone recognition in each HI ear. We propose to prescribe hearing aid amplification tailored to individual HI ears, based on the confusion pattern, the response from cue enhancement, and the response from frequency fine-tuning of the cue.

DEMoS: an Italian emotional speech corpus

We present DEMoS (Database of Elicited Mood in Speech), a new, large database with Italian emotional speech: 68 speakers, some 9 k speech samples. As Italian is under-represented in speech emotion research, for a comparison with the state-of-the-art, we model the ‘big 6 emotions’ and guilt. Besides making available this database for research, our contribution is three-fold: First, we employ a variety of mood induction procedures, whose combinations are especially tailored for specific emotions. Second, we use combinations of selection procedures such as an alexithymia test and self- and external assessment, obtaining 1,5 k (proto-) typical samples; these were used in a perception test (86 native Italian subjects, categorical identification and dimensional rating). Third, machine learning techniques—based on standardised brute-forced openSMILE ComParE features and support vector machine classifiers—were applied to assess how emotional typicality and sample size might impact machine learning efficiency. Our results are three-fold as well: First, we show that appropriate induction techniques ensure the collection of valid samples, whereas the type of self-assessment employed turned out not to be a meaningful measurement. Second, emotional typicality—which shows up in an acoustic analysis of prosodic main features—in contrast to sample size is not an essential feature for successfully training machine learning models. Third, the perceptual findings demonstrate that the confusion patterns mostly relate to cultural rules and to ambiguous emotions.

The Dynamic Association of Interest and Confusion: The Potential for Moderation by Utility Value and Gender

To better understand students’ experiences while learning in an online computer-science course, we adopted a dynamical systems analytic framework in which we tested linked patterns of confusion and interest over time and investigated whether and how those patterns might change as a function of students’ exposure to a utility value (UV) intervention and gender. We assessed interest and confusion via event contingent experience-sampled questionnaires that displayed immediately after students had engaged with key examples or exercises. Results indicated that students’ exposure to a UV intervention and gender affected the levels of confusion that students reported and the associations between interest and confusion levels over time. Implications for educators and researchers are discussed.

Shared neural representations of tactile roughness intensities by somatosensation and touch observation using an associative learning method

Previous human fMRI studies have reported activation of somatosensory areas not only during actual touch, but also during touch observation. However, it has remained unclear how the brain encodes visually evoked tactile intensities. Using an associative learning method, we investigated neural representations of roughness intensities evoked by (a) tactile explorations and (b) visual observation of tactile explorations. Moreover, we explored (c) modality-independent neural representations of roughness intensities using a cross-modal classification method. Case (a) showed significant decoding performance in the anterior cingulate cortex (ACC) and the supramarginal gyrus (SMG), while in the case (b), the bilateral posterior parietal cortices, the inferior occipital gyrus, and the primary motor cortex were identified. Case (c) observed shared neural activity patterns in the bilateral insula, the SMG, and the ACC. Interestingly, the insular cortices were identified only from the cross-modal classification, suggesting their potential role in modality-independent tactile processing. We further examined correlations of confusion patterns between behavioral and neural similarity matrices for each region. Significant correlations were found solely in the SMG, reflecting a close relationship between neural activities of SMG and roughness intensity perception. The present findings may deepen our understanding of the brain mechanisms underlying intensity perception of tactile roughness.

Tactile confusions of the fingers and toes.

Recent research has shown systematic patterns of confusions between digits of the hands and feet. The present study addressed whether such confusions arise from early somatosensory maps or higher level body representations. As the glabrous and hairy skin of the hands and feet have distinct representations in somatosensory cortex, an effect arising from early somatotopic maps may show distinct patterns on each skin surface. In contrast, if the effect arises from higher level body representations which represent the digits as volumetric units, similar patterns should be apparent regardless of which side of the digit is touched. We obtained confusion matrices showing the pattern of mislocalization on the glabrous and hairy skin surfaces of the toes (Experiment 1) and fingers (Experiment 2). Our results replicated the characteristic pattern of mislocalizations found on the glabrous skin reported in previous studies. Critically, these effects were highly similar on the hairy skin surface of both the toes and fingers. Despite the pattern of mislocalizations being highly stereotyped across participants, there were consistent individual differences in the pattern of confusions across the two skin surfaces. These results suggest that mislocalizations occur at the level of individual digits, consistent with their resulting from higher level body representations. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

동일본 대지진 초기혼란 사례가 우리에게 주는 교훈: 대도시를 포함하는 광역 지진 시 감재(減災) 대책을 중심으로

동일본 대지진 초기혼란 사례가 우리에게 주는 교훈: 대도시를 포함하는 광역 지진 시 감재(減災) 대책을 중심으로

Letter recognition in different fonts

Reading performance can vary markedly with font. Mansfield and colleagues (1996) showed that people with low vision read faster in Courier than in Times, especially at small print sizes. Here, we assess single-letter recognition in different fonts, and explore whether the font effect starts to emerge at the stage of letter recognition. Twenty-five normally-sighted young adults identified single letters at 10deg eccentricity left and right of the fixation. Three fonts were tested: Courier (monospaced, serif font), Times New Roman (proportionally spaced, serif font) and Century Gothic (proportionally spaced, sans-serif font). A letter size of 0.4° (defined by x-height) was used. For a given font, each of the 26 lowercase English letters was presented ten times at each testing position. The data were collapsed across the two testing positions and 25 subjects (a total of 500 trials per letter) to construct a confusion matrix for each font. We found that accuracy of letter recognition was always highest for Courier (82%), intermediate for Times New Roman (75%), and lowest for Century Gothic (71%). The pattern of confusions also varied with font. The comparisons among the confusion matrices provided valuable information on what and how letter features contribute to letter recognition. For instance, letter "e" was often reported as "c" for Times New Roman font (45%), but not for Courier (1%) and Century Gothic (2%). Letter "w" was frequently reported as "v" for Courier font (42%) but rarely for Times New Roman (2%) and Century Gothic (1%). Letter "a" was often reported as "o" for Century Gothic font (45%) but hardly for Times New Roman (< 0.5%) and Courier (2%). Our findings suggested that the font effect emerges early in text processing. These results also provided insights on how to design or select font for better reading performance. Meeting abstract presented at VSS 2018

English phoneme recognition of vocoded speech by adult Mandarin-speaking English-learners

The present study examined the joint influence of listeners’ language experience and the degree of spectral degradation of speech signals on English phoneme recognition of L2 listeners. The participants included 27 native English-speaking listeners and 43 native Mandarin-speaking listeners who used English as an L2. The L2 participants varied in chronological age, age of English learning, length of residency in the United States, and the amount of daily-based English usage. The speech stimuli included 12 English vowels embedded in a /hVd/ context produced by four speakers and 20 English consonants embedded in a /Ca/ context produced by two speakers. The speech stimuli were processed using 2-, 4-, 6-, 8-, and 12-channel noise vocoders. The processed and original stimuli were presented to the listeners for identification in a random order. The results showed that spectral degradation had more adverse effects on the L2 listeners but the L2 disadvantage became more evident as the number of frequency increased. The L2 listeners showed different confusion patterns from the L1 listeners, which was affected by the L2 listeners’ native language experience. Furthermore, the regression analysis revealed that the L2 listener’s length of residency in the United States was a significant predictor for their phoneme recognition outcomes.

Vowel confusion patterns in children and adults with cochlear implants

Suboptimal interfaces between cochlear implant (CI) electrodes and their target neurons may distort the spectral information that is important for vowel identification. The goal of this investigation was to relate vowel confusion patterns of CI users to their site-specific (i.e., frequency-specific) variation in electrode-neuron interface across the electrode array. The electrode-neuron interface is estimated using single-channel detection thresholds with a spatially focused electrode configuration; elevated thresholds indicate channels with relatively poorer frequency transmission. We hypothesized that higher focused thresholds on channels transmitting low frequency (&amp;lt;983 Hz) information would be associated with errors in vowel height, whereas higher thresholds on channels transmitting mid- and high-frequency (&amp;gt;984 Hz) information would be related to vowel advancement errors. Vowel recognition was assessed with a closed set of medial vowels in 4-talker babble noise. Focused thresholds were measured on electrodes 2–15. Vowel confusions were examined according to the direction of errors in 2-dimensional physical articulatory and acoustic F1-F2 space. Results showed that higher average focused thresholds on the 3 most basal electrodes tested (2327–4630 Hz) were associated with perception of front vowels as more back. These findings suggest that suboptimal electrode-neuron interfaces could systematically affect CI listeners’ perception of vowel characteristics.

Orientation information in encoding facial expressions

Previous research showed that we use different regions of a face to categorize different facial expressions, e.g. mouth region for identifying happy faces; eyebrows, eyes and upper part of nose for identifying angry faces. These findings imply that the spatial information along or close to the horizontal orientation might be more useful than others for facial expression recognition. In this study, we examined how the performance for recognizing facial expression depends on the spatial information along different orientations, and whether the pixel-level differences in the face images could account for subjects’ performance. Four facial expressions—angry, fearful, happy and sad—were tested. An orientation filter (bandwidth = 23°) was applied to restrict information within the face images, with the center of the filter ranged from 0° (horizontal) to 150° in steps of 30°. Accuracy for recognizing facial expression was measured for an unfiltered and the six filtered conditions. For all four facial expressions, recognition performance (normalized d′) was virtually identical for filter orientations of −30°, horizontal and 30°, and declined systematically as the filter orientation approached vertical. The information contained in mouth and eye regions is a significant predictor for subject’s response (based on the confusion patterns). We conclude that young adults with normal vision categorizes facial expression most effectively based on the spatial information around the horizontal orientation which captures primary changes of facial features across expressions. Across all spatial orientations, the information contained in mouth and eye regions contributes significantly to facial expression categorization.

Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks

Primates, including humans, can typically recognize objects in visual images at a glance despite naturally occurring identity-preserving image transformations (e.g., changes in viewpoint). A primary neuroscience goal is to uncover neuron-level mechanistic models that quantitatively explain this behavior by predicting primate performance for each and every image. Here, we applied this stringent behavioral prediction test to the leading mechanistic models of primate vision (specifically, deep, convolutional, artificial neural networks; ANNs) by directly comparing their behavioral signatures against those of humans and rhesus macaque monkeys. Using high-throughput data collection systems for human and monkey psychophysics, we collected more than one million behavioral trials from 1472 anonymous humans and five male macaque monkeys for 2400 images over 276 binary object discrimination tasks. Consistent with previous work, we observed that state-of-the-art deep, feedforward convolutional ANNs trained for visual categorization (termed DCNNIC models) accurately predicted primate patterns of object-level confusion. However, when we examined behavioral performance for individual images within each object discrimination task, we found that all tested DCNNIC models were significantly nonpredictive of primate performance and that this prediction failure was not accounted for by simple image attributes nor rescued by simple model modifications. These results show that current DCNNIC models cannot account for the image-level behavioral patterns of primates and that new ANN models are needed to more precisely capture the neural mechanisms underlying primate object vision. To this end, large-scale, high-resolution primate behavioral benchmarks such as those obtained here could serve as direct guides for discovering such models.SIGNIFICANCE STATEMENT Recently, specific feedforward deep convolutional artificial neural networks (ANNs) models have dramatically advanced our quantitative understanding of the neural mechanisms underlying primate core object recognition. In this work, we tested the limits of those ANNs by systematically comparing the behavioral responses of these models with the behavioral responses of humans and monkeys at the resolution of individual images. Using these high-resolution metrics, we found that all tested ANN models significantly diverged from primate behavior. Going forward, these high-resolution, large-scale primate behavioral benchmarks could serve as direct guides for discovering better ANN models of the primate visual system.