Target Speech Research Articles

Performance analysis of a dilated attention fast GAN for speech enhancement

Recent advancements in speech enhancement have witnessed the emergence of generator-based methodologies. However, several of these approaches exhibit complexity in handling input variations, either excelling at low signal-to-noise ratios (SNRs) by utilizing intricate representations of noisy and clean speech or demonstrating superior performance only at higher SNRs. In this work, we investigated speech enhancement using a Dilated Attention Fast Generative Adversarial Network (DAF-GAN). The proposed DAF-GAN framework achieves stability in performance across different SNR conditions by efficiently processing large-scale signal lengths. The DFS-GAN features a dilated discriminator model operating via patches. The generator architecture incorporates multi-decoding and attention gates facilitated through skip-connections, strategically integrated within the Fast-U-Net model to optimize processing speed. An ideal ratio mask was used in the test phase to further refine the enhanced signal by emphasizing target speech while suppressing residual noise or artifacts. The DAF-GAN performance was assessed using objective metrics such as PESQ on a number of noisy speech databases. Results revealed that the DAF-GAN performed modestly in comparison with the state-of-the-art models. For example, analyses of the VoiceBank-DEMAND dataset yielded a PESQ score of 2.50 for the DAF-GAN.

Perceptual consequences of reverberant environments on spatial unmasking

Spatial hearing provides access to auditory spatial cues that promote speech perception in noisy listening situations. However, reverberation degrades auditory spatial cues and limits listeners’ ability to utilize these cues for segregating target speech from competing babble. Hence, spatial unmasking—an intelligibility benefit from a spatial separation between a target and masker—is reduced in reverberant environments as compared to free field. To understand the perceptual consequences of poorer spatial unmasking in reverberation, we assessed three aspects of functional spatial hearing in virtual reverberant environments: perceived auditory source width, auditory spatial acuity, and spatial unmasking. Three auditory environments were simulated and auralized using ODEON to vary interaural coherence (IC): (1) a control anechoic environment, (2) a classroom designed to meet classroom acoustics standards (IC = 0.58), and (3) a classroom of the same size with more severe reverberation (IC = 0.37). Individually measured head-related transfer functions were used to binaurally reproduce the auralized signals over headphones. We hypothesize that interaural decorrelation, the result of increasing reverberation, will broaden the perceived auditory source width with a cascading effect of reduced auditory spatial acuity and subsequently poorer spatial unmasking. Preliminary data from normal-hearing adults will be presented. [Work supported by NIH R21DC020532.]

The theoretical maximum intelligibility improvement that can be provided by hearing aids

One goal of hearing aids is to increase the intelligibility of speech by improving the signal-to-noise ratio through the use of directional microphones and/or digital noise reduction. With this objective in mind, an ideal hearing aid would output amplified target speech with a signal-to-noise ratio of +∞. However, even in this ideal scenario, vent effects would allow background noise to enter the ear canal via the direct sound path while also allowing amplified sound to escape from the ear canal. In this experiment, a range of hearing aid fittings were simulated to account for these vent effects. Hearing-impaired listeners were tested using the ideal simulated hearing aid, and the results will show the maximum theoretical hearing aid benefit.

A Study of Blockchain and Machine Learning-Enabled IoT Security in Time-Delayed Neural Network Vocal Pattern Recognition to Improve Web-Based Vocal Teaching

With the development of information technology, online vocal teaching is becoming more and more popular, but the sound quality of teaching is also becoming more and more demanding. As online vocal instruction becomes more popular, the need for high-quality sound in these digital environments becomes more critical. This research tackles the problem of improving sound quality in real-time vocal teaching by integrating advanced technologies such as Blockchain and Machine Learning within the Internet of Things (IoT) security framework. We created a vocal recognition model using Time-Delay Neural Network (TDNN) and improved it with Generated Feature Vector (GFV). This integration yields a strong GTDNN vocal recognition system that is specifically designed to secure and optimize web-based vocal teaching. Our experiments show that GTDNN outperforms traditional TDNN and i-vector methods in feature vector extraction, adapting well to different speech environments. In various speech settings, GTDNN's Error Rates (EERs) are impressively low at 11.3%, 12.0%, 4.9%, 6.2%, and 6.1%, indicating superior performance over comparison models. GTDNN has an EER of 9.6% for short-duration speech and 2.3% for long-duration speech. Furthermore, the GTDNN system achieves an overall pass rate of 94% for target speech and an impressive rejection rate for non-target speech, ensuring high accuracy in a variety of speech environments.

Utilizing Translation to Enhance NLP Models in Offensive Language and Hate Speech Identification

The number of social media users in Indonesia has increased in recent years. The surge in social media users leads to more offensive language on these platforms. The use of offensive language can trigger conflicts between users. Therefore, it is necessary to identify the use of offensive language on social media. This study focused on identifying offensive language, hate speech, and hate speech targets on Twitter. The data used were obtained from previous research on identifying offensive language and hate speech. The amount of data is very influential on the performance of the classification. Therefore, data was added using translation in this study. Classical machine learning (SVM et al.) and deep learning (BiLSTM, CNN, and LSTM) algorithms are used as classification algorithms with word n-gram and word embedding as the features. Three scenarios were done based on the training data used in the classification model development. The result shows that scenario 3, which uses translation for data augmentation, can improve the classification model’s performance by 5%.

Zero-shot test time adaptation via knowledge distillation for personalized speech denoising and dereverberation.

A personalization framework to adapt compact models to test time environments and improve their speech enhancement (SE) performance in noisy and reverberant conditions is proposed. The use-cases are when the end-user device encounters only one or a few speakers and noise types that tend to reoccur in the specific acoustic environment. Hence, a small personalized model that is sufficient to handle this focused subset of the original universal SE problem is postulated. The study addresses a major data shortage issue: although the goal is to learn from a specific user's speech signals and the test time environment, the target clean speech is unavailable for model training due to privacy-related concerns and technical difficulty of recording noise and reverberation-free voice signals. The proposed zero-shot personalization method uses no clean speech target. Instead, it employs the knowledge distillation framework, where the more advanced denoising results from an overly large teacher work as pseudo targets to train a small student model. Evaluation on various test time conditions suggests that the proposed personalization approach can significantly enhance the compact student model's test time performance. Personalized models outperform larger non-personalized baseline models, demonstrating that personalization achieves model compression with no loss in dereverberation and denoising performance.

Processing manner under high cognitive pressure: Evidence from French–English and English–French simultaneous interpreting

Abstract The expression of manner has been extensively studied in the case of motion event descriptions, unveiling significant typological differences between satellite-framed and verb-framed languages and cognitive differences between speakers of these languages. However, far from being restricted to this semantic domain, the expression of manner extends to other types of event descriptions and across virtually all verb classes. In this paper, by considering all the means of expressing manner and grounding our research in a domain-independent definition of this component, we investigate the expression and the transfer of manner under high cognitive pressure as evidenced by corpus data from French–English (FE) and English–French (EF) simultaneous interpreting. Unexpectedly, both French and English displayed an overall cross-domain preference for the verbal-lexical coding of manner in event descriptions, while still differing in the degree to which it was favored. In addition, although our study does not allow a direct measure of cognitive load, the FE interpreters transfer more manner from the source to the target speeches than the EF interpreters do, despite high pressure on cognitive resources, supporting the claim that manner can be cognitively more salient and accessible for English than for French speakers, not only in the domain of motion but also at a more general level, potentially in any semantic domain.

Autistic traits and speech perception in social and non-social noises

Individuals with the autism spectrum disorder (ASD) experience difficulties in perceiving speech in background noises with temporal dips; they also lack social orienting. We tested two hypotheses: (1) the higher the autistic traits, the lower the performance in the speech-in-noise test, and (2) individuals with high autistic traits experience greater difficulty in perceiving speech, especially in the non-vocal noise, because of their attentional bias toward non-vocal sounds. Thirty-eight female Japanese university students participated in an experiment measuring their ability to perceive speech in the presence of noise. Participants were asked to detect Japanese words embedded in vocal and non-vocal background noises with temporal dips. We found a marginally significant effect of autistic traits on speech perception performance, suggesting a trend that favors the first hypothesis. However, caution is needed in this interpretation because the null hypothesis is not rejected. No significant interaction was found between the types of background noise and autistic traits, indicating that the second hypothesis was not supported. This might be because individuals with high autistic traits in the general population have a weaker attentional bias toward non-vocal sounds than those with ASD or to the explicit instruction given to attend to the target speech.

Keyword Guided Target Speech Recognition

Keyword Guided Target Speech Recognition

Digital Misogyny Unveiled: An Examination of Online Hate Speech Targeting Women in Bangladesh

Discriminatory and sexist language expresses bias towards men and projects that women are inferior around the world. It is also evident in the patriarchal society of Bangladesh. While they are often prevalent subtly in communication taking place in person, the anonymous nature of the internet has enabled people to be vile towards women in the virtual world. Online hate speech and misogynistic behavior have become pervasive issues in today's digital landscape. Most people take Islamic verses literally or believe the religious clerics without checking their words' authenticity. The socio-religious idea of women being subservient to men produces numerous hate speeches against women on social media platforms. The presence of numerous online religious lectures spewing offensive words against women made the situation worse in Bangladesh. They conform to the prevalent idea that men should control women's agency, and for that purpose, even violence is acceptable. This article aims to provide a comprehensive analysis of the phenomenon, specifically focusing on the context of Bangladesh. Through case studies, it looks into how religious leaders fuel online hatred towards women.It further explores the various forms, impacts, and underlying factors of misogynistic online speech. By delving into the complexities of this issue, it also discusses the challenges faced by women in seeking redress and proposes recommendations to mitigate online violence against women in Bangladesh.

Contribution of speech rhythm to understanding speech in noisy conditions: Further test of a selective entrainment hypothesis.

Previous work by McAuley et al. Attention, Perception, & Psychophysics, 82, 3222-3233, (2020), Attention, Perception & Psychophysics, 83, 2229-2240, (2021) showed that disruption of the natural rhythm of target (attended) speech worsens speech recognition in the presence of competing background speech or noise (a target-rhythm effect), while disruption of background speech rhythm improves target recognition (a background-rhythm effect). While these results were interpreted as support for the role of rhythmic regularities in facilitating target-speech recognition amidst competing backgrounds (in line with a selective entrainment hypothesis), questions remain about the factors that contribute to the target-rhythm effect. Experiment 1 ruled out the possibility that the target-rhythm effect relies on a decrease in intelligibility of the rhythm-altered keywords. Sentences from the Coordinate Response Measure (CRM) paradigm were presented with a background of speech-shaped noise, and the rhythm of the initial portion of these target sentences (the target rhythmic context) was altered while critically leaving the target Color and Number keywords intact. Results showed a target-rhythm effect, evidenced by poorer keyword recognition when the target rhythmic context was altered, despite the absence of rhythmic manipulation of the keywords. Experiment 2 examined the influence of the relative onset asynchrony between target and background keywords. Results showed a significant target-rhythm effect that was independent of the effect of target-background keyword onset asynchrony. Experiment 3 provided additional support for the selective entrainment hypothesis by replicating the target-rhythm effect with a set of speech materials that were less rhythmically constrained than the CRM sentences.

The Communication Success Screener: A Preliminary Investigation of Perceived Communicative Success Across Modalities, Environments, and Demands.

Behavioral intervention targeting speech, language, and communication concerns is an established therapeutic approach for patients with communication concerns across a range of acquired neurogenic disorders. The multidimensional factors that contribute to a person's self-identified communication challenges and strengths in their daily communication needs must be considered to provide functional and person-centered care. While assessments grounded in clinician observation or client self-report exist, there is a direct need for a screening tool that comprehensively evaluates the roles of modality (verbal, text, gesture) and environment (in-person, virtual) on self-reported success across communicative demands. In this study, we describe a preliminary approach to monitoring the progression of receptive and expressive communication skills in people with chronic poststroke aphasia in the context of communication practices of the 21st century, culminating in the development and exploratory implementation of a novel clinical instrument: the Communication Success Screener (COMSS). Thirty-three participants with aphasia due to stroke were recruited to complete and evaluate the COMSS via an online survey. Quantitative responses and open-ended participant feedback were collected to validate and propose adaptations to the COMSS. Group-level analyses and case presentations were used to highlight COMSS features and outcomes. Participant responses to the COMSS questionnaire suggest that this screening tool creates differentiated communicative success profiles based on self-report. Participant feedback also indicated that the COMSS appropriately evaluates self-reported success across modalities of verbal, text, and gesture in the context of in-person and virtual environments. The communication concerns experienced by people with aphasia are often heterogeneous and vary as a function of their daily activities of living, preferred language modalities, and environmental supports. We present preliminary feasibility analyses of the COMSS as a potential tool to support the monitoring of functional change by evaluating self-reported communicative success across form, modality, environment, and task demand. https://doi.org/10.23641/asha.24521044.

Any-to-One Non-Parallel Voice Conversion System Using an Autoregressive Conversion Model and LPCNet Vocoder

We present an any-to-one voice conversion (VC) system, using an autoregressive model and LPCNet vocoder, aimed at enhancing the converted speech in terms of naturalness, intelligibility, and speaker similarity. As the name implies, non-parallel any-to-one voice conversion does not require paired source and target speeches and can be employed for arbitrary speech conversion tasks. Recent advancements in neural-based vocoders, such as WaveNet, have improved the efficiency of speech synthesis. However, in practice, we find that the trajectory of some generated waveforms is not consistently smooth, leading to occasional voice errors. To address this issue, we propose to use an autoregressive (AR) conversion model along with the high-fidelity LPCNet vocoder. This combination not only solves the problems of waveform fluidity but also produces more natural and clear speech, with the added capability of real-time speech generation. To precisely represent the linguistic content of a given utterance, we use speaker-independent PPG features (SI-PPG) computed from an automatic speech recognition (ASR) model trained on a multi-speaker corpus. Next, a conversion model maps the SI-PPG to the acoustic representations used as input features for the LPCNet. The proposed autoregressive structure enables our system to produce the following prediction step outputs from the acoustic features predicted in the previous step. We evaluate the effectiveness of our system by performing any-to-one conversion pairs between native English speakers. Experimental results show that the proposed method outperforms state-of-the-art systems, producing higher speech quality and greater speaker similarity.

Predicting user engagement with anti-gender, homophobic and sexist social media posts – a choice-based conjoint study in Hungary and Germany

ABSTRACT Social media are used by populist radical right parties for anti-gender campaigns to mobilize against the ‘gender ideology’. Anti-gender hate speech targets women, sexual minorities, and feminist activists and therefore poses a threat to their participation in society. In this study, we examine the engagement with anti-gender hate speech on social media in Germany and Hungary, using anti-gender, homophobic, and sexist Facebook posts as examples. These countries were chosen because of the cultural differences in the way gender issues and discrimination are dealt with, as well as the position of populist radical right parties. Employing a choice-based conjoint design, we analyze which content and source characteristics as well as cultural and individual factors influence users’ engagement with such postings. Findings show country differences in the acceptance (likes, shares) and disapproval (flagging behavior) of anti-gender hate speech but also with respect to relevance of the three main topics of hate speech (anti-gender, homophobia, sexism).

The Role of Instructions in Motor Learning of Oral Versus Nasalized Speech Targets.

The purpose of this study was to investigate how general, implicit instructions with auditory-perceptual emphasis; specific, explicit instructions with biomechanical focus; or both affect learning of oral-nasal balance control in speech. Thirty healthy, vocally untrained participants were assigned to one of three instructional groups (i.e., implicit, explicit, and integrated) and learned to produce oral versus nasalized vowel-, syllable-, and phrase-level targets during once-weekly sessions over 4 weeks. Learning gains and performance variability were analyzed using nasometry. We observed a significant main effect of instruction type on learning gains at phrase level (p = .016). Specifically, the integrated group (M = 59.8%) significantly outperformed the explicit group (M = 37.9%) and numerically outperformed the implicit group (M = 45.1%). For nasalized phrase targets, results revealed a significant main effect of instruction type on performance variability (p = .042), but pairwise comparisons between instruction groups were not significant. The integration of implicit processes via auditory-perceptual modeling and explicit processes via relevant biomechanical directives resulted in larger motor learning gains, especially at higher levels of task complexity (i.e., phrase) compared to providing implicit or explicit instruction alone. The higher performance variability (i.e., less stable productions) that was sometimes induced by explicit instruction did not negatively impact learning when integrated with implicit instruction. Clinical implications for speech/voice therapy models are discussed.

Selective listening in checkerboard and interrupted speech stimuli with two talkers

A two-talker situation is tougher than a single-talker situation when one has to selectively listen to a target speech. To simplify experimental conditions, we focused on the situation that two talkers speak different sentences alternating with each other. At the same time, we introduced two types of alternations: alternation only in time (staggered speech) and alternation in time and frequency (two-talker checkerboard speech). To our knowledge, the intelligibility of two-talker checkerboard speech stimuli had been never investigated. Three steps of segment duration (20, 80, and 320 ms) for both stimulus types and three steps of the number of frequency bands (4, 8, and 20) for checkerboard speech stimuli were employed. Japanese native listeners (N = 10 or 22; age range: 21–34) successfully heard out a specific talker’s sentence while ignoring the other for both stimulus types. The intelligibility curves of two-talker checkerboard speech stimuli exhibited characteristic U-shaped curves as found in the previous investigations with one-talker checkerboard speech stimuli (although the intelligibility for two-talker checkerboard speech stimuli went down by 5–20%), suggesting that a common auditory grouping mechanism works in this case also.

Dissociated neural tracking as an index of speech understanding in background noise

Human brain activity has been shown to track hierarchical linguistic units embedded in connected speech and these responses can be directly modulated by changes in speech intelligibility caused by spectral degradation or prior knowledge. In this study, we introduce a background noise and manipulate its level relative to the target speech to test the hypothesis that the tracking responses are modulated differently by the variations in Signal-to-Noise Ratio (SNR) and changes in speech intelligibility. Electroencephalography (EEG) responses to speech in quiet and in noise (multi-talker babble) at three different levels were measured from 19 normal hearing participants. Driven by the reduction in intelligibility, cortical coherence to “abstract” linguistic units with no accompanying acoustic cues was reduced relative to the “speech only” condition, and lateralized to single cerebral hemispheres. In contrast, brain responses coherent to words, aligned with acoustic onsets, were bilateral and reduced systematically as noise level increased. Strength of the tracking response correlated with subjective ratings from each participant on how much they can understand the speech sentences at all different linguistic levels. These results provide an objective and sensitive neural marker of speech intelligibility which can be further developed into clinical applications for objective assessment of speech-in-noise understanding.

Measuring listeners’ distraction to spatially presented masking noise conditions

This study investigates listeners' biological responses to speech under different masking noise conditions during a spatial selective attention task. Ten participants with normal hearing took part in the experiment which measured both behavioral and biological responses during a speech identification task with varying noise streams. The target speech stream was played through a front-left speaker, while masking noise, consisting of music and non-intelligent speech, was presented through either the front-right or back-center speaker. Behavioral performance was assessed based on the correct answer ratio, while biological responses were analyzed using Event-Related Potentials (ERPs) for four distinct conditions. These conditions included two Signal-to-Noise Ratios (SNRs) and two spatial positions of the masking noise stream. Using an EEG measurement headset with dry electrodes, the study collected neural responses from the participants and calculated the ERPs for each of the four conditions. After excluding one outlier, the ANOVA result revealed a difference (F(1, 3.5), p = 0.0984) in this listener group's ERP response between the two masker positions, with a lower mean peak value observed for the rear masker. This finding suggests that they were less influenced by the rear-located masking noise.

Individual differences in speech-on-speech masking are correlated with cognitive and visual task performance.

Individual differences in spatial tuning for masked target speech identification were determined using maskers that varied in type and proximity to the target source. The maskers were chosen to produce three strengths of informational masking (IM): high [same-gender, speech-on-speech (SOS) masking], intermediate (the same masker speech time-reversed), and low (speech-shaped, speech-envelope-modulated noise). Typical for this task, individual differences increased as IM increased, while overall performance decreased. To determine the extent to which auditory performance might generalize to another sensory modality, a comparison visual task was also implemented. Visual search time was measured for identifying a cued object among "clouds" of distractors that were varied symmetrically in proximity to the target. The visual maskers also were chosen to produce three strengths of an analog of IM based on feature similarities between the target and maskers. Significant correlations were found for overall auditory and visual task performance, and both of these measures were correlated with an index of general cognitive reasoning. Overall, the findings provide qualified support for the proposition that the ability of an individual to solve IM-dominated tasks depends on cognitive mechanisms that operate in common across sensory modalities.

How well do hearing aids amplify “speech glimpses” in multitalker mixtures?

Reductions in speech audibility are more detrimental in the presence of competing sounds, where there is little redundant speech information, than in quiet. Indeed, our previous work suggests that insufficient audibility of “speech glimpses” may partly explain the poor performance of listeners with hearing loss in multitalker mixtures. An implication of those results is that restoring audibility across the spectrum may be especially critical in such situations. Here we asked whether current hearing-aid amplification strategies adequately restore the audibility of speech glimpses. We used ideal time-frequency separation to isolate target speech glimpses from unaided and aided speech mixtures. We then measured intelligibility for the isolated glimpses and for the original mixtures. Participants were young adults with bilateral sensorineural hearing loss, and individualized non-linear amplification (NAL-NL2) was provided using the Oldenburg open Master Hearing Aid. Results show that amplification generally improved the audibility and intelligibility of the target speech glimpses. For some participants, the intelligibility improvements carried over to the mixtures. For others, the improvements were diminished in the mixtures, suggesting that there were counteracting effects. This approach may provide a useful way to unpack positive and negative effects of hearing-aid processing in multitalker mixtures.