Target Speech Research Articles

Speech Separation Algorithm Using Gated Recurrent Network Based on Microphone Array

Speech separation is an active research topic that plays an important role in numerous applications, such as speaker recognition, hearing prosthesis, and autonomous robots. Many algorithms have been put forward to improve separation performance. However, speech separation in reverberant noisy environment is still a challenging task. To address this, a novel speech separation algorithm using gate recurrent unit (GRU) network based on microphone array has been proposed in this paper. The main aim of the proposed algorithm is to improve the separation performance and reduce the computational cost. The proposed algorithm extracts the sub-band steered response power-phase transform (SRP-PHAT) weighted by gammatone filter as the speech separation feature due to its discriminative and robust spatial position information. Since the GRU network has the advantage of processing time series data with faster training speed and fewer training parameters, the GRU model is adopted to process the separation features of several sequential frames in the same sub-band to estimate the ideal Ratio Masking (IRM). The proposed algorithm decomposes the mixture signals into time-frequency (TF) units using gammatone filter bank in the frequency domain, and the target speech is reconstructed in the frequency domain by masking the mixture signal according to the estimated IRM. The operations of decomposing the mixture signal and reconstructing the target signal are completed in the frequency domain which can reduce the total computational cost. Experimental results demonstrate that the proposed algorithm realizes omnidirectional speech separation in noisy and reverberant environments, provides good performance in terms of speech quality and intelligibility, and has the generalization capacity to reverberate.

The impact of directionality and speech event type on target speech compression/expansion in simultaneous interpreting

Simultaneous interpreting is a complex cognitive activity that can be influenced by several factors, including source speech features (e.g., delivery rate), contextual variables, working languages, and directionality (e.g., interpreting from/into one’s native or foreign language), among others. Owing to the time constraints inherent in this interpreting mode, simultaneous interpreters must make swift decisions on how to best deliver the original message into the target language. Although explicitation is considered a universal feature of translation and interpreting, it is also true that part of (redundant) information is eventually omitted. In fact, as opposed to translated texts, interpreting corpora show a general trend of interpreted speeches being shorter than source speeches (in terms of number of words). However, a closer look at the Directionality in Simultaneous Interpreting Corpus (DIRSI) partially disconfirms such a general trend. The DIRSI corpus consists of three medical conferences mediated by simultaneous interpreters (English/Italian). Each conference is analyzed in terms of speech length to ascertain to what extent directionality and speech event type may have an impact on the interpreters’ output. Results show that directionality cannot always be linked to target speech expansion, whereas the type of speech event is likely to play a role. In particular, this applies to the interpretation of source speeches under 500 words, as interpreters adopt optimization strategies to manage politeness, source speech ungrammaticality, and integrate contextual cues.

Adversarial Examples Protect Your Privacy on Speech Enhancement System

Speech is easily leaked imperceptibly. When people use their phones, the personal voice assistant is constantly listening and waiting to be activated. Private content in speech may be maliciously extracted through automatic speech recognition (ASR) technology by some applications on phone devices. To guarantee that the recognized speech content is accurate, speech enhancement technology is used to denoise the input speech. Speech enhancement technology has developed rapidly along with deep neural networks (DNNs), but adversarial examples can cause DNNs to fail. Considering that the vulnerability of DNN can be used to protect the privacy in speech. In this work, we propose an adversarial method to degrade speech enhancement systems, which can prevent the malicious extraction of private information in speech. Experimental results show that the generated enhanced adversarial examples can be removed most content of the target speech or replaced with target speech content by speech enhancement. The word error rate (WER) between the enhanced original example and enhanced adversarial example recognition result can reach 89.0%. WER of target attack between enhanced adversarial example and target example is low at 33.75%. The adversarial perturbation in the adversarial example can bring much more change than itself. The rate of difference between two enhanced examples and adversarial perturbation can reach more than 1.4430. Meanwhile, the transferability between different speech enhancement models is also investigated. The low transferability of the method can be used to ensure the content in the adversarial example is not damaged, the useful information can be extracted by the friendly ASR. This work can prevent the malicious extraction of speech.

Sampling Rate Offset Estimation and Compensation for Distributed Adaptive Node-Specific Signal Estimation in Wireless Acoustic Sensor Networks

Sampling rate offsets (SROs) between devices in a heterogeneous wireless acoustic sensor network (WASN) can hinder the ability of distributed adaptive algorithms to perform as intended when they rely on coherent signal processing. In this paper, we present an SRO estimation and compensation method to allow the deployment of the distributed adaptive node-specific signal estimation (DANSE) algorithm in WASNs composed of asynchronous devices. The signals available at each node are first utilised in a coherence-drift-based method to blindly estimate SROs which are then compensated for via phase shifts in the frequency domain. A modification of the weighted overlap-add (WOLA) implementation of DANSE is introduced to account for SRO-induced full-sample drifts, permitting per-sample signal transmission via an approximation of the WOLA process as a time-domain convolution. The performance of the proposed algorithm is evaluated in the context of distributed noise reduction for the estimation of a target speech signal in an asynchronous WASN.

STFT-Domain Neural Speech Enhancement With Very Low Algorithmic Latency

Deep learning based speech enhancement in the short-time Fourier transform (STFT) domain typically uses a large window length such as 32 ms. A larger window can lead to higher frequency resolution and potentially better enhancement. This however incurs an algorithmic latency of 32 ms in an online setup, because the overlap-add algorithm used in the inverse STFT (iSTFT) is also performed using the same window size. To reduce this inherent latency, we adapt a conventional dual-window-size approach, where a regular input window size is used for STFT but a shorter output window is used for overlap-add, for STFT-domain deep learning based frame-online speech enhancement. Based on this STFT-iSTFT configuration, we employ complex spectral mapping for frame-online enhancement, where a deep neural network (DNN) is trained to predict the real and imaginary (RI) components of target speech from the mixture RI components. In addition, we use the DNN-predicted RI components to conduct frame-online beamforming, the results of which are used as extra features for a second DNN to perform frame-online postfiltering. The frequency-domain beamformer can be easily integrated with our DNNs and is designed to not incur any algorithmic latency. Additionally, we propose a future-frame prediction technique to further reduce the algorithmic latency. Evaluation on noisy-reverberant speech enhancement shows the effectiveness of the proposed algorithms. Compared with Conv-TasNet, our STFT-domain system can achieve better enhancement performance for a comparable amount of computation, or comparable performance with less computation, maintaining strong performance at an algorithmic latency as low as 2 ms.

Streaming End-to-End Target-Speaker Automatic Speech Recognition and Activity Detection

Automatic speech recognition of a target speaker in the presence of interfering speakers remains a challenging issue. One approach to tackle this problem is target-speaker speech recognition, which conditions the recognition process on an embedding that characterizes the voice of the target speaker. This enables recognizing only the speech of the target speaker while ignoring interferences. In this work, we propose an end-to-end target-speaker speech recognition system based on a neural transducer architecture to allow streaming and on-device recognition. Moreover, a target-speaker speech recognition system should be able to detect when the target speaker is inactive and output nothing in such a case. We introduce training and decoding schemes to allow target-speaker activity detection within our proposed recognition system. We confirm experimentally that our proposed end-to-end system performs competitively to conventional cascade approaches of a target speech extraction module and a recognition module while reducing computation costs and allowing streaming decoding.

No Link Between Speech-in-Noise Perception and Auditory Sensory Memory - Evidence From a Large Cohort of Older and Younger Listeners.

A growing literature is demonstrating a link between working memory (WM) and speech-in-noise (SiN) perception. However, the nature of this correlation and which components of WM might underlie it, are being debated. We investigated how SiN reception links with auditory sensory memory (aSM) - the low-level processes that support the short-term maintenance of temporally unfolding sounds. A large sample of old (N = 199, 60-79 yo) and young (N = 149, 20-35 yo) participants was recruited online and performed a coordinate response measure-based speech-in-babble task that taps listeners' ability to track a speech target in background noise. We used two tasks to investigate implicit and explicit aSM. Both were based on tone patterns overlapping in processing time scales with speech (presentation rate of tones 20 Hz; of patterns 2 Hz). We hypothesised that a link between SiN and aSM may be particularly apparent in older listeners due to age-related reduction in both SiN reception and aSM. We confirmed impaired SiN reception in the older cohort and demonstrated reduced aSM performance in those listeners. However, SiN and aSM did not share variability. Across the two age groups, SiN performance was predicted by a binaural processing test and age. The results suggest that previously observed links between WM and SiN may relate to the executive components and other cognitive demands of the used tasks. This finding helps to constrain the search for the perceptual and cognitive factors that explain individual variability in SiN performance.

Mandarin Electro-Laryngeal Speech Enhancement Using Cycle-Consistent Generative Adversarial Networks

Electro-laryngeal (EL) speech has poor intelligibility and naturalness, which hampers the popular use of the electro-larynx. Voice conversion (VC) can enhance EL speech. However, if the EL speech to be enhanced is with complicated tone variation rules in Mandarin, the enhancement will be less effective. This is because the source speech (Mandarin EL speech) and the target speech (normal speech) are not strictly parallel. We propose using cycle-consistent generative adversarial networks (CycleGAN, a parallel-free VC framework) to enhance continuous Mandarin EL speech, which can solve the above problem. In the proposed framework, the generator is designed based on the neural networks of a 2D-Conformer-1D-Transformer-2D-Conformer. Then, we used Mel-Spectrogram instead of traditional acoustic features (fundamental frequency, Mel-Cepstrum parameters and aperiodicity parameters). At last, we converted the enhanced Mel-Spectrogram into waveform signals using WaveNet. We undertook both subjective and objective tests to evaluate the proposed approach. Compared with traditional approaches to enhance continuous Mandarin EL speech with variable tone (the average tone accuracy being 71.59% and average word error rate being 10.85%), our framework increases the average tone accuracy by 12.12% and reduces the average errors of word perception by 9.15%. Compared with the approaches towards continuous Mandarin EL speech with fixed tone (the average tone accuracy being 29.89% and the average word error rate being 10.74%), our framework increases the average tone accuracy by 42.38% and reduces the average errors of word perception by 8.59%. Our proposed framework can effectively address the problem that the source and target speech are not strictly parallel. The intelligibility and naturalness of Mandarin EL speech have been further improved.

Effects of Target and Masker Fundamental Frequency Contour Depth on School-Age Children's Speech Recognition in a Two-Talker Masker.

Maturation of the ability to recognize target speech in the presence of a two-talker speech masker extends into early adolescence. This study evaluated whether children benefit from differences in fundamental frequency (f o) contour depth between the target and masker speech, a cue that has been shown to improve recognition in adults. Speech stimuli were recorded from talkers using three speaking styles, with f o contour depths that were Flat, Normal, or Exaggerated. Targets were open-set, declarative sentences produced by a female talker, and maskers were two streams of concatenated sentences produced by a second female talker. Listeners were children (ages 5-17 years) and adults (ages 18-24 years) with normal hearing. Each listener was tested in one of the three masker styles paired with all three target styles. Speech recognition thresholds (SRTs) corresponding to 50% correct were estimated by fitting psychometric functions to adaptive track data. For adults, performance did not differ significantly across conditions with matched speaking styles. A mismatch benefit was observed when combining Flat targets with the Exaggerated masker and Exaggerated targets with the Flat masker, and for both Flat and Exaggerated targets paired with the Normal masker. For children, there was a significant effect of age in all conditions. Flat targets in the Flat masker were associated with lower SRTs than the other two matched conditions, and a mismatch benefit was observed for young children only when the target f o contour was less variable than the masker f o contour. Whereas child-directed speech often has exaggerated pitch contours, young children were better able to recognize speech with less variable f o. Age effects were observed in the benefit of mismatched speaking styles for some conditions, which could be related to differences in baseline SRTs rather than differences in segregation abilities.

Improved Speech Spatial Covariance Matrix Estimation for Online Multi-Microphone Speech Enhancement.

Online multi-microphone speech enhancement aims to extract target speech from multiple noisy inputs by exploiting the spatial information as well as the spectro-temporal characteristics with low latency. Acoustic parameters such as the acoustic transfer function and speech and noise spatial covariance matrices (SCMs) should be estimated in a causal manner to enable the online estimation of the clean speech spectra. In this paper, we propose an improved estimator for the speech SCM, which can be parameterized with the speech power spectral density (PSD) and relative transfer function (RTF). Specifically, we adopt the temporal cepstrum smoothing (TCS) scheme to estimate the speech PSD, which is conventionally estimated with temporal smoothing. Furthermore, we propose a novel RTF estimator based on a time difference of arrival (TDoA) estimate obtained by the cross-correlation method. Furthermore, we propose refining the initial estimate of speech SCM by utilizing the estimates for the clean speech spectrum and clean speech power spectrum. The proposed approach showed superior performance in terms of the perceptual evaluation of speech quality (PESQ) scores, extended short-time objective intelligibility (eSTOI), and scale-invariant signal-to-distortion ratio (SISDR) in our experiments on the CHiME-4 database.

Writing Instruction as an Authentic Context for Targeting Speech and Language Therapy Goals for Deaf and Hard of Hearing Children

Purpose: This tutorial describes how a speech-language pathologist (SLP) might incorporate writing-based principles into therapy sessions to target a variety of speech and language goals for deaf and hard of hearing (DHH) children in addition to writing. We present an illustrative example of one SLP's experience implementing Strategic and Interactive Writing Instruction (SIWI), an approach to writing instruction designed for DHH students, within a public elementary school setting. Method: We motivate this tutorial by first reviewing the literature related to the challenges for SLPs in targeting written language within therapy settings and then discuss writing and communication difficulties for DHH students. We describe the components of SIWI with illustrative examples of how one SLP applied these principles within her therapy sessions with DHH students. The SIWI instructional approach integrates well with the roles and responsibilities of an SLP in providing therapy across a variety of communication domains for the DHH students. This tutorial describes how the SLP scaffolds production of various morphological and syntactic linguistic structures as a natural part of cocreating text with her students. The highly interactive nature of SIWI allows for targeting pragmatic language goals with student–student and student–SLP interactions. Students also have opportunities for practicing articulation when generating or revising ideas for the cocreated text and when rereading the text. Conclusions: SIWI provides a framework to address DHH students' speech and language goals within authentic writing activities that may support increased generalization into classroom academic tasks. We provide suggestions about how an SLP can incorporate the principles of SIWI into therapy sessions to integrate writing instruction with the various speech and language goals they already target as a part of implementing a student's Individualized Education Program.

The effectiveness of computer-assisted interpreting

Abstract Facing a new technological turn, the field of interpreting is in great need of evidence on the effectiveness of computer-assisted interpreting. This study proposes a computer-assisted consecutive interpreting (CACI) mode incorporating speech recognition (SR) and machine translation (MT). First, the interpreter listens to the source speech and respeaks it into an SR system, creating an SR text which is then processed by an MT system. Second, the interpreter produces a target speech with reference to the SR and MT texts. Six students participated in training on CACI, after which they performed consecutive interpreting in both the conventional and the new mode. The study finds that CACI featured fewer pauses and reduced cognitive load. Moreover, the overall interpreting quality, especially the accuracy, was increased. The effectiveness of the new mode is found to be modulated by the interpreting direction.

Event-Related Potential Correlates of Learning to Produce Novel Foreign Phonemes.

Learning to pronounce a foreign phoneme requires an individual to acquire a motor program that enables the reproduction of the new acoustic target sound. This process is largely based on the use of auditory feedback to detect pronunciation errors to adjust vocalization. While early auditory evoked neural activity underlies automatic detection and adaptation to vocalization errors, little is known about the neural correlates of acquiring novel speech targets. To investigate the neural processes that mediate the learning of foreign phoneme pronunciation, we recorded event-related potentials when participants (N = 19) pronounced native or foreign phonemes. Behavioral results indicated that the participants' pronunciation of the foreign phoneme improved during the experiment. Early auditory responses (N1 and P2 waves, approximately 85-290 ms after the sound onset) revealed no differences between foreign and native phonemes. In contrast, the amplitude of the frontocentrally distributed late slow wave (LSW, 320-440 ms) was modulated by the pronunciation of the foreign phonemes, and the effect changed during the experiment, paralleling the improvement in pronunciation. These results suggest that the LSW may reflect higher-order monitoring processes that signal successful pronunciation and help learn novel phonemes.

Performance evaluation of speech masking design among listeners with varying language backgrounds

Speech masking is a technique used for protecting speech confidentiality, which is realised by adding maskers to cover target speech. Numerous studies have demonstrated the native and non-native differences in perceiving masked speech, but few have compared the effect of masker types from the perspective of effectiveness of speech privacy and their annoyance levels. This paper reports on the result of an English word perception task by native and non-native listeners where six types of maskers, both speech-like and non-speech-like, are implemented, as well as the annoyance levels of each masker. Natives, non-natives residing in an English-speaking country, and non-natives residing in a non-English-speaking country were tasked to write down sentences when masking sound was present, and also to rate the annoyance levels of the maskers. Results showed that language background significantly affected the performance of unmasked speech, the difference between the two non-native listeners became minimal in the presence of maskers, and maskers annoyed the listener groups differently, calling for a discussion in the selection of a universally effective masker to increase speech privacy.

Theta Band (4-8 Hz) Oscillations Reflect Online Processing of Rhythm in Speech Production.

How speech prosody is processed in the brain during language production remains an unsolved issue. The present work used the phrase-recall paradigm to analyze brain oscillation underpinning rhythmic processing in speech production. Participants were told to recall target speeches aloud consisting of verb-noun pairings with a common (e.g., [2+2], the numbers in brackets represent the number of syllables) or uncommon (e.g., [1+3]) rhythmic pattern. Target speeches were preceded by rhythmic musical patterns, either congruent or incongruent, created by using pure tones at various temporal intervals. Electroencephalogram signals were recorded throughout the experiment. Behavioral results in 2+2 target speeches showed a rhythmic priming effect when comparing congruent and incongruent conditions. Cerebral-acoustic coherence analysis showed that neural activities synchronized with the rhythmic patterns of primes. Furthermore, target phrases that had congruent rhythmic patterns with a prime rhythm were associated with increased theta-band (4-8 Hz) activity in the time window of 400-800 ms in both the 2+2 and 1+3 target conditions. These findings suggest that rhythmic patterns can be processed online. Neural activities synchronize with the rhythmic input and speakers create an abstract rhythmic pattern before and during articulation in speech production.

Effective Dereverberation with a Lower Complexity at Presence of the Noise

Adaptive beamforming and deconvolution techniques have shown effectiveness for reducing noise and reverberation. The minimum variance distortionless response (MVDR) beamformer is the most widely used for adaptive beamforming, whereas multichannel linear prediction (MCLP) is an excellent approach for the deconvolution. How to solve the problem where the noise and reverberation occur together is a challenging task. In this paper, the MVDR beamformer and MCLP are effectively combined for noise reduction and dereverberation. Especially, the MCLP coefficients are estimated by the Kalman filter and the MVDR filter based on the complex Gaussian mixture model (CGMM) is used to enhance the speech corrupted by the reverberation with the noise and to estimate the power spectral density (PSD) of the target speech required by the Kalman filter, respectively. The final enhanced speech is obtained by the Kalman filter. Furthermore, a complexity reduction method with respect to the Kalman filter is also proposed based on the Kronecker product. Compared to two advanced algorithms, the integrated sidelobe cancellation and linear prediction (ISCLP) method and the weighted prediction error (WPE) method, which are very effective for removing reverberation, the proposed algorithm shows better performance and lower complexity.

A Wearable Device of 360° Speech Tracking and Enhancement for Hearing Loss Students in Class

This research proposed a new wearable device with a microphone array and camera speech tracking and enhancement over a full range of 360° of a specific speaker. The proposed sensing method combined computer vision (CV) and voice techniques to track the speech target in a flexible manner. The device could scan the specific speech source over 360° and reduces the noise due to the interference, thus enhancing the autoscan hearing-impaired process; this helps students with hearing loss to concentrate more easily during the learning process in the class. The experimental results indicated that the proposed method could locate the noise source far from the target by more than 30°, regardless of whether the CV detected the human face or not. The measured positioning accuracy is greater than 83% within a range of 360°, and the speech could be enhanced by at least 5.2 dB. The proposed wearable design for schoolchildren with speech recording also has the potential to be a tool for a review class.

Informational Masking and Listening Effort in Speech Recognition in Noise: The Role of Working Memory Capacity and Inhibitory Control in Older Adults With and Without Hearing Impairment.

The study aimed to assess the relationship between (a) speech recognition in noise, mask type, working memory capacity (WMC), and inhibitory control and (b) self-rated listening effort, speech material, and mask type, in older adults with and without hearing impairment. It was of special interest to assess the relationship between WMC, inhibitory control, and speech recognition in noise when informational maskers masked target speech. A mixed design was used. A group (N = 24) of older (Mage = 69.7 years) individuals with hearing impairment and a group of age normal-hearing adults (Mage = 59.3 years, SD = 6.5) participated in the study. The participants were presented with auditory tests in a sound-attenuated room and with cognitive tests in a quiet office. The participants were asked to rate listening effort after being presented with energetic and informational background maskers in two different speech materials used in this study (i.e., Hearing In Noise Test and Hagerman test). Linear mixed-effects models were set up to assess the effect of the two different speech materials, energetic and informational maskers, hearing ability, WMC, inhibitory control, and self-rated listening effort. Results showed that WMC and inhibitory control were of importance for speech recognition in noise, even when controlling for pure-tone average 4 hearing thresholds and age, when the maskers were informational. Concerning listening effort, on the other hand, the results suggest that hearing ability, but not cognitive abilities, is important for self-rated listening effort in speech recognition in noise. Speech-in-noise recognition is more dependent on WMC for older adults in informational maskers than in energetic maskers. Hearing ability is a stronger predictor than cognition for self-rated listening effort. https://doi.org/10.23641/asha.21357648.

Speaking in the first-person singular or plural

Abstract Conventionally, the professional interpreter speaks in the first-person singular. Research in corpus-based discourse analysis has reported shifts from this norm towards first-person plural becoming the most frequent pronoun shift in political institutional interpreting, possibly signalling interpreter ‘alignment with the institution’. Nonetheless, few studies have teased apart the simultaneous constraints of social, cognitive, and linguistic factors on institutional interpreters’ preference for the plural. The present research adopts the usage-based theory to consider the three types of explanations together. It extends recent multivariate methodologies based on this theory to analyse 2,438 first-person cases in parallel interpreting and comparable speech corpora. Following robust context analyses and cross-linguistic prosodic transcription, this study weighs the strengths of 33 associates regarding the three explanations through regression analyses. The results show that first-person shifts are better explained by chunking effects when interpreters process complex forms and referents in the source and target speeches, and when they process zero-subject source inputs. The institutional alignment explanation fails to account for the extensive grammaticalisation of plural constructions in the interpreted speech. When all the interactive and additive effects are considered together, institutional alignment or monofactorial paradigms have little explanatory power. This study concludes by highlighting the relevance of usage-based multifactorial designs to interpreting research.

The role of visual cues indicating onset times of target speech syllables in release from informational or energetic masking

• Listeners benefit from visually guided cues for the timing of syllables in noise. • Visual cues are redundant when there are auditory temporal cues from speech rhythm. • Visual cues are more effective at reducing informational than energetic masking. This study examined the effect of visual cues that provide the timing information of syllables in nonsense target sentences on the recognition of target speech against either a speech-spectrum noise masker or a two-talker masker. When the target sentences with normally regular rhythm were used (Experiment 1), visual cues indicating the onset times of the target-sentence syllables did not significantly improve target-speech recognition against either the speech-spectrum noise masker or the two-talker masker. On the other hand, recognition of the last (third) keyword in these normal rhythmic sentences was significantly better than that of the first keyword against the two-talker masker, but not the noise masker. This word-position-related speech-recognition improvement indicates a beneficial role of regular speech rhythm in release from informational masking due to the temporal prediction of target-sentence syllables. When artificially irregular rhythms were introduced into target sentences (Experiment 2), visual cues indicating syllable onset times significantly improved target-speech recognition against both types of masker. Moreover, the visual-cue-induced improvement in speech recognition against the two-talker masker was significantly larger than that against the speech-spectrum noise masker. The results of this study suggested that visual cues indicating target-syllable timing information improve target-speech recognition against competing maskers only when the temporal rhythmic regularity of target speech sentences is reduced.