Speech Sources Research Articles

Spectral and Spatial Characteristics of the Activity of Brain Structures, Participating in the Perception and Production of Speech

Spectral and spatial characteristics of the encephalograms, registered while speech perception and production, are considered. Systematical bibliographical review is presented, including the articles studying the speech sources spectra and their location in the brain. Encephalography is selected as a basic experimental approach. Advantages of the magnetic encephalography, experimental difficulties and possible artifacts are noted. It is concluded that brain speech activity possesses a great variety of spectral and spatial features. The method of functional tomography based on magnetic encephalography data is proposed to quantitatively analyze this activity in detail. The method makes it possible to extract and precisely localize in space various spectral features of the brain activity studied in experiments on speech research.

Les enjeux de la traduction arabe des antithèses dans la presse française

Any translator has the mission to translate a text or interpret a speech while retaining the meaning and while remaining as faithful as possible to the source text or speech. The goal of the translator is to transfer the text from the source language to the target language so that the meaning and style of writing remain the same. However, because of the differences between languages, it is often difficult to preserve an exact translation of the source text. The translator is therefore confronted with issues in his translation work, especially when it comes to figurative meanings. This article discusses some of these issues in journalistic texts. We propose in this study to examine the translation of the antithesis in some French newspapers and its translation into Arabic, thus shedding light on the way in which the translators manage their work and the different types of translation that they may be confronted with.

Motion capture-based animated characters for the study of speech\u2013gesture integration

Digitally animated characters are promising tools in research studying how we integrate information from speech and visual sources such as gestures because they allow specific gesture features to be manipulated in isolation. We present an approach combining motion capture and 3D-animated characters that allows us to manipulate natural individual gesture strokes for experimental purposes, for example to temporally shift and present gestures in ecologically valid sequences. We exemplify how such stimuli can be used in an experiment investigating implicit detection of speech–gesture (a) synchrony, and discuss the general applicability of the workflow for research in this domain.

Sparsity-based phase spectrum compensation for single-channel speech source separation

This paper proposes a sparsity-based phase spectrum compensation (SPSC) function to improve the quality of reconstructed signals by using magnitude-based single-channel speech source separation. While conventional approaches to the reconstruction of separated sources use the input phase spectrum, the proposed SPSC function modifies each source's phase spectrum using the estimated magnitude spectra of multiple sources and their spectro-temporal sparsity. In particular, the spectro-temporal sparsity is estimated from the signal-to-interference ratio between the magnitude spectrum of a source to be separated and those of the other sources, including the background noise. The proposed SPSC function is first embedded into the reconstruction stage of magnitude-based speech source separation methods that employ deep recurrent neural network (DRNN) and sparse nonnegative matrix factorization (SNMF), respectively. Then, speech denoising is performed under four different noise conditions with signal-to-noise ratios (SNRs) in the range of 0–15 dB. Both objective and subjective tests show that both DRNN and SNMF-based speech separation methods employing the proposed SPSC function substantially outperform those employing the conventional PSC function for the speech denoising task. Moreover, the DRNN combined with the proposed SPSC function offers higher average perceptual evaluation of speech quality (PESQ) scores for speech denoising than SEGAN, which is based on a very large end-to-end neural networks model. Next, the proposed SPSC method is also applied on the reconstruction stage of a deep clustering-based speech separation method to examine its contribution on a multi-talker speech separation task, where spoken utterances of males and females are mixed at −3, 0, and 3 dB SNR. It is shown from the experiment that the proposed SPSC function further improves SDR and PESQ scores of the magnitude spectra processed by the deep clustering-based single-channel speech separation method.

Semi-Supervised Multichannel Speech Enhancement With a Deep Speech Prior

This paper describes a semi-supervised multichannel speech enhancement method that uses clean speech data for prior training. Although multichannel nonnegative matrix factorization (MNMF) and its constrained variant called independent low-rank matrix analysis (ILRMA) have successfully been used for unsupervised speech enhancement, the low-rank assumption on the power spectral densities (PSDs) of all sources (speech and noise) does not hold in reality. To solve this problem, we replace a low-rank speech model with a deep generative speech model, i.e., formulate a probabilistic model of noisy speech by integrating a deep speech model, a low-rank noise model, and a full-rank or rank-1 model of spatial characteristics of speech and noise. The deep speech model is trained from clean speech data in an unsupervised auto-encoding variational Bayesian manner. Given multichannel noisy speech spectra, the full-rank or rank-1 spatial covariance matrices and PSDs of speech and noise are estimated in an unsupervised maximum-likelihood manner. Experimental results showed that the full-rank version of the proposed method was significantly better than MNMF, ILRMA, and the rank-1 version. We confirmed that the initialization-sensitivity and local-optimum problems of MNMF with many spatial parameters can be solved by incorporating the precise speech model.

Integration of a Priori and Estimated Constraints Into an MVDR Beamformer for Speech Enhancement

Conventionally, the single constraint of the minimum variance distortionless response (MVDR) beamformer for speech enhancement has been defined using one of two approaches. Either it is based on a priori assumptions such as microphone characteristics, position, speech source location, and room acoustics, or on a relative transfer function (RTF) vector estimate using a data dependent method. Each approach has its respective merits and drawbacks and a decision usually has to be made between one of the approaches. In this paper, an alternative approach of using an integrated MVDR beamformer is investigated, where both the hard constraints from the two conventional approaches are softened to yield two tuning parameters. It will be shown that this integrated MVDR beamformer can be expressed as a convex combination of the conventional MVDR beamformers, a linearly constrained minimum variance (LCMV) beamformer, and an all-zero vector, with real, positive-valued coefficients. By analysing how the tuning parameters affect these coefficients, two tuning rules for a practical implementation of the integrated MVDR are subsequently proposed. An evaluation with simulated and recorded data demonstrates that the integrated MVDR beamformer can be beneficial as opposed to relying on either of the conventional MVDR beamformers.

An order-aware scheme for robust direction of arrival estimation in the spherical harmonic domain.

Direction of arrival (DOA) estimation of sound sources using a spherical microphone array is usually performed in the spherical harmonic (SH) domain. In a non-noisy environment, it suffices to use only the zeroth- and first-order spherical harmonic beams (SHBs) in the SH domain for DOA estimation. One such method is based on the pseudo-intensity vector (PIV), which is attractive due to its low computational complexity. To improve the performance of the PIV method in reverberant environments, some methods have been proposed recently to further exploit high-order SHBs. However, these methods ignore the effect of noise on high-order SHBs, which may lead to poor performance in low signal-to-noise ratio (SNR) environments. To address the problem, this paper proposes an order-aware scheme that is able to select the high-order SHBs reliable for robust DOA estimation of multiple speech sources. Simulation and real-world experimental results demonstrate that the order-aware scheme based methods outperform their existing counterparts with less computational complexity in terms of both accuracy and robustness of DOA estimation. Moreover, the performance improvement is more significant in low SNR environment and in a scenario with small angular separation of sources.

A Speech Synthesis Approach for High Quality Speech Separation and Generation

We propose a new method for source separation by synthesizing the source from a speech mixture corrupted by various environmental noise. Unlike traditional source separation methods which estimate the source from the mixture as a replica of the original source (e.g. by solving an inverse problem), our proposed method is a synthesis-based approach which aims to generate a new signal (i.e. “fake” source) that sounds similar to the original source. The proposed system has an encoder-decoder topology, where the encoder predicts intermediate-level features from the mixture, i.e. Mel-spectrum of the target source, using a hybrid recurrent and hourglass network, while the decoder is a state-of-the-art WaveNet speech synthesis network conditioned on the Mel-spectrum, which directly generates time-domain samples of the sources. Both objective and subjective evaluations were performed on the synthesized sources, and show great advantages of our proposed method for high-quality speech source separation and generation.

A class of multichannel sparse linear prediction algorithms for time delay estimation of speech sources

Time delay estimation (TDE), which is also called time difference of arrival estimation, is an important yet challenging problem in room acoustic environments where reverberation and noise coexist. The multichannel cross-correlation coefficient algorithm extends the traditional cross-correlation method from the two- to the multiple-channel cases and exploits the spatial information among all microphones to improve the robustness of TDE with respect to noise. The multichannel spatiotemporal prediction algorithm generalizes the multichannel cross-correlation coefficient algorithm by incorporating both the spatial and temporal information to make TDE robust to reverberation. This multichannel spatiotemporal prediction algorithm, however, is sensitive to noise. In this work, we attempt to improve the robustness of this algorithm by making it robust to both reverberation and noise. Based on the sparsity of the prediction coefficient matrix of speech signals, a class of multichannel sparse linear prediction algorithms, including the multichannel spatiotemporal sparse prediction and the multichannel spatiotemporal group sparse prediction, are developed for TDE. The multichannel cross-correlation coefficient and multichannel spatiotemporal prediction algorithms are unified from a TDE performance perspective via an F/ℓ1-norm (or F/ℓ1,2-norm) optimization model, which is solved by an alternating direction method of multipliers. The two new algorithms also respectively construct a set of time delay estimators, which make different tradeoffs between prewhitening and non-prewhitening by adjusting a regularization parameter.

Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translation

Speech translation has traditionally been approached through cascaded models consisting of a speech recognizer trained on a corpus of transcribed speech, and a machine translation system trained on parallel texts. Several recent works have shown the feasibility of collapsing the cascade into a single, direct model that can be trained in an end-to-end fashion on a corpus of translated speech. However, experiments are inconclusive on whether the cascade or the direct model is stronger, and have only been conducted under the unrealistic assumption that both are trained on equal amounts of data, ignoring other available speech recognition and machine translation corpora. In this paper, we demonstrate that direct speech translation models require more data to perform well than cascaded models, and although they allow including auxiliary data through multi-task training, they are poor at exploiting such data, putting them at a severe disadvantage. As a remedy, we propose the use of end- to-end trainable models with two attention mechanisms, the first establishing source speech to source text alignments, the second modeling source to target text alignment. We show that such models naturally decompose into multi-task–trainable recognition and translation tasks and propose an attention-passing technique that alleviates error propagation issues in a previous formulation of a model with two attention stages. Our proposed model outperforms all examined baselines and is able to exploit auxiliary training data much more effectively than direct attentional models.

Human Frequency Following Response Correlates of Spatial Release From Masking.

Purpose Speech recognition in complex listening environments is enhanced by the extent of spatial separation between the speech source and background competing sources, an effect known as spatial release from masking (SRM). The aim of this study was to investigate whether the phase-locked neural activity in the central auditory pathways, reflected in the frequency following response (FFR), exhibits SRM. Method Eighteen normal-hearing adults (8 men and 10 women, ranging in age from 20 to 42 years) with no known neurological disorders participated in this study. FFRs were recorded from the participants in response to a target vowel /u/ presented with spatially colocated and separated competing talkers at 3 ranges of signal-to-noise ratios (SNRs), with median SNRs of -5.4, 0.5, and 6.8 dB and for different attentional conditions (attention and no attention). Results Amplitude of the FFR at the fundamental frequency was significantly larger in the spatially separated condition as compared to the colocated condition for only the lowest (< -2.4 dB SNR) of the 3 SNR ranges tested. A significant effect of attention was found when subjects were actively focusing on the target stimuli. No significant interaction effects were found between spatial separation and attention. Conclusions The enhanced representation of the target stimulus in the separated condition suggests that the temporal pattern of phase-locked brainstem neural activity generating the FFR may contain information relevant to the binaural processes underlying SRM but only in challenging listening environments. Attention may modulate FFR fundamental frequency amplitude but does not seem to modulate spatial processing at the level of generating the FFR. Supplemental Material https://doi.org/10.23641/asha.9992597.

Evaluation of acoustical conditions for speech communication in active university classrooms

One can measure speech and noise levels separately but then the noise is not the same as experienced during the speech communication in active classrooms. In this study, the acoustical conditions in university classrooms during actual lecturing situations were measured and evaluated based on real recordings of combined speech and noise sounds. Acoustical measurements and recordings were made during actual 15 lectures in 11 university classrooms in Korea. A statistical method was used for obtaining octave band speech and noise levels in university classrooms during actual lectures from the recordings of combined speech and noise sounds. In the 11 active Korean university classrooms the adult students experienced on average: speech levels of 51.5 dBA (s.d. = 2.7 dBA), noise levels of 44.3 dBA (s.d. = 2.1 dBA), and a speech-to-noise ratio of 7.2 dBA (s.d. = 2.7 dBA). The mean speech source level of the 15 Korean instructors was 11 to 12 dBA lower than the two previous studies made in the 11 University of British Colombia classrooms (64.5 dBA, s.d. = 4.2 dBA) and 27 elementary school classrooms (65.3 dBA, s.d. = 2.5 dBA) in Canada. Both measures, the useful-to-detrimental ratio, U50(125–4 k), and the speech transmission index, STI, for speech intelligibility are highly correlated and essentially assess almost the same characteristics of university classrooms. The present results show that university classrooms having U50(125–4 k) values greater than +4.1 dB, will have STI values of 0.60 or higher, which indicates ‘Good’ acoustical conditions for speech intelligibility.

Structural Organization and Features of Speech and Music Perception in Language Functions’ Implementation with Timbre’s Perceptive Assessment

In cognitive science, the psychological and biological foundations of music are often considered in the context of comparison with the functions of speech and language. The article describes some aspects of the similarities and differences between music and speech in the implementation of language functions, as well as a typological study of its features according to Buhler, Jakobson, Halliday and Kiklewicz classifications. A general comparison of structural organization and main perceptual characteristics of music and speech signals is carried out additionally, with separate consideration of neurophysiological foundations of the auditory analysis of timbre characteristics of speech sources (speaker's voice) and music (instrument sounds). Some aspects of the practical application of auditory and musical training in the context of training and rehabilitation measures for sensory-cognitive dysfunctions of various genesis are explained, with the prospects of further research aimed at studying the influence of the timbre on the indices of spatial selectivity of musical perception in comparison with the characteristics of the spatial selectivity of speech hearing in the perception of complex acoustic scenes.

Underdetermined blind source separation using CapsNet

In this paper, we consider the problem of separating the speech source signal from the underdetermined convolutive mixture signals using capsule network (CapsNet). The objective of this paper is twofold. They are (1) to improve the underdetermined convolutive blind source separation algorithm in terms of signal-to-distortion ratio, signal-to-interference ratio and signal-to-artifact ratio; (2) to minimize the computational burden of the algorithm so that it is useful for applications like speech recognition system. The time–frequency points of the observed mixture signals are input to the first layer of CapsNet. In the first layer, single-source active point (SSP) is calculated using the ratio of mixtures. These SSPs are lower-level capsules in our system. In the second layer, we find a cluster center using a dynamic routing algorithm and these clusters are used to construct a binary mask. Finally, the algorithm solves the permutation problem by determining the correlation between the amplitudes of adjacent frequency bins. We test our algorithm on the live recording mixture signals obtained in the real environment and synthetically convoluted mixture signals. The test result shows the effectiveness of the proposed method when compared with the existing algorithms in terms of computational load, signal-to-distortion ratio and signal-to-interference ratio.

Deep neural network based direction of arrival estimation for hearing aid applications using smartphone

Deep neural network (DNN) techniques are gaining popularity due to performance boost in many applications. In this work we propose a DNN-based method for finding the direction of arrival (DOA) of speech source for hearing aid applications using smartphones. We consider the DOA estimation as a classification problem and use the magnitude and phase of speech signal as a feature set for DNN training stage and obtaining appropriate model. The model is trained and derived using real noisy speech data recorded on smartphone in different environments under low SNRs. The DNN-based DOA method, with the pre-trained model, is implemented and run on Android smartphone in real time and evaluated objectively and subjectively. The test results are presented showing the performance of proposed method versus other methods.

Using Phonetic Posteriorgram Based Frame Pairing for Segmental Accent Conversion

Accent conversion AC aims to transform non-native utterances to sound as if the speaker had a native accent. This can be achieved by mapping source speech spectra from a native speaker into the acoustic space of the target non-native speaker. In prior work, we proposed an AC approach that matches frames between the two speakers based on their acoustic similarity after compensating for differences in vocal tract length. In this paper, we propose a new approach that matches frames between the two speakers based on their phonetic rather than acoustic similarity. Namely, we map frames from the two speakers into a phonetic posteriorgram using speaker-independent acoustic models trained on native speech. We thoroughly evaluate the approach on a speech corpus containing multiple native and non-native speakers. The proposed algorithm outperforms the prior approach, improving ratings of acoustic quality 22% increase in mean opinion score and native accent 69% preference while retaining the voice quality of the non-native speaker. Furthermore, we show that the approach can be used in the reverse conversion direction, i.e., generating speech with a native speaker's voice quality and a non-native accent. Finally, we show that this approach can be applied to non-parallel training data, achieving the same accent conversion performance.

Non-parallel dictionary learning for voice conversion using non-negative Tucker decomposition

Voice conversion (VC) is a technique of exclusively converting speaker-specific information in the source speech while preserving the associated phonemic information. Non-negative matrix factorization (NMF)-based VC has been widely researched because of the natural-sounding voice it achieves when compared with conventional Gaussian mixture model-based VC. In conventional NMF-VC, models are trained using parallel data which results in the speech data requiring elaborate pre-processing to generate parallel data. NMF-VC also tends to be an extensive model as this method has several parallel exemplars for the dictionary matrix, leading to a high computational cost. In this study, an innovative parallel dictionary-learning method using non-negative Tucker decomposition (NTD) is proposed. The proposed method uses tensor decomposition and decomposes an input observation into a set of mode matrices and one core tensor. The proposed NTD-based dictionary-learning method estimates the dictionary matrix for NMF-VC without using parallel data. The experimental results show that the proposed method outperforms other methods in both parallel and non-parallel settings.

ACVAE-VC: Non-Parallel Voice Conversion With Auxiliary Classifier Variational Autoencoder

This paper proposes a non-parallel voice conversion (VC) method using a variant of the conditional variational autoencoder (VAE) called an auxiliary classifier VAE. The proposed method has two key features. First, it adopts fully convolutional architectures to construct the encoder and decoder networks so that the networks can learn conversion rules that capture the time dependencies in the acoustic feature sequences of source and target speech. Second, it uses information-theoretic regularization for the model training to ensure that the information in the attribute class label will not be lost in the conversion process. With regular conditional VAEs, the encoder and decoder are free to ignore the attribute class label input. This can be problematic since in such a situation, the attribute class label will have little effect on controlling the voice characteristics of input speech at test time. Such situations can be avoided by introducing an auxiliary classifier and training the encoder and decoder so that the attribute classes of the decoder outputs are correctly predicted by the classifier. We also present several ways to convert the feature sequence of input speech using the trained encoder and decoder and compare them in terms of audio quality through objective and subjective evaluations. We confirmed experimentally that the proposed method outperformed baseline non-parallel VC systems and performed comparably to an open-source parallel VC system trained using a parallel corpus in a speaker identity conversion task.

A duet and/or a concerto?

Abstract Working memory and interpreting expertise are generally acknowledged as two essential factors that affect the interpreting process. With reference to the compensation model, the independent influences model, and the rich-get-richer model proposed in cognitive psychology concerning the interplay between working memory and domain expertise, this experimental study, by varying the delivery speed of the source speech, examines the interplay between working memory and interpreting expertise in the setting of simultaneous interpreting. This study finds that (1) working memory and interpreting expertise can each have a positive impact on simultaneous interpreting quality; (2) other factors being equal, interpreting expertise can have a greater impact on simultaneous interpreting than working memory, and when the interpreting task proves more challenging, interpreting expertise can help relieve the pressure triggered by inadequate working memory; (3) working memory and interpreting expertise function differently in simultaneous interpreting in different stages: for beginner interpreting students, working memory’s contribution to the quality of simultaneous interpreting is more prominent, while for advanced interpreting students, interpreting expertise plays a more direct and prominent role, and the composite influence of working memory and interpreting expertise on simultaneous interpreting performance is weakened due to the increasing impact of other factors.

Time–Frequency Masking Based Online Multi-Channel Speech Enhancement With Convolutional Recurrent Neural Networks

This paper presents a time–frequency masking based online multi-channel speech enhancement approach that uses a convolutional recurrent neural network to estimate the mask. The magnitude and phase components of the short-time Fourier transform coefficients for multiple time frames are provided as an input such that the network is able to discriminate between the directional speech and the noise components based on the spatial characteristics of the individual signals as well as their spectro-temporal structure. The estimation of two different masks, namely, ideal ratio mask (IRM) and ideal binary mask (IBM), along with two different approaches for incorporating the mask to obtain the desired signal are discussed. In the first approach, the mask is directly applied as a real valued gain to a reference microphone signal, whereas in the second approach, the masks are used as an activity indicator for the recursive update of power spectral density (PSD) matrices to be used within a beamformer. The performance of the proposed system with the two different estimated masks utilized within the two different enhancement approaches is evaluated with both simulated as well as measured room impulse responses, where it is shown that the IBM is better suited as an indicator for the PSD updates while direct application of IRM as a real valued gain leads to a better improvement in terms of short term objective intelligibility. Analysis of the performance of the proposed system also demonstrates the robustness of the system to different angular positions of the speech source.