Joint Speech Research Articles

Decoupling and Interacting Multi-Task Learning Network for Joint Speech and Accent Recognition

Decoupling and Interacting Multi-Task Learning Network for Joint Speech and Accent Recognition

SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberation

SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberation

The empirical discovery of domains of assembly and communion

We consider chanting, or joint speech, which is ubiquitous, but not evenly distributed, in human activity. Taking an observational stance motivated by embodied cognitive science, we approach this topic without assumptions of the structure of persons, social formations, culture, or nature. This restrictive starting point motivates the use of a simple empirical definition of joint speech (the utterance by multiple persons of the same sounds at the same time) to allow us to induce four distinguished domains of assembly and communion among persons. These may be loosely indicated by the familiar terms of ritual, sports, protest and primary education. We use our empirical definition to induce these domains, and then consider how they might be regarded jointly. We are not aware of any social or psychological theory that would generate these four domains, and we suggest that our restricted mode of observation may be of use in the collective consideration of human patterning, without the common assumptions of cognitivism.

Multilingual Speech Recognition for Turkic Languages

The primary aim of this study was to contribute to the development of multilingual automatic speech recognition for lower-resourced Turkic languages. Ten languages—Azerbaijani, Bashkir, Chuvash, Kazakh, Kyrgyz, Sakha, Tatar, Turkish, Uyghur, and Uzbek—were considered. A total of 22 models were developed (13 monolingual and 9 multilingual). The multilingual models that were trained using joint speech data performed more robustly than the baseline monolingual models, with the best model achieving an average character and word error rate reduction of 56.7%/54.3%, respectively. The results of the experiment showed that character and word error rate reduction was more likely when multilingual models were trained with data from related Turkic languages than when they were developed using data from unrelated, non-Turkic languages, such as English and Russian. The study also presented an open-source Turkish speech corpus. The corpus contains 218.2 h of transcribed speech with 186,171 utterances and is the largest publicly available Turkish dataset of its kind. The datasets and codes used to train the models are available for download from our GitHub page.

A Hybrid Reverberation Model and Its Application to Joint Speech Dereverberation and Separation

A Hybrid Reverberation Model and Its Application to Joint Speech Dereverberation and Separation

A Joint Speech Enhancement and Self-Supervised Representation Learning Framework for Noise-Robust Speech Recognition

Though speech enhancement (SE) can be used to improve speech quality in noisy environments, it may also cause distortions that degrade the performance of automatic speech recognition (ASR) models. Self-supervised pre-training, on the other hand, has been shown to improve the noise robustness of ASR models. However, the potential of the (optimal) integration of SE and self-supervised pre-training still remains unclear. In this paper, we propose a novel self-supervised pre-training framework that incorporates SE to improve ASR performance in noisy environments. First, in the pre-training phase the original noisy waveform or the waveform obtained by SE is fed into the self-supervised model to learn the contextual representation, where the quantized clean speech acts as the target. Second, we propose a dual-attention fusion method to fuse the features of noisy and enhanced speech, which can compensate for the information loss caused by separately using individual modules. Due to the flexible exploitation of clean/noisy/enhanced branches, the proposed method turns out to be a generalization of some existing noise-robust ASR models, e.g., enhanced wav2vec2.0. Finally, experimental results on both synthetic and real noisy datasets show that the proposed joint training approach can improve the ASR performance under various noisy settings, leading to a stronger noise robustness.

Deep Learning-Based Speech Specific Source Localization by Using Binaural and Monaural Microphone Arrays in Hearing Aids

A deep learning-based method is proposed for jointly detecting and localizing speech sources in a complex acoustic scene by using microphones of a hearing aid. Motivated by the human auditory system, peripheral preprocessing is applied on the microphone signals to obtain auditory subband signals that serve as input to the proposed deep neural network for detecting and localizing speech sources. In the proposed neural network, a combination of residual and dense aggregation learning is utilized rather than the conventional residual learning to preserve and reuse the spatial representations at the output layers. This process is performed to improve the gradient flow in deeper layers, in the training stage. The learning curves show that the proposed residual-dense aggregation mapping do improve the speed and accuracy of the convergence. The proposed model shows good performance in joint speech source detection and localization using a binaural microphone array (i.e., three channels at each side) but also using a monaural microphone array (i.e., four channels at the right side) despite of the short distances between the microphones. The proposed methods also outperform neural networks that are directly using STFT components of the binaural or monaural microphone arrays. In addition, the proposed models extended with learnable peripheral processing show slightly improved in detection and localization scores compared to the proposed models using the plain auditory subband signals, in both the binaural and monaural microphone arrays but only so, when the learnable peripheral processing is initialized with parameters stemming from human peripheral processing.

The multimodal nature of communicative efficiency in social interaction

How does communicative efficiency shape language use? We approach this question by studying it at the level of the dyad, and in terms of multimodal utterances. We investigate whether and how people minimize their joint speech and gesture efforts in face-to-face interactions, using linguistic and kinematic analyses. We zoom in on other-initiated repair—a conversational microcosm where people coordinate their utterances to solve problems with perceiving or understanding. We find that efforts in the spoken and gestural modalities are wielded in parallel across repair turns of different types, and that people repair conversational problems in the most cost-efficient way possible, minimizing the joint multimodal effort for the dyad as a whole. These results are in line with the principle of least collaborative effort in speech and with the reduction of joint costs in non-linguistic joint actions. The results extend our understanding of those coefficiency principles by revealing that they pertain to multimodal utterance design.

Педагогическая риторика в обучении учителя и ученика

The article describes the experience of teaching pedagogical rhetoric as part of the subject of the culture of native Russian speech. The problem of the professional competence of the teacher of preschool and primary education in the field of speech activity is raised. The author briefly characterizes the history of the development of pedagogical rhetoric. A significant difference is emphasized in the quality of the language training of a teacher, educator and children, as well as ways to overcome this gap in the process of joint speech work. The styles and genres of speech inherent in the teacher as a carrier of the speech profession are characterized, methods and algorithms for their development are proposed. Additional sources of knowledge about listening and speaking as oral forms of speech of a teacher and a child, as well as speech samples from works of children's literature are named.

Педагогическая риторика в обучении учителя и ученика

The article describes the experience of teaching pedagogical rhetoric as part of the subject of the culture of native Russian speech. The problem of the professional competence of the teacher of preschool and primary education in the field of speech activity is raised. The author briefly characterizes the history of the development of pedagogical rhetoric. A significant difference is emphasized in the quality of the language training of a teacher, educator and children, as well as ways to overcome this gap in the process of joint speech work. The styles and genres of speech inherent in the teacher as a carrier of the speech profession are characterized, methods and algorithms for their development are proposed. Additional sources of knowledge about listening and speaking as oral forms of speech of a teacher and a child, as well as speech samples from works of children's literature are named.

Disentangling the Effects of Matching Content and Simultaneous Speech on Phonetic Adaptation

ABSTRACT Past studies demonstrated that interlocutors mutually affect each other’s vocal behavior in joint speech situations. However, findings on exactly how and when these adaptive processes occur are inconsistent. We propose a novel experimental paradigm, the ‘One-Voice Professor’ game, that explicitly separates content matching (“saying the same thing”) from simultaneity (“speaking at the same time”), in an effort to systematically investigate different joint speech conditions. In our first experiment, we analyzed the vocal characteristics of 96 participants who engaged in joint speech with a recorded voice, in a 2-by-2 experimental design. Results indicate significant effects on the fundamental vocal frequency (f0), with the strongest f0 adaptation in the alternating-and-matching condition, and f0 divergence in the simultaneous-and-matching condition. Since these effects could have been impacted by the fact that speakers could not mutually adapt with the recorded voice, we applied the same paradigm in our second experiment to an interactive joint speech situation. Same-gendered dyads (n = 66) improvised speech together in interviews designed to separate matching from simultaneity. The results show main effects of the conditions, with matching leading to stronger f0 adaptation and simultaneity to divergence on secondary voice characteristics (f1, f2, and harmonics-to-noise ratio (HNR)). These findings confirm the hypothesis that simultaneity and matching of speech influence phonetic adaptation differently.

Chorality’s Sonic-Social Relationships

The sonic-social relationships of people singing together, chanting, or engaged in group vocality are underrepresented in voice, sound, and music studies. Work on the voice tends to focus on individual voices, despite the human commonplace of group singing, choric chanting, and joint speech. This article brings into conversation practice-based, ethnographic, and theoretical perspectives on chorality to mark a noteworthy constellation of interdisciplinary work on voice; decolonial, antiracist, and LGBTQ+ activism; crowds and masses; intimate publics; and democratic politics. Writing in 2021 at a moment when voices joined in chants and anthems of protest assume tragic urgency, and when chanting and singing together risk physical and social violence and the transmission of COVID-19, the authors fix their attention on sonic-social relationships in chorality in order to set down how the precarity of this moment can translate into new thinking about joint voicing. In the case studies at the heart of this article, the authors offer four frames for approaching sonic-social relationships in chorality—activist choirs, collegiate a cappella, call-response singing, and virtual choirs. They advocate for a both-and approach to the sonic-social relationship in chorality: Sound qualities matter, often urgently, as does chorality’s social power to include, capture, and exclude. Ultimately, the article stresses that chorality is ethically neutral—a key methodological consideration in encountering chorality. What emerges from chorality’s sonic-social relationships, then, is the presence or absence of care—care for the effects of chorality’s uplift or harm, carelessness with chorality’s difference-leveling potentials, and chorality as an upwelling of care.

Pushing the envelope: Evaluating speech rhythm with different envelope extraction techniques.

The amplitude of the speech signal varies over time, and the speech envelope is an attempt to characterise this variation in the form of an acoustic feature. Although tacitly assumed, the similarity between the speech envelope-derived time series and that of phonetic objects (e.g., vowels) remains empirically unestablished. The current paper, therefore, evaluates several speech envelope extraction techniques, such as the Hilbert transform, by comparing different acoustic landmarks (e.g., peaks in the speech envelope) with manual phonetic annotation in a naturalistic and diverse dataset. Joint speech tasks are also introduced to determine which acoustic landmarks are most closely coordinated when voices are aligned. Finally, the acoustic landmarks are evaluated as predictors for the temporal characterisation of speaking style using classification tasks. The landmark that performed most closely to annotated vowel onsets was peaks in the first derivative of a human audition-informed envelope, consistent with converging evidence from neural and behavioural data. However, differences also emerged based on language and speaking style. Overall, the results show that both the choice of speech envelope extraction technique and the form of speech under study affect how sensitive an engineered feature is at capturing aspects of speech rhythm, such as the timing of vowels.

Autoregressive Moving Average Jointly-Diagonalizable Spatial Covariance Analysis for Joint Source Separation and Dereverberation

This paper describes a computationally-efficient statistical approach to joint (semi-)blind source separation and dereverberation for multichannel noisy reverberant mixture signals. A standard approach to source separation is to formulate a generative model of a multichannel mixture spectrogram that consists of source and spatial models representing the time-frequency power spectral densities (PSDs) and spatial covariance matrices (SCMs) of source images, respectively, and find the maximum-likelihood estimates of these parameters. A state-of-the-art blind source separation method in this thread of research is fast multichannel nonnegative matrix factorization (FastMNMF) based on the low-rank PSDs and jointly-diagonalizable full-rank SCMs. To perform mutually-dependent separation and dereverberation jointly, in this paper we integrate both moving average (MA) and autoregressive (AR) models that represent the early reflections and late reverberations of sources, respectively, into the FastMNMF formalism. Using a pretrained deep generative model of speech PSDs as a source model, we realize semi-blind joint speech separation and dereverberation. We derive an iterative optimization algorithm based on iterative projection or iterative source steering for jointly and efficiently updating the AR parameters and the SCMs. Our experimental results showed the superiority of the proposed ARMA extension over its AR- or MA-ablated version in a speech separation and/or dereverberation task.

Convergence in voice fundamental frequency during synchronous speech.

Joint speech behaviours where speakers produce speech in unison are found in a variety of everyday settings, and have clinical relevance as a temporary fluency-enhancing technique for people who stutter. It is currently unknown whether such synchronisation of speech timing among two speakers is also accompanied by alignment in their vocal characteristics, for example in acoustic measures such as pitch. The current study investigated this by testing whether convergence in voice fundamental frequency (F0) between speakers could be demonstrated during synchronous speech. Sixty participants across two online experiments were audio recorded whilst reading a series of sentences, first on their own, and then in synchrony with another speaker (the accompanist) in a number of between-subject conditions. Experiment 1 demonstrated significant convergence in participants' F0 to a pre-recorded accompanist voice, in the form of both upward (high F0 accompanist condition) and downward (low and extra-low F0 accompanist conditions) changes in F0. Experiment 2 demonstrated that such convergence was not seen during a visual synchronous speech condition, in which participants spoke in synchrony with silent video recordings of the accompanist. An audiovisual condition in which participants were able to both see and hear the accompanist in pre-recorded videos did not result in greater convergence in F0 compared to synchronisation with the pre-recorded voice alone. These findings suggest the need for models of speech motor control to incorporate interactions between self- and other-speech feedback during speech production, and suggest a novel hypothesis for the mechanisms underlying the fluency-enhancing effects of synchronous speech in people who stutter.

Improved spatialization performance for joint speech dereverberation and noise reduction in binaural hearing aids

ObjectiveIn this work we propose a unified solution for joint speech dereverberation, noise reduction and spatial preservation of either diffuse noise fields or single-point noise sources in binaural hearing aid applications, which is based on the multichannel Wiener filter (MWF) with preservation of the interaural coherence (MWF-IC) method. MethodsThe original interaural coherence (IC) auxiliary penalty cost function in the MWF-IC, which comprises information about both interfering noise and late reverberation component, is modified to consider the interference noise information only. It provides a new regularization form for preserving both interaural time difference (ITD) and IC binaural cues of the residual noise. This approach is theoretically supported by recent findings about the relationship between IC and ITD binaural cues for point acoustic sources. ResultsExtensive simulation results with objective measures, as well as psychoacoustic experiments with normal hearing volunteers indicate that the proposed method outperforms the preservation of spatial localization cues of its original version, keeping approximately the same performance in terms of noise-reduction and speech-dereverberation. ConclusionThe proposed scheme expands the MWF-IC original spatial cue preservation capability from only diffuse noise fields to either diffuse noise fields or a single-point noise source. SignificanceThis is the first MWF-based speech dereverberation and noise reduction method for binaural hearing aid that provides perceptually relevant preservation of acoustic scenarios comprised of one speech source and either a single-point noise source or a diffuse noise field.

Língua e fenômenos linguageiros de um ponto de vista fonético

We differentiate between language-as-system, as exemplified by such constructs as “English,” or “Mandarin,'” and languaging, understood as a rich set of affiliative and coordinative behaviours that involve speech. The former is the more familiar term, and has been constructed in a specific manner that is inextricably bound to literacy, writing and normative social practices. But we argue that only the latter can inform us about what it was that happened to the human species to so differentiate us from other primates. To draw out this distinction, we lean on the contrast between emic and etic approaches, introduced by Ken Pike and rooted in the distinction between phonology and phonetics. We argue that an etic approach to speech can reveal forms of languaging that are not addressed by language-as-system. Joint speech is put forward as an important form of languaging that can be thematised for study only if the emic/etic distinction is taken seriously. Consequences for the self-understanding of phonetics as a discipline are cautiously put forward.

Chaos-based joint speech encryption scheme using SHA-1

In this paper, the speech signal is encrypted using multiple chaotic maps and cryptographic protocols. The input signal is fragmented and scrambled into four segments using cubic map. In order to make the signal invulnerable against attacks, scrambled signal is processed through various one-dimensional chaotic maps like cubic, logistic, skew-tent, and quadratic map. All chaotic maps are employed to provide an encryption at transmitter side; however, reversal of the same is carried out at other end. To protect the various parameters of all chaotic techniques, blowfish algorithm is employed with the private key. On top of that, additional layer of security is provided through hashing algorithm for authentication of shared data and the blowfish key of system between two ends. The computed message digest of secure hash algorithm should be same for successful communication, which authenticates and verify the parameters of chaotic maps. Furthermore, several tests have been performed on 8 KHz secured speech signal such as signal-to-noise ratio, power-spectral density, peak signal-to-noise ratio, and correlation test in order to check the efficiency of proposed system. The various statistical tests and analysis of adversary model validates the encryption model and embraces stronger security, thus making it as invulnerable to the attacks of intruder. The result promises that presented technique is extremely secure, provides end-to-end encryption and can be utilized in a real time speech communication.

Single-channel speech enhancement with correlated spectral components: Limits-potential

In this paper, we investigate single-channel speech enhancement algorithms that operate in the short-time Fourier transform and take into account dependencies w.r.t. frequency. As a result of allowing for inter-frequency dependencies, the minimum mean square error optimal estimates of the short-time Fourier transform expansion coefficients are functions of complex-valued covariance matrices in general. The covariance matrices are not known a priori and have to be estimated from the observed data. This work is dedicated to analyzing how this affects the respective single-channel speech enhancement algorithms. We propose a statistical model that circumvents the need to estimate complex-valued second order statistics and derive a linear multidimensional short-time spectral amplitude estimator that is motivated by these assumptions. Further, we provide empirical evidence for the assumptions that form the basis of this model. We evaluate the potential of taking into account inter-frequency dependencies for single-channel speech enhancement and subsequently compare the estimator resulting from the proposed statistical model to relevant benchmark methods. The results indicate that estimators that consider inter-frequency dependencies are capable of pushing the limits of standard approaches in terms of joint speech quality and intelligibility improvement when the second order statistics are estimated from isolated speech data. The proposed linear multidimensional short-time spectral amplitude estimator preserves this trend in fully blind scenarios.

Speaking to a common tune: Between-speaker convergence in voice fundamental frequency in a joint speech production task.

Recent research on speech communication has revealed a tendency for speakers to imitate at least some of the characteristics of their interlocutor's speech sound shape. This phenomenon, referred to as phonetic convergence, entails a moment-to-moment adaptation of the speaker's speech targets to the perceived interlocutor's speech. It is thought to contribute to setting up a conversational common ground between speakers and to facilitate mutual understanding. However, it remains uncertain to what extent phonetic convergence occurs in voice fundamental frequency (F0), in spite of the major role played by pitch, F0's perceptual correlate, as a conveyor of both linguistic information and communicative cues associated with the speaker's social/individual identity and emotional state. In the present work, we investigated to what extent two speakers converge towards each other with respect to variations in F0 in a scripted dialogue. Pairs of speakers jointly performed a speech production task, in which they were asked to alternately read aloud a written story divided into a sequence of short reading turns. We devised an experimental set-up that allowed us to manipulate the speakers' F0 in real time across turns. We found that speakers tended to imitate each other's changes in F0 across turns that were both limited in amplitude and spread over large temporal intervals. This shows that, at the perceptual level, speakers monitor slow-varying movements in their partner's F0 with high accuracy and, at the production level, that speakers exert a very fine-tuned control on their laryngeal vibrator in order to imitate these F0 variations. Remarkably, F0 convergence across turns was found to occur in spite of the large melodic variations typically associated with reading turns. Our study sheds new light on speakers' perceptual tracking of F0 in speech processing, and the impact of this perceptual tracking on speech production.