Multiple Speakers Research Articles

Combined Keyword Spotting and Localization Network Based on Multi-Task Learning

The advent of voice assistance technology and its integration into smart devices has facilitated many useful services, such as texting and application execution. However, most assistive technologies lack the capability to enable the system to act as a human who can localize the speaker and selectively spot meaningful keywords. Because keyword spotting (KWS) and sound source localization (SSL) are essential and must operate in real time, the efficiency of a neural network model is crucial for memory and computation. In this paper, a single neural network model for KWS and SSL is proposed to overcome the limitations of sequential KWS and SSL, which require more memory and inference time. The proposed model uses multi-task learning to utilize the limited resources of the device efficiently. A shared encoder is used as the initial layer to extract common features from the multichannel audio data. Subsequently, the task-specific parallel layers utilize these features for KWS and SSL. The proposed model was evaluated on a synthetic dataset with multiple speakers, and a 7-module shared encoder structure was identified as optimal in terms of accuracy, direction of arrival (DOA) accuracy, DOA error, and latency. It achieved a KWS accuracy of 94.51%, DOA error of 12.397°, and DOA accuracy of 89.86%. Consequently, the proposed model requires significantly less memory owing to the shared network architecture, which enhances the inference time without compromising KWS accuracy, DOA error, and DOA accuracy.

Multi-microphone simultaneous speakers detection and localization of multi-sources for separation and noise reduction

This paper addresses the challenge of online blind speaker separation in a multi-microphone setting. The linearly constrained minimum variance (LCMV) beamformer is selected as the backbone of the separation algorithm due to its distortionless response and capacity to create a null towards interfering sources. A specific instance of the LCMV beamformer that considers acoustic propagation is implemented. In this variant, the relative transfer functions (RTFs) associated with each speaker of interest are utilized as the steering vectors of the beamformer. A control mechanism is devised to ensure robust estimation of the beamformer’s building blocks, comprising speaker activity detectors and direction of arrival (DOA) estimation branches. This control mechanism is implemented as a multi-task deep neural network (DNN). The primary task classifies each time frame based on speaker activity: no active speaker, single active speaker, or multiple active speakers. The secondary task is DOA estimation. It is implemented as a classification task, executed only for frames classified as single-speaker frames by the primary branch. The direction of the active speaker is classified into one of the multiple ranges of angles. These frames are also leveraged to estimate the RTFs using subspace estimation methods. A library of RTFs associated with these DOA ranges is then constructed, facilitating rapid acquisition of new speakers and efficient tracking of existing speakers. The proposed scheme is evaluated in both simulated and real-life recordings, encompassing static and dynamic scenarios. The benefits of the multi-task approach are showcased, and significant improvements are evident, even when the control mechanism is trained with simulated data and tested with real-life data. A comparison between the proposed scheme and the independent low-rank matrix analysis (ILRMA) algorithm reveals significant improvements in static scenarios. Furthermore, the tracking capabilities of the proposed scheme are highlighted in dynamic scenarios.

Acoustic design of an innovative classroom: the Piscopia Corner in Milan

This paper reports on the measurements, design, and testing of a highly innovative classroom: the Piscopia Corner - Augmented Education Lab, built in the new MIND (Milano Innovation District) district in Milan, Italy. This educational space is designed to leverage connectivity, artificial intelligence, and sensing to enhance sustainability, inclusivity, and efficacy. As key use cases of the Piscopia Corner pivot on real-time speech transcription and translation of interactions, acoustics plays a key role. Improving classroom acoustic conditions is crucial for enhancing speech clarity, promoting student focus, and relieving teachers of vocal effort. The Italian standard UNI 11532-2:2020 states proper requirements for multimedia classrooms such as the one involved in the present case. This space aims for superior speech intelligibility, accommodating multiple speakers and addressing the needs of individuals with hearing impairments or language barriers, with a high outward intelligibility. The Piscopia Corner is a modern classroom meeting the stringent requirements of interactive audio-video activities and inclusive principles. The treated suspended ceiling integrated with sensor technology required a proper mix of materials to balance the sound absorption in frequency and improve speech intelligibility to achieve category A4 limits.

Speaker variability, but not bilingualism, influences cross-situational word learning.

When learning new words, listeners must contend with multiple sources of ambiguity and variability. Research has revealed that learners can resolve referential ambiguity by tracking co-occurrence statistics between words and their referents across multiple exposures over time-a process termed cross-situational word learning (XSWL). However, the degree to which variability in the input, such as input from multiple speakers, and variability in learner experience, such as bilingual language experience, modulate XSWL remains unclear. In this study, we examined the effects of speaker variability in XSWL performance in monolingual adults and bilingual adults with a range of second language backgrounds and language acquisition histories. Results revealed above-chance word learning in both the single and the multiple-speaker conditions across language groups. An advantage for word learning was observed in the single-speaker condition, but the effects of bilingual language experience were null. This research adds to the limited body of work dedicated to extending theories of statistical learning to account for variations in both input and learner characteristics as well as their interactions.

Sparse representation of neurons for encoding complex sounds in the auditory cortex

Listening in complex sound environments requires rapid segregation of different sound sources, e.g., having a conversation with multiple speakers or other environmental sounds. Efficient processing requires fast encoding of inputs to adapt to target sounds and identify relevant information from past experiences. This adaptation process represents an early phase of implicit learning of the sound statistics to form auditory memory. The auditory cortex (ACtx) plays a crucial role in this implicit learning process, but the underlying circuits are unknown. In awake mice, we recorded neuronal responses in different ACtx subfields using in vivo 2-photon imaging of excitatory and inhibitory (parvalbumin; PV) neurons. We used a paradigm adapted from human studies that induced rapid implicit learning from passively presented complex sounds and imaged A1 Layer 4 (L4), A1 L2/3, and A2 L2/3. In this paradigm, a frozen spectro-temporally complex ‘Target’ sound randomly re-occurred within a stream of other random complex sounds. All ACtx subregions contained distinct groups of cells specifically responsive to complex acoustic sequences, indicating that even thalamocortical input layers (A1 L4) respond to complex sounds. Subgroups of excitatory and inhibitory cells in all subfields showed decreased responses for re-occurring Target sounds, indicating that ACtx is highly involved in the early implicit learning phase. At the population level, activity was more decorrelated to Target sounds independent of the duration of frozen token, subregions, and cell type. These findings suggest that ACtx and its input layers contribute to the early phase of auditory memory for complex sounds, suggesting a parallel strategy across ACtx areas and between excitatory and inhibitory neurons.

Languaging-as-practice in science education: an alternative to metaphors of language-as-tool

AbstractThis commentary proposes the metaphor of “languaging-as-practice” in science education as an alternative to “language-as-tool” metaphors. Describing language as a tool implicitly positions language as static and unchanging and assumes that named languages are distinct and bounded entities. In contrast, describing languaging as a practice acknowledges the multiple and advanced ways speakers draw on many aspects of their linguistic repertoires while often crossing purported boundaries between named languages. This approach is grounded in translanguaging theory and offers a dynamic understanding of language in educational settings. It envisions students constructing and expressing scientific knowledge by drawing on their complete linguistic repertoires. By advocating for this approach, we aim to promote practices that acknowledge and leverage the diverse linguistic backgrounds of students, fostering inclusive and effective learning environments. While this approach has gained some traction in language-education circles, we believe it needs more consideration in the science education community. We also critique the limitations of viewing language merely as a tool for reproducing predefined scientific concepts and discourses, proposing that a more expansive approach to language can enhance opportunities for scientific sense-making and knowledge construction. We urge science education researchers and practitioners to shift how we think and talk about language in the science classroom to adopt and act upon a metaphor of “languaging-as-practice.”

Speaker-Attributed Training for Multi-Speaker Speech Recognition Using Multi-Stage Encoders and Attention-Weighted Speaker Embedding

Automatic speech recognition (ASR) aims at understanding naturally spoken human speech to be used as text inputs to machines. In multi-speaker environments, where multiple speakers are talking simultaneously with a large amount of overlap, a significant performance degradation may occur with conventional ASR systems if they are trained by recordings of single talkers. This paper proposes a multi-speaker ASR method that incorporates speaker embedding information as an additional input. The embedding information for each of the speakers in the training set was extracted as numeric vectors, and all of the embedding vectors were stacked to construct a total speaker profile matrix. The speaker profile matrix from the training dataset enables finding embedding vectors that are close to the speakers of the input recordings in the test conditions, and it helps to recognize the individual speakers’ voices mixed in the input. Furthermore, the proposed method efficiently reuses the decoder from the existing speaker-independent ASR model, eliminating the need for retraining the entire system. Various speaker embedding methods such as i-vector, d-vector, and x-vector were adopted, and the experimental results show 0.33% and 0.95% absolute (3.9% and 11.5% relative) improvements without and with the speaker profile in the word error rate (WER).

Speech recognition in videos using features of different enhancement filters

Video Speech Recognition (VSR) is the ability of software to extract spoken text from input sources, such as offline videos and YouTube. It achieves a good recognition accuracy rate for videos with clean environments. However, with unwanted noisy elements in speech, this presents a major challenge. Depending on where the noise originates from, there are considerable differences in the impact of various noise kinds on video speech recognition, including background music, multiple speakers, microphone quality, and dialects. It is not possible to create a filter for every kind of noise because the sources of noise might differ greatly and the speech environment’s circumstances can change over time. Hence, instead of using a single filter for each type of noise, this study proposed to combine features from a small number of noise removal filters applied to the same video speech signal, resulting in a better VSR output. According to experimental results, using the proposed framework increases the accuracy of video speech recognition when evaluated on YouTube videos with different noise levels. In test experiments, the proposed framework was evaluated against state-of-the-art approaches. The accuracy mean of the relevant current approaches was improved by 16.67% and against the best accuracy of them by 7.78%. This research contributes to the video speech recognition topic as the proposed framework can facilitate the extraction of spoken phrases in millions of hours of video available online, indexing them, and then searching for them through their index.

Using deep learning to improve the intelligibility of a target speaker in noisy multi-talker environments for people with normal hearing and hearing loss.

Understanding speech in noisy environments is a challenging task, especially in communication situations with several competing speakers. Despite their ongoing improvement, assistive listening devices and speech processing approaches still do not perform well enough in noisy multi-talker environments, as they may fail to restore the intelligibility of a speaker of interest among competing sound sources. In this study, a quasi-causal deep learning algorithm was developed that can extract the voice of a target speaker, as indicated by a short enrollment utterance, from a mixture of multiple concurrent speakers in background noise. Objective evaluation with computational metrics demonstrated that the speaker-informed algorithm successfully extracts the target speaker from noisy multi-talker mixtures. This was achieved using a single algorithm that generalized to unseen speakers, different numbers of speakers and relative speaker levels, and different speech corpora. Double-blind sentence recognition tests on mixtures of one, two, and three speakers in restaurant noise were conducted with listeners with normal hearing and listeners with hearing loss. Results indicated significant intelligibility improvements with the speaker-informed algorithm of 17% and 31% for people without and with hearing loss, respectively. In conclusion, it was demonstrated that deep learning-based speaker extraction can enhance speech intelligibility in noisy multi-talker environments where uninformed speech enhancement methods fail.

Channel-Aware Decoupling Network for Multiturn Dialog Comprehension.

Training machines to understand natural language and interact with humans is one of the major goals of artificial intelligence. Recent years have witnessed an evolution from matching networks to pretrained language models (PrLMs). In contrast to the plain-text modeling as the focus of the PrLMs, dialog texts involve multiple speakers and reflect special characteristics, such as topic transitions and structure dependencies, between distant utterances. However, the related PrLM models commonly represent dialogs sequentially by processing the pairwise dialog history as a whole. Thus, the hierarchical information on either utterance interrelation or speaker roles coupled in such representations is not well addressed. In this work, we propose compositional learning for holistic interaction across the utterances beyond the sequential contextualization from PrLMs, in order to capture the utterance-aware and speaker-aware representations entailed in a dialog history. We decouple the contextualized word representations by masking mechanisms in transformer-based PrLM, making each word only focus on the words in the current utterance, other utterances, and two speaker roles (i.e., utterances of the sender and utterances of the receiver), respectively. In addition, we employ domain-adaptive training strategies to help the model adapt to the dialog domains. Experimental results show that our method substantially boosts the strong PrLM baselines in four public benchmark datasets, achieving new state-of-the-art performance over previous methods.

Wavoice: An mmWave-Assisted Noise-Resistant Speech Recognition System

As automatic speech recognition evolves, deployment of the voice user interface (VUI) has boomingly expanded. Especially since the COVID-19 pandemic, the VUI has gained more attention in online communication owing to its non-contact property. However, the VUI struggles to be applied in public scenes due to the degradation of received audio signals caused by various ambient noises. In this article, we propose Wavoice , the first noise-resistant multi-modal speech recognition system that fuses two distinct voices sensing modalities (i.e., millimeter-wave signals and audio signals from a microphone) together. One key contribution is to model the inherent correlation between millimeter-wave and audio signals. Based on it, Wavoice facilitates the real-time noise-resistant voice activity detection and user targeting from multiple speakers. Additionally, we elaborate on two novel modules for multi-modal fusion embedded into the neural network, leading to accurate speech recognition. Extensive experiments prove the effectiveness of Wavoice under adverse conditions—that is, the character recognition error rate below 1% in a range of 7 m. In terms of robustness and accuracy, Wavoice considerably outperforms existing audio-only speech recognition methods with lower character error and word error rates.

Summary of the DISPLACE challenge 2023-DIarization of SPeaker and LAnguage in Conversational Environments

In multi-lingual societies, where multiple languages are spoken in a small geographic vicinity, informal conversations often involve mix of languages. Existing speech technologies may be inefficient in extracting information from such conversations, where the speech data is rich in diversity with multiple languages and speakers. The DISPLACE (DIarization of SPeaker and LAnguage in Conversational Environments) challenge constitutes an open-call for evaluating and bench-marking the speaker and language diarization technologies on this challenging condition. To facilitate this challenge, a real-world dataset featuring multilingual, multi-speaker conversational far-field speech was recorded and distributed. The challenge entailed two tracks: Track-1 focused on speaker diarization (SD) in multilingual situations while, Track-2 addressed the language diarization (LD) in a multi-speaker scenario. Both the tracks were evaluated using the same underlying audio data. Furthermore, a baseline system was made available for both SD and LD task which mimicked the state-of-art in these tasks. The challenge garnered a total of 42 world-wide registrations and received a total of 19 combined submissions for Track-1 and Track-2. This paper describes the challenge, details of the datasets, tasks, and the baseline system. Additionally, the paper provides a concise overview of the submitted systems in both tracks, with an emphasis given to the top performing systems. The paper also presents insights and future perspectives for SD and LD tasks, focusing on the key challenges that the systems need to overcome before wide-spread commercial deployment on such conversations.

The utility of ChatGPT as a generative medical translator.

Large language models continue to dramatically change the medical landscape. We aimed to explore the utility of ChatGPT in providing accurate, actionable, and understandable generative medical translations in English, Spanish, and Mandarin pertaining to Otolaryngology. Responses of GPT-4 to commonly asked patient questions listed on official otolaryngology clinical practice guidelines (CPG) were evaluated with the Patient Education materials Assessment Tool-printable (PEMAT-P.) Additional critical elements were identified a priori to evaluate ChatGPT's accuracy and thoroughness in its responses. Multiple fluent speakers of English, Mandarin, and Spanish evaluated each response generated by ChatGPT. Total PEMAT-P scores differed between English, Mandarin, and Spanish GPT-4 generated responses depicting a moderate effect size of language, Eta-Square 0.07 with scores ranging from 73 to 77 (P-value = 0.03). Overall understandability scores did not differ between English, Mandarin, and Spanish depicting a small effect size of language, Eta-Square 0.02 scores ranging from 76 to 79 (P-value = 0.17), nor did overall actionability scores Eta-Square 0 score ranging 66-73 (P-value = 0.44). Overall a priori procedure-specific responses similarly did not differ between English, Spanish, and Mandarin Eta-Square 0.02 scores ranging 61-78 (P-value = 0.22). GPT-4 produces accurate, understandable, and actionable outputs in English, Spanish, and Mandarin. Responses generated by GPT-4 in Spanish and Mandarin are comparable to English counterparts indicating a novel use for these models within Otolaryngology, and implications for bridging healthcare access and literacy gaps. IV.

Evaluation of BLE-based audio broadcasting under probabilistic interference

Wireless Bluetooth audio communication has become an inherent part of everyday life, from listening to a podcast on our headphones to streaming music on multiple speakers. Up until now, broadcasting audio to multiple receivers has always required a proprietary implementation. Therefore, recent advances in the specification allow Broadcast Audio streams to be set up on top of Bluetooth Low Energy (BLE). To cope with the unpredictability of wireless media, audio frame retransmission opportunities are provided. However, determining the applicable number of retransmissions for a broadcast stream that is exposed to volatile environmental conditions, is a complex research challenge. This paper presents a model that is capable of simulating a BLE broadcast stream schedule, exposed to various environmental conditions while using a variable number of audio frame retransmissions. The evaluation employs several existing Packet Loss Concealment (PLC) techniques to cope with audio frame losses. The results provide insights into the impact of various frame loss patterns on the audio quality and intelligibility of broadcasted speech. The more advanced PLC techniques can handle a higher frame loss rate threshold. The analysis also shows that large audio frames requiring fragmentation exhibit a higher amount of frame loss for the same BLE packet loss rate and that overlap with in-use 802.11 channels can lead to a large variability in frame loss behavior. The model provides a baseline for the next research challenge related to continuous management of Broadcast Audio streams operating under volatile environmental conditions.

Is Syntactic Priming from Multiple Speakers Stronger?

Syntactic priming in dialogue occurs when exposure to a particular syntactic structure implicitly induces a speaker’s subsequent preference for the same syntactic structures in their own speech. Here, we asked whether this priming effect is boosted when individuals are primed by several different speakers as opposed to one. In an initial baseline session involving a picture description task, we assessed adult participants’ production of double object/DO (vs. prepositional/PO) dative and passive (vs. active) transitive structures. Subsequently, participants played a picture description and verification game, in turns, with six other players (confederates). During verification turns, confederates primed participants by using DO and passive utterances. Crucially, participants were primed either by a single confederate (single-speaker priming condition, SSP) or by five confederates (multi-speaker priming condition, MSP). Across conditions, the same priming stimuli were presented in the same order, leaving speaker source/variation as the only different feature. The degree to which participants were primed for the target structures compared to baseline was measured. Results indicated a robust priming effect in both conditions. Nevertheless, the increase in the target structures’ use did not differ significantly between the SSP and MSP conditions, suggesting that speaker variation did not promote stronger priming.

Speech detection models for effective communicable disease risk assessment in air travel environments

In environments characterized by elevated noise levels, such as airports or aircraft cabins, travelers often find themselves involuntarily speaking loudly and drawing closer to one another in an effort to enhance communication and speech intelligibility. Unfortunately, this unintentional behaviour increases the risk of respiratory particles dispersion, potentially carrying infectious agents like bacteria which makes the contagion control more challenging. The accurate characterization of the risk associated to speaking, in such a challenging noise environment with multiple overlapping speech sources, is therefore of outmost importance. Among the most advanced signal processing strategies that can be used to accurately determine who spoke when and with whom and for how long but most importantly how loudly, at one location, artificial intelligence-based speaker diarization approaches were considered and adapted for this task. This article details the implementation of speaker diarization algorithms, customized to extract speaker and speech parameters discreetly. Validation and preliminary study results are also provided. The algorithms calculate speech duration and sound pressure level for each sentence and speaker, aiding in assessing viral contaminant spread. The paper focuses on applying these algorithms in noisy environments, particularly in confined spaces with multiple speakers. The findings contribute to proactive measures for containing and managing communicable diseases in air travel settings.

Convolutional Neural Network Architectures for Gender, Emotional Detection from Speech and Speaker Diarization

This paper introduces three system architectures for speaker identification that aim to overcome the limitations of diarization and voice-based biometric systems. Diarization systems utilize unsupervised algorithms to segment audio data based on the time boundaries of utterances, but they do not distinguish individual speakers. On the other hand, voice-based biometric systems can only identify individuals in recordings with a single speaker. Identifying speakers in recordings of natural conversations can be challenging, especially when emotional shifts can alter voice characteristics, making gender identification difficult. To address this issue, the proposed architectures include techniques for gender, emotion, and diarization at either the segment or group level. The evaluation of these architectures utilized two speech databases, namely VoxCeleb and RAVDESS (Ryerson audio-visual database of emotional speech and song) datasets. The findings reveal that the proposed approach outperforms the strategy level in terms of recognition results, despite the real-time processing advantage of the latter. The challenge of identifying multiple speakers engaging in a conversation while considering emotional changes that impact speech is effectively addressed by the proposed architectures. The data indicates that the gender and emotion classification of diarization achieves an accuracy of over 98 percent. These results suggest that the proposed speech-based approach can achieve highly accurate speaker identification.

AS-Net: active speaker detection using deep audio-visual attention

Active Speaker Detection (ASD) aims at identifying the active speaker among multiple speakers in a video scene. Previous ASD models often seek audio and visual features from long video clips with a complex 3D Convolutional Neural Network (CNN) architecture. However, models based on 3D CNNs can generate discriminative spatial-temporal features, but this comes at the expense of computational complexity, and they frequently face challenges in detecting active speakers in short video clips. This work proposes the Active Speaker Network (AS-Net) model, a simple yet effective ASD method tailored for detecting active speakers in relatively short video clips without relying on 3D CNNs. Instead, it incorporates the Temporal Shift Module (TSM) into 2D CNNs, facilitating the extraction of dense temporal visual features without the need for additional computations. Moreover, self-attention and cross-attention schemes are introduced to enhance long-term temporal audio-visual synchronization, thereby improving ASD performance. Experimental results demonstrate that AS-Net outperforms state-of-the-art 2D CNN-based methods on the AVA-ActiveSpeaker dataset and remains competitive with the methods utilizing more complex architectures.

Multi-Speaker Tracking from Azure Kinect: Performance Evaluation using Generalized Optimal Sub-Pattern Assignment

The software and the hardware features of Azure Kinect show the potential for its application in multiple speaker tracking. We then evaluate the system’s capabilities by conducting tests using our given setup, including various scenarios to evaluate its tracking performance. The tracking performance is calculated using generalized optimal sub-pattern assignment (GOSPA) and multiple objects tracking accuracy (MOTA) metrics. It has been found that the body tracking algorithm can perform well in certain multi-speaker tracking conditions.

Neural Network-Based Approach to Detect and Filter Misleading Audio Segments in Classroom Automatic Transcription

Audio recording in classrooms is a common practice in educational research, with applications ranging from detecting classroom activities to analyzing student behavior. Previous research has employed neural networks for classroom activity detection and speaker role identification. However, these recordings are often affected by background noise that can hinder further analysis, and the literature has only sought to identify noise with general filters and not specifically designed for classrooms. Although the use of high-end microphones and environmental monitoring can mitigate this problem, these solutions can be costly and potentially disruptive to the natural classroom environment. In this context, we propose the development of a novel neural network model that specifically detects and filters out problematic audio sections in classroom recordings. This model is particularly effective in reducing transcription errors, achieving up to a 96% success rate in filtering out segments that could lead to incorrect automated transcriptions. The novelty of our work lies in its targeted approach for low-budget, aurally complex environments like classrooms, where multiple speakers are present. By allowing the use of lower-quality recordings without compromising analysis capability, our model facilitates data collection in natural educational settings and reduces the dependency on expensive recording equipment. This advancement not only demonstrates the practical application of specialized neural network filters in challenging acoustic environments but also opens new avenues for enhancing audio analysis in educational research and beyond.