Voice Activity Detection Research Articles

An Ensemble Approach for Speaker Identification from Audio Files in Noisy Environments

Automatic noise-robust speaker identification is essential in various applications, including forensic analysis, e-commerce, smartphones, and security systems. Audio files containing suspect speech often include background noise, as they are typically not recorded in soundproof environments. To this end, we address the challenges of noise robustness and accuracy in speaker identification systems. An ensemble approach is proposed combining two different neural network architectures including an RNN and DNN using softmax. This approach enhances the system’s ability to identify speakers even in noisy environments accurately. Using softmax, we combine voice activity detection (VAD) with a multilayer perceptron (MLP). The VAD component aims to remove noisy frames from the recording. The softmax function addresses these residual traces by assigning a higher probability to the speaker’s voice compared to the noise. We tested our proposed solution on the Kaggle speaker recognition dataset and compared it to two baseline systems. Experimental results show that our approach outperforms the baseline systems, achieving a 3.6% and 5.8% increase in test accuracy. Additionally, we compared the proposed MLP system with Long Short-Term Memory (LSTM) and Bidirectional LSTM (BiLSTM) classifiers. The results demonstrate that the MLP with VAD and softmax outperforms the LSTM by 23.2% and the BiLSTM by 6.6% in test accuracy.

Speech Perception Outcomes with the Anatomy-Based Fitting Map Among Experienced, Adult Cochlear Implant Users: A Longitudinal Study.

Anatomy-based fitting (ABF), a relatively new technique for cochlear implant programming, attempts to lessen the impact of the electrode insertion location-related frequency-to-place mismatch (FPM). This study aimed to compare vowels and consonant perception in quiet and in noise among experienced adult cochlear implant (CI) users using the ABF and the regular, conventional-based fitting (CBF) map (pre-ABF) over six months. Nine ears from eight experienced adult CI users were included in the experimental and longitudinal research. Using surgical planning software called Otoplan, post-operative computed CT scans were used to determine the locations of intracochlear electrodes and their angle of insertion. The anatomy-based frequency bands were produced by Maestro 9.0 CI fitting software using the Otoplan data. Nonsense syllables with consonant-vowel-consonant (CVC) recognition scores in quiet and noise (+5dB SNR) were compared at baseline, three, and six months after ABF. The vowels involved were /a, i, u/, while the consonants were voiced /b, d, g/ and voiceless /p, t, k/ plosives. Speech pieces were presented at 30 dB SL in a sound-treated room through a loudspeaker positioned at 0° azimuth. On average, the ABF maps shifted center frequency ranging from 0.46 semitones (0.04 octave) at (E12) to 23.94 semitones (1.99 octave) at (E1) as compared to the CBF maps. The mean vowel and consonant identification scores in quiet and in noise were significantly higher in ABF than in CBF (p<0.05) with a large effect size and the trend of improvement was seen with time. Voiced consonants had better scores than the voiceless consonants. The results demonstrated improved perception of vowels and consonants, particularly for sounds containing voicing cues after using the ABF maps. The results also suggested that ABF could be more effective for voice detection in noise. Overall, the findings indicate that correcting place mismatch with an ABF map may improve speech perception, at least among experienced adult cochlear implant users.

Exploiting speech tremors: machine learning for early diagnosis of amyotrophic lateral sclerosis

Abstract Neurodegenerative diseases pose significant challenges in healthcare, with Amyotrophic Lateral Sclerosis (ALS) being one such rare yet debilitating condition affecting motor neurons. Machine learning (ML) and artificial intelligence (AI) have emerged as powerful tools in healthcare, offering insights and solutions for various medical conditions. This study investigates the application of ML to enhance early ALS diagnosis through the analysis of tremors in sustained speech. By focusing on tremor detection as a diagnostic marker, the research employs ML algorithms to develop predictive models capable of distinguishing ALS patients from healthy controls. The dataset comprises 54 patients from the Republican Research and Clinical Centre of Neurology and Neurosurgery in Belarus, Minsk. The study adopts a two-faceted approach: (1) Exploratory voice analysis to identify tremors associated with ALS in speech samples. (2) Development of ML algorithms to construct predictive models for early ALS diagnosis based on the identified tremors. The ML models exhibit promising results in distinguishing ALS patients from healthy controls based on speech analysis. Tremor detection in sustained speech proves to be an effective marker for early ALS diagnosis. While initial findings are encouraging, larger-scale studies are required to validate the clinical applicability of this approach. The successful application of ML and AI in early ALS diagnosis by leveraging innovative approaches, such as tremor detection in sustained speech, we can enhance early diagnosis and improve patient outcomes in neurodegenerative diseases like ALS on a broader scale.

Multi-scale Information Aggregation for Spoofing Detection

Synthesis artifacts that span scales from small to large are important cues for spoofing detection. However, few spoofing detection models leverage artifacts across different scales together. In this paper, we propose a spoofing detection system built on SincNet and Deep Layer Aggregation (DLA), which leverages speech representations at different levels to distinguish synthetic speech. DLA is totally convolutional with an iterative tree-like structure. The unique topology of DLA makes possible compounding of speech features from convolution layers at different depths, and therefore the local and the global speech representations can be incorporated simultaneously. Moreover, SincNet is employed as the frontend feature extractor to circumvent manual feature extraction and selection. SincNet can learn fine-grained features directly from the input speech waveform, thus making the proposed spoofing detection system end-to-end. The proposed system outperforms the baselines when tested on ASVspoof LA and DF datasets. Notably, our single model surpasses all competing systems in ASVspoof DF competition with an equal error rate (EER) of 13.99%, which demonstrates the importance of multi-scale information aggregation for synthetic speech detection.

AFP-Conformer: Asymptotic Feature Pyramid Conformer for Spoofing Speech Detection

The existing spoofing speech detection methods mostly use either convolutional neural networks or Transformer architectures as their backbone, which fail to adequately represent speech features during feature extraction, resulting in poor detection and generalization performance of the models. To solve this limitation, we propose a novel spoofing speech detection method based on the Conformer architecture. This method integrates a convolutional module into the Transformer framework to enhance its capacity for local feature modeling, enabling to extract both local and global information from speech signals simultaneously. Besides, to mitigate the issue of semantic information loss or degradation in traditional feature pyramid networks during feature fusion, we propose a feature fusion method based on the asymptotic feature pyramid network (AFPN) to fuse multi-scale features and improve generalization of detecting unknown attacks. Our experiments conducted on the ASVspoof 2019 LA dataset demonstrate that our proposed method achieved the equal error rate (EER) of 1.61% and the minimum tandem detection cost function (min t-DCF) of 0.045, effectively improving the detection performance of the model while enhancing its generalization capability against unknown spoofing attacks. In particular, it demonstrates substantial performance improvement in detecting the most challenging A17 attack.

Deep learning for hate speech detection: a comparative study

Deep learning for hate speech detection: a comparative study

Research on multimodal hate speech detection based on self-attention mechanism feature fusion

Research on multimodal hate speech detection based on self-attention mechanism feature fusion

Emotion Classification from Speech Waveform Using Machine Learning and Deep Learning Techniques

Emotions play a key role in determining the human mental state and indirectly express an individual’s well- being. A speech emotion recognition system can extract a person’s emotions from his/her speech inputs. There are some universal emotions such as anger, disgust, fear, happiness, pleasantness, sadness and neutral. These emotions are of significance especially in a situation like the Covid pandemic, when the aged or sick are vulnerable to depression. In the current paper, we examined various classification models with finite computational strength and resources in order to determine the emotion of a person from his/her speech. Speech prosodic features like pitch, loudness, and tone of speech, and work spectral features such as Mel Frequency Capstral Coefficients (MFCCs) of the voice were used to analyze the emotions of a person. Although sequence to sequence state of the art models for speech detection that offer high levels of accuracy and precision are currently in use, the computational needs of such approaches are high and inefficient. Therefore, in this work, we emphasised analysis and comparison of different classification algorithms such as multi layer perceptron, decision tree, support vector machine, and deep neural networks such as convolutional neural network and long short term memory. Given an audio file, the emotions that were exhibited by the speaker were recognized using machine learning and deep learning techniques. A comparative study was performed to identify the most appropriate algorithms that could be used to recognize emotions. Based on the experiment results, the MLP classifier and convolutional neural network model offered better accuracy with smaller variations when compared with other models used for the study.

GASCOM: Graph-based Attentive Semantic Context Modeling for Online Conversation Understanding

Online conversation understanding is an important yet challenging NLP problem which has many useful applications (e.g., hate speech detection). However, online conversations typically unfold over a series of posts and replies to those posts, forming a tree structure within which individual posts may refer to semantic context from elsewhere in the tree. Such semantic cross-referencing makes it difficult to understand a single post by itself; yet considering the entire conversation tree is not only difficult to scale but can also be misleading as a single conversation may have several distinct threads or points, not all of which are relevant to the post being considered. In this paper, we propose a Graph-based Attentive Semantic COntext Modeling (GASCOM) framework for online conversation understanding. Specifically, we design two novel algorithms that utilize both the graph structure of the online conversation as well as the semantic information from individual posts for retrieving relevant context nodes from the whole conversation. We further design a token-level multi-head graph attention mechanism to pay different attentions to different tokens from different selected context utterances for fine-grained conversation context modelling. Using this semantic conversational context, we re-examine two well-studied problems: polarity prediction and hate speech detection. Our proposed framework significantly outperforms state-of-the-art methods on both tasks, improving macro-F1 scores by 4.5% for polarity prediction and by 5% for hate speech detection. The GASCOM context weights also enhance interpretability.

A Comparative Study of Transformer-based Models for Hate-Speech Detection in English-Kiswahili Code-Switched Social Media Text

The transformer architecture, first introduced in 2017 by researchers at Google, has revolutionized natural language processing in various tasks, including text classification. This architecture formed the basis of future models such as those used in hate speech detection in code-switched text. In this research, we conduct a comparative study of transformer-based models for hate speech detection in English-Kiswahili code-switched text. First, the models were compared as feature extractors using a traditional classifier and then as end-to-end classifiers. The three multilingual transformer-based models compared include mBERT, mDistilBERT and XLM-RoBERTa, using SVM as the traditional classifier for the extracted features. The HateSpeech_Kenya dataset, sourced from Kaggle, was utilized in this study. As a feature extractor, mBERT’s hidden states trained the highest-performing SVM with an accuracy of 0.5461 and a macro f1 score of 0.40. Among the three models evaluated, XLM-RoBERTa achieved the highest accuracy of 0.6069 and a macro f1 score of 0.49 on a balanced dataset. In contrast, mBERT achieved the highest accuracy of 0.7820 and a macro f1 score of 0.53 on an imbalanced dataset. The comparative study establishes that using transformer-based models as end-to-end classifiers generally performs better than using them as feature extractors with traditional classifiers. This is because directly training the models allows them to learn more task-specific features. Furthermore, the varying performance across balanced and imbalanced datasets highlights the need for careful model selection based on the dataset characteristics and specific task requirements.

Pathological voice detection using optimized deep residual neural network and explainable artificial intelligence

Pathological voice detection using optimized deep residual neural network and explainable artificial intelligence

Contrastive Learning based Speech Spoofing Detection for Multimedia Security in Edge Intelligence

Artificial intelligence (AI) empowered edge computing has given rise to a new paradigm and effectively facilitated the promotion and development of multimedia applications. The speech assistant is one of the significant services provided by multimedia applications, which aims to offer intelligent interactive experiences between humans and machines. However, malicious attackers may exploit spoofed speeches to deceive speech assistants, posing great challenges to the security of multimedia applications. The limited resources of multimedia terminal devices hinder their ability to effectively load speech spoofing detection models. Furthermore, processing and analyzing speech in the cloud can result in poor real-time performance and potential privacy risks. Existing speech spoofing detection methods rely heavily on annotated data and exhibit poor generalization capabilities for unseen spoofed speeches. To address these challenges, this paper first proposes the Coordinate Attention Network (CA2Net) that consists of coordinate attention blocks and Res2Net blocks. CA2Net can simultaneously extract temporal and spectral speech feature information and represent multi-scale speech features at a granularity level. Besides, a contrastive learning-based speech spoofing detection framework named GEMINI is proposed. GEMINI can be effectively deployed on edge nodes and autonomously learn speech features with strong generalization capabilities. GEMINI first performs data augmentation on speech signals and extracts conventional acoustic features to enhance the feature robustness. Subsequently, GEMINI utilizes the proposed CA2Net to further explore the discriminative speech features. Then, a tensor-based multi-attention comparison model is employed to maximize the consistency between speech contexts. GEMINI continuously updates CA2Net with contrastive learning, which enables CA2Net to effectively represent speech signals and accurately detect spoofed speeches. Extensive experiments on the ASVspoof2019 dataset show that GEMINI reduces the Equal Error Rate and tandem Detection Cost Function by up to 96.75% and 96.35% in the physical access scenario, and by up to 86.62% and 87.71% in the logical access scenario compared to peer methods.

Integrated noise suppression techniques for enhancing voice activity detection in degraded environments

Integrated noise suppression techniques for enhancing voice activity detection in degraded environments

Dual-stream Noise and Speech Information Perception based Speech Enhancement

In real-world scenarios, dynamic ambient noise often degrades speech quality, highlighting the need for advanced speech enhancement techniques. Traditional methods, which rely on static embeddings as auxiliary features, struggle to address the complexities of varying noise conditions. To overcome this, we propose a Dual-stream Noise and Speech Information Perception (DNSIP) approach that dynamically detects and processes both noise and speech through innovative information extraction and suppression mechanisms. Initially, non-speech segments predominantly contain environmental noise, while speech segments carry information about the intended speaker. To handle this dynamic nature, real-time voice activity detection (VAD) is employed to accurately differentiate between speech and noise components. Building on VAD estimates, we propose an innovative information extraction framework that selectively extracts relevant noise and speech features from the noisy input, establishing a dual-stream network for concurrent noise and speech learning. To account for the temporal and spectral variability of noise and speech, a frequency-sequence attention mechanism is integrated, enhancing the model’s ability to learn contextual and spectral dependencies. Additionally, an information suppression module is introduced to minimize cross-stream interference by attenuating noise within the speech stream and suppressing speech content within the noise stream. The derived noise and speech spectrograms are then utilized to formulate a minimum mean square error log-spectral amplitude (MMSE-LSA) estimator for robust speech enhancement. Experimental evaluations on the WSJ0 and VCTK+DEMAND datasets demonstrate that our DNSIP approach surpasses existing state-of-the-art methods, underscoring its efficacy in challenging acoustic environments.

Spoofing countermeasure for fake speech detection using brute force features

Due to the progress in deep learning technology, techniques that generate spoofed speech have significantly emerged. Such synthetic speech can be exploited for harmful purposes, like impersonation or disseminating false information. Researchers in the area investigate the useful features for spoof detection. This paper extensively investigates three problems in spoof detection in speech, namely, the imbalanced sample per class, which may negatively affect the performance of any detection models, the effect of the feature early and late fusion, and the analysis of unseen attacks on the model. Regarding the imbalanced issue, we have proposed two approaches (a Synthetic Minority Over Sampling Technique (SMOTE)-based and a Bootstrap-based model). We have used the OpenSMILE toolkit, to extract different feature sets, their results and early and late fusion of them have been investigated. The experiments are evaluated using the ASVspoof 2019 datasets which encompass synthetic, voice-conversion, and replayed speech samples. Additionally, Support Vector Machine (SVM) and Deep Neural Network (DNN) have been adopted in the classification. The outcomes from various test scenarios indicated that neither the imbalanced nature of the dataset nor a specific feature or their fusions outperformed the brute force version of the model as the best Equal Error Rate (EER) achieved by the Imbalance model is 6.67 % and 1.80 % for both Logical Access (LA) and Physical Access (PA) respectively.

Understanding hate speech: the HateInsights dataset and model interpretability

The persistence of hate speech continues to pose an obstacle in the realm of online social media. Despite the continuous evolution of advanced models for identifying hate speech, the critical dimensions of interpretability and explainability have not received proportional scholarly attention. In this article, we introduce the HateInsights dataset, a groundbreaking benchmark in the field of hate speech datasets, encompassing diverse aspects of this widespread issue. Within our dataset, each individual post undergoes thorough annotation from dual perspectives: firstly, conforming to the established 3-class classification paradigm that includes hate speech, offensive language, and normal discourse; secondly, incorporating rationales that outline specific segments of a post supporting the assigned label (categorized as hate speech, offensive language, or normal discourse). Our exploration yields a significant finding by harnessing cutting-edge state-of-the-art models: even models demonstrating exceptional proficiency in classification tasks yield suboptimal outcomes in crucial explainability metrics, such as model plausibility and faithfulness. Furthermore, our analysis underscores a promising revelation concerning models trained using human-annotated rationales. To facilitate scholarly progress in this realm, we have made both our dataset and codebase accessible to fellow researchers. This initiative aims to encourage collaborative involvement and inspire the advancement of the hate speech detection approach characterized by increased transparency, clarity, and fairness.

Smart Dispenser Using Voice Recognition as an Assistive Device for the Visually Impaired with a Second-Order IIR Filter Algorithm

This research developed a Smart Dispenser with Voice Recognition technology as a tool to help the visually impaired in taking drinking water. The system is designed using a Voice Recognition sensor, an ISD 1820 sensor, and a microcontroller to enable the operation of the dispenser via voice commands. The test was carried out to analyze the performance of the system in recognizing voice commands and removing water. The results showed a voice detection accuracy rate of 28%, with factors such as environmental noise and sound intonation affecting the effectiveness of the system. The study also identified several areas of improvement, including the use of a DC water pump to increase the flow rate, the addition of an automatic feature to stop the flow of water when a glass is taken, and the use of a DC fan to maintain the sensor's optimal temperature. Further development is needed to improve the accuracy and efficiency of the system so that it can more effectively help the visually impaired in daily life.

Exploring a Mobile Technology-Driven Model for Intercultural Communication Education

In the context of accelerating globalization, intercultural communication competence has become a crucial element for the success of individuals and organizations. The frequent international interactions have underscored the importance of enhancing methods for improving intercultural communication skills within the educational sector. The rapid advancement of mobile technology offers unprecedented opportunities for educational innovation. Its convenience and widespread use have made mobile device-based learning models highly attractive. Particularly, advancements in real-time speech processing technologies have provided new tools and methods for intercultural communication education. By leveraging mobile real-time speech detection and synthesis technologies, a more interactive and personalized learning experience can be achieved, thereby enhancing the efficiency and effectiveness of language learning. This study aims to explore a mobile technology-based model for intercultural communication education and is divided into three main parts: firstly, the investigation of mobile real-time speech detection technologies aimed at intercultural communication education to provide instant feedback and improvement suggestions; secondly, the exploration of mobile real-time speech synthesis technologies to generate high-quality speech samples for learners to practice against; and thirdly, the integration of the aforementioned technologies to develop a flexible, efficient, and highly interactive learning system based on mobile technology. This study is expected to not only improve the effectiveness of intercultural communication education but also provide significant references for the innovative application of educational technologies.

Real-time detection of spoken speech from unlabeled ECoG signals: A pilot study with an ALS participant.

Brain-Computer Interfaces (BCIs) hold significant promise for restoring communication in individuals with partial or complete loss of the ability to speak due to paralysis from amyotrophic lateral sclerosis (ALS), brainstem stroke, and other neurological disorders. Many of the approaches to speech decoding reported in the BCI literature have required time-aligned target representations to allow successful training - a major challenge when translating such approaches to people who have already lost their voice. In this pilot study, we made a first step toward scenarios in which no ground truth is available. We utilized a graph-based clustering approach to identify temporal segments of speech production from electrocorticographic (ECoG) signals alone. We then used the estimated speech segments to train a voice activity detection (VAD) model using only ECoG signals. We evaluated our approach using held-out open-loop recordings of a single dysarthric clinical trial participant living with ALS, and we compared the resulting performance to previous solutions trained with ground truth acoustic voice recordings. Our approach achieves a median error rate of around 0.5 seconds with respect to the actual spoken speech. Embedded into a real-time BCI, our approach is capable of providing VAD results with a latency of only 10 ms. To the best of our knowledge, our results show for the first time that speech activity can be predicted purely from unlabeled ECoG signals, a crucial step toward individuals who cannot provide this information anymore due to their neurological condition, such as patients with locked-in syndrome. ClinicalTrials.gov, registration number NCT03567213.

TABHATE: A Target-based hate speech detection dataset in Hindi

TABHATE: A Target-based hate speech detection dataset in Hindi