Speaker Recognition Research Articles

Flexible Hair‐Like Piezoelectric Acoustic Particle Velocity Sensor with Enhanced Sensitivity for Speaker Recognition

AbstractFlexible resonant acoustic sensors exhibit enhanced sensitivity near their resonant frequencies and have attracted increasing interest as essential components for next‐generation human‐machine speech interactions. However, membrane‐based resonant acoustic sensors are bulky owing to the inherently high bending stiffness of the sensing membrane. Inspired by the tiny hair‐based sensitive acoustic receptors of spiders, this study reports a hair‐like piezoelectric resonant acoustic particle velocity sensor (HP‐APVS) that enables highly sensitive sound sensing with a significantly reduced size. The HP‐APVS device is essentially a polyimide cantilever array fabricated using Nb0.02‐Pb(Zr0.6Ti0.4)O3 (PZT) on polyimide and self‐bending processes. The experimental results demonstrate that the HP‐APVS shows a linear response to acoustic particle velocity instead of acoustic pressure, with enhanced sensitivity in the resonance mode. Additionally, the proposed HP‐APVS exhibits an outstanding speaker recognition rate of 95.3% with an error rate reduction of 75% compared with that of a reference commercial microphone, indicating many potential applications in the realms of biometric authentication, voice user interfaces, and other intelligent acoustic electronics.

Backdoor Attacks against Voice Recognition Systems: A Survey

Voice Recognition Systems (VRSs) employ deep learning for speech recognition and speaker recognition. They have been widely deployed in various real-world applications, from intelligent voice assistance to telephony surveillance and biometric authentication. However, prior research has revealed the vulnerability of VRSs to backdoor attacks, which pose a significant threat to the security and privacy of VRSs. Unfortunately, existing literature lacks a thorough review on this topic. This paper fills this research gap by conducting a comprehensive survey on backdoor attacks against VRSs. We first present an overview of VRSs and backdoor attacks, elucidating their basic knowledge. Then we propose a set of evaluation criteria to assess the performance of backdoor attack methods. Next, we present a comprehensive taxonomy of backdoor attacks against VRSs from different perspectives and analyze the characteristic of different categories. After that, we comprehensively review existing attack methods and analyze their pros and cons based on the proposed criteria. Furthermore, we review classic backdoor defense methods and generic audio defense techniques. Then we discuss the feasibility of deploying them on VRSs. Finally, we figure out several open issues and further suggest future research directions to motivate the research of VRSs security.

Evaluating the effects of continuous pitch and speech tempo modifications on perceptual speaker verification performance by familiar and unfamiliar listeners

A study was conducted to evaluate the effects of continuous pitch and speech tempo modifications on perceptual speaker verification performance by familiar and unfamiliar naive listeners. Speech recordings made by twelve male, native-French speakers were organised into three groups of four (two in-set, one out-of-set). Two groups of listeners participated, where one group was familiar with one in-set speaker group, while both groups were unfamiliar with the remaining in- and out-of-set speaker groups. Pitch and speech tempo were continuously modified, such that the first 75% of words spoken were modified with percentages of modification beginning at 100% and decaying linearly to 0%. Pitch modifications began at ± 600 cents, while speech tempo modifications started with word durations scaled 1:2 or 3:2. Participants evaluated a series of “go/no-go” task trials, where they were presented a modified speech recording with a face and tasked to respond as quickly as possible if they judged the stimuli to be continuous. The major findings revealed listeners overcame higher percentages of modification when presented familiar speaker stimuli. Familiar listeners outperformed unfamiliar listeners when evaluating continuously modified speech tempo stimuli, however, this effect was speaker-specific for pitch modified stimuli. Contrasting effects of modification direction were also observed. The findings suggest pitch is more useful to listeners when verifying familiar and unfamiliar voices.

Open source platform for Estonian speech transcription

AbstractThis paper presents our progress in developing and maintaining a public speech and speaker recognition platform for the Estonian language. The platform consists of a speech processing pipeline and a web-based user interface for end-users, offering transcript post-editing functionality. It is offered for free as a public service and is in active use. The service provides significantly higher speech recognition accuracy than commercial alternatives. We discuss the switch to a workflow management system and how it has improved the core speech processing pipeline. The core systems behind the platform have been made available as open-source code and deployed internally by multiple public and private institutions.

Ensemble Adversarial Defenses and Attacks in Speaker Verification Systems

Ensemble Adversarial Defenses and Attacks in Speaker Verification Systems

Speaker Recognition System Using Hybrid of MFCC and RCNN with HCO Algorithm Optimization

Though there are advancements in speaker recognition technology, available systems often fail to correctly recognize speakers especially in noisy environments. The use of Mel-frequency cepstral coefficients (MFCC) has been improved using Convolutional Neural Networks (CNN) yet difficulties in achieving high accuracies still exists. Hybrid algorithms combining MFCC and Region-based Convolutional Neural Networks (RCNN) have been found to be promising. In this research features from speech signals were extracted for speaker recognition, to denoise the signals, design and develop a DFT-based denoising system using spectrum subtraction and to develop a speaker recognition method for the Verbatim Transcription using MFCC. The DFT was used to transform the sampled audio signal waveform into a frequency-domain signal. RCNN was used to model the characteristics of speakers based on their voice samples, and to classify them into different categories or identities. The novelty of the research was that it used MFCC integrated with RCNN and optimized with Host-Cuckoo Optimization (HCO) algorithm. HCO algorithm is capable of further weight optimization through the process of generating fit cuckoos for best weights. It also captured the temporal dependencies and long-term information. The system was tested and validated on audio recordings from different personalities from the National Assembly of Kenya. The results were compared with the actual identity of the speakers to confirm accuracy. The performance of the proposed approach was compared with two other existing speaker recognition the traditional approaches being MFCC-CNN and Linear Predictive Coefficients (LPC)-CNN. The comparison was based the Equal Error Rate (EER), False Rejection Rate (FRR), False Match Rate (FMR), and True Match Rate (TMR). Results show that the proposed algorithm outperformed the others in maintaining a lowest EER, FMR, FRR and highest TMR.

Construction of an effective telephone bandwidth specifically for speaker recognition  

Although the telephone bandwidth is 0-4 kHz, the speaker specific information is not evenly distributed within this range but extends beyond 4 kHz. This is because the hypopharynx, especially the piriform fossa, affects the higher frequency region and contributes more to inter-speaker variation. By effectively shifting the higher bands to a lower frequency region, where speaker specific information is reduced, an effective 4 kHz bandwidth can be constructed to enhance speaker recognition performance. To achieve this a method was already proposed, which is extended in this paper to experimentally demonstrate and validate with more experiments. Furthermore, this paper defines the theoretically possible frequency space for which the frequency shifting method can be applied. To validate the method for different combinations of bands, possible bands were shifted in various directions in small steps. Speaker recognition experiments were conducted at each step to compare the performance against the baseband without any frequency shifting. Using the results of these extensive experiments, an approximate frequency space was defined where this frequency shifting performed better than the conventional baseband of 0-4 kHz signal. A simplified frequency shifting method was also investigated. Finally, the speech intelligibility of the frequency shifted narrow band speech signal was analysed using objective speech quality measures. This showed that intelligibility was not significantly affected by the frequency shifting method.

Balancing validity and reliability as a function of sampling variability in forensic voice comparison

In forensic comparison sciences, experts are required to compare samples of known and unknown origin to evaluate the strength of the evidence assuming they came from the same- and different-sources. The application of valid (if the method measures what it is intended to) and reliable (if that method produces consistent results) forensic methods is required across many jurisdictions, such as the England & Wales Criminal Practice Directions 19A and UK Crown Prosecution Service and highlighted in the 2009 National Academy of Sciences report and by the President’s Council of Advisors on Science and Technology in 2016. The current study uses simulation to examine the effect of number of speakers and sampling variability and on the evaluation of validity and reliability using different generations of automatic speaker recognition (ASR) systems in forensic voice comparison (FVC). The results show that the state-of-the-art system had better overall validity compared with less advanced systems. However, better validity does not necessarily lead to high reliability, and very often the opposite is true. Better system validity and higher discriminability have the potential of leading to a higher degree of uncertainty and inconsistency in the output (i.e. poorer reliability). This is particularly the case when dealing with small number of speakers, where the observed data does not adequately support density estimation, resulting in extrapolation, as is commonly expected in FVC casework.

Conceptual model of the technology for calculating the similarity threshold of two audio sequences

The paper is focused on the pressing problem of speaker verification by means of voice time series comparison. The aim of this paper is to determine the orders of mel-frequency cepstral coefficients that most accurately describe the difference, between an authentic voice and an artificially generated copy for their further use as input to a neural network model in a resource-limited environment. To achieve this goal, the following tasks were accomplished: a conceptual model of the technology for determining the similarity threshold of two audio series was developed; the orders of fine-frequency cepstral coefficients with the most characteristic differences between the recording and the generated voice were determined on the basis of neural network analysis; an experimental study of the dependence of the execution time and computational load on the created feature vector when assessing the degree of similarity of two time series was conducted; and the optimal similarity threshold was determined on the basis of the chosen dataset. The developed model of the technology for determining the similarity threshold was tested on a dataset that is a combination of the DEEP-VOICE dataset and our own dataset. The demonstrated result of applying the developed technology showed an increase of 43% when using the specified MFCCs compared to using all of them. Based on experimental studies, the DTW acceptance threshold was set at 0.37.

MelCochleaGram-DeepCNN: Sequentially Fused Spectrogram and the DeepCNN Classifiers-based Audio Spoof Detection System

Automatic Speaker Verification (ASV) systems are crucial in various fields, enabling speaker identification for authentication, fraud detection, and forensic applications. While the simplicity and effectiveness of speech biometrics are driving the demand for ASV systems, their increasing popularity raises concerns about vulnerability to speech attacks. To enhance the security of these systems, the work in this paper proposes a spectrogram-based solution that leverages the robustness of spectrograms in audio analysis and feature extraction. The proposed model consists of two main components: frontend and backend. In the frontend, it introduces a novel spectrogram MelCochleaGram (MCG) by fusing Mel Spectrogram and Cochleagram, sequentially. For the backend implementation, pre-trained deep learning models, including ResNet50, ResNet50V2, and InceptionV3, are employed using the Keras framework. These models are individually paired with MCG to detect deepfake and replay attacks. To validate the effectiveness of the proposed system, thorough experimentation is conducted on two datasets: the DEepfake CROss-lingual (DECRO) evaluation dataset and the Voice Spoofing Detection Corpus (VSDC). The proposed combination of MCG with ResNet50 has achieved an Equal Error Rate (EER) of 0.2%, and 1.2% for deepfake detection over DECRO English and Chinese datasets, respectively. Also, for replay attack detection, the proposed combination has produced an EER of 1.4%.

Enhanced text-independent speaker recognition using MFCC, Bi-LSTM, and CNN-based noise removal techniques

Enhanced text-independent speaker recognition using MFCC, Bi-LSTM, and CNN-based noise removal techniques

Identification of Customer Through Voice Biometric System in Call Centres

In recent times, there has been a growing emphasis on adjusting communication strategies to foster strong customer relationships. This shift is driven by intensified competition, market maturation, and swift advancements in business technology. Consequently, companies have established call centers to efficiently handle customer support and fulfil customer inquiries. A pivotal aspect of enhancing service quality within these call centers involves accurately identifying customers during their interactions. The primary objective of this study is to introduce a methodology for identifying customers within call centers by analyzing their voice characteristics. Voice authentication (VA) has gained prominence in critical security operations, including banking transactions and conversations within call centers. The susceptibility of automatic speaker verification systems (ASVs) to deceptive spoofing attacks has prompted the development of countermeasures (CMs). These countermeasures are designed to differentiate between authentic and fabricated speech. ASVs and CMs collectively constitute contemporary VA systems, positioned as robust access control mechanisms. To achieve this goal, various customer identification systems within call centers have been examined, along with an analysis of audio signal attributes. Ultimately, the manuscript presents a novel approach to customer identification through voice biometrics. Notably, this method excels in recognizing customers even when provided with limited voice data. Empirical findings demonstrate that the suggested speaker identity confirmation method outperforms alternative techniques utilizing different algorithms, exhibiting a higher recognition rate. The present research work is based on two important perspectives of the call centres: a. call center agents experience and b. customer experience. The data collected separately from customers and agents for understanding the effective usage of voice biometric system in call centres. The data represented and satisfies the effectiveness of voice biometric system from both the perspectives. From the data it is also cleared that, the implementation of voice biometric system in call centres still have long way to go but will be a major technological change for the industries worldwide.

Decoding Silent Speech Cues From Muscular Biopotential Signals for Efficient Human‐Robot Collaborations

AbstractSilent speech interfaces offer an alternative and efficient communication modality for individuals with voice disorders and when the vocalized speech communication is compromised by noisy environments. Despite the recent progress in developing silent speech interfaces, these systems face several challenges that prevent their wide acceptance, such as bulkiness, obtrusiveness, and immobility. Herein, the material optimization, structural design, deep learning algorithm, and system integration of mechanically and visually unobtrusive silent speech interfaces are presented that can realize both speaker identification and speech content identification. Conformal, transparent, and self‐adhesive electromyography electrode arrays are designed for capturing speech‐relevant muscle activities. Temporal convolutional networks are employed for recognizing speakers and converting sensing signals into spoken content. The resulting silent speech interfaces achieve a 97.5% speaker classification accuracy and 91.5% keyword classification accuracy using four electrodes. The speech interface is further integrated with an optical hand‐tracking system and a robotic manipulator for human‐robot collaboration in both assembly and disassembly processes. The integrated system achieves the control of the robot manipulator by silent speech and facilitates the hand‐over process by hand motion trajectory detection. The developed framework enables natural robot control in noisy environments and lays the ground for collaborative human‐robot tasks involving multiple human operators.

Voice recognition by deep transfer learning and vision transformers to secure voice authentication

Speech recognition is crucial for ensuring the security of personal devices and financial transactions. Attaining high accuracy and robustness in voice authentication is challenging due to the presence of voice and environmental variability. Recent advancements in the field of deep learning, particularly in transfer learning and visual transformers, have the potential to enhance voice recognition systems. This study employs advanced deep transfer learning techniques, including Vision Transformers (ViT), VGG16, and a customized Convolutional Neural Network (CNN), to enhance the accuracy and security of speech authentication. The objective is to evaluate and contrast various solutions' voice recognition and authentication accuracy. The experiment included 3000 voice samples, with an equal distribution of 1500 samples from male participants and 1500 from female participants. The dataset was used to train Vision Transformers, VGG16 with transfer learning, and a custom CNN. The models were assessed based on their accuracy in identifying and authenticating voice samples. The VGG16 model achieved the highest level of accuracy in speech recognition, with a precision rate of 95%. The Vision Transformer and custom CNN exhibited satisfactory performance. However, VGG16 demonstrated higher accuracy. The most accurate voice authentication model studied is the VGG16 model based on transfer learning. This study suggests that the security and reliability of voice recognition systems can be enhanced through the use of deep learning techniques.

Serial Multimodal Biometrics Authentication and Liveness Detection Using Speech Recognition with Normalized Longest Word Subsequence Method

Biometric authentication aims to verify whether an entity matches the claimed identity based on biometric data. Despite its advantages, vulnerabilities, particularly those related to spoofing, still exist. Efforts to mitigate these vulnerabilities include multimodal approaches and liveness detection. However, these strategies may potentially increase resource requirements in the authentication process. This paper proposes a multimodal authentication process incorporating voice and facial recognition, with liveness detection applied to voice data using speech recognition. This paper introduces Normalized Longest Word Subsequence (NLWS), a combination of Intersection Over Union (IOU) and the longest common subsequence, to compare the prompted system sentence with the user's spoken sentence at speech recognition. Unlike the Word Error Rate (WER), NLWS has a measurable range between 1 and 0. Furthermore, the paper introduces decision-level fusion in the multimodal approach, employing two threshold levels in voice authentication. This approach aims to reduce resource requirements while enhancing the overall security of the authentication process. This paper uses cosine similarity, Euclidean distance, random forest, and extreme gradient boosting (XGBoost) to measure distance or similarity. The results show that the proposed method has better accuracy compared to unimodal approaches, achieving accuracies of 98.44%, 98.83%, 97.46%, and 99.22% using cosine similarity, Euclidean distance, random forest, and XGBoost calculations. The proposed method also demonstrates resource savings, reducing from 5.19 MB to 0.792 MB, from 7.3294 MB to 1.9437 MB, from 6.6512 MB to 1.3284 MB, and from 7.8632 MB to 2.1517 MB in different distance or similarity measurements

ResNet152: A Deep Learning Approach for Robust Spoof Detection in Speaker Verification Systems

In human life, we know that sound is the most important factor. From the normal perspective to the intelligent perspective, sound develops automated systems for various fields for several purposes. However, within contemporary conventional systems, there is significant abuse leading to the proliferation of forgery and other crimes, with sound often playing a central role. With the help of the latest technology such as deep learning, there comes a vast possibility of integrating with many systems for boosting the efficiency of existing systems. So, in this paper, we bring an effective classification of audio using ResNet152. The audio signals are converted to spectrogram images and are passed to a classifier for generating binary classification such as genuine or spoof. We also evaluated our model with existing methods such as VGG16, CNN, VGG19, and AlexNet under performance measures such as Accuracy, EER, and t-DCF in which the proposed model outperforms with 92.2% testing accuracy and 82.2% inference accuracy.

Learning incremental audio–visual representation for continual multimodal understanding

Deep learning methods have demonstrated remarkable success in processing static datasets for various video tasks. However, when confronted with continuous data streams, these approaches often encounter the challenge of catastrophic forgetting. This phenomenon leads to a significant decline in overall performance when learning new classes incrementally. Moreover, existing methods tend to overlook the correlation between audio and visual modalities in video incremental learning, despite their joint significance in scene comprehension. How to continuously learn from new classes while maintaining the knowledge of old videos with limited storage and computing resources is becoming imperative in the field of multimodal learning. In this paper, we introduce CavRL, a pioneering benchmark for audio–visual representation learning under class incremental scenarios. To mitigate catastrophic forgetting, we propose a rehearsal-based training approach that leverages a small exemplar set from previous classes. Our approach constrains the memory buffer within strict storage limits, optimizing exemplar selection by learning correlative audio–visual representations. Additionally, we employ a distillation method to mitigate forgetting in a self-supervised manner. Evaluations on two prevalent multimodal tasks: audio–visual event classification and audio–visual speaker recognition, which demonstrate that CavRL outperforms existing state-of-the-art incremental learning methods across various settings. We anticipate that CavRL will significantly advance research in continual multimodal learning.

Foreign Accent and Forensic Speaker Identification in Voice Lineups: The Influence of Acoustic Features Based on Prosody.

This research aims to examine both the prosodic-acoustic features and the perceptual correlates of foreign-accented English and foreign-accented Brazilian Portuguese and check how the speakers' productions of foreign and native accents are correlated to the listeners' perception. In the Methodology, we conducted a speech production procedure with a group of American speakers of L2 Brazilian Portuguese and a group of Brazilian speakers of L2 English, and a speech perception procedure in which we performed voice lineups for both languages.For the speech production statistical analysis, we ran Generalized Additive Models to evaluate the effect of the language groups on each class (metric or prosodic-acoustic) of features controlled for the smoothing effect of the covariate(s) of the opposite class. For the speech perception statistical analysis, we ran a Kruskal-Wallis test and a post-hoc Dunn's test to evaluate the effect of the voices of the lineups on the scores judged by the listeners. We nevertheless conducted acoustic (voice) similarity tests based on Cosine and Euclidean distances. Results showed significant acoustic differences between the language groups in terms of variability of the f0, duration, and voice quality. For the lineups, the results indicated that prosodic features of f0, intensity, and voice quality correlated to the listeners' perceived judgments.

Contrastive Speaker Representation Learning with Hard Negative Sampling for Speaker Recognition.

Speaker recognition is a technology that identifies the speaker in an input utterance by extracting speaker-distinguishable features from the speech signal. Speaker recognition is used for system security and authentication; therefore, it is crucial to extract unique features of the speaker to achieve high recognition rates. Representative methods for extracting these features include a classification approach, or utilizing contrastive learning to learn the speaker relationship between representations and then using embeddings extracted from a specific layer of the model. This paper introduces a framework for developing robust speaker recognition models through contrastive learning. This approach aims to minimize the similarity to hard negative samples-those that are genuine negatives, but have extremely similar features to the positives, leading to potential mistaken. Specifically, our proposed method trains the model by estimating hard negative samples within a mini-batch during contrastive learning, and then utilizes a cross-attention mechanism to determine speaker agreement for pairs of utterances. To demonstrate the effectiveness of our proposed method, we compared the performance of a deep learning model trained with a conventional loss function utilized in speaker recognition with that of a deep learning model trained using our proposed method, as measured by the equal error rate (EER), an objective performance metric. Our results indicate that when trained with the voxceleb2 dataset, the proposed method achieved an EER of 0.98% on the voxceleb1-E dataset and 1.84% on the voxceleb1-H dataset.

ViT-LSTM synergy: a multi-feature approach for speaker identification and mask detection

ViT-LSTM synergy: a multi-feature approach for speaker identification and mask detection