Short Utterances Research Articles

Toward comprehensive short utterances manipulations detection in videos

AbstractIn a landscape increasingly populated by convincing yet deceptive multimedia content generated through generative adversarial networks, there exists a significant challenge for both human interpretation and machine learning algorithms. This study introduces a shallow learning technique specifically tailored for analyzing visual and auditory components in videos, targeting the lower face region. Our method is optimized for ultra-short video segments (200-600 ms) and employs wavelet scattering transforms for audio and discrete cosine transforms for video. Unlike many approaches, our method excels at these short durations and scales efficiently to longer segments. Experimental results demonstrate high accuracy, achieving 96.83% for 600 ms audio segments and 99.87% for whole video sequences on the FakeAVCeleb and DeepfakeTIMIT datasets. This approach is computationally efficient, making it suitable for real-world applications with constrained resources. The paper also explores the unique challenges of detecting deepfakes in ultra-short sequences and proposes a targeted evaluation strategy for these conditions.

"Just to Jog My Memory": An Examination of Forensic Interviewers' Note-taking Behaviors and Perceptions of Notes With Child Witnesses.

In the current study, we surveyed forensic interviewers (N = 137) on their note-taking practices, perceptions of note-taking, and note-taking training. Many forensic interviewers surveyed (81%) reported that they take notes during forensic interviews. Of those, the most common reason for note-taking was to assist with remembering what the interviewee reported during the interview (89%) and to guide the formulation of follow-up questions (87%). Note-taking style was also reported upon, with most respondents indicating that they write down keywords that may be used again in the interview (78%), as well as short utterances or sentences related to the presenting narrative (61%). Finally, the majority (50%) of respondents who take notes reported always taking notes, although 29% reported taking notes most of the time. Of those respondents who reported not taking notes during forensic interviews, the majority listed the reasons as being that it distracts the child from the interview (85%) and causes them to break eye contact with the child (46%). Overall, many respondents endorsed the benefits of note-taking to the interviewing process, whereas a small minority reported some perceived risks or concerns with note-taking during interviews. Perhaps most notably, forensic interviewers, both of whom take notes and those who do not, reported low rates of note-taking training and a desire for more information on note-taking practices within the field. These results underscore the need for further research and best practice guidelines regarding note-taking during forensic interviews.

Analysis of Deep Generative Model Impact on Feature Extraction and Dimension Reduction for Short Utterance Text-Independent Speaker Verification

Speaker verification is a biometric-based method for individual authentication. However, there are still several challenging problems in achieving high performance in short utterance text-independent conditions, maybe for weak speaker-specific features. Recently, deep learning algorithms have been used extensively in speech processing. This manuscript uses a deep belief network (DBN) as a deep generative method for feature extraction in speaker verification systems. This study aims to show the impact of using the proposed method in various challenging issues, including short utterances, text independence, language variation, and large-scale speaker verification. The proposed DBN uses MFCC as input and tries to extract more efficient features. This new representation of speaker information is evaluated in two popular speaker verification systems: GMM-UBM and i-vector-PLDA methods. The results show that, for the i-vector-PLDA system, the proposed feature decreases the EER considerably from 15.24 to 10.97%. In another experiment, DBN is used to reduce feature dimension and achieves significant results in decreasing computational time and increasing system response speed. In a case study, all the evaluations are performed for 1270 speakers of the NIST SRE2008 dataset. We show deep belief networks can be used in state-of-the-art acoustic modeling methods and more challenging datasets.

Short Utterance Speaker Recognition Based on Speech High Frequency Information Compensation and Dynamic Feature Enhancement Methods

This work aims to further compensate for the weaknesses of feature sparsity and insufficient discriminative acoustic features in existing short-duration speaker recognition. To address this issue, we propose the Bark-scaled Gauss and the linear filter bank superposition cepstral coefficients (BGLCC), and the multidimensional central difference (MDCD) acoustic feature extracted method. The Bark-scaled Gauss filter bank focuses on low-frequency information, while linear filtering is uniformly distributed, therefore, the filter superposition can obtain more discriminative and richer acoustic features of short-duration audio signals. In addition, the multi-dimensional central difference method captures better dynamics features of speakers for improving the performance of short utterance speaker verification. Extensive experiments are conducted on short-duration text-independent speaker verification datasets generated from the VoxCeleb, SITW, and NIST SRE corpora, respectively, which contain speech samples of diverse lengths, and different scenarios. The results demonstrate that the proposed method outperforms the existing acoustic feature extraction approach by at least 10% in the test set. The ablation experiments further illustrate that our proposed approaches can achieve substantial improvement over prior methods.

Leveraging BERT to Improve Spoken Language Identification of Code-Switching Speech

Language identification (LID) involves automatically determining the language being spoken in a given segment. In code-switching speech, there are rapid switches between two or more languages within a single conversation. Despite LID attaining high accuracy on medium or long utterances, the performance on short utterances of code-switching speech is currently unsatisfactory. We propose a bidirectional encoder representation from transformers (BERT)-based LID system (BERT-LID) to enhance LID performance, especially for short-duration code-switching speech segments. The original BERT model is expanded by incorporating phonetic posterior grams (PPGs) extracted from a front-end phone recognizer as input. Then it is followed by the deployment of an optimal deep classifier for LID. Our BERT-LID model demonstrates a significant improvement of approximately 6.5% in accuracy for long-segment identification and 19.9% for short-segment identification, thereby demonstrating its effectiveness in code-switching speech LID tasks.

Multi-Fisher and Triple-Domain Feature Enhancement-Based Short Utterance Speaker Verification for IoT Smart Service

Multi-Fisher and Triple-Domain Feature Enhancement-Based Short Utterance Speaker Verification for IoT Smart Service

Fisher ratio-based multi-domain frame-level feature aggregation for short utterance speaker verification

As the durations of the short utterances are small, it is difficult to learn sufficient information to distinguish the person, thus, short utterance speaker recognition is highly challenging. In this paper, we propose a multi-domain frame-level feature joint learning method to aggregate the discriminative information from multiple dimensions and domain, which is different domains of the speech, time-domain, frequency-domain, and spectral-domain, represent distinct physical characteristics and provide different dimension information, the time domain captures information about the temporal aspect of the physical signal, the frequency domain represents the signal strength in different frequency ranges, and the spectral domain reflects the overall information of the speech, then, based on the extracted multi-domain frame-level features, using the Multi-Fisher criterion aggregates feature parameters categorically and match the corresponding Multi-Fisher ratio weights to the feature parameters as a way to achieve effective feature aggregation and to preserve more effective information, termed Firm-Domain. Extensive experiments are carried out on short-duration text-independent speaker verification datasets derived from the VoxCeleb, SITW, and NIST SRE corpora, which contain speech samples of varying lengths and scenarios. The results demonstrate that the proposed method outperforms the state-of-the-art deep learning architectures by at least 13%, respectively, in the test set. The results of the ablation experiments demonstrate that our proposed methods can significantly outperform previous approaches.

The vocal side of empathy: neural correlates of pain perception in spoken complaints.

In the extensive neuroimaging literature on empathy for pain, few studies have investigated how this phenomenon may relate to everyday social situations such as spoken interactions. The present study used functional Magnetic Resonance Imaging (fMRI) to assess how complaints, as vocal expressions of pain, are empathically processed by listeners and how these empathic responses may vary based on speakers' vocal expression and cultural identity. Twenty-four French participants listened to short utterances describing a painful event, which were either produced in a neutral-sounding or complaining voice by both in-group (French) and out-group (French Canadian) speakers. Results suggest that the perception of suffering from a complaining voice increased activity in the emotional voice areas, composed of voice-sensitive temporal regions interacting with prefrontal cortices and the amygdala. The Salience and Theory of Mind networks, associated with affective and cognitive aspects of empathy, also showed prosody-related activity and specifically correlated with behavioral evaluations of suffering by listeners. Complaints produced by in- vs out-group speakers elicited sensorimotor and default mode activity, respectively, suggesting accent-based changes in empathic perspective. These results, while reaffirming the role of key networks in tasks involving empathy, highlight the importance of vocal expression information and social categorization processes when perceiving another's suffering during social interactions.

Experimental studies for improving the performance of children's speaker verification system using short utterances

The task of developing an automatic speaker verification (ASV) system for children's speech is a formidable one due to a number of reasons. The dearth of domain-specific data is one among them. The challenge further intensifies with the introduction of short utterances of speech, a relatively unexplored domain in the case of children's ASV. Voice-based biometric systems suffers miserably when speech data, inadequate both in volume as well as in duration, is used either for enrollment or verification. To circumvent the issue arising due to data scarcity, the work in this paper extensively explores in-domain as well as various out-of-domain data augmentation techniques. A data augmentation approach is proposed that encompasses both in-domain and out-of-domain data augmentation techniques. The in-domain data augmentation approach, incorporates speed perturbation of children's speech. The out-of-domain data used are from adult speakers which are known to have acoustic attributes in stark contrast to child speakers. The acoustic characteristics of the adult speech data in this study are altered on two fronts namely speech waveform modification and feature-level modification, in order to modify the adult acoustic features and render it acoustically similar to children's speech prior to augmentation. While the speech waveform modification involves various signal processing techniques like prosody modification, formant modification and voice-conversion. The feature-level modification on the other hand involves Vocal-tract length normalization technique (VTLN) which explicitly models and compensates for the ill-effects of variations in vocal tract length by linearly warping the frequency axis of speech signals. The proposed data augmentation approach helps not only in increasing the amount of training data but also in effectively capturing the missing target attributes which helps in boosting the verification performance. A relative improvement of 48.01% in equal error rate (EER) with respect to the baseline system is a testimony of it. Furthermore, the conventionally used Mel-frequency cepstral coefficients (MFCC) are known to average out the higher-frequency components. Prior literary works have shown that a significant amount of relevant acoustic information is available in the higher-frequency region of the children's speech. Therefore, effective preservation of higher-frequency contents in children's speech is of paramount importance which must be appropriately tackled for the development of a reliable and robust children's ASV system. In this regard, frame-level concatenation of the MFCC features with the Inverse-Mel-frequency cepstral coefficient (IMFCC) features is undertaken. The feature concatenation of MFCC and IMFCC is carried out with the sole intention of effectively preserving the higher-frequency contents in the children's speech data. The low canonical correlation existing between the MFCC and the IMFCC feature vectors provides the necessary impetus to go with their feature fusion. The feature concatenation approach, when combined with proposed data augmentation, helps in further improvement of the verification performance. The experimental results testify our claims, wherein we have achieved an overall relative reduction of 50.15% for equal error rate.

Speaker Identification in Multiple Languages: Regional, Indonesian, and English with Short Utterance

One of the authentication models that are currently often used is based on biometrics, such as eye retina, fingerprint, and speech recognition. Moreover, textindependent speaker identification is one of the domains of speech recognition that has been widely studied. Short speech duration in the speaker identification process is one of the challenges in the field of speaker recognition. Accuracy is a great issue when speech duration shorter, besides identification system has to be general enough to process various languages with different dialects which have their own characteristic based on tribe and region. Therefore, the author of this study introduces the speaker identification system in multi languages that comprise of regional, Indonesian, and English with short utterance. Researchers used MFCC technique to extract voice features and CNN as the classification model. There are two kinds of dataset used, open dataset for regional and English language, and own dataset for Indonesian. Own dataset used is a voice recording of 18 persons of different gender who each read the text in several paragraphs of sentences in Indonesian. Whereas public dataset of regional language used consisted of 80 speakers, 41 Sundanese and 39 Javanese. As for English dataset, 126 male speakers and 125 female speakers were taken from LibriSpeech. Tests are carried out separately with variety of language and speech duration, about 3 seconds in English and regional languages, 1 and 3 seconds in Indonesian. The result, best accuracy obtained by each dataset is 95% (regional dataset), 94% (English dataset), and 98% (private dataset).

Application of Reading Skills to Children with Special Needs through NaraMedia at SMPN 8 Tamban

This study aims to describe the application of accompanying teachers in developing reading skills for children with needs special at SMPN 8 Tamban. This qualitative research study uses guidelines for interviews, observation, and documentation. The subject of the study is two students in Class VII. The data collection techniques are interviews, observation, tests, and documentation. The research results show that the teacher's efforts in developing reading skills with a manner special and guided through NaraMedia. The application of reading skills to children with special needs as a whole by teachers has used simple or short utterances (consisting of one or two words). The teacher's approach to the three children slowly made them know, understand, and understand when reading.

CAST: Context-association architecture with simulated long-utterance training for mandarin speech recognition

End-to-end (E2E) models are widely used because they significantly improve the performance of automatic speech recognition (ASR). However, based on the limitations of existing hardware computing devices, previous studies mainly focus on short utterances. Typically, utterances used for ASR training do not last much longer than 15 s, and therefore the models often fail to generalize to longer utterances at inference time. To address the challenge of long-form speech recognition, we propose a novel Context-Association Architecture with Simulated Long-utterance Training (CAST), which consists of a Context-Association RNN-Transducer (CARNN-T) and a simulating long utterance training (SLUT) strategy. The CARNN-T obtains the sentence-level contextual information by paying attention to the cross-sentence historical utterances and adds it in the inference stage, which improves the robustness of long-form speech recognition. The SLUT strategy simulates long-form audio training by updating the recursive state, which can alleviate the length mismatch between training and testing utterances. Experiments on the test of the Aishell-1 and aidatatang_200zh synthetic corpora show that our model has the best recognition performer on long utterances with the character error rate (CER) of 12.0%/12.6%, respectively.

Speech With Pauses Sounds Deceptive to Listeners With and Without Hearing Impairment.

Communication is as much persuasion as it is the transfer of information. This creates a tension between the interests of the speaker and those of the listener, as dishonest speakers naturally attempt to hide deceptive speech and listeners are faced with the challenge of sorting truths from lies. Listeners with hearing impairment in particular may have differing levels of access to the acoustical cues that give away deceptive speech. A greater tendency toward speech pauses has been hypothesized to result from the cognitive demands of lying convincingly. Higher vocal pitch has also been hypothesized to mark the increased anxiety of a dishonest speaker. Listeners with or without hearing impairments heard short utterances from natural conversations, some of which had been digitally manipulated to contain either increased pausing or raised vocal pitch. Listeners were asked to guess whether each statement was a lie in a two-alternative forced-choice task. Participants were also asked explicitly which cues they believed had influenced their decisions. Statements were more likely to be perceived as a lie when they contained pauses, but not when vocal pitch was raised. This pattern held regardless of hearing ability. In contrast, both groups of listeners self-reported using vocal pitch cues to identify deceptive statements, though at lower rates than pauses. Listeners may have only partial awareness of the cues that influence their impression of dishonesty. Listeners with hearing impairment may place greater weight on acoustical cues according to the differing degrees of access provided by hearing aids. https://doi.org/10.23641/asha.24052446.

On Training Targets and Activation Functions for Deep Representation Learning in Text-Dependent Speaker Verification

Deep representation learning has gained significant momentum in advancing text-dependent speaker verification (TD-SV) systems. When designing deep neural networks (DNN) for extracting bottleneck (BN) features, the key considerations include training targets, activation functions, and loss functions. In this paper, we systematically study the impact of these choices on the performance of TD-SV. For training targets, we consider speaker identity, time-contrastive learning (TCL), and auto-regressive prediction coding, with the first being supervised and the last two being self-supervised. Furthermore, we study a range of loss functions when speaker identity is used as the training target. With regard to activation functions, we study the widely used sigmoid function, rectified linear unit (ReLU), and Gaussian error linear unit (GELU). We experimentally show that GELU is able to reduce the error rates of TD-SV significantly compared to sigmoid, irrespective of the training target. Among the three training targets, TCL performs the best. Among the various loss functions, cross-entropy, joint-softmax, and focal loss functions outperform the others. Finally, the score-level fusion of different systems is also able to reduce the error rates. To evaluate the representation learning methods, experiments are conducted on the RedDots 2016 challenge database consisting of short utterances for TD-SV systems based on classic Gaussian mixture model-universal background model (GMM-UBM) and i-vector methods.

Joint filter combination-based central difference feature extraction and attention-enhanced Dense-Res2Block network for short-utterance speaker recognition

Speech authentication in smart services typically involves short utterances. However, due to the short duration of these utterances (e.g. less than 3 s) and the limited enrolment and/or test data available, it is difficult to learn enough information to accurately distinguish the person. As a result, speaker recognition from short utterances is very challenging. In this paper, in the acoustic end. we propose a novel Bark-scaled Gaussian and linear filter bank cepstral coefficients (BGLCC) and multi-dimensional central difference (MDCD) acoustic features extraction method. Also, to enhance the discriminative embedding, in the network end, a novel attention-enhanced Dense-Res2Block network is proposed. First, the rich low-frequency information is extracted based on the high distribution density of the Bark-scaled Gaussian filter bank in the low-frequency domain, and more high-frequency information is extracted based on the linear filter bank uniformly distributed in the high-frequency domain. This means that the combined filters can produce more discriminative and richer acoustic features from short-duration audio signals. In addition, the multi-dimensional central difference method captures better speaker dynamic features in the relative BGLCC domain to improve the performance of short utterance speaker recognition. Finally, an attention-enhanced Dense-Res2Block architecture can obtain a variety of feature expressions of different scale combinations to enhance features. Extensive analysis of a variety of datasets, which speech samples of different types, diverse lengths, etc., demonstrate the superiority of the proposed feature extraction method and model over existing acoustic feature extraction methods and speaker recognition models, including those based on MFCCs, LPCCs, and fusion features, and X-vector-PLDA, BLSTM-ResNet, ResNet34-SP, ECAPA-TDNN respectively. The experimental results show that the proposed method achieves the best performance compared to the existing approach.

Effective preservation of higher-frequency contents in the context of short utterance based children’s speaker verification system

Developing an automatic speaker verification (ASV) system for children is extremely challenging due to the unavailability of children’s speech corpora. The challenges are further exacerbated in the case of short utterances. Voice-based biometric systems require adequate amount of speech data for enrollment and verification; otherwise the performance considerably degrades. In this paper, we have focussed on data paucity and preserving the higher-frequency contents in order to enhance the performance of a short-utterance based children’s speaker verification system. To deal with data scarcity, several out-of-domain data augmentation techniques have been utilized. Since the out-of-domain data used is from adult speakers which are acoustically very different from children’s speech, we have resorted to techniques like prosody modification, formant modification and voice-conversion in order to render it acoustically similar to children’s speech prior to augmentation. This helps in not only increasing the amount of training data but also in effectively capturing the missing target attributes. A relative improvement of 33.6% in equal error rate (EER) with respect to the baseline system trained solely on child data-set is achieved when the proposed data augmentation technique is applied. Further to that, for the preservation of the higher-frequency contents, we have resorted to concatenation of the classical Mel-frequency cepstral coefficients (MFCC) features with the linear-frequency cepstral coefficient (LFCC) or with the inverse-Mel-frequency cepstral coefficient (IMFCC) features. The use of Mel-filter-bank leads to poor resolution of higher-frequency components. On the other hand, linear- or inverse-Mel-filter-banks yield better resolution of higher-frequency components. Moreover, MFCC and IMFCC features exhibit low canonical correlation. Consequently, the frame-level concatenation of MFCC and LFCC or IMFCC features leads to better resolution of both lower- as well as higher-frequency components. Therefore, the EER considerably reduces when either LFCC features or IMFCC features are concatenated with MFCC features. The EER for the full test set shows a relative reduction of 10.56% (with respect to the EER for the MFCC features) when IMFCC features are concatenated with the MFCC features. This novel approach of incorporating data augmentation followed by frame-level feature concatenation helps in achieving an overall reduction of 40.6% in EER.

Sinc‐attention feature extraction for trivial‐event based speaker verification

AbstractHuman speech contains trivial events that are non‐subjectively controlled and are limited by physical characteristics. These events are represented by a series of extremely short utterances, such as the ‘hmm’ phrase used for expressing doubt or confirmation. In speaker verification (SV), such events can be a robust choice to verify the real speaker from disguised speech since they are less affected by the randomness of pronunciation. However, trivial events like ‘hmm’ contain little linguistic information and are extremely short, thus the performance of SV systems will decrease drastically. In this letter, a Sinc‐Attention feature extraction method is proposed to extract more discriminative speech features from speech signals to achieve a robust SV system for trivial events. Learnable filters are utilized to obtain low‐level representations of the speech for SV. Moreover, a novel adaptive weighting of features inspired by attention mechanisms is proposed, which effectively improves the representation capability of speaker features. The experiments on different models prove that our method helps SV systems achieve a lower equal error rate (EER) than hand‐crafted feature‐based systems.

Few-shot short utterance speaker verification using meta-learning.

Short utterance speaker verification (SV) in the actual application is the task of accepting or rejecting the identity claim of a speaker based on a few enrollment utterances. Traditional methods have used deep neural networks to extract speaker representations for verification. Recently, several meta-learning approaches have learned a deep distance metric to distinguish speakers within meta-tasks. Among them, a prototypical network learns a metric space that may be used to compute the distance to the prototype center of speakers, in order to classify speaker identity. We use emphasized channel attention, propagation and aggregation in TDNN (ECAPA-TDNN) to implement the necessary function for the prototypical network, which is a nonlinear mapping from the input space to the metric space for either few-shot SV task. In addition, optimizing only for speakers in given meta-tasks cannot be sufficient to learn distinctive speaker features. Thus, we used an episodic training strategy, in which the classes of the support and query sets correspond to the classes of the entire training set, further improving the model performance. The proposed model outperforms comparison models on the VoxCeleb1 dataset and has a wide range of practical applications.

A deep learning approach for text-independent speaker recognition with short utterances

A deep learning approach for text-independent speaker recognition with short utterances

ASTT: acoustic spatial-temporal transformer for short utterance speaker recognition

ASTT: acoustic spatial-temporal transformer for short utterance speaker recognition