TIMIT Corpus Research Articles

Recurrent neural network and long short-term memory models for audio copy-move forgery detection: a comprehensive study

One of the most pressing challenges in audio forgery detection—a major topic of signal analysis and digital forensics research—is detecting copy-move forgery in audio data. Because audio data are used in numerous sectors, including security, but increasingly tampered with and manipulated, studies dedicated to detecting forgery and verifying voice data have intensified in recent years. In our study, 2189 fake audio files were produced from 2189 audio recordings on the TIMIT corpus, for a total of 4378 audio files. After the 4378 files were preprocessed to detect silent and unsilent regions in the signals, a Mel-frequency-based hybrid feature data set was obtained from the 4378 files. Next, RNN and LSTM deep learning models were applied to detect audio forgery in the data set in four experimental setups—two with RNN and two with LSTM—using the AdaGrad and AdaDelta optimizer algorithms to identify the optimum solution in the unlinear systems and minimize the loss rate. When the experimental results were compared, the accuracy rate of detecting forgery in the hybrid feature data was 76.03%, and the hybrid model, in which the features are used together, demonstrated high accuracy even with small batch sizes. This article thus reports the first-ever use of RNN and LSTM deep learning models to detect audio copy-move forgery. Moreover, because the proposed method does not require adjusting threshold values, the resulting system is more robust than other systems described in the literature.

Using auditory texture statistics for domain-neutral removal of background sounds

IntroductionHuman communication often occurs under adverse acoustical conditions, where speech signals mix with interfering background noise. A substantial fraction of interfering noise can be characterized by a limited set of statistics and has been referred to as auditory textures. Recent research in neuroscience has demonstrated that humans and animals utilize these statistics for recognizing, classifying, and suppressing textural sounds.MethodsHere, we propose a fast, domain-free noise suppression method exploiting the stationarity and spectral similarity of sound sources that make up sound textures, termed Statistical Sound Filtering (SSF). SSF represents a library of spectrotemporal features of the background noise and then compares this against instants in speech-noise-mixtures to subtract contributions that are statistically consistent with the interfering noise.ResultsWe evaluated the performance of SSF using multiple quality measures and human listeners on the standard TIMIT corpus of speech utterances. SSF improved the sound quality across all performance metrics, capturing different aspects of the sound. Additionally, human participants reported reduced background noise levels as a result of filtering, without any significant damage to speech quality. SSF executes rapidly (~100× real-time) and can be retrained rapidly and continuously in changing acoustic contexts.DiscussionSSF is able to exploit unique aspects of textural noise and therefore, can be integrated into hearing aids where power-efficient, fast, and adaptive training and execution are critical.

Design of a robust MVDR beamforming method with Low-Latency by reconstructing covariance matrix for speech enhancement

Aiming at solving the problems of the conventional minimum variance distortionless response (MVDR) beamformer in practical applications, such as the sensibility of the steering vector mismatch and beampattern distortion, a robust broadband MVDR beamforming method with low-latency by reconstructing covariance matrix is proposed and applied to speech enhancement with a linear microphone array in this paper. In this work, some important steps are optimized, and the main contribution is to consider the problem of correlation terms generated by the low latency. Firstly, the direction of arrival (DOA) is corrected and the steering vector is estimated based on the sparsity of the DOAs corresponding to the sound sources, which improves the ability of anti-mismatches in the steering vector. Secondly, the correlation terms between the sound sources and noise are estimated and eliminated by the Capon power within the eigen-subspace, and the indirect dominant method is used to eliminate the correlation terms between the sound sources, so that the covariance matrix is reconstructed to obtain a more robust MVDR beamformer. Thirdly, the problem of white noise amplification at low frequency bins is analyzed, and a white noise gain (WNG) modification method is proposed to obtain a compromise between the interference suppression and WNG. In the experiments, the TIMIT corpus is used to generate the multi-channel speech data set, and the performance of the proposed method is evaluated with different DOAs and input signal to interference plus noise ratios (SINRs). The experimental results show that the proposed method can effectively suppress the interferences and reduce the noise with strong robustness.

Using Hybrid Penalty and Gated Linear Units to Improve Wasserstein Generative Adversarial Networks for Single-Channel Speech Enhancement

Recently, speech enhancement methods based on Generative Adversarial Networks have achieved good performance in time-domain noisy signals. However, the training of Generative Adversarial Networks has such problems as convergence difficulty, model collapse, etc. In this work, an end-to-end speech enhancement model based on Wasserstein Generative Adversarial Networks is proposed, and some improvements have been made in order to get faster convergence speed and better generated speech quality. Specifically, in the generator coding part, each convolution layer adopts different convolution kernel sizes to conduct convolution operations for obtaining speech coding information from multiple scales; a gated linear unit is introduced to alleviate the vanishing gradient problem with the increase of network depth; the gradient penalty of the discriminator is replaced with spectral normalization to accelerate the convergence rate of the model; a hybrid penalty term composed of L1 regularization and a scale-invariant signal-to-distortion ratio is introduced into the loss function of the generator to improve the quality of generated speech. The experimental results on both TIMIT corpus and Tibetan corpus show that the proposed model improves the speech quality significantly and accelerates the convergence speed of the model.

Detection of audio copy-move-forgery with novel feature matching on Mel spectrogram

Audio copy-move-forgery created by copying one or more segments of an audio file and pasting it in a different position within the same audio is one of the most widely used methods in the field of audio forensics. This type of forgery is easy to apply but difficult to detect in the case of post-processing operations applied to forged speech to hide traces of forgeries. This paper proposes a robust method for the detection and localization of the audio copy-move forgery using a keypoint-based approach to the Mel spectrogram representation of audio. In the proposed method, first, the Mel spectrogram image is created from the input audio. Then, SIFT keypoints are obtained from each RGB color channel of this image. The obtained keypoints from each channel are matched via feature vectors to reveal the clues of the forgery regions, and the image sub-blocks whose keypoints are determined to be the center are labeled as forged blocks. Then the blocks in the neighborhood of the forged blocks are investigated whether forged or not. The proposed post-processing stage completes the determination of the forged regions. This stage eliminates the possible false positives and marks the forged areas in the spectrogram image. The forged segments are marked in the audio file by utilizing the positions of the forged regions in the spectrogram image. Experimental studies are carried out on two pitch-based datasets, using TIMIT and Arabic Speech Corpus. The paper presents the detailed performance results of popular referenced studies on these datasets. The performance results prove that the proposed method is more robust against common post-processing operations such as noise addition, filtering operation, and especially compression operation.

Robust voice activity detection based on weighted average of long-term quadratic Renyi and differential entropies

We propose two novel voice activity detection (VAD) algorithms by employing i) the Rényi entropy estimate, and ii) a weighted combination of Rényi and differential entropies, at each frequency bin of power spectral estimates, by considering overlapping frames in the received signal samples. The Bartlett-Welch method is employed to estimate the power spectrum from a suitably chosen number of past frames in the frequency domain. Exploiting the known fact that a long-term averaging enhances the performance of VAD, we first propose a novel frequency domain long-term quadratic Rényi entropy (FLQRE) feature, which is extracted by summing the estimated Rényi entropy values across the frequency bins over each signal frame. Later, we propose a weighted combination of the frequency domain long-term differential entropy (FLDE) and FLQRE feature for VAD, termed as a weighted quadratic and differential entropy (WQDE) feature. We evaluate the performance of the proposed FLQRE and WQDE VAD schemes, by considering twelve types of noises from NOISEX-92 at five different SNR values, artificially added to speech samples from SWITCHBOARD and TIMIT corpora. We present an extensive performance comparison study to establish the utility of our proposed VAD algorithms over state-of-the-art short- and long-term VAD techniques such as ITU-T G.729B, likelihood ratio test, long-term signal variability, and long-term spectral flatness measure-based algorithms. We show that the proposed algorithms yield the best average accuracy and noise-hit rate under the SWITCHBOARD corpus, and yield a comparable performance under the TIMIT corpus.

Temporal and spectral characteristics of conversational versus read fricatives in American English.

The present study compares the production of fricatives in conversational versus read speech in American English. The goal is to examine which parameters contribute to the identification of fricatives across the two speech styles. The study surveys over 162 000 fricative tokens from the Buckeye Corpus [Pitt, Johnson, Hume, Kiesling, and Raymond (2005). Speech Commun. 45, 89-95] and the TIMIT Corpus [Zue and Seneff (1996). Recent Research towards Advanced Man-Machine Interface through Spoken Language (Elsevier, Amsterdam, the Netherlands), pp. 515-525]. A total of 18 different temporal and spectral measures are tested, including segment duration, preceding and following phone duration, spectral moments (at onset, midpoint, and/or offset), spectral peak frequency, etc. Results show that segment duration and midpoint spectral moments make the most prominent contribution to the categorization of fricatives for both speech styles. Spectral measures are more important for conversational speech, whereas duration plays a greater role for read speech. At the same time, the magnitude of the differences across speech styles is often low and many of the observed effects may be attributable to methodological differences across the corpora. Results may indicate that reduction of fricatives in conversational speech is more limited compared to the reduction of other types of speech sounds, such as plosives.

Phonetic Segmentation using a Wavelet-based Speech Cepstral Features and Sparse Representation Classifier

Speech segmentation is the process of dividing speech signal into distinct acoustic blocks that could be words, syllables or phonemes. Phonetic segmentation is about finding the exact boundaries for the different phonemes that composes a specific speech signal. This problem is crucial for many applications, i.e. automatic speech recognition (ASR). In this paper we propose a new model-based text independent phonetic segmentation method based on wavelet packet speech parametrization features and using the sparse representation classifier (SRC). Experiments were performed on two datasets, the first is an English one derived from TIMIT corpus, while the second is an Arabic one derived from the Arabic speech corpus. Results showed that the proposed wavelet packet de composition features outperform the MFCC features in speech segmentation task, in terms of both F1-score and R-measure on both datasets. Results also indicate that the SRC gives higher hit rate than the famous k-Nearest Neighbors (k-NN) classifier on TIMIT dataset.

DESIGN AN AUTOMATIC SPEECH VERIFICATION BASED ON WORD DETECTION

Based on word identification and knowledge-based verification, we offer a new technique to automated speech recognition (ASR). Assuming we know the vocabulary, we begin by creating word detectors for each word in the utterance. Pruning techniques are employed to weed out improbable word candidates. These words are then grouped together to form word strings. To construct a bottom-up, detection-based voice recognition system that incorporates knowledge of acoustics, speech, and language into pruning and rescoring, the suggested strategy differs from the typical maximal a posteriori decoding method. Using phone models learned from the TIMIT corpus, the suggested method was tested on a connected digit task. Even though no digit samples were provided to train the detectors and recognizers, the suggested detection-based framework performed well when compared to current linked digit recognition methods. This detection-based technique can incorporate other knowledge-based limitations, such as the manner and location of articulation detectors, to enhance the overall system's robustness and performance.

Harmonic Differences Method for Robust Fundamental Frequency Detection in Wideband and Narrowband Speech Signals

In this article, a novel pitch determination algorithm based on harmonic differences method (HDM) is proposed. Most of the algorithms today rely on autocorrelation, cepstrum, and lastly convolutional neural networks, and they have some limitations (small datasets, wideband or narrowband, musical sounds, temporal smoothing, etc.), accuracy, and speed problems. There are very rare works exploiting the spacing between the harmonics. HDM is designed for both wideband and exclusively narrowband (telephone) speech and tries to find the most repeating difference between the harmonics of speech signal. We use three vowel databases in our experiments, namely, Hillenbrand Vowel Database, Texas Vowel Database, and Vowels from the TIMIT corpus. We compare HDM with autocorrelation, cepstrum, YIN, YAAPT, CREPE, and FCN algorithms. Results show that harmonic differences are reliable and fast choice for robust pitch detection. Also, it is superior to others in most cases.

Analysis of vowel addition or deletion in Continuous Speech

In order to improve the recognition performance, the articulation of the transcription is very important in the process of training. For continuous speech, the essential characteristics of various speakers are pronunciation variation, over focused or inadequately highlighted words can results the waveform misalignment in the sub word unit margin. Because of the deviation in the articulation leads into misalignment when this is compared with articulation dictionary. So the deletion or insertion of the sub word is necessary. This happens because for each expression, the transcription is not precise. This paper presents the corrections in the transcription at the sub word level utilizing sound prompts that are presented in the waveform. The transcription of a word is fixed Utilizing sentence-level transcriptions with reference to the phonemes that create the word. Specifically, it clarifies that vowels are either deleted or inserted. To help the proposed contention, errors in persistent discourse are validated utilizing machine learning and signal processing tools. A programmed information driven annotator abusing the inductions drawn from the examination is utilized to address transcription errors. The outcomes show that rectified pronunciations lead to higher probability for train expressions in the TIMIT corpus.

Sequential routing framework: Fully capsule network-based speech recognition

Capsule networks (CapsNets) have recently gotten attention as a novel neural architecture. This paper presents the sequential routing framework which we believe is the first method to adapt a CapsNet-only structure to sequence-to-sequence recognition. Input sequences are capsulized then sliced by a window size. Each slice is classified to a label at the corresponding time through iterative routing mechanisms. Afterwards, losses are computed by connectionist temporal classification (CTC). During routing, the required number of parameters can be controlled by the window size regardless of the length of sequences by sharing learnable weights across the slices. We additionally propose a sequential dynamic routing algorithm to replace traditional dynamic routing. The proposed technique can minimize decoding speed degradation caused by the routing iterations since it can operate in a non-iterative manner without dropping accuracy. The method achieves a 1.1% lower word error rate at 16.9% on the Wall Street Journal corpus compared to bidirectional long short-term memory-based CTC networks. On the TIMIT corpus, it attains a 0.7% lower phone error rate at 17.5% compared to convolutional neural network-based CTC networks (Zhang et al., 2016).

CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networks

How can deep neural networks encode information that corresponds to words in human speech into raw acoustic data? This paper proposes two neural network architectures for modeling unsupervised lexical learning from raw acoustic inputs: ciwGAN (Categorical InfoWaveGAN) and fiwGAN (Featural InfoWaveGAN). These combine Deep Convolutional GAN architecture for audio data (WaveGAN; Donahue et al., 2019) with the information theoretic extension of GAN – InfoGAN (Chen et al., 2016) – and propose a new latent space structure that can model featural learning simultaneously with a higher level classification and allows for a very low-dimension vector representation of lexical items. In addition to the Generator and Discriminator networks, the architectures introduce a network that learns to retrieve latent codes from generated audio outputs. Lexical learning is thus modeled as emergent from an architecture that forces a deep neural network to output data such that unique information is retrievable from its acoustic outputs. The networks trained on lexical items from the TIMIT corpus learn to encode unique information corresponding to lexical items in the form of categorical variables in their latent space. By manipulating these variables, the network outputs specific lexical items. The network occasionally outputs innovative lexical items that violate training data, but are linguistically interpretable and highly informative for cognitive modeling and neural network interpretability. Innovative outputs suggest that phonetic and phonological representations learned by the network can be productively recombined and directly paralleled to productivity in human speech: a fiwGAN network trained on suit and dark outputs innovative start, even though it never saw start or even a [st] sequence in the training data. We also argue that setting latent featural codes to values well beyond training range results in almost categorical generation of prototypical lexical items and reveals underlying values of each latent code. Probing deep neural networks trained on well understood dependencies in speech bears implications for latent space interpretability and understanding how deep neural networks learn meaningful representations, as well as potential for unsupervised text-to-speech generation in the GAN framework.

Deep Learning-Based Detection of Articulatory Features in Arabic and English Speech.

This study proposes using object detection techniques to recognize sequences of articulatory features (AFs) from speech utterances by treating AFs of phonemes as multi-label objects in speech spectrogram. The proposed system, called AFD-Obj, recognizes sequence of multi-label AFs in speech signal and localizes them. AFD-Obj consists of two main stages: firstly, we formulate the problem of AFs detection as an object detection problem and prepare the data to fulfill requirement of object detectors by generating a spectral three-channel image from the speech signal and creating the corresponding annotation for each utterance. Secondly, we use annotated images to train the proposed system to detect sequences of AFs and their boundaries. We test the system by feeding spectrogram images to the system, which will recognize and localize multi-label AFs. We investigated using these AFs to detect the utterance phonemes. YOLOv3-tiny detector is selected because of its real-time property and its support for multi-label detection. We test our AFD-Obj system on Arabic and English languages using KAPD and TIMIT corpora, respectively. Additionally, we propose using YOLOv3-tiny as an Arabic phoneme detection system (i.e., PD-Obj) to recognize and localize a sequence of Arabic phonemes from whole speech utterances. The proposed AFD-Obj and PD-Obj systems achieve excellent results for Arabic corpus and comparable to the state-of-the-art method for English corpus. Moreover, we showed that using only one-scale detection is suitable for AFs detection or phoneme recognition.

Robust vowel region detection method for multimode speech

The aim of this paper is to explore a robust method for vowel region detection from multimode speech. In realistic scenario, speech can be classified into three modes namely; conversation, extempore, and read. The existing method detects the vowel form the speech recorded in clean environment which may not be appropriate for the multimode speech tasks. To address this issue, we proposed an approach based on continuous wavelet transform coefficients and phone boundaries for detecting the vowel regions from different modes of the speech signal. For evaluation of the proposed vowel region (VR) detection technique, TIMIT (read speech) and Bengali (read, extempore, and conversation speech) corpora are used. The proposed VR detection technique is compared to the state-of-the-art methods. The experiments has recorded significant gain in the performance of the proposed technique than the state-of-the-art methods. The efficiency of the proposed technique is shown by extracting vocal tract and excitation source features from automatically detected VRs for developing the multilingual speech mode classification (MSMC) model. The evaluation results report that the performance of the MSMC model is significantly improved when features are extracted from the vowel regions than the entire speech utterance.

Towards Constructing HMM Structure for Speech Recognition With Deep Neural Fenonic Baseform Growing

For decades, acoustic models in speech recognition systems pivot on Hidden Markov Models (HMMs), e.g., Gaussian Mixture Model-HMM system, Deep Neural Network-HMM system, etc., and achieve remarkable results. However, the popular HMM model is the three-state left-to-right structure, without the superiority certainty. There are multiple studies on the HMM structure’s optimization, but none of them addresses this problem leveraging deep learning algorithms. For the first time, this paper proposes a new training method based on Deep Neural Fenonic Baseform Growing to optimize the HMM structure, which is concisely designed and computationally cheap. Moreover, this data-driven method customizes the HMM structure for each phone precisely without external assumptions concerning the number of states or transition patterns. Experimental results on both TIMIT and TEDliumv2 corpora indicate that the proposed HMM structure improves both the monophone system and the triphone system substantially. Besides, its adoption further improves state-of-the-art speech recognition systems with remarkably reduced parameters.

IDANet: An Information Distillation and Aggregation Network for Speech Enhancement

Speech enhancement aims to restore clean speech from noisy environments. In recent years, skip connections have shown great promise in improving speech enhancement performance. Although directly transmitting low-level information is helpful for reconstructing the spectrum, the noise components negatively impact denoising results. In this letter, we propose IDANet, an end-to-end framework that incorporates an information distillation and aggregation unit to store fine-grained features and filter out noisy components through continuous distillation and recalibration. In addition, we design a novel decoding block equipped with deformable convolution and dynamic attention mechanism to further improve the capability of the reconstruction unit. Experimental results conducted on TIMIT corpus demonstrate that the proposed IDANet is efficient yet effective, e.g., the parameters of our model against the state-of-the-art model are 0.68 M vs. 1.23 M, and the performance boost on STOI and PESQ is 0.81% and 3.73%.

Speaker Recognition using Random Forest

Speaker identification has become a mainstream technology in the field of machine learning that involves determining the identity of a speaker from his/her speech sample. A person’s speech note contains many features that can be used to discriminate his/her identity. A model that can identify a speaker has wide applications such as biometric authentication, security, forensics and human-machine interaction. This paper implements a speaker identification system based on Random Forest as a classifier to identify the various speakers using MFCC and RPS as feature extraction techniques. The output obtained from the Random Forest classifier shows promising result. It is observed that the accuracy level is significantly higher in MFCC as compared to the RPS technique on the data taken from the well-known TIMIT corpus dataset.

Dual-Transform Source Separation Using Sparse Nonnegative Matrix Factorization

In this article, we propose a new source separation method in which the dual-tree complex wavelet transform (DTCWT) and short-time Fourier transform (STFT) algorithms are used sequentially as dual transforms and sparse nonnegative matrix factorization (SNMF) is used to factorize the magnitude spectrum. STFT-based source separation faces issues related to time and frequency resolution because it cannot exactly determine which frequencies exist at what time. Discrete wavelet transform (DWT)-based source separation faces a time-variation-related problem (i.e., a small shift in the time-domain signal causes significant variation in the energy of the wavelet coefficients). To address these issues, we utilize the DTCWT, which comprises two-level trees with different sets of filters and provides additional information for analysis and approximate shift invariance; these properties enable the perfect reconstruction of the time-domain signal. Thus, the time-domain signal is transformed into a set of subband signals in which low- and high-frequency components are isolated. Next, each subband is passed through the STFT and a complex spectrogram is constructed. Then, SNMF is applied to decompose the magnitude part into a weighted linear combination of the trained basis vectors for both sources. Finally, the estimated signals can be obtained through a subband binary ratio mask by applying the inverse STFT (ISTFT) and the inverse DTCWT (IDTCWT). The proposed method is examined on speech separation tasks utilizing the GRID audiovisual and TIMIT corpora. The experimental findings indicate that the proposed approach outperforms the existing methods.

A landmark-cue-based approach to analyzing the acoustic realizations of American English intervocalic flaps.

This study examines the acoustic realizations of American English intervocalic flaps in the TIMIT corpus, using the landmark-critical feature-cue-based framework. Three different acoustic patterns of flaps are described: (i) both closure and release landmarks present, (ii) only the closure landmark present, and (iii) both landmarks deleted. The patterns occur consistently across several phonological and morphological conditions but vary with sociolinguistic factors, including speaker dialect and gender. This method of analysing speech at the level of acoustic landmarks and other individual cues to distinctive features contributes to a deeper understanding of how speakers and listeners employ systematic variation in phonetic detail in speech processing.