Time-scale Modification Research Articles

Excitation of proton firehose instability in magnetospheric cold and hot proton plasma: a quasilinear approach

Abstract Unstable states of different charged species in the solar wind and Earth’s magnetosphere are governed with the collective and collisional processes. For these dilute plasmas, the contribution of microinstabilities driven by the anisotropic particle distribution and heat flux becomes important in defining the stable/equilibrium states of electrons and ions/protons. The present paper highlights the key role of proton firehose instability to regulate an unchecked rise in the temperature anisotropy in these solar wind and magnetospheric environments. Right-handed circularly polarized proton firehose mode becomes unstable when the temperature condition of T ‖p > T ⊥p is satisfied, where the directional subscripts denote directions with respect to the ambient magnetic field. Based on the observations of magnetospheric multi-scale (MMS) space mission, we assume the bi-Maxwellian nature of the model distribution for the multi-component proton plasma. To study the time evolution of the unstable mode, we further allow the time variation in the cold and hot proton temperatures. For the choice of the initial conditions related with observations, we unveil the wave properties (growth and unstable wave number domain) corresponding to the cold/hot proton temperature anisotropy and the plasma betas of constituents proton components. In the back action of proton firehose instability, we highlight the time-scale modifications and saturation of initial bi-Maxwellian distributions and resulting wave-energy densities for various choices of initial cold-hot temperature anisotropy and plasma betas.

Listening to the Magnetosphere: How Best to Make ULF Waves Audible

Observations across the heliosphere typically rely on in situ spacecraft observations producing time-series data. While often this data is analysed visually, it lends itself more naturally to our sense of sound. The simplest method of converting oscillatory data into audible sound is audification—a one-to-one mapping of data samples to audio samples—which has the benefit that no information is lost, thus is a true representation of the original data. However, audification can make some magnetospheric ULF waves observations pass by too quickly for someone to realistically be able to listen to effectively. For this reason, we detail various existing audio time scale modification techniques developed for music, applying these to ULF wave observations by spacecraft and exploring how they affect the properties of the resulting audio. Through a public dialogue we arrive at recommendations for ULF wave researchers on rendering these waves audible and discuss the scientific and educational possibilities of these new methods.

Audio-lossless robust watermarking against desynchronization attacks

Audio-lossless robust watermarking (ALRW) is a new research issue in the field of information hiding, which can resist those content-preserving manipulations while can restore the cover audio files completely in case of no attacks. Most ALRW countermeasures proposed in the literature usually focus on additive noise-like manipulations such as MP3 compression, low-pass filtering and Gaussian additive noise, but few are resistant to challenging desynchronization attacks such as random cropping and time-scale modification (TSM). The main reason is that the exploited robust features in the existing ALRW algorithms are related to the sample position. In this paper, we propose an ALRW method by efficiently embedding a watermark into the histogram shape of the low-frequency component after the discrete wavelet transform (DWT) and reversibly hiding the distortion due to the robust watermark embedding as the compensation information for restoration of the cover audio file. The use of the histogram shape in the DWT low-frequency component as the exploited robust feature is: 1) mathematically invariant to TSM; 2) robust to random cropping attacks since an audio clip before and after the cropping often has similar distribution, and 3) resistant to interpolation errors during time scaling and those common audio processing operations. To reduce the compensation information, the robust watermarking process is elaborately and luminously designed to represent the difference between the original and the robust watermarked audio. As a result, an ALRW system against desynchronization attacks is achieved with satisfactory performance. Experimental results show that the proposed ALRW scheme can effectively reduce the compensation information and provide strong robustness to those common manipulations including TSM, random cropping, MP3 lossy compression and other noise-like manipulations. In case of no attacks, the cover audio signal can be recovered without any loss.

Voice privacy using CycleGAN and time-scale modification

Extensive use of Intelligent Personal Assistants (IPA) and biometrics in our day-to-day life asks for privacy preservation while dealing with personal data. To that effect, efforts have been made to preserve the personally identifiable characteristics from human voice using different speaker anonymization techniques. In this paper, we propose Cycle Consistent Generative Adversarial Network (CycleGAN) to modify (transform) the speaker’s gender as well as the other prosodic aspects using their Mel cepstral coefficients (MCEPs) and fundamental frequency (i.e., F0). For effective anonymization in the context of voice privacy, we propose two-level (i.e., double) anonymization, where first-level anonymization is done using CycleGAN, followed by second-level anonymization using time-scale modification. The speaker anonymization and intelligibility are measured objectively using the automatic speaker verification (ASV) and automatic speech recognition (ASR) experiments, respectively, on development and test sets of Librispeech and VCTK datasets. For CycleGAN-based anonymization, the average % EERs (% WERs) are 40.3% (8.89%) and 40.95% (9.37%) with original enrollments and anonymized trials of the development and test datasets, respectively. The average % EERs (% WERs) for double anonymization are 46.19% (9.95%) and 44.76% (10.34%) with original enrollments and anonymized trials of the development and test datasets, respectively. For the voice privacy evaluation , the performance of ASV system is much important, when the enrollments and trials both are anonymized (called as A-A case), which is also briefly discussed in this work. The average % EERs for A-A case (test set) are 24.29% and 2.81% using CycleGAN-based anonymization and double anonymization, respectively. Objective evaluation for more advanced attack model (i.e., attacker having anonymized data) is also explored in this study. The performance reflected the robustness of proposed anonymization approach towards voice privacy. The subjective tests using 101 listeners and corresponding analysis of variance (ANOVA) and Tukey–Kramer-based Ad-hoc tests are also carried out in order to quote statistical significance of our results. The subjective test show that the CycleGAN and double anonymization approaches give better naturalness, intelligibility, and speaker dissimilarity than the state-of-the-art x-vector-based baseline system.

Speech Time-Scale Modification With GANs

While listening to spoken content, it is often desired to vary the speech rate while preserving the speaker’s timbre and pitch. To date, advanced signal processing techniques are used to address this task, but it still remains a challenge to maintain a high speech quality at all time-scales. Inspired by the success of speech generation using Generative Adversarial Networks (GANs), we propose a novel unsupervised learning algorithm for time-scale modification (TSM) of speech, called ScalerGAN. The model is trained using a set of speech utterances, where no time-scales are provided. The ScalerGAN algorithm is composed of a generator that gets as input speech with the desired rate and outputs a time-adjusted speech; a discriminator that works on various spectrum scales; and a decoder that converts the time-adjusted signal back to the original rate to maintain consistency. Using an A/B test and conditional A/B test, human listeners were asked to compare ScalerGAN with other state-of-the-art TSM methods. The results showed that the speech quality of ScalerGAN outperforms all other methods.

Using Data Augmentation and Time-Scale Modification to Improve ASR of Children’s Speech in Noisy Environments

Current ASR systems show poor performance in recognition of children’s speech in noisy environments because recognizers are typically trained with clean adults’ speech and therefore there are two mismatches between training and testing phases (i.e., clean speech in training vs. noisy speech in testing and adult speech in training vs. child speech in testing). This article studies methods to tackle the effects of these two mismatches in recognition of noisy children’s speech by investigating two techniques: data augmentation and time-scale modification. In the former, clean training data of adult speakers are corrupted with additive noise in order to obtain training data that better correspond to the noisy testing conditions. In the latter, the fundamental frequency (F0) and speaking rate of children’s speech are modified in the testing phase in order to reduce differences in the prosodic characteristics between the testing data of child speakers and the training data of adult speakers. A standard ASR system based on DNN–HMM was built and the effects of data augmentation, F0 modification, and speaking rate modification on word error rate (WER) were evaluated first separately and then by combining all three techniques. The experiments were conducted using children’s speech corrupted with additive noise of four different noise types in four different signal-to-noise (SNR) categories. The results show that the combination of all three techniques yielded the best ASR performance. As an example, the WER value averaged over all four noise types in the SNR category of 5 dB dropped from 32.30% to 12.09% when the baseline system, in which no data augmentation or time-scale modification were used, was replaced with a recognizer that was built using a combination of all three techniques. In summary, in recognizing noisy children’s speech with ASR systems trained with clean adult speech, considerable improvements in the recognition performance can be achieved by combining data augmentation based on noise addition in the system training phase and time-scale modification based on modifying F0 and speaking rate of children’s speech in the testing phase.

An objective measure of quality for time-scale modification of audio.

Objective evaluation of audio processed with time-scale modification (TSM) remains an open problem. Recently, a dataset of time-scaled audio with subjective quality labels was published and used to create an initial objective measure of quality (OMOQ). In this paper, an improved OMOQ for time-scaled audio is proposed. The measure uses handcrafted features and a fully connected network to predict subjective mean opinion scores (SMOS). Basic and advanced perceptual evaluation of audio quality features are used in addition to nine features specific to TSM artefacts. Six methods of alignment are explored with interpolation of the reference magnitude spectrum to the length of the test magnitude spectrum giving the best performance. The proposed measure achieves a mean root mean square error of 0.490 and a mean Pearson correlation of 0.864 to SMOS, equivalent to the 97th and 82nd percentiles of the subjective sessions, respectively. The proposed measure is used to evaluate TSM algorithms, finding that Elastique gives the highest objective quality for solo instrument and voice signals, whereas the identity phase-locking phase vocoder gives the highest objective quality for music signals and the best overall quality. The objective measure is available online at https://www.github.com/zygurt/TSM.

Robust and Blind Audio Watermarking Algorithm in Dual Domain for Overcoming Synchronization Attacks

How to effectively resist synchronization attacks is the most challenging topic in the research of robust watermarking algorithms. A robust and blind audio watermarking algorithm for overcoming synchronization attacks is proposed in dual domain by considering time domain and transform domain. Based on analysing the characteristics of synchronization attacks, an implicit synchronization mechanism (ISM) is developed in the time domain, which can effectively track the appropriate region for embedding and extracting watermarks. The data in this region will be subjected to discrete cosine transform (DCT) and singular value decomposition (SVD) in turn to obtain the eigenvalue that can be utilized to carry watermarks. In order to extract the watermark blindly, the eigenvalue will be quantized. Genetic algorithm (GA) is utilized to optimize the quantization step to balance both transparency and robustness. The experimental results confirm that the proposed algorithm not only withstands various conventional signal processing operations but also resists malicious synchronization attacks, such as time scale modification (TSM), pitch-shifting modification (PSM), jittering, and random cropping. Especially, it can overcome TSM with strength from −30% to +30%, which is much higher than the standard of the International Federation of the Phonographic Industry (IFPI) and far superior to the other algorithms in related papers.

A time-scale modification dataset with subjective quality labels.

Time Scale Modification (TSM) is a well-researched field; however, no effective objective measure of quality exists. This paper details the creation, subjective evaluation, and analysis of a dataset for use in the development of an objective measure of quality for TSM. Comprised of two parts, the training component contains 88 source files processed using six TSM methods at 10 time scales, while the testing component contains 20 source files processed using three additional methods at four time scales. The source material contains speech, solo harmonic and percussive instruments, sound effects, and a range of music genres. Ratings (42 529) were collected from 633 sessions using laboratory and remote collection methods. Analysis of results shows no correlation between age and quality of rating; expert and non-expert listeners to be equivalent; minor differences between participants with and without hearing issues; and minimal differences between testing modalities. A comparison of published objective measures and subjective scores shows the objective measures to be poor indicators of subjective quality. Initial results for a retrained objective measure of quality are presented with results approaching average root mean squared error loss and Pearson correlation values of subjective sessions. The labeled dataset is available at http://ieee-dataport.org/1987.

RFPS: A Robust Feature Points Detection of Audio Watermarking for Against Desynchronization Attacks in Cyber Security

Digital watermarking is an effective covert communication technique in the military field, which can solve the problems in traditional encryption. In general, the Spread Spectrum (SS) algorithm can well satisfy the requirements of concealment and anti-interference in the covert communication process. However, it has a natural defect in resisting desynchronization attacks. In this paper, we propose a Robust Feature Points Scheme (RFPS) against desynchronization attacks by calculating the maximum response value of the second-order derivative of the original audio signal. The maximum response value will be used as a feature point and the audio segments centering at the detected feature points are extracted for both watermark embedding and extraction process. This study mainly focuses on cropping attacks and pitch invariant Time-Scale Modification (TSM) attacks. The experimental results show that under non-uniform cropping attacks, the proposed method can extract the watermark information without any bit error rate. Even in the case of the pitch-invariant TSM attacks, the proposed scheme has great improvement compared with the existing works.

Haptic Exploration During Fast Video Playback: Vibrotactile Support for Event Search in Robot Operation Videos

Fast playback allows quick video exploration, but it also decreases the saliency of short events. We propose a haptic exploration for detection of short events during fast video playback, considering that event-related information in vibrotactile feedback can be preserved during fast playback using Time Scale Modification (TSM) methods developed for audio. We evaluate our proposal in two collision detection experiments using first-person view (FPV) videos. In the first experiment, viewers watched at a fixed playback speed, i.e., 1× or 2×, videos recorded with a camera mounted on a platform cart. In this experiment, event-related vibrations were measured at the back of the camera. In the second experiment, viewers used a media controller to adjust the playback speed in videos simulating an exploration with a mobile robot. In this experiment, event-related vibrations were generated from the measurements used in the first experiment. We show that a haptic exploration improves collision awareness under either constant or adjustable playback speed. In both experiments, the number of collisions reported without vibrotactile feedback deviated the greatest from the actual number of collisions in a video. Moreover, collision detection performance with vibrations time-scaled without Time Scale Modification (TSM) methods was not significantly different from performance without vibrations.

Significance of Pitch-Based Spectral Normalization for Children's Speech Recognition

It is well known from the literature that due to several acoustic mismatches, the recognition performances of children's speech using adult-trained-acoustic models get deteriorated. The differences in pitch and speaking rate are the two major factors that cause the acoustic mismatch between two groups of speakers. This work proposes to incorporate pitch information into an automatic speech recognition (ASR) system by exploiting the correlation between pitch and formants. By using the pitch-based spectrum normalization module in the front-end feature extraction process, the performance of mismatch ASR system is improved for children of different age groups. Further, fuzzy-based time scale modification is applied to study the effect of speaking-rate normalization on the proposed feature. The proposed feature results in relative improvement of $\text{30}{\%}$ and $\text{33}{\%}$ on DLSTM-based ASR system over the MFCC baseline without and with speaking-rate normalization, respectively.

Audio watermarking algorithm against synchronization attacks using global characteristics and adaptive frame division

Achieving resistance to synchronization attacks on audio watermarking is a challenging task. Available synchronization methods based on local features utilize sample alignments. The local features vanish easily after operation, hindering resynchronization. This study proposes a new audio watermarking method that divides frames adaptively; it is based on the global characteristics of the frame and achieves indirect synchronization by employing a corresponding frame sequence number. The experimental results reveal that the proposed scheme exhibits effective robustness against conventional signal processing manipulation and synchronization attack processes such as jittering, time scale modification, and pitch shift modification.

Short Utterance Based Speech Language Identification in Intelligent Vehicles With Time-Scale Modifications and Deep Bottleneck Features

Conversations in the intelligent vehicles are usually short utterance. As the durations of the short utterances are small (e.g., less than 3 s), it is difficult to learn sufficient information to distinguish the type of languages. In this paper, we propose an end-to-end short utterances based speech language identification (SLI) approach, which is especially suitable for the short utterance based language identification. This approach is implemented with a long short-term memory (LSTM) neural network, which is designed for the SLI application in intelligent vehicles. The features used for LSTM learning are generated by a transfer learning method. The bottleneck features of a deep neural network, which are obtained for a mandarin acoustic-phonetic classifier, are used for the LSTM training. In order to improve the SLD accuracy with short utterances, a phase vocoder based time-scale modification method is utilized to reduce/increase the speech rate of the test utterance. By connecting the normal, speech rate reduced, and speech rate increased utterances, we can extend the length of the test utterances such that the performance of the SLI system is improved. The experimental results on the AP17-OLR database demonstrate that the proposed method can improve the performance of SLD, especially on short utterance. The proposed SLI has robust performance under the vehicular noisy environment.

An Experimental Study on the Significance of Variable Frame-Length and Overlap in the Context of Children’s Speech Recognition

It is well known that the recognition performance of an automatic speech recognition (ASR) system is affected by intra-speaker as well inter-speaker variability. The differences in the geometry of vocal organs, pitch and speaking-rate among the speakers are some such inter-speaker variabilities affecting the recognition performance. A mismatch between the training and test data with respect to any of those aforementioned factors leads to increased error rates. An example of acoustically mismatched ASR is the task of transcribing children’s speech on adult data-trained system. A large number of studies have been reported earlier that present a myriad of techniques for addressing acoustic mismatch arising from differences in pitch and dimensions of vocal organs. At the same time, only a few works on speaking-rate adaptation employing timescale modification have been reported. Furthermore, those studies were performed on ASR systems developed using Gaussian mixture models. Motivated by these facts, speaking-rate adaptation is explored in this work in the context of children’s ASR system employing deep neural network-based acoustic modeling. Speaking-rate adaptation is performed by changing the frame-length and overlap during front-end feature extraction process. Significant reductions in errors are noted by speaking-rate adaptation. In addition to that, we have also studied the effect of combining speaking-rate adaptation with vocal-tract length normalization and explicit pitch modification. In both the cases, additive improvements are obtained. To summarize, relative improvements in 15–20% over the baselines are obtained by varying the frame-length and frame-overlap.

Constraint-Based Time-Scale Modification of Music Recordings for Noise Beautification

In magnetic resonance imaging (MRI), a patient is exposed to beat-like knocking sounds, often interrupted by periods of silence, which are caused by pulsing currents of the MRI scanner. In order to increase the patient’s comfort, one strategy is to play back ambient music to induce positive emotions and to reduce stress during the MRI scanning process. To create an overall acceptable acoustic environment, one idea is to adapt the music to the locally periodic acoustic MRI noise. Motivated by this scenario, we consider in this paper the general problem of adapting a given music recording to fulfill certain temporal constraints. More concretely, the constraints are given by a reference time axis with specified time points (e.g., the time positions of the MRI scanner’s knocking sounds). Then, the goal is to temporally modify a suitable music recording such that its beat positions align with the specified time points. As one technical contribution, we model this alignment task as an optimization problem with the objective to fulfill the constraints while avoiding strong local distortions in the music. Furthermore, we introduce an efficient algorithm based on dynamic programming for solving this task. Based on the computed alignment, we use existing time-scale modification procedures for locally adapting the music recording. To illustrate the outcome of our procedure, we discuss representative synthetic and real-world examples, which can be accessed via an interactive website. In particular, these examples indicate the potential of automated methods for noise beautification within the MRI application scenario.

Explicit Pitch Mapping for Improved Children’s Speech Recognition

Recognizing children’s speech on automatic speech recognition (ASR) systems developed using adults’ speech is a very challenging task. As reported by several earlier works, a severely degraded recognition performance is observed in such ASR tasks. This is mainly due to the gross mismatch in the acoustic and linguistic attributes between those two groups of speakers. One among the various identified sources of mismatch is that the vocal organs of the adult and child speakers are of significantly different dimensions. Feature-space normalization techniques are noted to effectively address the ill-effects arising from those differences. Two most commonly used approaches are the vocal-tract length normalization and the feature-space maximum-likelihood linear regression. Another important mismatch factor is the large variation in the average pitch values across the adult and child speakers. Addressing the ill-effects introduced by the pitch differences is the primary focus of the presented study. In this regard, we have explored the feasibility of explicitly changing the pitch of the children’s speech so that observed pitch differences between the two groups of speaker are reduced. In general, speech data from children is high-pitched in comparison with that from the adults’. Consequently, in this study, the pitch of the adults’ speech used for training the ASR system is kept unchanged while that for the children’s test speech data is reduced. Significant improvement in the recognition performance is noted by this explicit reduction of pitch. To conserve the critical spectral information and to avoid introducing perceptual artifacts, we have exploited timescale modification techniques for explicit pitch mapping. Furthermore, we also presented two schemes to automatically determine the factor by which the pitch of the given test data should be varied. Automatically determining the compensation factor is critical since an ASR system is expected to be accessed by both adult and child speakers. The effectiveness of proposed techniques is evaluated on adult data trained ASR systems employing different acoustic modeling approaches, viz. Gaussian mixture modeling (GMM), subspace GMM and deep neural networks (DNN). The proposed techniques are found to be highly effective in all the explored modeling paradigms. To further study the effectiveness of the proposed approaches, another DNN-based ASR system is developed on a mix of speech data from adult as well as child speakers. The use of pitch reduction is observed to be effective even in this case.

A Novel Mobility-Aware Playout Algorithm for VoIP Services

The latest explosive growth in mobile networks has resulted in an increasing interest in optimisation techniques for mobile services. Resent advances in mobile wireless networks incorporate link-layer intelligence in order to enhance the performance of network and application layers. The Media Independent Handover (MIH) standard provides a framework that can make such link-layer intelligence available to upper layers. In this paper, a novel MIH-enabled playout algorithm for Voice over Internet Protocol applications is presented that aims to compensate for the degradation of voice quality caused by handovers. To that end, link-layer triggers in conjunction with speech time-scale modification techniques are exploited to mitigate the increase in delay and jitter induced by the handover process. Results of subjective listening tests show typical gains of 0.3 on a 5-point scale of the Mean Opinion Score with respect to existing playout scheduling schemes.

Efficient and robust frame-synchronized blind audio watermarking by featuring multilevel DWT and DCT

This paper presents a self-synchronizing blind audio watermarking method developed based on the distinctive features of multilevel discrete wavelet transform (MDWT) and discrete cosine transform (DCT). In the proposed design, the repetitive pattern hidden in the 11th approximation subband is used to control frame synchronization, and the wideband signal covering the first to ninth detail subbands is used to hide binary information. Tracing the zero-crossings across the extracted synchronous signal makes it possible to recalibrate frame positions and achieve synchronous watermarking on a rational dither modulation (RDM) framework. We also developed a windowed vector modulation (WVM) scheme suited for the time as well as DCT domain to enhance watermark imperceptivity. The proposed watermarking method currently achieves a payload capacity of 86 bits per second. The results of PEAQ testing confirm that the watermarked audio signal retains quality nearly indistinguishable from the original. Compared with two existing synchronized methods designed specifically to cope with playback speed modification, the proposed MDWT---DCT---RDM---WVM method demonstrates superior robustness to commonly encountered attacks. When encountering desynchronization attacks, such as cropping and resampling time-scale modification, the proposed technique is capable of retrieving the watermark bits with a high degree of accuracy.

Commutative Watermarking-Encryption of Audio Data with Minimum Knowledge Verification

We present a scheme for commutative watermarking-encryption (CWE) of audio data and demonstrate its robustness against an important class of attacks, Time-Scale Modifications (TSM). In addition, we show how the proposed CWE scheme can be integrated into a cryptographic protocol enabling public verification of the embedded mark without disclosing the mark or the watermarking key used for embedding.