Speech Encoding Research Articles

A Robust Chaotic Map and Its Application to Speech Encryption in Dual Frequency Domain

When chaotic systems are used for speech encryption, their chaotic performance largely determines the security of speech encryption. However, traditional chaotic systems have problems such as parameter discontinuity, easy occurrence of chaos degradation, low complexity, and the existence of periodic windows in chaotic intervals. In real applications, chaotic mappings may fall into periodic windows, which is extremely unfavorable for security. In this paper, a new chaotic mapping 2D-LMSM is proposed by improving the chaotic logistic and sine mappings, and applied to speech encryption. Performance evaluation shows that this map can effectively generate robust chaotic signals in a wide parameter range. The 2D-LMSM achieves better robustness and desired chaotic properties than several existing two-dimensional chaotic maps. We propose a novel speech encryption algorithm using this map. First, it performs Fast Fourier Transform (FFT) on the input speech signal to obtain real and imaginary values, which are encrypted by one-time scrambling encryption and XOR diffusion encryption with pseudorandom numbers generated by chaos; then, it performs secondary scrambling encryption by Discrete Wavelet Transform (DWT) and 2D-LMSM; finally, it obtains encrypted speech data by Discrete Wavelet Inverse Transform (IDWT) and Fast Fourier Inverse Transform (IFFT). Experimental results show that this algorithm has good encryption and decryption performances and ensures system security.

Distinct neural encoding of glimpsed and masked speech in multitalker situations.

Humans can easily tune in to one talker in a multitalker environment while still picking up bits of background speech; however, it remains unclear how we perceive speech that is masked and to what degree non-target speech is processed. Some models suggest that perception can be achieved through glimpses, which are spectrotemporal regions where a talker has more energy than the background. Other models, however, require the recovery of the masked regions. To clarify this issue, we directly recorded from primary and non-primary auditory cortex (AC) in neurosurgical patients as they attended to one talker in multitalker speech and trained temporal response function models to predict high-gamma neural activity from glimpsed and masked stimulus features. We found that glimpsed speech is encoded at the level of phonetic features for target and non-target talkers, with enhanced encoding of target speech in non-primary AC. In contrast, encoding of masked phonetic features was found only for the target, with a greater response latency and distinct anatomical organization compared to glimpsed phonetic features. These findings suggest separate mechanisms for encoding glimpsed and masked speech and provide neural evidence for the glimpsing model of speech perception.

Deficits in neural encoding of speech in preterm infants

Preterm children show developmental cognitive and language deficits that can be subtle and sometimes undetectable until later in life. Studies of brain development in children who are born preterm have largely focused on vascular and gross anatomical characteristics rather than pathophysiological processes that may contribute to these developmental deficits. Neural encoding of speech as reflected in EEG recordings is predictive of future language development and could provide insights into those pathophysiological processes. We recorded EEG from 45 preterm (≤ 34 weeks of gestation) and 45 term (≥ 38 weeks) Chinese-learning infants 0–12 months of (corrected) age during natural sleep. Each child listened to three speech stimuli that differed in lexically meaningful pitch (2 native and 1 non-native speech categories). EEG measures associated with synchronization and gross power of the frequency following response (FFR) were examined. ANCOVAs revealed no main effect of stimulus nativeness but main effects of age, consistent with earlier studies. A main effect of prematurity also emerged, with synchronization measures showing stronger group differences than power. By detailing differences in FFR measures related to synchronization and power, this study brings us closer to identifying the pathophysiological pathway to often subtle language problems experienced by preterm children.

Metamodal Coupling of Vibrotactile and Auditory Speech Processing Systems through Matched Stimulus Representations.

It has been postulated that the brain is organized by "metamodal," sensory-independent cortical modules capable of performing tasks (e.g., word recognition) in both "standard" and novel sensory modalities. Still, this theory has primarily been tested in sensory-deprived individuals, with mixed evidence in neurotypical subjects, thereby limiting its support as a general principle of brain organization. Critically, current theories of metamodal processing do not specify requirements for successful metamodal processing at the level of neural representations. Specification at this level may be particularly important in neurotypical individuals, where novel sensory modalities must interface with existing representations for the standard sense. Here we hypothesized that effective metamodal engagement of a cortical area requires congruence between stimulus representations in the standard and novel sensory modalities in that region. To test this, we first used fMRI to identify bilateral auditory speech representations. We then trained 20 human participants (12 female) to recognize vibrotactile versions of auditory words using one of two auditory-to-vibrotactile algorithms. The vocoded algorithm attempted to match the encoding scheme of auditory speech while the token-based algorithm did not. Crucially, using fMRI, we found that only in the vocoded group did trained-vibrotactile stimuli recruit speech representations in the superior temporal gyrus and lead to increased coupling between them and somatosensory areas. Our results advance our understanding of brain organization by providing new insight into unlocking the metamodal potential of the brain, thereby benefitting the design of novel sensory substitution devices that aim to tap into existing processing streams in the brain.SIGNIFICANCE STATEMENT It has been proposed that the brain is organized by "metamodal," sensory-independent modules specialized for performing certain tasks. This idea has inspired therapeutic applications, such as sensory substitution devices, for example, enabling blind individuals "to see" by transforming visual input into soundscapes. Yet, other studies have failed to demonstrate metamodal engagement. Here, we tested the hypothesis that metamodal engagement in neurotypical individuals requires matching the encoding schemes between stimuli from the novel and standard sensory modalities. We trained two groups of subjects to recognize words generated by one of two auditory-to-vibrotactile transformations. Critically, only vibrotactile stimuli that were matched to the neural encoding of auditory speech engaged auditory speech areas after training. This suggests that matching encoding schemes is critical to unlocking the brain's metamodal potential.

Encoding of speech in convolutional layers and the brain stem based on language experience

Comparing artificial neural networks with outputs of neuroimaging techniques has recently seen substantial advances in (computer) vision and text-based language models. Here, we propose a framework to compare biological and artificial neural computations of spoken language representations and propose several new challenges to this paradigm. The proposed technique is based on a similar principle that underlies electroencephalography (EEG): averaging of neural (artificial or biological) activity across neurons in the time domain, and allows to compare encoding of any acoustic property in the brain and in intermediate convolutional layers of an artificial neural network. Our approach allows a direct comparison of responses to a phonetic property in the brain and in deep neural networks that requires no linear transformations between the signals. We argue that the brain stem response (cABR) and the response in intermediate convolutional layers to the exact same stimulus are highly similar without applying any transformations, and we quantify this observation. The proposed technique not only reveals similarities, but also allows for analysis of the encoding of actual acoustic properties in the two signals: we compare peak latency (i) in cABR relative to the stimulus in the brain stem and in (ii) intermediate convolutional layers relative to the input/output in deep convolutional networks. We also examine and compare the effect of prior language exposure on the peak latency in cABR and in intermediate convolutional layers. Substantial similarities in peak latency encoding between the human brain and intermediate convolutional networks emerge based on results from eight trained networks (including a replication experiment). The proposed technique can be used to compare encoding between the human brain and intermediate convolutional layers for any acoustic property and for other neuroimaging techniques.

Speech Files Cryptography using Indices Keys and Blocking

Protecting speech file is a vital issue. In this paper research a simple method of speech cryptography will be introduced. The method will use indices key to perform simple replacement operations to apply speech encryption and decryption, the replacement operation will use the indexes and the indexes contents to apply speech cryptography. The proposed method will protect the speech signal by increasing the security level, the method will use a color image as an image_key, this key will be kept in secret between the sender and receiver and it will be used to generate the indices key. The proposed will sensitive to the selected image_key, changing this image will be considered as a hacking attempt by producing a damaged decrypted speech signal. The proposed method will be flexible, changing the image_key will not cause any changes in the proposed method algorithms. To increase the efficiency of the proposed method by decreasing the processing time the speech file will be divided into blocks, and each block will be treated separately. The proposed method will be tested and implemented using various speech files, the obtained results will be used to prove the quality, sensitivity and efficiency of the proposed method.

Perceptual warping exposes categorical representations for speech in human brainstem responses

The brain transforms continuous acoustic events into discrete category representations to downsample the speech signal for our perceptual-cognitive systems. Such phonetic categories are highly malleable, and their percepts can change depending on surrounding stimulus context. Previous work suggests these acoustic-phonetic mapping and perceptual warping of speech emerge in the brain no earlier than auditory cortex. Here, we examined whether these auditory-category phenomena inherent to speech perception occur even earlier in the human brain, at the level of auditory brainstem. We recorded speech-evoked frequency following responses (FFRs) during a task designed to induce more/less warping of listeners’ perceptual categories depending on stimulus presentation order of a speech continuum (random, forward, backward directions). We used a novel clustered stimulus paradigm to rapidly record the high trial counts needed for FFRs concurrent with active behavioral tasks. We found serial stimulus order caused perceptual shifts (hysteresis) near listeners’ category boundary confirming identical speech tokens are perceived differentially depending on stimulus context. Critically, we further show neural FFRs during active (but not passive) listening are enhanced for prototypical vs. category-ambiguous tokens and are biased in the direction of listeners’ phonetic label even for acoustically-identical speech stimuli. These findings were not observed in the stimulus acoustics nor model FFR responses generated via a computational model of cochlear and auditory nerve transduction, confirming a central origin to the effects. Our data reveal FFRs carry category-level information and suggest top-down processing actively shapes the neural encoding and categorization of speech at subcortical levels. These findings suggest the acoustic-phonetic mapping and perceptual warping in speech perception occur surprisingly early along the auditory neuroaxis, which might aid understanding by reducing ambiguity inherent to the speech signal.

Cross-modal dynamic sentiment annotation for speech sentiment analysis

Traditionally, one single hard label determines the sentiment label of an entire utterance for speech sentiment analysis. It obviously ignores the inherent dynamic and ambiguity of speech sentiments. Moreover, there are few segment-level ground truth labels in the most existing sentiment corpora, due to the label ambiguity and annotation cost. In this work, to capture segment-level sentiment fluctuations across one utterance, we propose sentiment profiles (SPs) to express segment-level soft labels. Meanwhile, we introduce massive multi-modal wild video data to solve the data shortage problem, and facial expression knowledge is used to guide audio segments generate soft labels through the Cross-modal Sentiment Annotation Module. Then, we design a Speech Encoder Module to encode audio segments into SPs. We further exploit the sentiment profile purifier (SPP) to iteratively improve the accuracy of SPs. Numerous experiments show that our model achieves state-of-the-art performance on CH-SIMS and IEMOCAP datasets with unlabeled data respectively.

TTS-Guided Training for Accent Conversion Without Parallel Data

Accent Conversion (AC) seeks to change the accent of speech from one (source) to another (target) while preserving the speech content and speaker identity. However, many existing AC approaches rely on source-target parallel speech data during training or reference speech at run-time. We propose a novel accent conversion framework without the need for either parallel data or reference speech. Specifically, a text-to-speech (TTS) system is first pretrained with target-accented speech data. This TTS model and its hidden representations are expected to be associated only with the target accent. Then, a speech encoder is trained to convert the accent of the speech under the supervision of the pretrained TTS model. In doing so, the source-accented speech and its corresponding transcription are forwarded to the speech encoder and the pretrained TTS, respectively. The output of the speech encoder is optimized to be the same as the text embedding in the TTS system. At run-time, the speech encoder is combined with the pretrained speech decoder to convert the source-accented speech toward the target. In the experiments, we converted English with two source accents (Chinese/Indian) to the target accent (American/British/Canadian). Both objective metrics and subjective listening tests successfully validate that the proposed approach generates speech samples that are close to the target accent with high speech quality.

A New Speech Encoder Based on Dynamic Framing Approach

Latent information is difficult to get from the text in speech synthesis. Studies show that features from speech can get more information to help text encoding. In the field of speech encoding, a lot of work has been conducted on two aspects. The first aspect is to encode speech frame by frame. The second aspect is to encode the whole speech to a vector. But the scale in these aspects is fixed. So, encoding speech with an adjustable scale for more latent information is worthy of investigation. But current alignment approaches only support frame-by-frame encoding and speech-to-vector encoding. It remains a challenge to propose a new alignment approach to support adjustable scale speech encoding. This paper presents the dynamic speech encoder with a new alignment approach in conjunction with frame-by-frame encoding and speech-to-vector encoding. The speech feature from our model achieves three functions. First, the speech feature can reconstruct the origin speech while the length of the speech feature is equal to the text length. Second, our model can get text embedding from speech, and the encoded speech feature is similar to the text embedding result. Finally, it can transfer the style of synthesis speech and make it more similar to the given reference speech.

Time-Domain Speech Separation Networks With Graph Encoding Auxiliary

End-to-end time-domain speech separation with masking strategy has shown its performance advantage, where a 1-D convolutional layer is used as the speech encoder to encode a sliding window of waveform to a latent feature representation, i.e. an embedding vector. A large window leads to low resolution in the speech processing, on the other hand, a small window offers high resolution but at the expense of high computational cost. In this work, we propose a graph encoding technique to model the fine structural knowledge of speech samples in a window of reasonable size. Specifically, we build a graph representation for each latent representation, and encode the structural details with a graph convolutional network encoder. The encoded graph feature representation complements the original latent feature representation and benefits the separation and reconstruction of speech. Experiments on various models and datasets show that our proposed encoding technique significantly improves the speech quality over other time-domain speech encoders.

An Unknown Case of Speech Encryption

An Unknown Case of Speech Encryption

Comparable Encoding, Comparable Perceptual Pattern: Acoustic and Electric Hearing.

Perception with electric neuroprostheses is sometimes expected to be simulated using properly designed physical stimuli. Here, we examined a new acoustic vocoder model for electric hearing with cochlear implants (CIs) and hypothesized that comparable speech encoding can lead to comparable perceptual patterns for CI and normal hearing (NH) listeners. Speech signals were encoded using FFT-based signal processing stages including band-pass filtering, temporal envelope extraction, maxima selection, and amplitude compression and quantization. These stages were specifically implemented in the same manner by an Advanced Combination Encoder (ACE) strategy in CI processors and Gaussian-enveloped Tones (GET) or Noise (GEN) vocoders for NH. Adaptive speech reception thresholds (SRTs) in noise were measured using four Mandarin sentence corpora. Initial consonant (11 monosyllables) and final vowel (20 monosyllables) recognition were also measured. Naïve NH listeners were tested using vocoded speech with the proposed GET/GEN vocoders as well as conventional vocoders (controls). Experienced CI listeners were tested using their daily-used processors. Results showed that: 1) there was a significant training effect on GET vocoded speech perception; 2) the GEN vocoded scores (SRTs with four corpora and consonant and vowel recognition scores) as well as the phoneme-level confusion pattern matched with the CI scores better than controls. The findings suggest that the same signal encoding implementations may lead to similar perceptual patterns simultaneously in multiple perception tasks. This study highlights the importance of faithfully replicating all signal processing stages in the modeling of perceptual patterns in sensory neuroprostheses. This approach has the potential to enhance our understanding of CI perception and accelerate the engineering of prosthetic interventions. The GET/GEN MATLAB program is freely available at https://github.com/BetterCI/GETVocoder.

Effects of Unilateral Hearing Loss on Speech-Evoked Auditory Brainstem Responses and Reading: A Preliminary Study.

The purpose of this study was to determine whether the speech-evoked auditory brainstem responses (sABRs) obtained by stimulating the ear with normal sensitivity in children with unilateral hearing loss (UHL) were different from that of children with normal hearing (NH), and to explore correlations between the sABR findings and measures of reading. Eleven children with UHL and 11 children with NH were tested via the BioMARK sABR protocol using the syllable /da/; latency and amplitudes of Waves V, A, C, D, E, F, and O were measured. Participants also were tested on the Phonemic Synthesis Test (PST) and the Woodcock Reading Mastery Test-Revised (WRMT-R), particularly the Reading Readiness, Basic Skills, and Comprehension subtests. Multivariate analysis of variance testing showed a significantly higher amplitude for Wave A for the NH group as compared to the UHL group. Separate ANOVAs also found significantly lower scores for the UHL group compared to the NH group on Basic Skills and Comprehension subtests of the Woodcock. Significant positive Spearman rho correlations were found for the UHL group between wave amplitudes for V, A, and O and the Reading Readiness score, and between wave amplitudes for V, A, D, and O and the Reading Comprehension score. A significant correlation also was found between the Total Reading score and wave amplitudes for V and A. No such correlations were found between wave amplitude and Woodcock scores for the NH group. Further testing of the UHL data found significant relationships between the pure-tone average of the hearing loss ear and the Basic Skills, Comprehension, and Total Reading scores from the WRMT-R. There was also a significant correlation between the PST score and Wave C amplitude. There may be a connection between speech encoding and measures of reading in children with UHL.

Differences in neural encoding of speech in noise between cochlear implant users with and without preserved acoustic hearing

Cochlear implants (CIs) have evolved to combine residual acoustic hearing with electric hearing. It has been expected that CI users with residual acoustic hearing experience better speech-in-noise perception than CI-only listeners because preserved acoustic cues aid unmasking speech from background noise. This study sought neural substrate of better speech unmasking in CI users with preserved acoustic hearing compared to those with lower degree of acoustic hearing. Cortical evoked responses to speech in multi-talker babble noise were compared between 29 Hybrid (i.e., electric acoustic stimulation or EAS) and 29 electric-only CI users. The amplitude ratio of evoked responses to speech and noise, or internal SNR, was significantly larger in the CI users with EAS. This result indicates that CI users with better residual acoustic hearing exhibit enhanced unmasking of speech from background noise.

The possible role of early-stage phase-locked neural activities in speech-in-noise perception in human adults across age and hearing loss

Ageing affects auditory neural phase-locked activities which could increase the challenges experienced during speech-in-noise (SiN) perception by older adults. However, evidence for how ageing affects SiN perception through these phase-locked activities is still lacking. It is also unclear whether influences of ageing on phase-locked activities in response to different acoustic properties have similar or different mechanisms to affect SiN perception. The present study addressed these issues by measuring early-stage phase-locked encoding of speech under quiet and noisy backgrounds (speech-shaped noise (SSN) and multi-talker babbles) in adults across a wide age range (19–75 years old). Participants passively listened to a repeated vowel whilst the frequency-following response (FFR) to fundamental frequency that has primary subcortical sources and cortical phase-locked response to slowly-fluctuating acoustic envelopes were recorded. We studied how these activities are affected by age and age-related hearing loss and how they are related to SiN performances (word recognition in sentences in noise). First, we found that the effects of age and hearing loss differ for the FFR and slow-envelope phase-locking. FFR was significantly decreased with age and high-frequency (≥ 2 kHz) hearing loss but increased with low-frequency (< 2 kHz) hearing loss, whilst the slow-envelope phase-locking was significantly increased with age and hearing loss across frequencies. Second, potential relationships between the types of phase-locked activities and SiN perception performances were also different. We found that the FFR and slow-envelope phase-locking positively corresponded to SiN performance under multi-talker babbles and SSN, respectively. Finally, we investigated how age and hearing loss affected SiN perception through phase-locked activities via mediation analyses. We showed that both types of activities significantly mediated the relation between age/hearing loss and SiN perception but in distinct manners. Specifically, FFR decreased with age and high-frequency hearing loss which in turn contributed to poorer SiN performance but increased with low-frequency hearing loss which in turn contributed to better SiN performance under multi-talker babbles. Slow-envelope phase-locking increased with age and hearing loss which in turn contributed to better SiN performance under both SSN and multi-talker babbles. Taken together, the present study provided evidence for distinct neural mechanisms of early-stage auditory phase-locked encoding of different acoustic properties through which ageing affects SiN perception.

Working memory asymmetrically modulates auditory and linguistic processing of speech

Working memory load can modulate speech perception. However, since speech perception and working memory are both complex functions, it remains elusive how each component of the working memory system interacts with each speech processing stage. To investigate this issue, we concurrently measure how the working memory load modulates neural activity tracking three levels of linguistic units, i.e., syllables, phrases, and sentences, using a multiscale frequency-tagging approach. Participants engage in a sentence comprehension task and the working memory load is manipulated by asking them to memorize either auditory verbal sequences or visual patterns. It is found that verbal and visual working memory load modulate speech processing in similar manners: Higher working memory load attenuates neural activity tracking of phrases and sentences but enhances neural activity tracking of syllables. Since verbal and visual WM load similarly influence the neural responses to speech, such influences may derive from the domain-general component of WM system. More importantly, working memory load asymmetrically modulates lower-level auditory encoding and higher-level linguistic processing of speech, possibly reflecting reallocation of attention induced by mnemonic load.

Attentional Demand of Motor Speech Encoding: Evidence From Parkinson's Disease.

While the involvement of attention in utterance planning is well established at the conceptual and lexical levels, the attentional demands of postlexical processes are still debated. This study investigates the involvement of attentional resources on motor speech encoding during utterance production in the context of Parkinson's disease (PD), a population allowing to assess if the attentional demands observed in a dual-task paradigm (the dual-task costs [DTCs]) are explained by postlexical difficulties and not solely by executive impairment. Speech production was analyzed in a dual-task paradigm with 30 participants presenting with motor speech disorders due to hypokinetic dysarthria in the context of PD. The dual-task comprised an automatic speech task in which participants recited the days of the week and two nonverbal tasks evaluating processing speed and inhibition. The severity of dysarthria and performance in several executive tests (inhibition, verbal fluency, and cognitive shifting) were used as potential predictors of the DTCs. Individuals with PD exhibited a DTC on the nonverbal tasks and on the speech task when the secondary task was inhibition (the most difficult one). Additionally, the severity of dysarthria and a poorer performance in cognitive shifting predicted a more severe DTC on speech rate. Finally, modulation of the magnitude of the DTCs was observed, depending on the difficulty of the nonverbal secondary task. The results suggest that, in PD, postlexical processes require attentional resources and cognitive shifting is related to dual-task performance in speech. https://doi.org/10.23641/asha.21265893.

SAMU-XLSR: Semantically-Aligned Multimodal Utterance-Level Cross-Lingual Speech Representation

We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation learning framework. Unlike previous works on speech representation learning, which learns multilingual contextual speech embedding at the resolution of an acoustic frame (10-20ms), this work focuses on learning multimodal (speech-text) multilingual speech embedding at the resolution of a sentence (5-10s) such that the embedding vector space is semantically aligned across different languages. We combine state-of-the-art multilingual acoustic frame-level speech representation learning model XLS-R with the Language Agnostic BERT Sentence Embedding (LaBSE) model to create an utterance-level multimodal multilingual speech encoder SAMU-XLSR. Although we train SAMU-XLSR with only multilingual transcribed speech data, cross-lingual speech-text and speech-speech associations emerge in its learned representation space. To substantiate our claims, we use SAMU-XLSR speech encoder in combination with a pre-trained LaBSE text sentence encoder for cross-lingual speech-to-text translation retrieval, and SAMU-XLSR alone for cross-lingual speech-to-speech translation retrieval. We highlight these applications by performing several cross-lingual text and speech translation retrieval tasks across several datasets.

Non-native speech sound learning as a window to individual differences in the neural encoding of speech

Non-native speech sound learning in adulthood is a skill marked by enormous individual differences, especially when compared to acquisition of speech sounds in childhood. This has led to the proposal that adult speech sound acquisition might be achieved through different mechanisms (or relying on fundamentally altered neural circuitry) compared to sound learning in infancy and childhood. One possibility we explore in a recent line of work is that individual differences in non-native speech learning are attributable to differences in the structure of brain areas responsible for processing speech, especially the transverse temporal gyrus. Of interest, the morphology of these same regions also predicts individual differences in native language speech perception, suggesting that sound acquisition (whether in childhood or adulthood) is limited by (likely innate) differences in neural architecture supporting sound processing. However, evidence that the best non-native perceivers are also the most finely-tuned native-sound perceivers is limited. Taken together, these results suggest that differences in brain structure are a soft constraint on the precision of acoustic-phonetic encoding, but that native language speech category structure is not the explanation for the difficulty of non-native sound learning.