Speech Reconstruction Research Articles

LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading.

The aim of this work is to investigate the impact of crossmodal self-supervised pre-training for speech reconstruction (video-to-audio) by leveraging the natural co-occurrence of audio and visual streams in videos. We propose LipSound2 that consists of an encoder-decoder architecture and location-aware attention mechanism to map face image sequences to mel-scale spectrograms directly without requiring any human annotations. The proposed LipSound2 model is first pre-trained on ∼2400 -h multilingual (e.g., English and German) audio-visual data (VoxCeleb2). To verify the generalizability of the proposed method, we then fine-tune the pre-trained model on domain-specific datasets (GRID and TCD-TIMIT) for English speech reconstruction and achieve a significant improvement on speech quality and intelligibility compared to previous approaches in speaker-dependent and speaker-independent settings. In addition to English, we conduct Chinese speech reconstruction on the Chinese Mandarin Lip Reading (CMLR) dataset to verify the impact on transferability. Finally, we train the cascaded lip reading (video-to-text) system by fine-tuning the generated audios on a pre-trained speech recognition system and achieve the state-of-the-art performance on both English and Chinese benchmark datasets.

Lip2Speech: Lightweight Multi-Speaker Speech Reconstruction with Gabor Features

In environments characterised by noise or the absence of audio signals, visual cues, notably facial and lip movements, serve as valuable substitutes for missing or corrupted speech signals. In these scenarios, speech reconstruction can potentially generate speech from visual data. Recent advancements in this domain have predominantly relied on end-to-end deep learning models, like Convolutional Neural Networks (CNN) or Generative Adversarial Networks (GAN). However, these models are encumbered by their intricate and opaque architectures, coupled with their lack of speaker independence. Consequently, achieving multi-speaker speech reconstruction without supplementary information is challenging. This research introduces an innovative Gabor-based speech reconstruction system tailored for lightweight and efficient multi-speaker speech restoration. Using our Gabor feature extraction technique, we propose two novel models: GaborCNN2Speech and GaborFea2Speech. These models employ a rapid Gabor feature extraction method to derive lowdimensional mouth region features, encompassing filtered Gabor mouth images and low-dimensional Gabor features as visual inputs. An encoded spectrogram serves as the audio target, and a Long Short-Term Memory (LSTM)-based model is harnessed to generate coherent speech output. Through comprehensive experiments conducted on the GRID corpus, our proposed Gabor-based models have showcased superior performance in sentence and vocabulary reconstruction when compared to traditional end-to-end CNN models. These models stand out for their lightweight design and rapid processing capabilities. Notably, the GaborFea2Speech model presented in this study achieves robust multi-speaker speech reconstruction without necessitating supplementary information, thereby marking a significant milestone in the field of speech reconstruction.

Imagined Speech Reconstruction From Neural Signals—An Overview of Sources and Methods

Imagined Speech Reconstruction From Neural Signals—An Overview of Sources and Methods

Argumentative indicators in the adjudication of Russia-Ukraine dispute. A pragma-dialectical study

The context of Russia–Ukraine war has given sufficient reasons to consider any standpoint of argumentation significant and potentially groundbreaking in dealing with threat and conflict. This article reports the findings of a pragma–dialectical study of argumentative indicators in the adjudication of a Russia–Ukraine dispute which mark the judgement–based understanding and arguments–infused processes that give solid grounds to establish the strategy of dealing with war. Linguistic choices in the form of argumentative indicators constitute keystones in the critical discussion, allowing the reconstruction and identification of speech act moves that are to be found in the patterned route of argumentation. As this analysis shows, the organisation of FTAs parameters and propositional attitude indicators significantly contribute to the sequentiality and complementariness of the argumentation process which proves to be highly effective and reasonable. Given the specificity of an adjudication as a type of a genre, its judiciary contextualisation, and legislative power, it is suggested that this argumentative practice makes for a threat and risk management strategy.

Optical Microphone-Based Speech Reconstruction System With Deep Learning for Individuals With Hearing Loss.

Although many speech enhancement (SE) algorithms have been proposed to promote speech perception in hearing-impaired patients, the conventional SE approaches that perform well under quiet and/or stationary noises fail under nonstationary noises and/or when the speaker is at a considerable distance. Therefore, the objective of this study is to overcome the limitations of the conventional speech enhancement approaches. This study proposes a speaker-closed deep learning-based SE method together with an optical microphone to acquire and enhance the speech of a target speaker. The objective evaluation scores achieved by the proposed method outperformed the baseline methods by a margin of 0.21-0.27 and 0.34-0.64 in speech quality (HASQI) and speech comprehension/intelligibility (HASPI), respectively, for seven typical hearing loss types. The results suggest that the proposed method can enhance speech perception by cutting off noise from speech signals and mitigating interference caused by distance. The results of this study show a potential way that can help improve the listening experience in enhancing speech quality and speech comprehension/intelligibility for hearing-impaired people.

Assistive Magnetic Skin System for Speech Reconstruction: An Empowering Technology for Aphonic Individuals

Individuals with a voice disorder or speech sound disorder (SSD) (i.e., aphonic or mute people) can have any combination of difficulties with perception, articulation/motor production, and phonotactics, which may impact their speech intelligibility and acceptability, thus finding challenging to communicate with the public. As a result, many individuals suffer from frustration, isolation, and depression. Natural verbal communication for SSD patients is now more feasible than ever thanks to advancements in wearable artificial skins and machine learning. This article presents an assistive magnetic skin system for speech reconstruction (AM2S‐SR), which enables SSD‐afflicted people to communicate with their mouths. Using magnetic field sensors integrated into Magnetphones, the system reads the movement of the mouth by tracking the movement of magnetic skin patches attached next to the bottom lip. The measured magnetic field signals data are then processed using a Fine K‐Nearest Neighbor machine learning classifier. The classified data are then exported verbally on speakers, or visually on a display. AM2S‐SR successfully identifies the full English alphabets with average success rate of 94.96%, thus enabling SSD people to talk using the mouth and have a more natural conversation with others.

Auditory stimulus reconstruction from ECoG with DNN and self-attention modules

Auditory stimulus reconstruction is a technique that predicts sound from neural activities. Decoding speech from human brains using neuronal networks allows us to understand auditory information processing and study hearing abilities effectively. However, current approaches produce low-quality reconstruction of speech which is problematic for applications of the reconstruction technology to practical uses such as evaluation of hearing characteristics of individuals or functional preservation during neurosurgical procedures. To tackle this issue, we used a self-attention (SA) module which are showing promising advances in deep neural network studies. We investigated the effect of including neural activity from a longer span on the reconstruction accuracy, and the learning behavior of self-attention modules. Our results showed that self-attention modules that focus on the presence or absence of auditory stimuli used relationships between features over a longer period (over 1000 ms) and produced better reconstructions than a convoluted neural network (CNN) model using features over a shorter period (300 ms) and a multilayer perceptron (MLP) model not using the features and reconstructed with attention to presence and absence of auditory stimuli. These findings imply that long and sparse relationships of time-series information on neural activities derived from SA modules improve the reconstruction performance.

Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models.

Objective.Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable results is crucial for advancing the field.Approach.In this paper, we optimized and validated a decoding approach based on speech reconstruction directly from high-density electrocorticography recordings from sensorimotor cortex during a speech production task.Main results.We show that (1) dedicated machine learning optimization of reconstruction models is key for achieving the best reconstruction performance; (2) individual word decoding in reconstructed speech achieves 92%-100% accuracy (chance level is 8%); (3) direct reconstruction from sensorimotor brain activity produces intelligible speech.Significance.These results underline the need for model optimization in achieving best speech decoding results and highlight the potential that reconstruction-based speech decoding from sensorimotor cortex can offer for development of next-generation BCI technology for communication.

A new speech enhancement method based on Swin-UNet model

U-shaped Network (UNet) has shown excellent performance in a variety of speech enhancement tasks. However, because of the intrinsic limitation of convolutional operation, traditional UNet built with convolutional neural network (CNN) cannot learn global and long-term information well. In this work, we propose a new Swin-UNet-based speech enhancement method. Unlike the traditional UNet model, the CNN blocks are all replaced with Swin-Transformer blocks to explore more multi-scale contextual information. The Swin-UNet model employs shifted window mechanism which not only overcomes the defect of high computational complexity of the Transformer but also enhances global information interaction by utilizing the powerful global modeling capability of the Transformer. Through hierarchical Swin-Transformer blocks, global and local speech features can be fully leveraged to improve speech reconstruction ability. Experimental results confirm that the proposed method can eliminate more background noise while maintaining good objective speech quality.

Overt speech decoding from cortical activity: a comparison of different linear methods.

Speech BCIs aim at reconstructing speech in real time from ongoing cortical activity. Ideal BCIs would need to reconstruct speech audio signal frame by frame on a millisecond-timescale. Such approaches require fast computation. In this respect, linear decoder are good candidates and have been widely used in motor BCIs. Yet, they have been very seldomly studied for speech reconstruction, and never for reconstruction of articulatory movements from intracranial activity. Here, we compared vanilla linear regression, ridge-regularized linear regressions, and partial least squares regressions for offline decoding of overt speech from cortical activity. Two decoding paradigms were investigated: (1) direct decoding of acoustic vocoder features of speech, and (2) indirect decoding of vocoder features through an intermediate articulatory representation chained with a real-time-compatible DNN-based articulatory-to-acoustic synthesizer. Participant's articulatory trajectories were estimated from an electromagnetic-articulography dataset using dynamic time warping. The accuracy of the decoders was evaluated by computing correlations between original and reconstructed features. We found that similar performance was achieved by all linear methods well above chance levels, albeit without reaching intelligibility. Direct and indirect methods achieved comparable performance, with an advantage for direct decoding. Future work will address the development of an improved neural speech decoder compatible with fast frame-by-frame speech reconstruction from ongoing activity at a millisecond timescale.

Time-Domain Speech Separation Networks With Graph Encoding Auxiliary

End-to-end time-domain speech separation with masking strategy has shown its performance advantage, where a 1-D convolutional layer is used as the speech encoder to encode a sliding window of waveform to a latent feature representation, i.e. an embedding vector. A large window leads to low resolution in the speech processing, on the other hand, a small window offers high resolution but at the expense of high computational cost. In this work, we propose a graph encoding technique to model the fine structural knowledge of speech samples in a window of reasonable size. Specifically, we build a graph representation for each latent representation, and encode the structural details with a graph convolutional network encoder. The encoded graph feature representation complements the original latent feature representation and benefits the separation and reconstruction of speech. Experiments on various models and datasets show that our proposed encoding technique significantly improves the speech quality over other time-domain speech encoders.

Semantic-Preserved Communication System for Highly Efficient Speech Transmission

Deep learning (DL) based semantic communication methods have been explored for the efficient transmission of images, text, and speech in recent years. In contrast to traditional wireless communication methods that focus on the transmission of abstract symbols, semantic communication approaches attempt to achieve better transmission efficiency by only sending the semantic-related information of the source data. In this paper, we consider semantic-oriented speech transmission which transmits only the semantic-relevant information over the channel for the speech recognition task, and a compact additional set of semantic-irrelevant information for the speech reconstruction task. We propose a novel end-to-end DL-based transceiver which extracts and encodes the semantic information from the input speech spectrums at the transmitter and outputs the corresponding transcriptions from the decoded semantic information at the receiver. In particular, we employ a soft alignment module and a redundancy removal module to extract only the text-related semantic features while dropping semantically redundant content, greatly reducing the amount of semantic redundancy compared to existing methods. We also propose a semantic correction module to further correct the predicted transcription with semantic knowledge by leveraging a pretrained language model. For the speech to speech transmission, we further include a CTC alignment module that extracts a small number of additional semantic-irrelevant but speech-related information, such as duration, pitch, power and speaker identification of the speech for the better reconstruction of the original speech signals at the receiver. We also introduce a two-stage training scheme which speeds up the training of the proposed DL model. The simulation results confirm that our proposed method outperforms current methods in terms of the accuracy of the predicted text for the speech to text transmission and the quality of the recovered speech signals for the speech to speech transmission, and significantly improves transmission efficiency. More specifically, the proposed method only sends 16% of the amount of the transmitted symbols required by the existing methods while achieving about a 10% reduction in WER for the speech to text transmission. For the speech to speech transmission, it results in an even more remarkable improvement in terms of transmission efficiency with only 0.2% of the amount of the transmitted symbols required by the existing method while preserving the comparable quality of the reconstructed speech signals.

A Speech Enhancement Algorithm for Speech Reconstruction Based on Laser Speckle Images.

In the optical system for reconstructing speech signals based on laser speckle images, the resonance between the sound source and nearby objects leads to frequency response problem, which seriously affects the accuracy of reconstructed speech. In this paper, we propose a speech enhancement algorithm to reduce the frequency response. The results show that after using the speech enhancement algorithm, the frequency spectrum correlation coefficient between the reconstructed sinusoidal signal and the original sinusoidal signal is improved by up to 82.45%, and the real speech signal is improved by up to 56.40%. This proves that the speech enhancement algorithm is a valuable tool for solving the frequency response problem and improving the accuracy of reconstructed speech.

Meta-reinforcement learning based few-shot speech reconstruction for non-intrusive speech quality assessment

Meta-reinforcement learning based few-shot speech reconstruction for non-intrusive speech quality assessment

Multimodal audio-visual information fusion using canonical-correlated Graph Neural Network for energy-efficient speech enhancement

This paper proposes a novel multimodal self-supervised architecture for energy-efficient audio-visual (AV) speech enhancement that integrates Graph Neural Networks with canonical correlation analysis (CCA-GNN). The proposed approach lays its foundations on a state-of-the-art CCA-GNN that learns representative embeddings by maximizing the correlation between pairs of augmented views of the same input while decorrelating disconnected features. The key idea of the conventional CCA-GNN involves discarding augmentation-variant information and preserving augmentation-invariant information while preventing capturing of redundant information. Our proposed AV CCA-GNN model deals with multimodal representation learning context. Specifically, our model improves contextual AV speech processing by maximizing canonical correlation from augmented views of the same channel and canonical correlation from audio and visual embeddings. In addition, it proposes a positional node encoding that considers a prior-frame sequence distance instead of a feature-space representation when computing the node’s nearest neighbors, introducing temporal information in the embeddings through the neighborhood’s connectivity. Experiments conducted on the benchmark ChiME3 dataset show that our proposed prior frame-based AV CCA-GNN ensures a better feature learning in the temporal context, leading to more energy-efficient speech reconstruction than state-of-the-art CCA-GNN and multilayer perceptron.

A review of Melodic Intonation Therapy (MIT) and pilot of Dynamic Adaptive Speech Reconstruction (DASR) imaging

IntroductionMaking mistakes is at the heart of learning. This principle is central to “re‐learning” how to speak and, when applied with music, can elicit a robust brain activation. As simple as a search for the right note provokes a plastic process in normal language development since it activates the right hemisphere homolog of Broca’s area, which “takes over” language functions. This review and pilot experiment examines language neuroplasticity by observing how Melodic Intonation Therapy (MIT), a treatment modality that combines elements of speech, rhythm, and melody, plays a role in language re‐learning and how its limitations can be overcome with a new tool our lab is developing ‐ Dynamic Adaptive Speech Reconstruction (DASR). MIT’s design in combining task modalities of speaking, singing, and tapping to activate Broca’s area is limited by its well‐intentioned, yet subjective interpretation by speech‐language pathologists. DASR, when fully developed, overcomes subjectivity by using an AI algorithm that incorporates MIT data in real‐time in combination with functional Near‐Infrared Spectroscopy (fNIRS) to observe areas of the brain that are actively recruited when a patient attempts to form phrases. The DASR algorithm objectively analyzes patients’ responses, deconstructs the response, associates it with the fNIRS brain activation, measured as the Blood Oxygenation Level Dependent (BOLD) response, and suggests the next request the therapist should ask of the patient. We tested two aspects of DASR in recruiting the right hemisphere homologue of Broca’s area.MethodsUsing fNIRS in two age‐matched, middle‐aged subjects, we measured activation of Broca’s area in response to a set of randomized repeated phrases. Participants were fitted with an electrode skull cap that measures the BOLD response to the cerebral cortex. Each participant performed a series of 21 tasks (20 seconds each) per trial (420 seconds). Each of the 21 tasks consisted of a recited prompt, which the subject was to repeat as precisely as possible. Subjects were asked to speak, sing, intone with an augmented 4th note (added per DASR protocol), and tap a specific rhythm to “happy birthday to you” and to subsequently more difficult analogs of the phrase: “Hippo birdseed two ewes,” then “Ep say feef now jam bob,” and finally “happy birthday to you.” The set of 21 tasks were repeated in a pseudo‐random order for each subject five times. Block design statistical parametric mapping was used to visualize activation.ResultsBroca’s area showed the greatest activation when task modalities were combined, in particular when singing augmented 4th phrases. It is notable that when tapping is added to singing the augmented 4th, Broca’s area activation appears to decrease, but activation of the right lateral frontal lobe increases with each successive trial. Subjects repeating the phrase, “happy birthday to you”, after going through a series of increasingly complex phrases showed recruitment of the right inferior frontal lobe after the subject completes just one series of trials.DiscussionThis review and pilot study demonstrates that fNIRS can be used to visual brain activation of Broca’s area, particularly in conditions of increasing linguistic complexity, and shows immediate recruitment of the right hemisphere homologue of Broca’s area.

Two-level optimization approach with accelerated proximal gradient for objective measures in sparse speech reconstruction

<p style='text-indent:20px;'>Compressive speech enhancement makes use of the sparseness of speech and the non-sparseness of noise in time-frequency representation to perform speech enhancement. However, reconstructing the sparsest output may not necessarily translate to a good enhanced speech signal as speech distortion may be at risk. This paper proposes a two level optimization approach to incorporate objective quality measures in compressive speech enhancement. The proposed method combines the accelerated proximal gradient approach and a global one dimensional optimization method to solve the sparse reconstruction. By incorporating objective quality measures in the optimization process, the reconstructed output is not only sparse but also maintains the highest objective quality score possible. In other words, the sparse speech reconstruction process is now quality sparse speech reconstruction. Experimental results in a compressive speech enhancement consistently show score improvement in objectives measures in different noisy environments compared to the non-optimized method. Additionally, the proposed optimization yields a higher convergence rate with a lower computational complexity compared to the existing methods.</p>

A Nested U-Net With Self-Attention and Dense Connectivity for Monaural Speech Enhancement

With the development of deep neural networks, speech enhancement technology has been vastly improved. However, commonly used speech enhancement approaches cannot fully leverage contextual information from different scales, which limits performance improvement. To address this problem, we propose a nested U-Net with self-attention and dense connectivity (SADNUNet) for monaural speech enhancement in the time domain. SADNUNet is an encoder-decoder structure with skip connections. In SADNUNet, the multi-scale aggregation block is proposed to explore more contextual information from different scales. By this means, the advantage of global and local speech features can be fully utilized to improve speech reconstruction ability. Furthermore, dense connectivity and self-attention are incorporated in the network for better feature extraction and utterance level context aggregation. The experimental results demonstrate that the proposed approach achieves on-par or better performance than other models in objective speech intelligibility and quality scores.

Near-ear sound pressure level distribution in everyday life considering the user’s own voice and privacy

Recently, exploring acoustic conditions of people in their everyday environments has drawn a lot of attention. One of the most important and disturbing sound sources is the test participant’s own voice. This contribution proposes an algorithm to determine the own-voice audio segments (OVS) for blocks of 125 ms and a method for measuring sound pressure levels (SPL) without violating privacy laws. The own voice detection (OVD) algorithm here developed is based on a machine learning algorithm and a set of acoustic features that do not allow for speech reconstruction. A manually labeled real-world recording of one full day showed reliable and robust detection results. Moreover, the OVD algorithm was applied to 13 near-ear recordings of hearing-impaired participants in an ecological momentary assessment (EMA) study. The analysis shows that the grand mean percentage of predicted OVS during one day was approx. 10% which corresponds well to other published data. These OVS had a small impact on the median SPL over all data. However, for short analysis intervals, significant differences up to 30 dB occurred in the measured SPL, depending on the proportion of OVS and the SPL of the background noise.

Envelope reconstruction of speech and music highlights stronger tracking of speech at low frequencies.

The human brain tracks amplitude fluctuations of both speech and music, which reflects acoustic processing in addition to the encoding of higher-order features and one's cognitive state. Comparing neural tracking of speech and music envelopes can elucidate stimulus-general mechanisms, but direct comparisons are confounded by differences in their envelope spectra. Here, we use a novel method of frequency-constrained reconstruction of stimulus envelopes using EEG recorded during passive listening. We expected to see music reconstruction match speech in a narrow range of frequencies, but instead we found that speech was reconstructed better than music for all frequencies we examined. Additionally, models trained on all stimulus types performed as well or better than the stimulus-specific models at higher modulation frequencies, suggesting a common neural mechanism for tracking speech and music. However, speech envelope tracking at low frequencies, below 1 Hz, was associated with increased weighting over parietal channels, which was not present for the other stimuli. Our results highlight the importance of low-frequency speech tracking and suggest an origin from speech-specific processing in the brain.