Speech Restoration Research Articles

Implanted cortical neuroprosthetics for speech and movement restoration.

Implanted cortical neuroprosthetics (ICNs) are medical devices developed to replace dysfunctional neural pathways by creating information exchange between the brain and a digital system which can facilitate interaction with the external world. Over the last decade, researchers have explored the application of ICNs for diverse conditions including blindness, aphasia, and paralysis. Both transcranial and endovascular approaches have been used to record neural activity in humans, and in a laboratory setting, high-performance decoding of the signals associated with speech intention has been demonstrated. Particular progress towards a device which can move into clinical practice has been made with ICNs focussed on the restoration of speech and movement. This article provides an overview of contemporary ICNs for speech and movement restoration, their mechanisms of action and the unique ethical challenges raised by the field.

End‐to‐end speech‐denoising deep neural network based on residual‐attention gated linear units

AbstractIn this letter, an improved gated linear unit (GLU) structure for end‐to‐end (E2E) speech enhancement is proposed. In the U‐Net structure, which is widely used as the foundational architecture for E2E deep neural network‐based speech denoising, the input noisy speech signal undergoes multiple layers of encoding and is compressed into essential potential representative information at the bottleneck. The latent information is then transmitted to the decoder stage for the restoration of the target clean speech. Among these approaches, CleanUNet, a prominent state‐of‐the‐art (SOTA) method, enhances temporal attention in latent space by employing multi‐head self‐attention. However, unlike the approach of applying the attention mechanism to the potentially compressed representative information of the bottleneck layer, the proposed method instead assigns the attention module to the GLU of each encoder/decoder block layer. The proposed method is validated by measuring short‐term objective speech intelligibility and sound quality. The objective evaluation results indicated that the proposed method using residual‐attention GLU outperformed existing methods using SOTA models such as FAIR‐denoiser and CleanUNet across signal‐to‐noise ratios ranging from 0 to 15 dB.

Rehabilitation Program for Postlaryngectomy Patients Following Ileocolon Flap Transfer for Voice Reconstruction: An Essential Part of Success.

Speech restoration is important for communication and social activities after pharyngolaryngectomy in head and neck cancer or corrosive injury. Several techniques of voice restoration have been developed to improve life quality. The aim of this paper was to focus on the microsurgical transfer of ileocolon flap and outcome of further voice rehabilitation. From 2010 to 2022, 69 patients had ileocolon flap at our hospital with postoperative speech training and regular follow-up for over 1 year. The patients received deglutition training first, followed by voice rehabilitation. Voice outcomes were evaluated at an interval of 3 months and finally at 12 months of voice training rehabilitation. Among other examinations, the speech function was evaluated using a 4-point Likert scale and senior surgeon (H-c.C.) scoring system. The results showed that speech function reached 13.1% of excellent voice, 65.1% of good voice, 13.1% of fair result, and 8.7% of poor result by Likert scales. Meanwhile, the senior surgeon (H-c.C.) score showed 17.4% of excellent, 63.8% of moderate, and 18.8% of poor results. About voice laboratory results, maximal phonation time was 11.0 seconds, and the average number counted in one breath was 15. Loudness and frequency showed 56.0 dB and 105.0 Hz, respectively. The study showed that after voice reconstruction with ileocolon flap followed by the voice rehabilitation program, the patients would have a better understanding of the altered anatomical structures and practice in a more efficient way. Adequate recommendation by the therapists to plastic surgeons for revision surgeries optimized voice function of the patients.

Peech diagnostics as an objective criterion for theeffectiveness of speech restoration in patients withoropharyngeal cancer

peech diagnostics as an objective criterion for theeffectiveness of speech restoration in patients withoropharyngeal cancer

Lip2Speech: Lightweight Multi-Speaker Speech Reconstruction with Gabor Features

In environments characterised by noise or the absence of audio signals, visual cues, notably facial and lip movements, serve as valuable substitutes for missing or corrupted speech signals. In these scenarios, speech reconstruction can potentially generate speech from visual data. Recent advancements in this domain have predominantly relied on end-to-end deep learning models, like Convolutional Neural Networks (CNN) or Generative Adversarial Networks (GAN). However, these models are encumbered by their intricate and opaque architectures, coupled with their lack of speaker independence. Consequently, achieving multi-speaker speech reconstruction without supplementary information is challenging. This research introduces an innovative Gabor-based speech reconstruction system tailored for lightweight and efficient multi-speaker speech restoration. Using our Gabor feature extraction technique, we propose two novel models: GaborCNN2Speech and GaborFea2Speech. These models employ a rapid Gabor feature extraction method to derive lowdimensional mouth region features, encompassing filtered Gabor mouth images and low-dimensional Gabor features as visual inputs. An encoded spectrogram serves as the audio target, and a Long Short-Term Memory (LSTM)-based model is harnessed to generate coherent speech output. Through comprehensive experiments conducted on the GRID corpus, our proposed Gabor-based models have showcased superior performance in sentence and vocabulary reconstruction when compared to traditional end-to-end CNN models. These models stand out for their lightweight design and rapid processing capabilities. Notably, the GaborFea2Speech model presented in this study achieves robust multi-speaker speech reconstruction without necessitating supplementary information, thereby marking a significant milestone in the field of speech reconstruction.

Neurosurgery and the Brain-Computer Interface.

Brain-computer interfaces (BCIs) are devices that connect the human brain to an effector via a computer and electrode interface. BCIs may also transmit sensory data to the brain. We describe progress with the many types of surgically implanted BCIs, in which electrodes contact or penetrate the cerebral cortex. BCIs developed for restoration of movement in paralyzed limbs or control a robotic arm; restoration of somatic sensation, speech, vision, memory, hearing, and olfaction are also presented. Most devices remain experimental. Commercialization is costly, incurs financial risk, and is time consuming. There are many ethical principles that should be considered by neurosurgeons and by all those responsible for the care of people with serious neurological disability. These considerations are also paramount when the technology is used in for the purpose of enhancement of normal function and where commercial gain is a factor. A new regulatory and legislative framework is urgently required. The evolution of BCIs is occurring rapidly with advances in computer science, artificial intelligence, electronic engineering including wireless transmission, and materials science. The era of the brain-"cloud" interface is approaching.

Restoration of Bone-Conducted Speech With U-Net-Like Model and Energy Distance Loss

Restoration of Bone-Conducted Speech With U-Net-Like Model and Energy Distance Loss

A Two-Stage Approach to Quality Restoration of Bone-Conducted Speech

A Two-Stage Approach to Quality Restoration of Bone-Conducted Speech

The Role of Computer-Aided Diagnosis in Speech Restoration in Patients with Cancer of the Oral Cavity and Oropharynx after Hemiglosectomy: a Prospective Comparative Study

INTRODUCTION. Oral cavity and oropharyngeal cancers usually require extensive surgeries accompanied by damages to anatomical structures and impaired speech function.
 AIM. To compare the effectiveness and terms of speech rehabilitation in patients with cancer of the oral cavity and oropharynx using various speech restoration techniques.
 MATERIALS AND METHODS. We studied the effectiveness and length of speech rehabilitation in 140 patients with stages II–IV oral cavity and oropharyngeal cancer. The age of the patients ranged from 24 to 70 years. All the patients underwent hemiglosectomy. Combined modality treatment and postoperative speech rehabilitation were performed according to the technique developed in the Department of Head and Neck Tumors of the Oncology Research Institute. Speech rehabilitation effectiveness was studied on the basis of subjective and objective evaluation of speech using the computer-software complex (OnkoSpeech v 1.0).
 RESULTS AND DISCUSSION. Impaired speech function of patients was observed before starting combined modality treatment and manifested as a change in the prosodic system of speech and subjective communication difficulties associated with the presence of a tumor. After surgery, all the patients demonstrated impaired speech function, ranging from distorted pronunciation to the complete absence of verbal communication. Speech rehabilitation started 8 to 29 days after surgery: in the study group I, Me = 9.6, in the study group II, Me = 9.8, values did not have statistically significant differences (p 0.05). At the beginning of speech rehabilitation, to restore activity and coordinate the work of articulatory apparatus muscles and speech expiration, special complexes of articulation and breathing exercises followed by sound pronunciation correction were performed. Postoperative speech rehabilitation resulted in the improvement of speech function in 100 % of cases. In group II, where the correction of sound pronunciation was performed on the basis of speech diagnosis data using the OnkoSpeech v 1.0 computer-software complex, it was possible to achieve a better result of sound pronunciation correction of all the studied sounds, except for hissing (lingual-frontal) [sh], [zh], [shch], [ch].
 CONCLUSION. The use of the OnkoSpeech v1. ± software package for speech computer-aided diagnosis made it possible to objectively evaluate and quantify the effectiveness of the correction of the sound pronunciation of six phonemes of the Russian language, improve the results of the correction of sound pronunciation and reduce the time of speech therapy.

OPPORTUNITIES FOR TREATMENT WITH A TEMPORARY OBTURATOR MADE OF THERMOPLASTIC FOIL

Prosthetic treatment with a temporary obturator is an intermediate stage in the overall prosthetic rehabilitation of patients with maxillary resection. According to prevailing opinions, treatment with a temporary obturator should start around 2 weeks to 2-3 months after resection and to continue until the final completion of the healing processes in the defect. Due to the dynamic changes in the defect, this treatment stage is associated with many difficulties, requiring frequent correction of the obturator and up to 12-14 clinical visits on average. The aim of the research is to track the opportunity for fabrication of a temporary obturator of thermoplastic foil with vacuum forming apparatus and its efficiency for restoring speech and fluid intake in patients with maxillary resection. For the aim of the study we conducted a treatment of a 45-year-old patient operated a week ago for maxillary carcinoma. As a result, the left alveolar bone and a part of the hard palate with a diameter of 10 mm were resected, with teeth 13 to 23, 16 and 17 preserved. The defect was padded with gauze and an impression with an irreversible hydrocolloid impression material was taken. In the laboratory a grade 4 plaster working model was cast on which the borders of the palatal plate were outlined. For the frontal teeth, these were the cervical areas of the teeth, and for the distal - their equators. In the area of the defect and the missing teeth the border reached the transitional fold, and distally - the soft palate. The plate was made from foil of 1 mm diameter which was drawn onto the model with a vacuum forming apparatus and was cut along the indicated borders. The palatal plate - obturator successfully closed the comunication to the nasal cavity and restored the opportunity for fluid intake, which was the main problem for the patient. The use of a foil with a thickness of 1 mm provided stability and resistance of the construction. As a result of this, the necessary hermetic seal of the defect was achieved and the speech of the patient was normalised. The described treatment methodology with a temporary obturator allows the successful restoration of speech and fluid intake during the healing processes in the defect. The main advantages of the method are the fast and easy fabrication, the opportunity for multiple corrections and the short period of adaptation.

Breakthroughs in Brain Implants.

In 2023, Brain implants got a big boost. Neuralink, Elon Musk's startup, began recruiting for its first clinical trial for paralysis patients; Precision Neuroscience received breakthrough designation from the Food and Drug Administration (FDA), allowing it to fast-track its own trials. Meanwhile, scientific advances are revealing promising new applications for such devices, including more comprehensive restoration of speech, sight, hearing, and movement to people who have lost those abilities, as well as help for psychiatric conditions, such as eating disorders and depression.

Interrelationship of Psychotherapy, Psychocorrection and Psychoconsultation

Modern neurorehabilitation is focused primarily on the restoration of speech and cognitive mental processes, considering the rehabilitation of the individual as a task of psychotherapy. The article shows that a gentle attitude, a change in "body image" and neurotic experiences, characteristic of most patients with organic brain lesions, significantly affect the effectiveness of neuropsychological rehabilitation. Neurorehabilitation focuses on the restoration of speech and cognitive processes, considering the rehabilitation of the individual as an object of psychotherapy. On the basis of clinical cases, it has been shown that the weakened attitude, body image changes, and neurotic experiences characteristic of most patients with idiophrenic disease affect the effectiveness of neuropsychological rehabilitation. The article examines the theoretical and methodological principles and defines the main concepts of the topic. The relevance of the article is determined by the new innovative social needs in solving problems of a psychological nature, taking into account the neurological features of human thinking. Scientific intelligence in the field of neurorehabilitation provides grounds for determining the most effective methods of psychocorrection. The purpose of the study is to determine the relationship between psychotherapy, psychocorrection and psycho-counseling based on neurorehabilitation. After studying the data, it was determined that the ischemic attack of the brain was unspecified, it was suggested that the observed neurological symptoms (disorders of speech, movements, emotional sphere) are related to residual (residual) symptoms after a transient ischemic attack, which requires a detailed study.

Теоретичний аналіз проблеми жестової стимуляції в роботі з пацієнтами з постінсультною афазією

The article contains the analyses of theoretical aspects of gesture usage as a therapeutic tool in the speech therapy rehabilitation of patients with aphasia. The paper characterizes studies that indicate a close relation between verbal and non-verbal communication. There are defined main types of gestures and their classification, as well as the communicative function of gesticulation for people with preserved speech and the ones with aphasia. There is a description of gesture functioning peculiarities for people with aphasia and their impact on the process of speech therapy rehabilitation. There were analyzed results of previously conducted studies on the effectiveness of gestural stimulation as a method of restoring communication in patients with aphasia. It allowed to state following. Most studies indicate that the use of gestures in impaired communication contributes to the restoration of verbal speech. As well as improves the search for needed vocabulary (both nouns and verbs), helps the patient to remember words better, and in the case of unintelligible speech to convey and recognize the content of the message. It also provides an opportunity for the patient to maintain contact with the surrounding people. Difficulties identified while the research are the use of gestural stimulation in remedial learning, namely motor disorders, particularly limb apraxia, which is common in stroke patients. The article contains the description of tests generally accepted in the neurological clinic, with the help of which it is possible to assess the preservation of complex forms of praxis in patients with organic lesions of the central nervous system. Furthermore, there is the analysis of studies that deny the positive influence of gestures on communication in people with aphasia, or determine the effectiveness of their use only in working with patients with a mild degree of aphasia.

A survey of an inverse halftoning technique toward speech restoration of printed spectrogram images

A survey of an inverse halftoning technique toward speech restoration of printed spectrogram images

Decoding and synthesizing tonal language speech from brain activity.

Recent studies have shown that the feasibility of speech brain-computer interfaces (BCIs) as a clinically valid treatment in helping nontonal language patients with communication disorders restore their speech ability. However, tonal language speech BCI is challenging because additional precise control of laryngeal movements to produce lexical tones is required. Thus, the model should emphasize the features from the tonal-related cortex. Here, we designed a modularized multistream neural network that directly synthesizes tonal language speech from intracranial recordings. The network decoded lexical tones and base syllables independently via parallel streams of neural network modules inspired by neuroscience findings. The speech was synthesized by combining tonal syllable labels with nondiscriminant speech neural activity. Compared to commonly used baseline models, our proposed models achieved higher performance with modest training data and computational costs. These findings raise a potential strategy for approaching tonal language speech restoration.

Technique of improving speech restoration from spectrograms of short windows for the Griffin–Lim algorithm

Technique of improving speech restoration from spectrograms of short windows for the Griffin–Lim algorithm

Using Automatic Speech Recognition to Measure the Intelligibility of Speech Synthesized from Brain Signals.

Brain-computer interfaces (BCIs) can potentially restore lost function in patients with neurological injury. A promising new application of BCI technology has focused on speech restoration. One approach is to synthesize speech from the neural correlates of a person who cannot speak, as they attempt to do so. However, there is no established gold-standard for quantifying the quality of BCI-synthesized speech. Quantitative metrics, such as applying correlation coefficients between true and decoded speech, are not applicable to anarthric users and fail to capture intelligibility by actual human listeners; by contrast, methods involving people completing forced-choice multiple-choice questionnaires are imprecise, not practical at scale, and cannot be used as cost functions for improving speech decoding algorithms. Here, we present a deep learning-based "AI Listener" that can be used to evaluate BCI speech intelligibility objectively, rapidly, and automatically. We begin by adapting several leading Automatic Speech Recognition (ASR) deep learning models - Deepspeech, Wav2vec 2.0, and Kaldi - to suit our application. We then evaluate the performance of these ASRs on multiple speech datasets with varying levels of intelligibility, including: healthy speech, speech from people with dysarthria, and synthesized BCI speech. Our results demonstrate that the multiple-language ASR model XLSR-Wav2vec 2.0, trained to output phonemes, yields superior performance in terms of speech transcription accuracy. Notably, the AI Listener reports that several previously published BCI output datasets are not intelligible, which is consistent with human listeners.

A multi-hole resonator enhanced acoustic energy harvester for ultra-high electrical output and machine-learning-assisted intelligent voice sensing

Acoustic energy harvesters are promising platforms for noise-to-electricity conversion in distributed environmental power supply and sound sensing in intuitive voice user interface. However, their efficacy in real-world applications is frustrated by the low electric output. We here report a multi-hole acoustic resonator enhanced triboelectric nanogenerator (MHAR-TENG) that couples a perforated plate resonator and a pressure differential acoustic receiver. It generated open-circuit voltage of 347 V, short-circuit current of 95 µA, and record-high power density of 8.9 W/m2. The numerical simulation confirmed the sound pressure amplification for enhanced output performance. Besides, the high signal-to-noise ratio and precise high-frequency response of electrical signals enabled the high-quality restoration of music and human speech. Assisted by machine learning, the MHAR-TENG realized self-powered voiceprint identification and emotion recognition with a remarkably improved accuracy over 90 %. Our MHAR-TENG holds great potential for efficient acoustic-to-electrical conversion in the fields of noise recycling, biometric authentication, and human-machine interaction.

PROSTHODONTIC REHABILITATION OF HEMIMAXILLECTOMY PATIENT WITH CAST PARTIAL DENTURE AND HOLLOW BULB OBTURATOR: A CASE REPORT

Obturator prosthesis is required for the restoration of speech, deglutition, and improvement of esthetics in patient after maxillectomy. The presence of oral tumors necessitates the surgical removal of all or part of the maxilla, leaving the patient with a defect that compromises the integrity and function of the oral cavity. The maxillofacial Prosthodontist, as a member of the surgical team, plays an important role in the recovery and rehabilitation of maxillectomy patient by fabricating and placing a surgical obturator. This clinical report describes the use of definitive hollow bulb obturator with cast partial denture framework for the treatment of a 65-year-oldpatient with hemimaxillectomy. This technique will improve the speech, mastication, swallowing and esthetics for the patient.

Real-Time Speech Enhancement Based on Convolutional Recurrent Neural Network

Speech enhancement is the task of taking a noisy speech input and producing an enhanced speech output. In recent years, the need for speech enhancement has been increased due to challenges that occurred in various applications such as hearing aids, Automatic Speech Recognition (ASR), and mobile speech communication systems. Most of the Speech Enhancement research work has been carried out for English, Chinese, and other European languages. Only a few research works involve speech enhancement in Indian regional Languages. In this paper, we propose a two-fold architecture to perform speech enhancement for Tamil speech signal based on convolutional recurrent neural network (CRN) that addresses the speech enhancement in a real-time single channel or track of sound created by the speaker. In the first stage mask based long short-term memory (LSTM) is used for noise suppression along with loss function and in the second stage, Convolutional Encoder-Decoder (CED) is used for speech restoration. The proposed model is evaluated on various speaker and noisy environments like Babble noise, car noise, and white Gaussian noise. The proposed CRN model improves speech quality by 0.1 points when compared with the LSTM base model and also CRN requires fewer parameters for training. The performance of the proposed model is outstanding even in low Signal to Noise Ratio (SNR).