Silent Speech Recognition Research Articles

SVIT‐SSR: A sEMG‐based vision transformer approach for silent speech recognition

SVIT‐SSR: A sEMG‐based vision transformer approach for silent speech recognition

A simplified adversarial architecture for cross-subject silent speech recognition using electromyography

Objective. The decline in the performance of electromyography (EMG)-based silent speech recognition is widely attributed to disparities in speech patterns, articulation habits, and individual physiology among speakers. Feature alignment by learning a discriminative network that resolves domain offsets across speakers is an effective method to address this problem. The prevailing adversarial network with a branching discriminator specializing in domain discrimination renders insufficiently direct contribution to categorical predictions of the classifier.Approach. To this end, we propose a simplified discrepancy-based adversarial network with a streamlined end-to-end structure for EMG-based cross-subject silent speech recognition. Highly aligned features across subjects are obtained by introducing a Nuclear-norm Wasserstein discrepancy metric on the back end of the classification network, which could be utilized for both classification and domain discrimination. Given the low-level and implicitly noisy nature of myoelectric signals, we devise a cascaded adaptive rectification network as the front-end feature extraction network, adaptively reshaping the intermediate feature map with automatically learnable channel-wise thresholds. The resulting features effectively filter out domain-specific information between subjects while retaining domain-invariant features critical for cross-subject recognition.Main results. A series of sentence-level classification experiments with 100 Chinese sentences demonstrate the efficacy of our method, achieving an average accuracy of 89.46% tested on 40 new subjects by training with data from 60 subjects. Especially, our method achieves a remarkable 10.07% improvement compared to the state-of-the-art model when tested on 10 new subjects with 20 subjects employed for training, surpassing its result even with three times training subjects.Significance. Our study demonstrates an improved classification performance of the proposed adversarial architecture using cross-subject myoelectric signals, providing a promising prospect for EMG-based speech interactive application.

Lipwatch: Enabling Silent Speech Recognition on Smartwatches using Acoustic Sensing

Silent Speech Interfaces (SSI) on mobile devices offer a privacy-friendly alternative to conventional voice input methods. Previous research has primarily focused on smartphones. In this paper, we introduce Lipwatch, a novel system that utilizes acoustic sensing techniques to enable SSI on smartwatches. Lipwatch leverages the inaudible waves emitted by the watch's speaker to capture lip movements and then analyzes the echo to enable SSI. In contrast to acoustic sensing-based SSI on smartphones, our development of Lipwatch takes into full consideration the specific scenarios and requirements associated with smartwatches. Firstly, we elaborate a wake-up-free mechanism, allowing users to interact without the need for a wake-up phrase or button presses. The mechanism utilizes the inertial sensors on the smartwatch to detect gestures, in combination with acoustic signals that detecting lip movements to determine whether SSI should be activated. Secondly, we design a flexible silent speech recognition mechanism that explores limited vocabulary recognition to comprehend a broader range of user commands, even those not present in the training dataset, relieving users from strict adherence to predefined commands. We evaluate Lipwatch on 15 individuals using a set of the 80 most common interaction commands on smartwatches. The system achieves a Word Error Rate (WER) of 13.7% in user-independent test. Even when users utter commands containing words absent in the training set, Lipwatch still demonstrates a remarkable 88.7% top-3 accuracy. We implement a real-time version of Lipwatch on a commercial smartwatch. The user study shows that Lipwatch can be a practical and promising option to enable SSI on smartwatches.

Ecofriendly and high-performance flexible pressure sensor derived from natural plant materials for intelligent audible and silent speech recognition

Flexible pressure sensors (FPSs) are widely used in human–computer interaction (HCI) and medical diagnoses, but their further development faces several challenges. Integrating artificial intelligence (AI) with FPSs is indispensable for advancing the cognitive capabilities of HCI systems, but the FPS must also be ecofriendly, cost-effective, and high-performance to meet the demands of environmental protection and sustainable development. Here, a biodegradable capacitive FPS is presented that is fabricated from natural plant materials (i.e., cornstarch and lignocellulose) while exhibiting remarkable attributes such as high sensitivity (2.427 kPa−1 at 0–2 kPa), rapid response/recovery time (20/24 ms), extremely low detection limit (∼10 Pa), outstanding stability (> 6000 fatigue tests), excellent biocompatibility (survival rate exceeding 99 % in cell viability assays), and prominent biodegradability (complete degradation within 24 h). Prepared sensors were integrated with optimized AI algorithms, embedded systems, and wireless transmission technologies to develop a dual-functional (i.e., audible and silent) speech recognition system that achieved recognition rates exceeding 95 %. The developed sensor is expected to play a crucial role in future HCI and medical assistance applications.

An novel SDA-CNN few shot domain adaptation framework for silent speech recognition

In BCI (brain-computer interface) applications, it is difficult to obtain enough well-labeled EEG data because of the expensive annotation and time-consuming data capture procedure. Conventional classification techniques that repurpose EEG data across domains and subjects lead to significant decreases in silent speech recognition classification accuracy. This research provides a supervised domain adaptation using Convolutional Neural Network framework (SDA-CNN) to tackle this problem. The objective is to provide a solution for the distribution divergence issue in the categorization of speech recognition across domains. The suggested framework involves taking raw EEG data and deriving deep features from it and the proposed feature selection method also retrieves the statistical features from the corresponding channels. Moreover, it attempts to minimize the distribution divergence caused by variations in people and settings by aligning the correlation of both the source and destination EEG characteristic dissemination. In order to obtain minimal feature distribution divergence and discriminative classification performance, the last stage entails simultaneously optimizing the loss of classification and adaption loss. The usefulness of the suggested strategy in reducing distributed divergence among the source and target Electroencephalography (EEG) data is demonstrated by extensive experiments carried out on KaraOne datasets. The suggested method achieves an average accuracy for classification of 87.4% for single-subject classification and a noteworthy average class accuracy of 88.6% for cross-subject situations, which shows that it surpasses existing cutting-edge techniques in thinking tasks. Regarding the speaking task, the model’s median classification accuracy for single-subject categorization is 86.8%, while its average classification accuracy for cross-subject classification is 87.8%. These results underscore the innovative approach of SDA-CNN to mitigating distribution discrepancies while optimizing classification performance, offering a promising avenue to enhance accuracy and adaptability in brain-computer interface applications.

Design and implementation of a silent speech recognition system based on sEMG signals: A neural network approach

Silent speech recognition (SSR) is a system that can recognize verbal expressions when speech signals are not accessible. It has found versatile applications not only in helping those who suffer a speech impediment, but also in human–computer interaction. In this paper, we propose an SSR system based on surface electromyography (sEMG) signals. We first design a Mandarin corpus which contains 135-class utterances and collect 8-channel sEMG data from 12 subjects. After preprocessing the raw sEMG signals, we extract a time–frequency hybrid feature map for each sample. Then, we use two data augmentation strategies to obtain more data in order to improve the performance of our model. Finally, we test our proposed models on two tasks: the multi-subject mixed classification task and the single subject classification task. We use a 5-fold cross-validation to evaluate our model on both tasks. The GRU model performs best, achieving a recognition accuracy of 88.01% on the multi-subject mixed classification task and a maximum recognition accuracy of 97.19% on the single subject classification task. Experiments results demonstrate the effectiveness of our proposed system.

Data augmentation method based on three-dimensional measurement for silent speech recognition

Data augmentation method based on three-dimensional measurement for silent speech recognition

A A Novel BCI - based Silent Speech Recognition using Hybrid Feature Extraction Techniques and Integrated Stacking Classifier

A A Novel BCI - based Silent Speech Recognition using Hybrid Feature Extraction Techniques and Integrated Stacking Classifier

Silent speech recognition using data augmentation based on a three-dimensional lip model

In this study, we proposed a data augmentation method using a three-dimensional model of a speaker's face for machine lip reading, which estimates the content of speech without using speech data. The proposed method converted a speaker's face into a three-dimensional model using DECA (Detailed Expression Capture and Animation), and generated a large amount of learning data by rotating the three-dimensional model in different directions. Then we obtained the facial features around the lips as time series data using dlib, which is an OpenCV module. The time series data is fed into a recognition model. We introduced end-to-end recognition model which is used in continuous speech recognition. The recognition model is based on DeepSpeech2, and we modified the filter size of CNN (convolutional neural network) and the number of layers of BiGRU (bidirectional gated recurrent unit) to adapt to our goal. We evaluated the proposed method on ten ordinary Japanese words. We used the phoneme error rate as the evaluation index. As a result of the evaluation, the error rate of about 0.32 was achieved even for data where the speaker of the evaluation data was not included in the speaker of the training data.

Analyzing lower half facial gestures for lip reading applications: Survey on vision techniques

Lip reading has gained popularity due to the proliferation of emerging real-world applications. This article provides a comprehensive review of benchmark datasets available for lip-reading applications and pioneering works that analyze lower facial cues for lip-reading applications. A comprehensive review of lip reading applications is broadly classified into five distinct applications: Lip Reading Biometrics (LRB), Audio Visual Speech Recognition (AVSR), Silent Speech Recognition (SSR), Voice from Lips, and Lip HCI (Human–computer interaction). LRB entails extensive research in the fields of authentication and liveness detection. AVSR covers key findings that have contributed significantly to applications such as voice assistants, video-to-text transcription, hearing aids, and pronunciation-correcting systems. SSR analyzes the efforts made for silent-video-to-text transcription and surveillance camera applications. The voice from lips section discusses applications such as voice for the voiceless and vision-infused speech inpainting. In lip HCI, LR-HCI for smartphones, smart TVs, computers, robots, and musical instruments is reviewed in detail. Comprehensive coverage is given to cutting-edge techniques in computer vision, signal processing, machine learning, and deep learning. The advancements that aid the system in learning to lip-read and authenticate lip gestures, generate text transcription, synthesize voice based on lip movements, and control systems via lip movements (lip HCI) are covered. The work concludes by highlighting the limitations of existing frameworks, the road maps of each application illustrating the evolution of techniques employed over time, and future research avenues in lip-reading applications.

MagTrack: A Wearable Tongue Motion Tracking System for Silent Speech Interfaces.

Current electromagnetic tongue tracking devices are not amenable for daily use and thus not suitable for silent speech interface and other applications. We have recently developed MagTrack, a novel wearable electromagnetic articulograph tongue tracking device. This study aimed to validate MagTrack for potential silent speech interface applications. We conducted two experiments: (a) classification of eight isolated vowels in consonant-vowel-consonant form and (b) continuous silent speech recognition. In these experiments, we used data from healthy adult speakers collected with MagTrack. The performance of vowel classification was measured by accuracies. The continuous silent speech recognition was measured by phoneme error rates. The performance was then compared with results using data collected with commercial electromagnetic articulograph in a prior study. The isolated vowel classification using MagTrack achieved an average accuracy of 89.74% when leveraging all MagTrack signals (x, y, z coordinates; orientation; and magnetic signals), which outperformed the accuracy using commercial electromagnetic articulograph data (only y, z coordinates) in our previous study. The continuous speech recognition from two subjects using MagTrack achieved phoneme error rates of 73.92% and 66.73%, respectively. The commercial electromagnetic articulograph achieved 64.53% from the same subject (66.73% using MagTrack data). MagTrack showed comparable results with the commercial electromagnetic articulograph when using the same localized information. Adding raw magnetic signals would improve the performance of MagTrack. Our preliminary testing demonstrated the potential for silent speech interface as a lightweight wearable device. This work also lays the foundation to support MagTrack's potential for other applications including visual feedback-based speech therapy and second language learning.

Silent Speech Recognition Based on High-Density Surface Electromyogram Using Hybrid Neural Networks

This article presents a silent speech recognition approach based on high-density (HD) surface electromyogram (sEMG) using hybrid neural networks that support anomaly detection. In the hybrid networks, both a convolutional long short-term memory module and an autoencoder module were designed to extract discriminative spatio-temporal features and potentially identify any anomaly patterns, respectively. To verify the effectiveness of the proposed method, experimental data were recorded using HD-sEMG arrays with 64 channels from 11 subjects subvocalizing 33 Chinese words and articulating 9 anomaly patterns. The proposed method significantly outperformed other comparison methods ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</i> < 0.05) and achieved the highest anomaly detection rate of 90.61% while maintaining a high level of target word-pattern classification accuracy of 82.30%. These findings demonstrate the effectiveness of the proposed method for improving the robustness of the SSR approach based on HD-sEMG recordings against anomaly muscular activities. This article also provides a novel solution for building practical and robust sEMG-based SSR systems with broad applications, such as instant messaging and human-computer interaction.

MSilent

Silent speech recognition (SSR) allows users to speak to the device without making a sound, avoiding being overheard or disturbing others. Compared to the video-based approach, wireless signal-based SSR can work when the user is wearing a mask and has fewer privacy concerns. However, previous wireless-based systems are still far from well-studied, e.g. they are only evaluated in corpus with highly limited size, making them only feasible for interaction with dozens of deterministic commands. In this paper, we present mSilent, a millimeter-wave (mmWave) based SSR system that can work in the general corpus containing thousands of daily conversation sentences. With the strong recognition capability, mSilent not only supports the more complex interaction with assistants, but also enables more general applications in daily life such as communication and input. To extract fine-grained articulatory features, we build a signal processing pipeline that uses a clustering-selection algorithm to separate articulatory gestures and generates a multi-scale detrended spectrogram (MSDS). To handle the complexity of the general corpus, we design an end-to-end deep neural network that consists of a multi-branch convolutional front-end and a Transformer-based sequence-to-sequence back-end. We collect a general corpus dataset of 1,000 daily conversation sentences that contains 21K samples of bi-modality data (mmWave and video). Our evaluation shows that mSilent achieves a 9.5% average word error rate (WER) at a distance of 1.5m, which is comparable to the performance of the state-of-the-art video-based approach. We also explore deploying mSilent in two typical scenarios of text entry and in-car assistant, and the less than 6% average WER demonstrates the potential of mSilent in general daily applications.

Force-induced ion generation in zwitterionic hydrogels for a sensitive silent-speech sensor

Human-sensitive mechanosensation depends on ionic currents controlled by skin mechanoreceptors. Inspired by the sensory behavior of skin, we investigate zwitterionic hydrogels that generate ions under an applied force in a mobile-ion-free system. Within this system, water dissociates as the distance between zwitterions reduces under an applied pressure. Meanwhile, zwitterionic segments can provide migration channels for the generated ions, significantly facilitating ion transport. These combined effects endow a mobile-ion-free zwitterionic skin sensor with sensitive transduction of pressure into ionic currents, achieving a sensitivity up to five times that of nonionic hydrogels. The signal response time, which relies on the crosslinking degree of the zwitterionic hydrogel, was ~38 ms, comparable to that of natural skin. The skin sensor was incorporated into a universal throat-worn silent-speech recognition system that transforms the tiny signals of laryngeal mechanical vibrations into silent speech.

Decoding Silent Speech Based on High-Density Surface Electromyogram Using Spatiotemporal Neural Network

Finer-grained decoding at a phoneme or syllable level is a key technology for continuous recognition of silent speech based on surface electromyogram (sEMG). This paper aims at developing a novel syllable-level decoding method for continuous silent speech recognition (SSR) using spatio-temporal end-to-end neural network. In the proposed method, the high-density sEMG (HD-sEMG) was first converted into a series of feature images, and then a spatio-temporal end-to-end neural network was applied to extract discriminative feature representations and to achieve syllable-level decoding. The effectiveness of the proposed method was verified with HD-sEMG data recorded by four pieces of 64-channel electrode arrays placed over facial and laryngeal muscles of fifteen subjects subvocalizing 33 Chinese phrases consisting of 82 syllables. The proposed method outperformed the benchmark methods by achieving the highest phrase classification accuracy (97.17 ± 1.53%, ), and lower character error rate (3.11 ± 1.46%, ). This study provides a promising way of decoding sEMG towards SSR, which has great potential applications in instant communication and remote control.

Extracting Spatial Muscle Activation Patterns in Facial and Neck Muscles for Silent Speech Recognition Using High-Density sEMG

Human is capable of producing versatile and precise speech owing to the complex neuromuscular systems that control the movement of facial and neck muscle groups. Silent speech recognition based on surface electromyography (sEMG) has been widely researched, which enables speech communication in the absence of an audible acoustic signal. However, the differences in the activities of facial and neck muscle groups under different phoneme pronunciation conditions have not been well investigated. The present work aims to quantify the macroscopic spatial patterns of the muscle activations at the phoneme level during the movement of facial and neck speech articulator muscle groups (SAMGs) using 320-channel high-density (HD) electrode arrays. 10 subjects performed 14 vowel speech tasks and 15 consonant speech tasks in audible speech and silent speech modes, respectively. Specifically, the root-mean-square heat maps of sEMG signals were used to characterize the SAMGs activation patterns. The results show that the muscle activation regions are overlapped under different phoneme pronunciation conditions and symmetric between the left and right sides of face and neck with the average correlation coefficient > 0.9. In addition, the best result can achieve a classification accuracy of 85.78% for consonant and 79.42% for vowel in silent speech mode. The overall results can provide a reference to the hardware or electrode design of portable device for practical use in future work.

Silent Speech Recognition Based on Surface Electromyography Using a Few Electrode Sites Under the Guidance From High-Density Electrode Arrays

Although surface electromyogram recorded from high-density electrode array is believed to carry sufficient spatial information that can benefit the decoding of motor intentions, the complexity of using the array hindered its widespread applications especially in wearable devices. This study is aimed to develop a non-acoustic modality of silent speech recognition that transfers knowledge learned from high-density array to a system using a few channels, with both high portability and performance. A convolutional neural network was established for recognizing a vocabulary of 33 Chinese words during subvocal speech production. The network was trained by the data recorded from face and neck muscles using two arrays with 64 channels in the source domain. Then it was calibrated through a transfer learning approach to grant its adaption to a new target domain with the data recorded by 8 separated electrodes, while its good capability of characterizing subvocal speech word patterns is expected to be maintained. The proposed method significantly outperformed three common classification approaches and the baseline approach without transfer learning (a network trained with data just from the target domain). Under conditions of electrode shift and cross-user variability, it still obtained performance improvements. The method is demonstrated to be viable for transfer learning across domains of electrode settings and it facilitates to improve the performance of silent speech recognition systems using separate electrode sites under the guidance from high-density of arrays.

Decoding silent speech commands from articulatory movements through soft magnetic skin and machine learning.

Silent speech interfaces have been pursued to restore spoken communication for individuals with voice disorders and to facilitate intuitive communications when acoustic-based speech communication is unreliable, inappropriate, or undesired. However, the current methodology for silent speech faces several challenges, including bulkiness, obtrusiveness, low accuracy, limited portability, and susceptibility to interferences. In this work, we present a wireless, unobtrusive, and robust silent speech interface for tracking and decoding speech-relevant movements of the temporomandibular joint. Our solution employs a single soft magnetic skin placed behind the ear for wireless and socially acceptable silent speech recognition. The developed system alleviates several concerns associated with existing interfaces based on face-worn sensors, including a large number of sensors, highly visible interfaces on the face, and obtrusive interconnections between sensors and data acquisition components. With machine learning-based signal processing techniques, good speech recognition accuracy is achieved (93.2% accuracy for phonemes, and 87.3% for a list of words from the same viseme groups). Moreover, the reported silent speech interface demonstrates robustness against noises from both ambient environments and users' daily motions. Finally, its potential in assistive technology and human-machine interactions is illustrated through two demonstrations - silent speech enabled smartphone assistants and silent speech enabled drone control.

IR-UWB Radar-Based Contactless Silent Speech Recognition of Vowels, Consonants, Words, and Phrases

IR-UWB Radar-Based Contactless Silent Speech Recognition of Vowels, Consonants, Words, and Phrases

Silent Speech Recognition with Strain Sensors and Deep Learning Analysis of Directional Facial Muscle Movement.

Silent communication based on biosignals from facial muscle requires accurate detection of its directional movement and thus optimally positioning minimum numbers of sensors for higher accuracy of speech recognition with a minimal person-to-person variation. So far, previous approaches based on electromyogram or pressure sensors are ineffective in detecting the directional movement of facial muscles. Therefore, in this study, high-performance strain sensors are used for separately detecting x- and y-axis strain. Directional strain distribution data of facial muscle is obtained by applying three-dimensional digital image correlation. Deep learning analysis is utilized for identifying optimal positions of directional strain sensors. The recognition system with four directional strain sensors conformably attached to the face shows silent vowel recognition with 85.24% accuracy and even 76.95% for completely nonobserved subjects. These results show that detection of the directional strain distribution at the optimal facial points will be the key enabling technology for highly accurate silent speech recognition.