Silent Speech Research Articles

Toward Silent-Speech Control of Consumer Wearables

Systems that recognize silent speech can enable fast, hands-free communication. Two prototypes let users control Google Glass with tongue movements and jaw gestures, requiring no additional equipment except a tongue-mounted magnet or consumer earphones augmented with embedded proximity sensors.

Integrating Dynamic Information with Multi-stream HMM in Ultrasound-based Silent Speech Interface

Integrating Dynamic Information with Multi-stream HMM in Ultrasound-based Silent Speech Interface

A review and meta-analysis of tongue and palatal implants for post carcinoma corrections

Tongue and palatal implants, prosthesis are recently being used for improvising patients medical condition. This paper reviews recent advances and research to overcome post carcinoma treatments for aiding vital processes such as deglutition, mastication and speech, using palatal augmentation prosthesis, food guiding prosthesis and tongue replacement prosthesis. Silent speech can be overcome by using magnetic implants and sensors. Restoration of voice box post laryngectomy and glossectomy can be achieved by Electrolarynx (EL) and Transoesophageal Voice Prosthesis (TEP). The success of tongue and palatal implants are discussed in the clinical report.

Articulatory constraints on spontaneous entrainment between speech and manual gesture

The present study examined the extent to which speech and manual gestures spontaneously entrain in a non-communicative task. Participants had to repeatedly utter nonsense /CV/ syllables while continuously moving the right index finger in flexion/extension. No instructions to coordinate were given. We manipulated the type of syllable uttered (/ba/ vs. /sa/), and vocalization (phonated vs. silent speech). Assuming principles of coordination dynamics, a stronger entrainment between the fingers oscillations and the jaw motion was predicted (1) for /ba/, due to expected larger amplitude of jaw motion and (2) in phonated speech, due to the auditory feedback. Fifteen out of twenty participants showed simple ratios of speech to finger cycles (1:1, 1:2 or 2:1). In contrast with our predictions, speech–gesture entrainment was stronger when vocalizing /sa/ than /ba/, also more widely distributed on an in-phase mode. Furthermore, results revealed a spatial anchoring and an increased temporal variability in jaw motion when producing /sa/. We suggested that this indicates a greater control of the speech articulators for /sa/, making the speech performance more receptive to environmental forces, resulting in the greater entrainment observed to gesture oscillations. The speech–gesture coordination was maintained in silent speech, suggesting a somatosensory basis for their endogenous coupling.

Repetitive speech elicits widespread deactivation in the human cortex: the "Mantra" effect?

BackgroundMantra (prolonged repetitive verbal utterance) is one of the most universal mental practices in human culture. However, the underlying neuronal mechanisms that may explain its powerful emotional and cognitive impact are unknown. In order to try to isolate the effect of silent repetitive speech, which is used in most commonly practiced Mantra meditative practices, on brain activity, we studied the neuronal correlates of simple repetitive speech in nonmeditators – that is, silent repetitive speech devoid of the wider context and spiritual orientations of commonly practiced meditation practices.MethodsWe compared, using blood oxygenated level-dependent (BOLD) functional magnetic resonance imaging (fMRI), a simple task of covertly repeating a single word to resting state activity, in 23 subjects, none of which practiced meditation before.ResultsWe demonstrate that the repetitive speech was sufficient to induce a widespread reduction in BOLD signal compared to resting baseline. The reduction was centered mainly on the default mode network, associated with intrinsic, self-related processes. Importantly, contrary to most cognitive tasks, where cortical-reduced activation in one set of networks is typically complemented by positive BOLD activity of similar magnitude in other cortical networks, the repetitive speech practice resulted in unidirectional negative activity without significant concomitant positive BOLD. A subsequent behavioral study showed a significant reduction in intrinsic thought processes during the repetitive speech condition compared to rest.ConclusionsOur results are compatible with a global gating model that can exert a widespread induction of negative BOLD in the absence of a corresponding positive activation. The triggering of a global inhibition by the minimally demanding repetitive speech may account for the long-established psychological calming effect associated with commonly practiced Mantra-related meditative practices.

Abraham Abulafia's Mystical Theology of the Divine Name and its Philosophical Revision in Walter Benjamin

The thirteenth century kabbalist Abraham Abulafia held Hebrew to be the divine language, designed by God as an ontological aspect of reality. Through meditating on and deconstructing names into their letters, one could then engage in the process of reunifying reality into the primordial Name of God, the Tetragrammaton. This paper offers an original analysis, aligning Abulafia with Walter Benjamin's writings on language wherein the world speaks itself phenomenally to human beings, who must take an active role in translating this silent speech. Here the human, as name-giver, is finalizer of reality; by translating base phenomena into named entities humanity elevates them into a new realm, beyond the merely real into the true. This epistemic-soteriological theory speaks to common speculative questions about language and the world, offering a new way of understanding how human beings can come to know the world as-it-is, and as-it-should-be; a way that is philosophical, yet based firmly in Medieval Jewish theology.

Statistical conversion of silent articulation into audible speech using full-covariance HMM

This article investigates the use of statistical mapping techniques for the conversion of articulatory movements into audible speech with no restriction on the vocabulary, in the context of a silent speech interface driven by ultrasound and video imaging. As a baseline, we first evaluated the GMM-based mapping considering dynamic features, proposed by Toda et al. (2007) for voice conversion. Then, we proposed a ‘phonetically-informed’ version of this technique, based on full-covariance HMM. This approach aims (1) at modeling explicitly the articulatory timing for each phonetic class, and (2) at exploiting linguistic knowledge to regularize the problem of silent speech conversion. Both techniques were compared on continuous speech, for two French speakers (one male, one female). For modal speech, the HMM-based technique showed a lower spectral distortion (objective evaluation). However, perceptual tests (transcription and XAB discrimination tests) showed a better intelligibility of the GMM-based technique, probably related to its less fluctuant quality. For silent speech, a perceptual identification test revealed a better segmental intelligibility for the HMM-based technique on consonants.

Dispositional mindfulness and semantic integration of emotional words: Evidence from event-related brain potentials

Initial research shows that mindfulness training can enhance attention and modulate the affective response. However, links between mindfulness and language processing remain virtually unexplored despite the prominent role of overt and silent negative ruminative speech in depressive and anxiety-related symptomatology. Here, we measured dispositional mindfulness and recorded participants’ event-related brain potential responses to positive and negative target words preceded by words congruent or incongruent with the targets in terms of semantic relatedness and emotional valence. While the low mindfulness group showed similar N400 effect pattern for positive and negative targets, high dispositional mindfulness was associated with larger N400 effect to negative targets. This result suggests that negative meanings are less readily accessible in people with high dispositional mindfulness. Furthermore, high dispositional mindfulness was associated with reduced P600 amplitudes to emotional words, suggesting less post-analysis and attentional effort which possibly relates to a lower inclination to ruminate. Overall, these findings provide initial evidence on associations between modifications in language systems and mindfulness.

Decoding Silent Speech in Japanese from Single Trial EEGS: Preliminary Results

We propose a new scheme for speaker-dependent silent speech recognition systems (SSRSs) using both single-trial electroencephalograms (EEGs) scalp-recorded and speech signals measured during overtly and covertly speaking “janken” and “season” in Japanese. This scheme consists of two phases. The learning phase specifies a Kalman filter using spectrograms of the speech signals and independent components (ICs), whose equivalent current dipole source localization (ECDL) solutions were located mainly at the Broca’s area, of the EEGs during the actual speech. In case of the “season” task, the speech signals were transformed into vowel and consonant sequences, and these relationships were learned by hidden Markov model (HMM) with Gaussian mixture densities. The decoding phase predicts spectrograms for the silent “janken” and “season” using the Kalman filter with the EEGs during the silent speech. For the silent “season”, the predicted spectrograms were inputted to the HMM, and which “season” was silently spoken was determined by the maximal log-likelihood among each HMM. Our preliminary results as training steps are as follows: the silent “jankens” were correctly discriminated; the silent “season”-HMMs worked well, suggesting that this scheme might be applied to the discrimination between all the pairs of the hiraganas

Imitatio Christi: The Poetics of Piety in Early Modern England by Nandra Perry (review)

Reviewed by: Imitatio Christi: The Poetics of Piety in Early Modern England by Nandra Perry Rebecca Huffman Nandra Perry, Imitatio Christi: The Poetics of Piety in Early Modern England (Notre Dame: University of Notre Dame Press 2014) 288 pp. Nandra Perry’s holistic approach to literary and religious imitatio from Sidney to Milton persuasively explores questions and categories of embodiment, kingship, private/public spheres, and the instability of language while also, in a most worthwhile step, seeking “to move more freely across the period, gender, generic, and confessional boundaries that often delimit treatments of early modern English piety” (11). This wide methodological lens enables Perry to draw together an ideologically diverse group of texts for case studies, all connected back to Sidney, to determine what good imitatio means in varying contexts and how it adapts over the course of this period. Perry engages questions of imitatio’s literariness alongside its religious purposes and tensions to work through its role in a variety of sixteenth- and seventeenth-century orthodoxies and heterodoxies. In the introduction, she uses Erasmus’s writings on the subject to set out a debate that will remain prominent throughout the book: what differentiates imitatio from mimicry, or worse, idolatry? In the range of ideologies in the chapters that follow, this hazard is a shared motivator in developing models of good imitatio. Beginning with Sidney’s 1580 Defense of Poesy and Thomas Roger’s translation of Thomas à Kempis’s Imitation of Christ, Perry suggests that Rogers develops a Protestant form of imitatio Christi that is especially connected to Sidney’s vision of imitatio’s literary qualities. Both Rogers and Sidney imagine Protestantism on a transnational scale, where Rogers in his translation of Kempis foregrounds the experience and presence of the Word. In place of Kempis’s culminating depiction of the Eucharist, Rogers offers the Word as the Protestant community’s sustenance. Rogers must also deal with the inherent fallibility of human words and human imitatio in relation to the sacred Word as well as “language’s seemingly irredeemable fleshliness” (56). In this reading, Rogers grounds good imitatio in the text and in reading, where despite its fallibilities language could connect to the sacred. While Roger’s long career and the chapter’s subsequent exploration of his later writings show how his work on imitatio played out in an increasingly factionalized English Protestantism rather than his hoped-for transnational Protestant community, Sidney’s short career is succeeded by his own Protestant martyrology; here, Sidney’s martyred body and “personal imitatio Christi is the best measure of the distance and difference separating his innovating model of poetic imitation from” modern models of Shuger or Habermas. Perry contends that Sidney’s body marks both an end and a beginning in the period studied in this volume, where the instability of words and physical bodies would eventually close down “the dream of an organic connection between language and the sacred body visible” (63). The second chapter moves into more heterodox territory as it delves into notions of Christian eloquence and tensions surrounding the instability of language with the writings and Life of Elizabeth Tanfield Cary. Here and in the [End Page 280] following chapter, Perry’s movement across conventional boundaries is especially rewarding. Perry examines the ways in which The Tragedy of Mariam and The Lady Falkland: Her Life by One of Her Daughters offer other kinds of bodies (beyond Sidney’s male, aristocratic body) for discussions of incarnational poetics. Reading Mariam’s confrontation with Herod as a Reformation confrontation between believer and magistrate shows models of female piety that align with the meekness and stoicism advocated for and by Catholic martyrs. Intervening in scholarship that draws clear parallels between Cary herself and Mariam, Perry suggests that instead Graphina, whose privatized “silent speech” offers the kind of exemplum that sustained Catholicism in England more effectively than martyrdom, is a more fitting model. From Graphina to Cary herself and subsequently to the daughter who authored her biography, “the Life explores the epistemological and expressive outer limits of this cross-confessional, post-Reformation vision of imitatio Christi” (96). The final two chapters turn to kingship and one body’s imitatio in particular: Charles I in...

Embedded wavelet analysis of non-audible signals

The analysis of non-audible signals has gained a significant importance due to their many fields of application, among them, speech synthesis for people with speech disabilities. This analysis can be used to acquire information from the vocal apparatus without the need of speaking in order to produce a phonetic expression. The analysis of a Wavelet transformation of Spanish words recorded through a non-audible murmur microphone in order to achieve an embedded silent speech recognition system of Spanish language is proposed. A non-audible murmur microphone is used as sensor of non-vocal speech. Coding of the input data is done through a Wavelet transform using a fourth-order Daubechies function. The acquisition, processing and transmission system is applied through a STM32F4-Discovery evaluation board. The used vocabulary consists of command words aimed to control mobile robots or human-machine interfaces. The Wavelet transformation of four Spanish words, each of them having five independent samples, was accomplished. An analysis of the resulting data was performed, and features as average, peaks and frequency were distinguished. The processing of the signals is performed successfully and further work in speech activity detection and features classifiers is proposed.

Mouth State Detection From Low-Frequency Ultrasonic Reflection

This paper develops, simulates and experimentally evaluates a novel method based on non-contact low frequency (LF) ultrasound which can determine, from airborne reflection, whether the lips of a subject are open or closed. The method is capable of accurately distinguishing between open and closed lip states through the use of a low-complexity detection algorithm, and is highly robust to interfering audible noise. A novel voice activity detector is implemented and evaluated using the proposed method and shown to detect voice activity with high accuracy, even in the presence of high levels of background noise. The lip state detector is evaluated at a number of angles of incidence to the mouth and under various conditions of background noise. The underlying mouth state detection technique relies upon an inaudible LF ultrasonic excitation, generated in front of the face of a user, either reflecting back from their face as a simple echo in the closed mouth state or resonating inside the open mouth and vocal tract, affecting the spectral response of the reflected wave when the mouth is open. The difference between echo and resonance behaviours is used as the basis for automated lip opening detection, which implies determining whether the mouth is open or closed at the lips. Apart from this, potential applications include use in voice generation prosthesis for speech impaired patients, or as a hands-free control for electrolarynx and similar rehabilitation devices. It is also applicable to silent speech interfaces and may have use for speech authentication.

Super-Audible Voice Activity Detection

In this paper, reflected sound of frequency just above the audible range is used to detect speech activity. The active signal used is inaudible to humans, readily generated by the typical audio circuitry and components found in mobile telephones, and is robust to background sounds such as nearby voices. In use, the system relies upon a wideband excitation signal emitted from a loudspeaker located near the lips, which reflects from the mouth region and is then captured by a nearby microphone. The state of the lip opening is evaluated periodically by tracking the resonance patterns in the reflected excitation signal. When the lips are open, deep and complex resonances are formed as energy propagates into and then reflects out from the open mouth and vocal tract, with resonance depth being related to the open lip area. When the lips are closed, these resonance patterns are absent. The presence of the resonances can thus serve as a low complexity detection measure. The technique is evaluated for multiple users in terms of sensitivity to source placement and sensor placement. Voice activity detection performance using this measure is further evaluated in the presence of realistic wideband acoustic background noise, as well as artificially added noise. The system is shown to be relatively insensitive to sensor placement, highly insensitive to background noise, and able to achieve greater than 90% voice activity detection accuracy. The technique is even suitable when a subject is whispering in the presence of much louder multi-speaker babble. The technique has potential for speech-based systems operating in high noise environments as well as in silent speech interfaces, whisper-input systems and voice prostheses for speech-impaired users.

Correction: Effective Cerebral Connectivity during Silent Speech Reading Revealed by Functional Magnetic Resonance Imaging

Correction: Effective Cerebral Connectivity during Silent Speech Reading Revealed by Functional Magnetic Resonance Imaging

Vocal frequency estimation and voicing state prediction with surface EMG pattern recognition

The majority of laryngectomees use the electrolarynx as their primary mode of verbal communication after total laryngectomy surgery. However, the archetypal electrolarynx suffers from a monotonous tone and the inconvenience of requiring manual control. This paper presents the potential of pattern recognition to support electrolarynx use by predicting fundamental frequency (F0) and voicing state (VS) from surface EMG of the infrahyoid and suprahyoid muscles, as well as from a respiratory trace. In this study, surface EMG signals from the infrahyoid and suprahyoid muscle groups and respiratory trace were collected from 10 able-bodied, adult males (18–60years old). Participants performed three kinds of vocal tasks – tones, legatos and phrases. Signal features were extracted from the EMG and respiratory trace, and a Support Vector Machine (SVM) classifier with radial basis function kernels was employed to predict F0 and voicing state. An average root mean squared error of 2.81±0.6 semitones was achieved for the estimation of vocal frequency in the range of 90–360Hz. An average cross-validation (CV) accuracy of 78.05±6.3% was achieved for the prediction of voicing state from EMG and 65.24±7.8% from the respiratory trace. The proposed method has the advantage of being non-invasive compared with studies that relied on intramuscular electrodes (invasive), while still maintaining an accuracy above chance. Pattern classification of neck-muscle surface EMG has merit in the prediction of fundamental frequency and voicing state during vocalization, encouraging further study of automatic pitch modulation for electrolarynges and silent speech interfaces.

Tackling speaking mode varieties in EMG-based speech recognition.

An electromyographic (EMG) silent speech recognizer is a system that recognizes speech by capturing the electric potentials of the human articulatory muscles, thus enabling the user to communicate silently. After having established a baseline EMG-based continuous speech recognizer, in this paper, we investigate speaking mode variations, i.e., discrepancies between audible and silent speech that deteriorate recognition accuracy. We introduce multimode systems that allow seamless switching between audible and silent speech, investigate different measures which quantify speaking mode differences, and present the spectral mapping algorithm, which improves the word error rate (WER) on silent speech by up to 14.3% relative. Our best average silent speech WER is 34.7%, and our best WER on audibly spoken speech is 16.8%.

Alaryngeal Speech Enhancement Based on One-to-Many Eigenvoice Conversion

In this paper, we present novel speaking-aid systems based on one-to-many eigenvoice conversion (EVC) to enhance three types of alaryngeal speech: esophageal speech, electrolaryngeal speech, and body-conducted silent electrolaryngeal speech. Although alaryngeal speech allows laryngectomees to utter speech sounds, it suffers from the lack of speech quality and speaker individuality. To improve the speech quality of alaryngeal speech, alaryngeal-speech-to-speech (AL-to-Speech) methods based on statistical voice conversion have been proposed. In this paper, one-to-many EVC capable of flexibly controlling the converted voice quality by adapting the conversion model to given target natural voices is further implemented for the AL-to-Speech methods to effectively recover speaker individuality of each type of alaryngeal speech. These proposed systems are compared with each other from various perspectives. The experimental results demonstrate that our proposed systems are capable of effectively addressing the issues of alaryngeal speech, e.g., yielding significant improvements in speech quality of each type of alaryngeal speech.

Silent Speech Recognition with Arabic and English Words for Vocally Disabled Persons

This paper presents the results of our research in silent speech recognition (SSR) using Surface Electromyography (sEMG); which is the technology of recording the electric activation potentials of the human articulatory muscles by surface electrodes in order to recognize speech. Though SSR is still in the experimental stage, a number of potential applications seem evident. Persons who have undergone a laryngectomy, or older people for whom speaking requires a substantial effort, would be able to mouth (vocalize) words rather than actually pronouncing them. Our system has been trained with 30 utterances from each of the three subjects we had on a testing vocabulary of 4 phrases, and then tested for 15 new utterances that were not part of the training list. The system achieved an average of 91.11% word accuracy when using Support Vector Machine (SVM) classifier while the base language is English, and an average of 89.44% word accuracy using the Standard Arabic language.

Silent Speech Recognition with Arabic and English Words for Vocally Disabled Persons

Silent Speech Recognition with Arabic and English Words for Vocally Disabled Persons

Classification of silent speech using support vector machine and relevance vector machine

To provide speech prostheses for individuals with severe communication impairments, we investigated a classification method for brain computer interfaces (BCIs) using silent speech. Event-related potentials (ERPs) were recorded using scalp electrodes when five subjects imagined the vocalization of Japanese vowels, /a/, /i/, /u/, /e/, and /o/ in order and in random order, while the subjects remained silent and immobilized.For actualization, we tried to apply relevance vector machine (RVM) and RVM with Gaussian kernel (RVM-G) instead of support vector machine with Gaussian kernel (SVM-G) to reduce the calculation cost in the use of 19 channels, common special patterns (CSPs) filtering, and adaptive collection (AC). Results show that using RVM-G instead of SVM-G reduced the ratio of the number of efficient vectors to the number of training data from 97% to 55%. At this time, the averaged classification accuracies (CAs) using SVM-G and RVM-G were, respectively, 77% and 79%, showing no degradation. However, the calculation cost was more than that using SVM-G because RVM-G necessitates high calculation costs for optimization. Furthermore, results show that CAs using RVM-G were weaker than SVM-G when the training data were few. Additionally, results showed that nonlinear classification was necessary for silent speech classification.This paper serves as a beginning of feasibility study for speech prostheses using an imagined voice. Although classification for silent speech presents great potential, many feasibility problems remain.