Continuous Speech Recognition Research Articles

Deep Learning Based Lipreading for Video Captioning

Visual speech recognition, often referred to as lipreading, has garnered significant attention in recent years due to its potential applications in various fields such as human-computer interaction, accessibility technology, and biometric security systems. This paper explores the challenges and advancements in the field of lipreading, which involves deciphering speech from visual cues, primarily movements of the lips, tongue, and teeth. Despite being an essential aspect of human communication, lipreading presents inherent difficulties, especially in noisy environments or when contextual information is limited. The McGurk effect, where conflicting audio and visual cues lead to perceptual illusions, highlights the complexity of lipreading. Human lipreading performance varies widely, with hearing-impaired individuals achieving relatively low accuracy rates. Automating lipreading using machine learning techniques has emerged as a promising solution, with potential applications ranging from silent dictation in public spaces to biometric authentication systems. Visual speech recognition methods can be broadly categorized into those that focus on mimicking words and those that model visemes, visually distinguishable phonemes. While word-based approaches are suitable for isolated word recognition, viseme-based techniques are better suited for continuous speech recognition tasks. This study proposes a novel deep learning architecture for lipreading, leveraging Conv3D layers for spatiotemporal feature extraction and bidirectional LSTM layers for sequence modelling. The proposed model demonstrates significant improvements in lipreading accuracy, outperforming traditional methods on benchmark datasets. The practical implications of automated lipreading extend beyond accessibility technology to include biometric identity verification, security surveillance, and enhanced communication aids for individuals with hearing impairments. This paper provides insights into the advancements, challenges, and future directions of visual speech recognition research, paving the way for innovative applications in diverse domains.

Continuous lipreading based on acoustic temporal alignments

Visual speech recognition (VSR) is a challenging task that has received increasing interest during the last few decades. Current state of the art employs powerful end-to-end architectures based on deep learning which depend on large amounts of data and high computational resources for their estimation. We address the task of VSR for data scarcity scenarios with limited computational resources by using traditional approaches based on hidden Markov models. We present a novel learning strategy that employs information obtained from previous acoustic temporal alignments to improve the visual system performance. Furthermore, we studied multiple visual speech representations and how image resolution or frame rate affect its performance. All these experiments were conducted on the limited data VLRF corpus, a database which offers an audio-visual support to address continuous speech recognition in Spanish. The results show that our approach significantly outperforms the best results achieved on the task to date.

Convolutional Neural Networks to Facilitate the Continuous Recognition of Arabic Speech with Independent Speakers

Automatic speech recognition (ASR) is a field of research that focuses on the ability of computers to process and interpret speech feedback from humans and to provide the highest degree of accuracy in recognition. Speech is one of the simplest ways to convey a message in a basic context, and ASR refers to the ability of machines to process and accept speech data from humans with the greatest degree of accuracy. As the human-to-machine interface continues to evolve, speech recognition is expected to become increasingly important. However, the Arabic language has distinct features that set it apart from other languages, such as the dialect and the pronunciation of words. Until now, insufficient attention has been devoted to continuous Arabic speech recognition research for independent speakers with a limited database. This research proposed two techniques for the recognition of Arabic speech. The first uses a combination of convolutional neural network (CNN) and long short-term memory (LSTM) encoders, and an attention-based decoder, and the second is based on the Sphinx-4 recognizer, which includes pocket sphinx, base sphinx, and sphinx train, with various types and number of features to be extracted (filter bank and mel frequency cepstral coefficients (MFCC)) based on the CMU Sphinx tool, which generates a language model for different sentences spoken by different speakers. These approaches were tested on a dataset containing 7 hours of spoken Arabic from 11 Arab countries, covering the Levant, Gulf, and African regions, which make up the Arab world, and achieved promising results. CNN-LSTM achieved a word error rate (WER) of 3.63% using 120 features for filter bank and 4.04% WER using 39 features for MFCC, respectively, while the Sphinx-4 recognizer technique achieved 8.17% WER and an accuracy of 91.83% using 25 features for MFCC and 8 Gaussian mixtures, respectively, when tested on the same benchmark dataset.

Event-Driven Spiking Learning Algorithm Using Aggregated Labels.

Traditional spiking learning algorithm aims to train neurons to spike at a specific time or on a particular frequency, which requires precise time and frequency labels in the training process. While in reality, usually only aggregated labels of sequential patterns are provided. The aggregate-label (AL) learning is proposed to discover these predictive features in distracting background streams only by aggregated spikes. It has achieved much success recently, but it is still computationally intensive and has limited use in deep networks. To address these issues, we propose an event-driven spiking aggregate learning algorithm (SALA) in this article. Specifically, to reduce the computational complexity, we improve the conventional spike-threshold-surface (STS) calculation in AL learning by analytical calculating voltage peak values in spiking neurons. Then we derive the algorithm to multilayers by event-driven strategy using aggregated spikes. We conduct comprehensive experiments on various tasks including temporal clue recognition, segmented and continuous speech recognition, and neuromorphic image classification. The experimental results demonstrate that the new STS method improves the efficiency of AL learning significantly, and the proposed algorithm outperforms the conventional spiking algorithm in various temporal clue recognition tasks.

Real-Time Automatic Continuous Speech Recognition System for Kannada Language/Dialects

Real-Time Automatic Continuous Speech Recognition System for Kannada Language/Dialects

Artificial Intelligence Technology Enabling Innovation in Museum Public Cultural Service Models

Abstract In this paper, we constructed an artificial intelligence model of environment perception and human body perception, and after filtering the irrelevant information in the environment, we collected and analyzed the three-dimensional information of the environment to realize the intelligent perception of the environment. Aiming at the transient noise that may occur in the robot’s operating environment, a time threshold is added on the basis of the traditional double threshold endpoint detection algorithm, which effectively eliminates the influence of transient noise on endpoint detection and realizes non-contact continuous speech recognition. Then the data of the perceived human body was analyzed to understand the perception of the human body by artificial intelligence. Finally, a model for museum public cultural innovation services was constructed based on AI, and the effects of AI on public cultural services were analyzed. The results show that people’s overall perception of intelligent services is 4.305 points, and the perceived functionality, perceived ease of use, perceived pleasantness, and level of intelligence have a significant positive correlation with the quality of public cultural services, with β-values of 0.286, 0.206, 0.068, and 0.378, respectively, and p<0.05.

Retracted: English Phrase Speech Recognition Based on Continuous Speech Recognition Algorithm and Word Tree Constraints

Retracted: English Phrase Speech Recognition Based on Continuous Speech Recognition Algorithm and Word Tree Constraints

Frontier Research on Low-Resource Speech Recognition Technology.

With the development of continuous speech recognition technology, users have put forward higher requirements in terms of speech recognition accuracy. Low-resource speech recognition, as a typical speech recognition technology under restricted conditions, has become a research hotspot nowadays because of its low recognition rate and great application value. Under the premise of low-resource speech recognition technology, this paper reviews the research status of feature extraction and acoustic models, and conducts research on resource expansion. Especially in terms of the technical challenges faced by this technology, solutions are proposed, and future research directions are prospected.

English Pronunciation Quality Evaluation System Based on Continuous Speech Recognition Technology for Multi-Terminal

This paper presents a novel approach for evaluating the pronunciation quality of English speech using continuous speech recognition technology. The research focuses on the application of artificial intelligence in speech recognition, utilizing web browsers on various terminal devices such as computers, mobile phones, and tablets to allow users to read the provided text aloud. The web program captures audio input from the microphone, records it in MP3 format, and uploads it to the server. The server employs the Whisper model to transcribe the audio into semantic text, which is then compared with the displayed text. By calculating the semantic distance and assessing the accuracy of pronunciation, the system provides an evaluation of pronunciation quality, marking correct and incorrect words. To achieve real-time processing, the compact tiny model is employed, and further optimization is performed using Ctranslate 2, resulting in significant performance improvements.

Silent speech recognition using data augmentation based on a three-dimensional lip model

In this study, we proposed a data augmentation method using a three-dimensional model of a speaker's face for machine lip reading, which estimates the content of speech without using speech data. The proposed method converted a speaker's face into a three-dimensional model using DECA (Detailed Expression Capture and Animation), and generated a large amount of learning data by rotating the three-dimensional model in different directions. Then we obtained the facial features around the lips as time series data using dlib, which is an OpenCV module. The time series data is fed into a recognition model. We introduced end-to-end recognition model which is used in continuous speech recognition. The recognition model is based on DeepSpeech2, and we modified the filter size of CNN (convolutional neural network) and the number of layers of BiGRU (bidirectional gated recurrent unit) to adapt to our goal. We evaluated the proposed method on ten ordinary Japanese words. We used the phoneme error rate as the evaluation index. As a result of the evaluation, the error rate of about 0.32 was achieved even for data where the speaker of the evaluation data was not included in the speaker of the training data.

An Efficient Voice Authentication Approach Using Improved Deep Convolutional Neural Networks with LSTM Networks

Voice authentication and verification play an essential role in forensic science and security systems in today's technological environment. A challenging issue is making precise voice authentication and verification. This document aims to identify scammers using machine learning techniques to find the best speaker identification and classification combination. Traditionally, researchers have approached voice authentication and speech comparison using separate classification and comparison models. However, the accuracy of training with large amounts of data has made deep learning more popular. In this study, we predict speech recognition in speech datasets using an improved deep convolutional neural network (IDCNN) and long short-term memory (LSTM) network structure. The proposed approach entails three main steps: pretreatment, feature extraction, and classification. We could predict voice authentication and verification in this study with an accuracy of 99.3%. Aside from comparing classification performance, performance metrics for accuracy, sensitivity, and specificity were also computed. The newly introduced methods offer superior error rates, precision, F-score accuracy, and recall compared to conventional algorithms. This research is regarded as a pioneer in the field as a quick and effective way to enhance continuous speech recognition activity.

Swahili Speech Dataset Development and Improved Pre-training Method for Spoken Digit Recognition

Speech dataset is an essential component in building commercial speech applications. However, low-resource languages such as Swahili lack such a resource that is vital for spoken digit recognition. For languages where such resources exist, they are usually insufficient. Thus, pre-training methods have been used with external resources to improve continuous speech recognition. However, to the best of our knowledge, no study has investigated the effect of pre-training methods specifically for spoken digit recognition. This study aimed at addressing these problems. First, we developed a Swahili spoken digit dataset for Swahili spoken digit recognition. Then, we investigated the effect of cross-lingual and multi-lingual pre-training methods on spoken digit recognition. Finally, we proposed an effective language-independent pre-training method for spoken digit recognition. The proposed method has the advantage of incorporating target language data during the pre-training stage that leads to an optimal solution when using less training data. Experiments on Swahili (being developed), English, and Gujarati datasets show that our method achieves better performance compared with all the baselines listed in this study.

Deep convolutional neural networks-based features for Indonesian large vocabulary speech recognition

There are great interests in developing speech recognition using deep learning technologies due to their capability to model the complexity of pronunciations, syntax, and language rules of speech data better than the traditional hidden Markov model (HMM) do. But, the availability of large amount of data is necessary for deep learning-based speech recognition to be effective. While this is not a problem for mainstream languages such as English or Chinese, this is not the case for non-mainstream languages such as Indonesian. To overcome this limitation, we present deep features based on convolutional neural networks (CNN) for Indonesian large vocabulary continuous speech recognition in this paper. The CNN is trained discriminatively which is different from usual deep learning implementations where the networks are trained generatively. Our evaluations show that the proposed method on Indonesian speech data achieves 7.26% and 9.01% error reduction rates over the state-of-the-art deep belief networks-deep neural networks (DBN-DNN) for large vocabulary continuous speech recognition (LVCSR), with Mel frequency cepstral coefficients (MFCC) and filterbank (FBANK) used as features, respectively. An error reduction rate of 6.13% is achieved compared to CNN-DNN with generative training.

Online quantum time series processing with random oscillator networks

Reservoir computing is a powerful machine learning paradigm for online time series processing. It has reached state-of-the-art performance in tasks such as chaotic time series prediction and continuous speech recognition thanks to its unique combination of high computational power and low training cost which sets it aside from alternatives such as traditionally trained recurrent neural networks, and furthermore is amenable to implementations in dedicated hardware, potentially leading to extremely compact and efficient reservoir computers. Recently the use of random quantum systems has been proposed, leveraging the complexity of quantum dynamics for classical time series processing. Extracting the output from a quantum system without disturbing its state too much is problematic however, and can be expected to become a bottleneck in such approaches. Here we propose a reservoir computing inspired approach to online processing of time series consisting of quantum information, sidestepping the measurement problem. We illustrate its power by generalizing two paradigmatic benchmark tasks from classical reservoir computing to quantum information and introducing a task without a classical analogue where a random system is trained to both create and distribute entanglement between systems that never directly interact. Finally, we discuss partial generalizations where only the input or only the output time series is quantum.

MagTrack: A Wearable Tongue Motion Tracking System for Silent Speech Interfaces.

Current electromagnetic tongue tracking devices are not amenable for daily use and thus not suitable for silent speech interface and other applications. We have recently developed MagTrack, a novel wearable electromagnetic articulograph tongue tracking device. This study aimed to validate MagTrack for potential silent speech interface applications. We conducted two experiments: (a) classification of eight isolated vowels in consonant-vowel-consonant form and (b) continuous silent speech recognition. In these experiments, we used data from healthy adult speakers collected with MagTrack. The performance of vowel classification was measured by accuracies. The continuous silent speech recognition was measured by phoneme error rates. The performance was then compared with results using data collected with commercial electromagnetic articulograph in a prior study. The isolated vowel classification using MagTrack achieved an average accuracy of 89.74% when leveraging all MagTrack signals (x, y, z coordinates; orientation; and magnetic signals), which outperformed the accuracy using commercial electromagnetic articulograph data (only y, z coordinates) in our previous study. The continuous speech recognition from two subjects using MagTrack achieved phoneme error rates of 73.92% and 66.73%, respectively. The commercial electromagnetic articulograph achieved 64.53% from the same subject (66.73% using MagTrack data). MagTrack showed comparable results with the commercial electromagnetic articulograph when using the same localized information. Adding raw magnetic signals would improve the performance of MagTrack. Our preliminary testing demonstrated the potential for silent speech interface as a lightweight wearable device. This work also lays the foundation to support MagTrack's potential for other applications including visual feedback-based speech therapy and second language learning.

A Helium Speech Unscrambling Algorithm Based on Deep Learning

Helium speech, the language spoken by divers in the deep sea who breathe a high-pressure helium–oxygen mixture, is almost unintelligible. To accurately unscramble helium speech, a neural network based on deep learning is proposed. First, an isolated helium speech corpus and a continuous helium speech corpus in a normal atmosphere are constructed, and an algorithm to automatically generate label files is proposed. Then, a convolution neural network (CNN), connectionist temporal classification (CTC) and a transformer are combined into a speech recognition network. Finally, an optimization algorithm is proposed to improve the recognition of continuous helium speech, which combines depth-wise separable convolution (DSC), a gated linear unit (GLU) and a feedforward neural network (FNN). The experimental results show that the accuracy of the algorithm, upon combining the CNN, CTC and the transformer, is 91.38%, and the optimization algorithm improves the accuracy of continuous helium speech recognition by 9.26%.

Perspectivas sobre la traducción automática del habla

En Japón se han llevado a cabo muchas actividades de investigación acerca de la traducción automática del habla. Este artículo pretende ofrecer una visión general de dichas actividades y presentar las que se han realizado más recientemente. El sistema S2ST está formado básicamente por tres módulos: el reconocimiento automático del habla continua y de amplios vocabularios (Automatic Speech Recognition, ASR), la traducción automática de textos (Machine translation, MT) y la conversión de texto a voz (Text-to-Speech Synthesis, TTS). Todos los módulos deben ser plurilingües, por lo cual se requieren discursos y corpus multilingües para los modelos de formación. El rendimiento del sistema S2ST mejora considerablemente por medio de un aprendizaje profundo y grandes corpus formativos. Sin embargo, todavía hace falta tratar diversos aspectos, com la simultaneidad, la paralingüística, la dependencia del contexto y de la situación, la intención y la dependencia cultural. Por todo ello, repasaremos las actividades de investigación actuales y discutiremos varias cuestiones relacionadas con la traducción automática del habla de última generación.

Development of Language Models for Continuous Uzbek Speech Recognition System.

Automatic speech recognition systems with a large vocabulary and other natural language processing applications cannot operate without a language model. Most studies on pre-trained language models have focused on more popular languages such as English, Chinese, and various European languages, but there is no publicly available Uzbek speech dataset. Therefore, language models of low-resource languages need to be studied and created. The objective of this study is to address this limitation by developing a low-resource language model for the Uzbek language and understanding linguistic occurrences. We proposed the Uzbek language model named UzLM by examining the performance of statistical and neural-network-based language models that account for the unique features of the Uzbek language. Our Uzbek-specific linguistic representation allows us to construct more robust UzLM, utilizing 80 million words from various sources while using the same or fewer training words, as applied in previous studies. Roughly sixty-eight thousand different words and 15 million sentences were collected for the creation of this corpus. The experimental results of our tests on the continuous recognition of Uzbek speech show that, compared with manual encoding, the use of neural-network-based language models reduced the character error rate to 5.26%.

Streaming Punctuation: A Novel Punctuation Technique Leveraging Bidirectional Context for Continuous Speech Recognition

Streaming Punctuation: A Novel Punctuation Technique Leveraging Bidirectional Context for Continuous Speech Recognition

Decoding Silent Speech Based on High-Density Surface Electromyogram Using Spatiotemporal Neural Network

Finer-grained decoding at a phoneme or syllable level is a key technology for continuous recognition of silent speech based on surface electromyogram (sEMG). This paper aims at developing a novel syllable-level decoding method for continuous silent speech recognition (SSR) using spatio-temporal end-to-end neural network. In the proposed method, the high-density sEMG (HD-sEMG) was first converted into a series of feature images, and then a spatio-temporal end-to-end neural network was applied to extract discriminative feature representations and to achieve syllable-level decoding. The effectiveness of the proposed method was verified with HD-sEMG data recorded by four pieces of 64-channel electrode arrays placed over facial and laryngeal muscles of fifteen subjects subvocalizing 33 Chinese phrases consisting of 82 syllables. The proposed method outperformed the benchmark methods by achieving the highest phrase classification accuracy (97.17 ± 1.53%, ), and lower character error rate (3.11 ± 1.46%, ). This study provides a promising way of decoding sEMG towards SSR, which has great potential applications in instant communication and remote control.