Real-time Speech Recognition System Research Articles

Real-time multilingual speech recognition and speaker diarization system based on Whisper segmentation.

This research presents the development of a cutting-edge real-time multilingual speech recognition and speaker diarization system that leverages OpenAI's Whisper model. The system specifically addresses the challenges of automatic speech recognition (ASR) and speaker diarization (SD) in dynamic, multispeaker environments, with a focus on accurately processing Mandarin speech with Taiwanese accents and managing frequent speaker switches. Traditional speech recognition systems often fall short in such complex multilingual and multispeaker contexts, particularly in SD. This study, therefore, integrates advanced speech recognition with speaker diarization techniques optimized for real-time applications. These optimizations include handling model outputs efficiently and incorporating speaker embedding technology. The system was evaluated using data from Taiwanese talk shows and political commentary programs, featuring 46 diverse speakers. The results showed a promising word diarization error rate (WDER) of 2.68% in two-speaker scenarios and 11.65% in three-speaker scenarios, with an overall WDER of 6.96%. This performance is comparable to that of non-real-time baseline models, highlighting the system's ability to adapt to various complex conversational dynamics, a significant advancement in the field of real-time multilingual speech processing.

Real-Time Automatic Continuous Speech Recognition System for Kannada Language/Dialects

Real-Time Automatic Continuous Speech Recognition System for Kannada Language/Dialects

DEVELOPMENT OF A REAL-TIME SPEECH RECOGNITION SYSTEM FOR THE AZERBAIJANI LANGUAGE

This paper investigates the development of a real-time automatic speech recognition system dedicated to the Azerbaijani language, focusing on addressing the prevalent gap in speech recognition system for underrepresented languages. Our research integrates a hybrid acoustic modeling approach that combines Hidden Markov Model and Deep Neural Network to interpret the complexities of Azerbaijani acoustic patterns effectively. Recognizing the agglutinative nature of Azerbaijani, the ASR system employs a syllable-based n-gram model for language modeling, ensuring the system accurately captures the syntax and semantics of Azerbaijani speech. To enable real-time capabilities, we incorporate WebSocket technology, which facilitates efficient bidirectional communication between the client and server, necessary for processing streaming speech data instantly. The Kaldi and SRILM toolkits are used for the training of acoustic and language models, respectively, contributing to the system's robust performance and adaptability. We have conducted comprehensive experiments to test the effectiveness of our system, the results of which strongly corroborate the utility of the syllable-based subword modeling approach for Azerbaijani language recognition. Our proposed ASR system shows superior performance in terms of recognition accuracy and rapid response times, outperforming other systems tested on the same language data. The system's success not only proves beneficial for Azerbaijani language recognition but also provides a valuable framework for potential future applications in other agglutinative languages, thereby contributing to the promotion of linguistic diversity in automatic speech recognition technology.

Voice Assistant Notepad

Abstract: The notepad is made on the basic concept of a real-time voice to text conversion technology that translates said words into text exactly as the user pronounces them. We developed a real-time speech recognition system and tested it in normal surroundings. The system is made up of two parts: the first is for processing an acoustic signal acquired by a microphone, and the second is for interpreting the processed signal and translating it to words. We want a voice recognition system that is reliable and inexpensive and with a good efficiency in performance. This system allows us to take notes faster which helps us to increase productivity and maintain good work life balance at the same time. This helps people of all age groups such as kids to take down notes who find writing difficult, adults to note down paragraphs or important points more easily and elderly persons to make use of technology who find typing hard. The software program was created using an object-oriented analysis and design methodology, and it accomplishes Speech Recognition by detecting and also capturing the audio using the microphone on the device. Along with additional advantages, the suggested system decreased the note making duration to more than 50% depending on the user's speed. To address the present issues with note taking, we decided to take on and do the project.

Design and Implementation of Energy-Efficient Floating Point MFCC Extraction Architecture for Speech Recognition Systems

Abstract: This brief presents an energy-efficient architecture to extract mel-frequency cepstrum coefficients (MFCCs) for realtime speech recognition systems. Based on the algorithmic property of MFCC feature extraction, the architecture is designed with floating-point arithmetic units to cover a wide dynamic range with a small bit-width. Moreover, various operations required in the MFCC extraction are examined to optimize operational bit-width and lookup tables needed to compute nonlinear functions, such as trigonometric and logarithmic functions. In addition, the dataflow of MFCC extraction is tailored to minimize the computation time. As a result, the energy consumption is considerably reduced compared with previous MFCC extraction systems

Design and Implementation of Embedded Real-Time English Speech Recognition System Based on Big Data Analysis

This article uses Field Programmable Gate Array (FPGA) as a carrier and uses IP core to form a System on Programmable Chip (SOPC) English speech recognition system. The SOPC system uses a modular hardware system design method. Except for the independent development of the hardware acceleration module and its control module, the other modules are implemented by software or IP provided by Xilinx development tools. Hardware acceleration IP adopts a top-down design method, provides parallel operation of multiple operation components, and uses pipeline technology, which speeds up data operation, so that only one operation cycle is required to obtain an operation result. In terms of recognition algorithm, a more effective training algorithm is proposed, Genetic Continuous Hidden Markov Model (GA_CHMM), which uses genetic algorithm to directly train CHMM model. It is to find the optimal model by encoding the parameter values of the CHMM and performing operations such as selection, crossover, and mutation according to the fitness function. The optimal parameter value after decoding corresponds to the CHMM model, and then the English speech recognition is performed through the CHMM algorithm. This algorithm can save a lot of training time, thereby improving the recognition rate and speed. This paper studies the optimization of embedded system software. By studying the fixed-point software algorithm and the optimization of system storage space, the real-time response speed of the system has been reduced from about 10 seconds to an average of 220 milliseconds. Through the optimization of the CHMM algorithm, the real-time performance of the system is improved again, and the average time to complete the recognition is significantly shortened. At the same time, the system can achieve a recognition rate of over 90% when the English speech vocabulary is less than 200.

Practical Study and Implementation of an Isolated Word Speech Recognition System.(Dept.E)

The art and science of speech recognition have been advanced to the state where it is now possible to communicate reliably with a machine by speaking to it in a disciplined manner using a vocabulary of moderate size. It is the purpose of this paper to present a hardware implementation of a simple real time speech recognition system for isolated words. The system determines the energy of the spoken word through a bank of four channels band pass filters. The output from each filter is passed through a rectifier followed by a low pass filter to provide an average level which is proportional to the total signal energy within the particular filter band. The energy measurements from each channel is then passed to a personal computer where a recognition algorithm program is then executed. The program detects the starting and ending points of each spoken word. compares the energy features with the reference stored in the library, and chooses the most close pattern that matches the pattern of the spoken word. Results from experiments conducted by simulation and by the hardware prototype using Arabic speech show that the system provides recognition score between 76 % and 94%.

RETRACTED: Real time speech recognition algorithm on embedded system based on continuous Markov model

Abstract Real time speech recognition technology, as a key cross technology in the field of artificial intelligence in recent years, has been widely used in the fields of intelligent voice toys, industrial control and intelligent rehabilitation. Because the real-time speech recognition technology based on embedded technology has obvious advantages in the volume, power consumption and research and development cost of the system, it has become a hot carrier to achieve efficient speech recognition technology. In order to realize a simple and practical real-time speech recognition system based on embedded system, this paper designs a basic framework of machine learning based on Markov random field theory combined with machine learning theory, and studies the algorithm of real-time speech vocabulary matching recognition based on this framework. In detail, the algorithm proposed in this paper will process the speech signal from the aspects of preprocessing, signal detection, feature extraction and quantization. Finally, this paper will build a real-time speech recognition system based on DSP processor. The experimental results show that the real-time speech recognition algorithm proposed in this paper can improve the real-time recognition speed of the system. The corresponding speed changes from about 12 s to about 200 ms, and the corresponding real-time accuracy rate increases to about 95%.

Real-time transcription, keyword spotting, archival and retrieval for telugu TV news using ASR

The main objective of this paper is to describe the system developed for transcription, keyword spotting and alerting, archival, and retrieval for broadcasted Telugu TV news. Real-time automatic speech recognition system is developed for searching the given keywords from the transcribed text of the broadcasted Telugu TV news. The user can set the required keywords and the system will continuously track the audio and alert in real time the user as and when the keywords appeared in the audio. The system will transcribe and index every word with time. As and when a user wanted to retrieve the data with a particular word, it will search the transcribed text in the archives and play the corresponding audio, video along with scrolling of transcribed text of audio. The system is developed fully with GUI for giving the keywords either through microphone or keyboard by transliteration in Telugu. The ASR system is developed using the annotated speech corpus of 65 h duration with two hundred thousand unique words. The acoustic models are developed using subspace Gaussian mixture model. The language modeling is developed with Witten Bell and Kneser–Ney smoothing techniques

A Robust Feature Extraction Method for Real-Time Speech Recognition System on a Raspberry Pi 3 Board

The development of a real-time automatic speech recognition system (ASR) better adapted to environmental variabilities, such as noisy surroundings, speaker variations and accents has become a high priority. Robustness is required, and it can be performed at the feature extraction stage which avoids the need for other pre-processing steps. In this paper, a new robust feature extraction method for real-time ASR system is presented. A combination of Mel-frequency cepstral coefficients (MFCC) and discrete wavelet transform (DWT) is proposed. This hybrid system can conserve more extracted speech features which tend to be invariant to noise. The main idea is to extract MFCC features by denoising the obtained coefficients in the wavelet domain by using a median filter (MF). The proposed system has been implemented on Raspberry Pi 3 which is a suitable platform for real-time requirements. The experiments showed a high recognition rate (100%) in clean environment and satisfying results (ranging from 80% to 100%) in noisy environments at different signal to noise ratios (SNRs).

, ,

There have being great efforts made in the development of automated Instrumentation system for speech recognition (AISR) to provide a two-way communication between deaf and vocal people. This system performance achievable with the output of current real-time speech recognition systems would be extremely poor relative to normal speech reception. An alternate application of AISR technology to aid the hearing impaired would derive cues from the acoustical speech signal that could be used to supplement speechreading. We propose a study of highly trained receivers of speech signal that indicates that nearly perfect reception of everyday connected speech materials can be achieved at near normal speaking rates. To understand the accuracy that might be achieved with automatically generated cue symbols for visual representation. The system uses (HMM) for recognition of voiced data & Euclidian distance approach for sign language. The proposed task is a complementary work to the ongoing research work for recognizing the finger movement of a vocally disabled person to speech signal called. A New communication Paradigm: “Action-To-Speech”

Question answering in conversations: Query refinement using contextual and semantic information

This paper introduces a query refinement method applied to questions asked by users to a system during a meeting or a conversation that they have with other users. To answer the questions, the proposed method leverages the local context of the conversation along with semantic resources, either WordNet or word embeddings from word2vec. The method first represents the local context by extracting keywords from the transcript of the conversation, which is obtained from a real-time Automatic Speech Recognition (ASR) system and may contain noise. It then expands the queries with keywords that best represent the topic of the query, i.e. expansion keywords accompanied by weights indicating their topical similarity to the query. Finally, semantically related terms are added, using two options: either synonymous terms drawn from WordNet or similar words based on distributed representations in a low-dimensional word embedding space learned using word2vec. To evaluate the system, we introduce a dataset (named AREX for AMI Requests for Explanations) and an evaluation metric based on relevance judgments collected by crowdsourcing. We compare our query expansion approach with other methods, over queries from the AREX dataset, showing the superiority of our method when either manual or automatic transcripts of the AMI Meeting Corpus are used.

Energy-Efficient Floating-Point MFCC Extraction Architecture for Speech Recognition Systems

This brief presents an energy-efficient architecture to extract mel-frequency cepstrum coefficients (MFCCs) for real-time speech recognition systems. Based on the algorithmic property of MFCC feature extraction, the architecture is designed with floating-point arithmetic units to cover a wide dynamic range with a small bit-width. Moreover, various operations required in the MFCC extraction are examined to optimize operational bit-width and lookup tables needed to compute nonlinear functions, such as trigonometric and logarithmic functions. In addition, the dataflow of MFCC extraction is tailored to minimize the computation time. As a result, the energy consumption is considerably reduced compared with previous MFCC extraction systems.

FPGA-Based Hardware Accelerator for Feature Extraction in Automatic Speech Recognition

We describe in this paper a hardware-based improvement scheme of a real-time automatic speech recognition (ASR) system with respect to speed by designing a parallel feature extraction algorithm on a Field-Programmable Gate Array (FPGA). A computationally intensive block in the algorithm is identified implemented in hardware logic on the FPGA. One such block is mel-frequency cepstrum coefficient (MFCC) algorithm used for feature extraction process. We demonstrate that the FPGA platform may perform efficient feature extraction computation in the speech recognition system as compared to the generalpurpose CPU including the ARM processor. The Xilinx Zynq-7000 System on Chip (SoC) platform is used for the MFCC implementation. From this implementation described in this paper, we confirmed that the FPGA platform is approximately 500× faster than a sequential CPU implementation and 60× faster than a sequential ARM implementation. We thus verified that a parallelized and optimized MFCC architecture on the FPGA platform may significantly improve the execution time of an ASR system, compared to the CPU and ARM platforms.

Real-Time Arabic Speech Recognition

Speech recognition system needs to perform a high complex calculation and short time to complete it. This is a big challenge for the real-time systems. However, using a simple and fast algorithm may do this task for the slow systems. Thus, the main objective of this paper is to design and implement a Real-Time Arabic Speech Recognition system using MATLAB environment. It is capable of accurately identifying some letters while remaining simple and fast. It uses the Mel-Frequency Cepstral Coefficients (MFCCs) as a feature extraction and Euclidean distance to compare the test sound and the database. A recognition rate of 89.6% has been reached. General Terms Speech Recognition.

Robust Automatic Speech recognition System Implemented in a Hybrid Design DSP-FPGA

The aim of this work is to reduce the burden task on the DSP processor by transferring a parallel computation part on a configurable circuits FPGA, in automatic speech recognition module design, signal pre-processing, feature selection and optimization, models construction and finally classification phase are necessary. LMS filter algorithm that contains more parallelism and more MACs (multiply and Accumulate) operations is implemented on FPGA Virtex 5 by Xilings , MFCCs features extraction and DTW( dynamic time wrapping ) method is used as a classifier. Major contribution of this work are hybrid solution DSP and FPGA in real time speech recognition system design, the optimization of number of MAC-core within the FPGA this result is obtained by sharing MAC resources between two operation phases: computation of output filter and updating LMS filter coefficients. The paper also provides a hardware solution of the filter with detailed description of asynchronous interface of FPGA circuit and TMS320C6713-EMIF component. The results of simulation shows an improvement in time computation and by optimizing the implementation on the FPGA a gain in space consumption is obtained.

Optimized Acoustic Likelihoods Computation for NVIDIA and ATI/AMD Graphics Processors

In this paper, we describe an optimized version of a Gaussian-mixture-based acoustic model likelihood evaluation algorithm for graphical processing units (GPUs). The evaluation of these likelihoods is one of the most computationally intensive parts of automatic speech recognizers, but it can be parallelized and offloaded to GPU devices. Our approach offers a significant speed-up over the recently published approaches, because it utilizes the GPU architecture in a more effective manner. All the recent implementations have been intended only for NVIDIA graphics processors, programmed either in CUDA or OpenCL GPU programming frameworks. We present results for both CUDA and OpenCL. Further, we have developed an OpenCL implementation optimized for ATI/AMD GPUs. Results suggest that even very large acoustic models can be used in real-time speech recognition engines on computers equipped with a low-end GPU or laptops. In addition, the completely asynchronous GPU management provides additional CPU resources for the decoder part of the LVCSR. The optimized implementation enables us to apply fusion techniques together with evaluating many (10 or even more) speaker-specific acoustic models. We apply this technique to a real-time parliamentary speech recognition system where the speaker changes frequently.

Design of a real time automatic speech recognition system using Modified One Against All SVM classifier

In this paper, Texas Instruments TMS320C6713 DSP based real-time speech recognition system using Modified One Against All Support Vector Machine (SVM) classifier is proposed. The major contributions of this paper are: the study and evaluation of the performance of the classifier using three feature extraction techniques and proposal for minimizing the computation time for the classifier. From this study, it is found that the recognition accuracies of 93.33%, 98.67% and 96.67% are achieved for the classifier using Mel Frequency Cepstral Coefficients (MFCC) features, zerocrossing (ZC) and zerocrossing with peak amplitude (ZCPA) features respectively. To reduce the computation time required for the systems, two techniques – one using optimum threshold technique for the SVM classifier and another using linear assembly are proposed. The ZC based system requires the least computation time and the above techniques reduce the execution time by a factor of 6.56 and 5.95 respectively. For the purpose of comparison, the speech recognition system is also implemented using Altera Cyclone II FPGA with Nios II soft processor and custom instructions. Of the two approaches, the DSP approach requires 87.40% less number of clock cycles. Custom design of the recognition system on the FPGA without using the soft-core processor would have resulted in less computational complexity. The proposed classifier is also found to reduce the number of support vectors by a factor of 1.12–3.73 when applied to speaker identification and isolated letter recognition problems. The techniques proposed here can be adapted for various other SVM based pattern recognition systems.

Online Speech/Music Segmentation Based on the Variance Mean of Filter Bank Energy

This paper presents a novel feature for online speech/music segmentation based on the variance mean of filter bank energy (VMFBE). The idea that encouraged the feature's construction is energy variation in a narrow frequency sub-band. The energy varies more rapidly, and to a greater extent for speech than formusic. Therefore, an energy variance in such a sub-band is greater for speech than for music. The radio broadcast database and the BNSI broadcast news database were used for feature discrimination and segmentation ability evaluation. The calculation procedure of the VMFBE feature has 4 out of 6 steps in common with the MFCC feature calculation procedure. Therefore, it is a very convenient speech/music discriminator for use in real-time automatic speech recognition systems based on MFCC features, because valuable processing time can be saved, and computation load is only slightly increased. Analysis of the feature's speech/music discriminative ability shows an average error rate below 10% for radio broadcast material and it outperforms other features used for comparison, by more than 8%. The proposed feature as a standalone speech/music discriminator in a segmentation system achieves an overall accuracy of over 94% on radio broadcast material.

Mutual Information Based Dynamic Integration of Multiple Feature Streams for Robust Real-Time LVCSR

We present a novel method of integrating the likelihoods of multiple feature streams, representing different acoustic aspects, for robust speech recognition. The integration algorithm dynamically calculates a frame-wise stream weight so that a higher weight is given to a stream that is robust to a variety of noisy environments or speaking styles. Such a robust stream is expected to show discriminative ability. A conventional method proposed for the recognition of spoken digits calculates the weights from the entropy of the whole set of HMM states. This paper extends the dynamic weighting to a real-time large-vocabulary continuous speech recognition (LVCSR) system. The proposed weight is calculated in real-time from mutual information between an input stream and active HMM states in a search space without an additional likelihood calculation. Furthermore, the mutual information takes the width of the search space into account by calculating the marginal entropy from the number of active states. In this paper, we integrate three features that are extracted through auditory filters by taking into account the human auditory system's ability to extract amplitude and frequency modulations. Due to this, features representing energy, amplitude drift, and resonant frequency drifts, are integrated. These features are expected to provide complementary clues for speech recognition. Speech recognition experiments on field reports and spontaneous commentary from Japanese broadcast news showed that the proposed method reduced error words by 9.2% in field reports and 4.7% in spontaneous commentaries relative to the best result obtained from a single stream.