Robust Speech Recognition System Research Articles

Refining maritime Automatic Speech Recognition by leveraging synthetic speech

Maritime transport serves as a critical component of global trade and logistics, enabling the movement of goods and resources across oceans and waterways. Especially in busy waterways and ports, effective and accurate communication is essential, as it ensures the seamless exchange of information and the coordinated execution of port activities. However, comprehensibility is often hindered by factors such as poor audio quality, background noise, and diverse languages and accents. Automatic Speech Recognition (ASR) systems can mitigate these issues by providing real-time transcription and enabling the implementation of automated, value-adding services to enhance situational awareness. While pre-trained ASR models excel on general speech, maritime ASR faces unique challenges due to a lack of annotated data, diverse accents, and specialized terminology.To this end, we focus on improving the transcription quality of pre-trained ASR models for maritime communication with a particular focus on accurately recognizing maritime-specific terminology such as vessel and location names. Due to the scarcity of transcribed maritime communication, we create a synthetic training dataset tailored to regional maritime terminology. The synthetic audio is augmented with general human speech and used to fine-tune an end-to-end ASR model under various settings. The evaluation of the models employs a proprietary dataset of regional maritime radio communication from the port of Hamburg.The experimental results demonstrate a notable enhancement in ASR performance. Specifically, our approach yields an absolute improvement over the pre-trained baseline of 13.46% Word-Error-Rate and an increase of 41.57% recall for vessel names and 38.65% recall for locations. Our findings underscore the efficacy of integrating synthetic training data to address the challenges encountered in maritime ASR, paving the way for more robust and accurate speech recognition systems tailored to maritime applications.

Boosting Low-Resource Speech Recognition in Air Traffic Communication via Pretrained Feature Aggregation and Multi-Task Learning

Developing a robust Automatic Speech Recognition (ASR) system usually requires a large amount of well-annotated samples which is extremely hard to build in the Air Traffic Control (ATC) due to domain-specific knowledge. In this work, we present a novel approach to improve ASR performance in the ATC domain by integrating Self-Supervised Learning and Multi-Task learning into a unified framework. Specifically, the proposed framework follows a two-stage training paradigm, i.e., (a) learning universal acoustic representations by employing the wav2vec 2.0 model and (b) jointly finetuning the model by the ASR, Speaker Role Identification, and Language Identification tasks. To capture the task-specific representations, an attention-guided feature aggregation module is dedicatedly designed to disentangle the discriminative representations from the pretrained features. In addition, the uncertainty-based loss combination strategy is employed to balance the loss weights for each task in a learnable manner. Finally, we conduct experiments to validate the technical improvements in a real-world ATC dataset. Experimental results demonstrated that the proposed framework outperforms competitive baselines among all tasks.

The Application of Classifier Model Based on FA-SVM Algorithm in Speech Recognition

In recent years, speech recognition technology has been increasingly widely used. How to improve the robustness of speech recognition systems has become the focus of researchers. This paper proposes an accent recognition model based on the Firefly algorithm to optimize the support vector machine. Firstly, a set of penalty factors and kernel parameters in the support vector machine are taken as firefly individuals, and the important parameters affecting the performance of the support vector machine are optimized by the firefly algorithm. A Firefly-Support vector machine (FA-SVM) model is established and applied to accent recognition. The example analysis shows that the FA-SVM accent recognition model has taller classification accuracy and faster running speed than the PSO-SVM and GA-SVM classifier models.

A Study on the Impact of Lombard Effect on Recognition of Hindi Syllabic Units Using CNN Based Multimodal ASR Systems

Research work on the design of robust multimodal speech recognition systems making use of acoustic and visual cues, extracted using the relatively noise robust alternate speech sensors is gaining interest in recent times among the speech processing research fraternity. The primary objective of this work is to study the exclusive influence of Lombard effect on the automatic recognition of the confusable syllabic consonant-vowel units of Hindi language, as a step towards building robust multimodal ASR systems in adverse environments in the context of Indian languages which are syllabic in nature. The dataset for this work comprises the confusable 145 consonant-vowel (CV) syllabic units of Hindi language recorded simultaneously using three modalities that capture the acoustic and visual speech cues, namely normal acoustic microphone (NM), throat microphone (TM) and a camera that captures the associated lip movements. The Lombard effect is induced by feeding crowd noise into the speaker’s headphone while recording. Convolutional Neural Network (CNN) models are built to categorise the CV units based on their place of articulation (POA), manner of articulation (MOA), and vowels (under clean and Lombard conditions). For validation purpose, corresponding Hidden Markov Models (HMM) are also built and tested. Unimodal Automatic Speech Recognition (ASR) systems built using each of the three speech cues from Lombard speech show a loss in recognition of MOA and vowels while POA gets a boost in all the systems due to Lombard effect. Combining the three complimentary speech cues to build bimodal and trimodal ASR systems shows that the recognition loss due to Lombard effect for MOA and vowels reduces compared to the unimodal systems, while the POA recognition is still better due to Lombard effect. A bimodal system is proposed using only alternate acoustic and visual cues which gives a better discrimination of the place and manner of articulation than even standard ASR system. Among the multimodal ASR systems studied, the proposed trimodal system based on Lombard speech gives the best recognition accuracy of 98%, 95%, and 76% for the vowels, MOA and POA, respectively, with an average improvement of 36% over the unimodal ASR systems and 9% improvement over the bimodal ASR systems.

Computational Approaches to Phonology: Advances in Speech Recognition and Synthesis

This article explores the field of computational approaches to phonology, with a particular focus on recent increases in popularity and speech synthesis. Phonology, as a department of linguistics, deals with the observation of sounds and their work in language. Recent improvements in this area have resulted in major improvements in accuracy and efficiency, towards the integration of deep recognition strategies together with recurrent neural networks (RNNs) and convolutional neural networks (CNNs). the models have tested the ability to capture complex phonetic patterns and decorate the overall performance of computerized speech reputation structures. These advances have seen packages in multiple domains, such as virtual assistants, text-to-speech structures, and language acquisition tools. This article looks at the main computational procedures used in phonology, along with characteristic-based representations, rule-based modes, and statistical methods. It explores demanding situations related to phonological evaluation, which include phoneme segmentation, pronunciation variants, and language-specific problems. In addition, this paper discusses mixing phonological knowledge with device learning strategies, which enables the development of robust and correct speech recognition and synthesis systems. As usual, this research represents remarkable advances from computational methods to phonology, particularly in the geographical area of speech recognition and synthesis. the combination of technology and linguistic insights has increased the accuracy, naturalness, and efficiency in processing spoken language. These enhancements open up new opportunities to enhance human-pc interaction, language recognition, and many other packages that rely on speech-based technologies.

Robust Automatic Speech Recognition System for the Recognition of Continuous Kannada Speech Sentences in the Presence of Noise

Robust Automatic Speech Recognition System for the Recognition of Continuous Kannada Speech Sentences in the Presence of Noise

HindiSpeech-Net: a deep learning based robust automatic speech recognition system for Hindi language

HindiSpeech-Net: a deep learning based robust automatic speech recognition system for Hindi language

Developing sequentially trained robust Punjabi speech recognition system under matched and mismatched conditions

Development of a native language robust ASR framework is very challenging as well as an active area of research. Although an urge for investigation of effective front-end as well as back-end approaches are required for tackling environment differences, large training complexity and inter-speaker variability in achieving success of a recognition system. In this paper, four front-end approaches: mel-frequency cepstral coefficients (MFCC), Gammatone frequency cepstral coefficients (GFCC), relative spectral-perceptual linear prediction (RASTA-PLP) and power-normalized cepstral coefficients (PNCC) have been investigated to generate unique and robust feature vectors at different SNR values. Furthermore, to handle the large training data complexity, parameter optimization has been performed with sequence-discriminative training techniques: maximum mutual information (MMI), minimum phone error (MPE), boosted-MMI (bMMI), and state-level minimum Bayes risk (sMBR). It has been demonstrated by selection of an optimal value of parameters using lattice generation, and adjustments of learning rates. In proposed framework, four different systems have been tested by analyzing various feature extraction approaches (with or without speaker normalization through Vocal Tract Length Normalization (VTLN) approach in test set) and classification strategy on with or without artificial extension of train dataset. To compare each system performance, true matched (adult train and test—S1, child train and test—S2) and mismatched (adult train and child test—S3, adult + child train and child test—S4) systems on large adult and very small Punjabi clean speech corpus have been demonstrated. Consequently, gender-based in-domain data augmented is used to moderate acoustic and phonetic variations throughout adult and children’s speech under mismatched conditions. The experiment result shows that an effective framework developed on PNCC + VTLN front-end approach using TDNN-sMBR-based model through parameter optimization technique yields a relative improvement (RI) of 40.18%, 47.51%, and 49.87% in matched, mismatched and gender-based in-domain augmented system under typical clean and noisy conditions, respectively.

Integrated Design of Financial Self-Service Terminal Based on Artificial Intelligence Voice Interaction.

Integrated design of financial self-service terminal based on artificial intelligence voice interaction with the rapid development of science and technology, artificial intelligence technology is deepening in the field of intelligence and automation. The financial industry is the lifeblood of a country’s economy, with great growth potential and high growth rate. The integrated design of intelligent financial self-service terminal has become an important topic in the field of rapid development of social economy and science and technology. Therefore, this paper designs the integration of financial self-service terminal based on artificial intelligence voice interaction. First, this paper introduces the meaning and composition of financial self-service terminal integration, then studies the voice interaction principle based on artificial intelligence technology, and designs the integrated structure of financial self-service terminal with voice interaction. After that, this paper makes a series of tests on voice interaction technology, user experience, and the performance of financial self-service terminal. Finally, the test results of voice interaction are as follows: the delay estimation results of voice interaction of the terminal are relatively accurate, and the error points are basically within five sampling points, which indicate that the delay estimation algorithm is practical. The endpoint detection method based on CO complexity can effectively overcome the impact of noise environment on speech endpoint detection system and is suitable for the requirements of robust speech recognition system. Considering that the actual application scenario of voice positioning can judge the speaker’s position and turn to the speaker’s direction during human–computer interaction, the azimuth error is acceptable within a few degrees to meet the application requirements. The direction angle error is acceptable within a few degrees to meet the application requirements. The accuracy of the improved algorithm is improved in intercepting effective speech signals. The terminal has short running time and delay time, small memory, and central processing unit (CPU) occupation and can meet the needs of users. The speech recognition accuracy of the financial self-service terminal basically reaches more than 80%, which can basically meet the daily needs.

Frequency domain analysis of MFCC feature extraction in children’s speech recognition system

Abstract —The research on speech recognition systems currently focuses on the analysis of robust speech recognition systems. When the speech signals are combined with noise, the recognition system becomes distracted, struggling to identify the speech sounds. Therefore, the development of a robust speech recognition system continues to be carried out. The principle of a robust speech recognition system is to eliminate noise from the speech signals and restore the original information signals. In this paper, researchers conducted a frequency domain analysis on one stage of the Mel Frequency Cepstral Coefficients (MFCC) process, the Fast Fourier Transform (FFT), in children's speech recognition system. The FTT analysis in the feature extraction process determined the effect of frequency value characteristics utilized in the FFT output on the noise disruption. The analysis method was designed into three scenarios based on the value of the employed FFT points. The differences between scenarios were based on the number of shared FFT points. All FFT points were divided into four, three, and two parts in the first, second, and third scenarios, respectively. This study utilized children's speech data from the isolated TIDIGIT English digit corpus. As comparative data, the noise was added manually to simulate real-world conditions. The results showed that using a particular frequency portion following the scenario designed on MFCC affected the recognition system performance, which was relatively significant on the noisy speech data. The designed method in the scenario 3 (C1) version generated the highest accuracy, exceeded the accuracy of the conventional MFCC method. The average accuracy in the scenario 3 (C1) method increased by 1% more than all the tested noise types. Using various noise intensity values (SNR), the testing process indicates that scenario 3 (C1) generates a higher accuracy than conventional MFCC in all tested SNR values. It proves that the selection of specific frequency utilized in MFCC feature extraction significantly affects the recognition accuracy in a noisy speech.

A new multi-stream approach using acoustic and visual features for robust speech recognition system

Building a robust Automatic Speech Recognition (ASR) system and improving recognition accuracy in adverse conditions is still a challenging task. One way to improve the robustness of an ASR system is combining information from multiple sources (streams). A multi-stream approach which handles the multiple inputs at the model level is the key contribution of our work. Standard mic (Sm), Throat mic (Tm), and Lip reading (Lr) are the various source streams that have been used. This work explores a static weighted two stream HMM (TSH) and multi-stream HMM (MSH) model for the bimodal and multimodal systems. A syllabic units of the Hindi language database categorized into three categories – Vowel, Place of Articulation (POA), and Manner of Articulation (MOA) are used for training and testing. In this study, four types of TSH are proposed for the combination of bimodal ((Sm+Tm), (Tm+Lr), (Sm+Lr), (Lm+Lm)) systems and one type MSH is proposed for multimodal (Sm+Tm+Lr) system in a synchronous and asynchronous manner. Mel Frequency Cepstral Coefficient (MFCC) features are used for Sm and Tm signals. Combined pixel-motion based features (DCT/DWT-MHI) are used for Lr signals. Among these two features, DWT outperforms than DCT and used as a feature for visual speech. Experiments were conducted for bimodal and multimodal system. The proposed MSH approach shows improvements of 1.36%, 6.21%, and 5.8% in recognition accuracy for Vowel, POA and MOA category respectively, as compared to bimodal systems.

A multimodal Lombard speech recognition system for the confusable Hindi syllabic units

Research work on the design of robust multimodal speech recognition systems making use of acoustic and visual cues, extracted using the relatively noise robust alternate speech sensors is gaining interest in recent times among the speech processing research fraternity. The primary objective of this work is to study the exclusive influence of Lombard effect on Automatic Speech Recognition (ASR) systems towards building robust multimodal ASR systems in adverse environments in the context of Indian languages which are syllabic in nature. The dataset for this work comprises the confusable 145 Consonant-Vowel (CV) syllabic units of Hindi language recorded simultaneously using three modalities that capture the acoustic and visual speech cues, namely Normal acoustic Microphone (NM), Throat Microphone (TM) and a camera that captures the associated lip movements. The Lombard effect is induced by feeding crowd noise into the speaker’s headphone while recording. HMM models are built to categorize the CV units based on their Place of Articulation (POA), Manner Of Articulation (MOA) and vowels (under clean and Lombard conditions). Unimodal ASR systems built using each speech cue show a recognition loss in all the systems due to Lombard effect. To overcome this loss, the complimentary speech cues taken from normal and throat microphone Lombard speech as well as from visual Lombard speech are used to build three bimodal and one trimodal ASR systems. Among the ASR systems studied, the trimodal system gives the best recognition accuracy of 98%, 95% and 76% for the vowels, MOA and POA, respectively, with an average improvement of 36% over the unimodal ASR systems and 9% improvement over the bimodal ASR systems.

Attention mechanism based LSTM in classification of stressed speech under workload

In order to improve the robustness of speech recognition systems, this study attempts to classify stressed speech caused by the psychological stress under multitasking workloads. Due to the transient nature and ambiguity of stressed speech, the stress characteristics is not represented in all the segments in stressed speech as labeled. In this paper, we propose a multi-feature fusion model based on the attention mechanism to measure the importance of segments for stress classification. Through the attention mechanism, each speech frame is weighted to reflect the different correlations to the actual stressed state, and the multi-channel fusion of features characterizing the stressed speech to classify the speech under stress. The proposed model further adopts SpecAugment in view of the feature spectrum for data augment to resolve small sample sizes problem among stressed speech. During the experiment, we compared the proposed model with traditional methods on CASIA Chinese emotion corpus and Fujitsu stressed speech corpus, and results show that the proposed model has better performance in speaker-independent stress classification. Transfer learning is also performed for speaker-dependent classification for stressed speech, and the performance is improved. The attention mechanism shows the advantage for continuous speech under stress in authentic context comparing with traditional methods.

Audio-Visual Speech Recognition Using LSTM and CNN

Background: Multimodal speech recognition is proved to be one of the most promising solutions for robust speech recognition, especially when the audio signal is corrupted by noise. As the visual speech signal not affected by audio noise, it can be used to obtain more information used to enhance the speech recognition accuracy in noisy system. The critical stage in designing robust speech recognition system is choosing of reliable classification method from large variety of available classification techniques. Deep learning is well-known as a technique that has the ability to classify a nonlinear problem, and takes into consideration the sequential characteristic of the speech signal. Numerous researches have been done in applying deep learning to overcome Audio-Visual Speech Recognition (AVSR) problems due to its amazing achievements in both speech and image recognition. Even though optimistic results have been obtained from the continuous studies, researches on enhancing accuracy in noise system and selecting the best classification technique are still gaining lots of attention. Objective: This paper aims to build AVSR system that uses both acoustic combined with visual speech information and use classification technique based on deep learning to improve the recognition performance in a clean and noisy environment. Methods: Mel Frequency Cepstral Coefficient (MFCC) and Discrete Cosine Transform (DCT) are used to extract the effective features from audio and visual speech signal respectively. The audio feature rate is greater than the visual feature rate, so that linear interpolation is needed to obtain equal feature vectors size then early integrating them to get combined feature vector. Bidirectional Long-Short Term Memory (BiLSTM), one of the Deep learning techniques, are used for classification process and compare the obtained results to other classification techniques like Convolution Neural Network (CNN) and the traditional Hidden Markov Models (HMM). The effectiveness of the proposed model is proved by using two multi-speaker AVSR datasets termed AVletters and GRID. Results: The proposed model gives promising results where the obtained results In case of GRID, using integrated audio-visual features achieved highest recognition accuracy of 99.07% and 98.47% , with enhancement up to 9.28% and 12.05% over audio-only for clean and noisy data respectively. For AVletters, the highest recognition accuracy is 93.33% with enhancement up to 8.33% over audio- only. Conclusion: Based on the obtained results, we can conclude that increasing the size of audio feature vector from 13 to 39 doesn’t give effective enhancement for the recognition accuracy in clean environment, but in noisy environment, it gives better performance. BiLSTM is considered to be the optimal classifier for a robust speech recognition system when compared to CNN and traditional HMM, because it takes into consideration the sequential characteristic of the speech signal (audio and visual). The proposed model gives great improvement in the recognition accuracy and decreasing the loss value for both clean and noisy environments than using audio-only features. Comparing the proposed model to previously obtain results which using the same datasets, we found that our model gives higher recognition accuracy and confirms the robustness of our model.

Gsdnet: Gated Self-Supervised Denoising Speech Control Network

The cost of labeling data remains high, even with the effective implementation of deep neural networks in speech recognition. At the same time, noise still hampers the performance of speech-recognition methods. Thus, it is still challenging to make full use of data sets to enhance the robustness of recognition systems. In this letter, we construct GSDNet, a gated self-supervised denoising speech control network that consists of three parts (a denoising feature-extraction frontend, a speech recognition encoder, and a decoder based on gated convolutionary neural networks with self-supervised regression), to provide a low-cost method for training a robust speech recognition system, and we apply it to equipment-control tasks. Finally, the experimental results with the THCH30 and AISHELL data sets for equipment control show that the word error rate is less than 0.2 without a language model.

A Modified MFCC for Improved Wavelet-Based Denoising on Robust Speech Recognition

Research on the current speech recognition system leads to the creation of a noise-resistant system. The Mel Frequency Cepstral Coefficients (MFCC) extraction method becomes a popular method in the speech recognition system. In this paper, the MFCC's weakness of noise interference is the main reason underlies the accomplishment of a robust speech recognition system. Development was carried out by improving the denoising performance using a wavelet transform. Modifications were carried out by analyzing the weakness of the wavelet denoising process on the recognition system using the MFCC method. The analysis was conducted at one of the MFCC stages, the Fast Fourier Transform (FFT) stage. The proposed method was conducted by performing the denoising process using Wavelet only on the noise-related data based on the FFT process' analysis results. The study utilized speech data in the form of eleven isolated words in English added with noise with several different characteristics. Results showed that the proposed method was capable of generating a better accuracy than conventional wavelet denoising methods on the signal to noise ratio (SNR) of 10dB, 15dB, and 20dB using a Fejer Korovkin 6 wavelet type. The highest accuracy increase of the proposed method was in signal to noise ratio (SNR) of 15dB with a rise of 4.63%, followed by a 3.96% increase at 20dB intensity, and 2.3% at 10dB intensity. The performance of the proposed method is then compared with other methods. The results show that the proposed method has the best performance on clean speech and noisy speech at SNR intensities of 10dB, 15dB, and 20dB.

HMM-based phoneme speech recognition system for the control and command of industrial robots

In recent years, the integration of human-robot interaction with speech recognition has gained a lot of pace in the manufacturing industries. Conventional methods to control the robots include semi-autonomous, fully-autonomous, and wired methods. Operating through a teaching pendant or a joystick is easy to implement but is not effective when the robot is deployed to perform complex repetitive tasks. Speech and touch are natural ways of communicating for humans and speech recognition, being the best option, is a heavily researched technology. In this study, we aim at developing a stable and robust speech recognition system to allow humans to communicate with machines (roboticarm) in a seamless manner. This paper investigates the potential of the linear predictive coding technique to develop a stable and robust HMM-based phoneme speech recognition system for applications in robotics. Our system is divided into three segments: a microphone array, a voice module, and a robotic arm with three degrees of freedom (DOF). To validate our approach, we performed experiments with simple and complex sentences for various robotic activities such as manipulating a cube and pick and place tasks. Moreover, we also analyzed the test results to rectify problems including accuracy and recognition score.

Does Speech Enhancement of Publicly Available Data Help Build Robust Speech Recognition Systems? (Student Abstract)

Automatic speech recognition(ASR) systems play a key role in many commercial products including voice assistants. Typically, they require large amounts of high quality speech data for training which gives an undue advantage to large organizations which have tons of private data. We investigated if speech data obtained from publicly available sources can be further enhanced to train better speech recognition models. We begin with noisy/contaminated speech data, apply speech enhancement to produce 'cleaned' version and use both the versions to train the ASR model. We have found that using speech enhancement gives 9.5% better word error rate than training on just the original noisy data and 9% better than training on just the ground truth 'clean' data. It's performance is also comparable to the ideal case scenario when trained on noisy and it's ground truth 'clean' version.

Noise robust speech recognition system using multimodal audio-visual approach using different deep learning classification techniques

Noise robust speech recognition system using multimodal audio-visual approach using different deep learning classification techniques

Audio–Visual (Multimodal) Speech Recognition System Using Deep Neural Network

Abstract Recognizing the voice of a person by a smart device has become the need of the hour. Our cell phones are equipped with such features that converts speech to text, receives a command as a voice instruction instead of screen touch or pressing of a button. There are numerous intelligent machines such as automatic cars, robots, smart phones that use speech recognition features. These features are also very useful for people who are disabled and cannot use the smart devices with their hands. There are numerous techniques available to implement a speech recognition system. The need of such a system is to accurately recognize human voice at any circumstances, because the accent of each human differs for a single language, and also, the human voice is never devoid of noise and other emotional elements. So, the aim of such systems is to recognize human voices of different accents, even in a noisy environment. Because of this, an effective and robust speech recognition system has become important for machine and human interaction. Much voice-recognizing software depends purely on audio to recognize a voice, and one cannot guarantee that the audio will be noise free. In order to improve the accuracy and performance of traditional speech recognition system in a noisy environment, a new, nontraditional approach called a multimodal speech recognition system that uses both the audio and video features such as lip movement facial expression to recognize a speech is used. This article discusses the advantage of using video features along with the audio features with an experiment that proves it.