Speech Translation System Research Articles

Optimization and Performance Analysis of Real-time Speech Translation Systems Based on Mobile Technology

As globalization deepens and mobile technology rapidly advances, the demand for crosslinguistic communication has been steadily increasing, making real-time speech translation systems a research focus. However, given the limited computational capacity and storage space of mobile devices, optimizing system performance while maintaining translation quality has become a critical challenge. Current optimization approaches for real-time speech translation systems primarily focus on improvements to model architectures and hardware acceleration, often neglecting a systematic study of model compression. This is particularly evident when handling real-time data, where achieving both high efficiency and translation accuracy remains difficult. To address these challenges, a model compression method based on the connectionist temporal classification (CTC) criterion was proposed, along with an in-depth study of parameter compression tailored for mobile applications. The research focuses on two key areas: first, model compression techniques based on the CTC criterion were explored to enhance the efficiency of real-time speech translation; second, parameter compression methods were investigated to significantly reduce resource consumption in mobile applications while preserving translation quality. The aim of this study is to improve the performance and user experience of real-time speech translation systems on mobile devices.

Multilingual Speech Transcription and Translation System

This project implements a multilingual speech recognition and translation system using Python. The system leverages libraries such as `pygame` for audio playback, `gTTS` for text-to-speech conversion, `speech_recognition` for capturing and recognizing spoken language, and Machine Translation model for translating text between languages. Users interact with the system via a Streamlit-based web interface, where they can select the input and target languages and control the start and stop of the translation process. The program can auto-detect the input language if the user chooses, and it provides real-time feedback by displaying the recognized text, translating it into the chosen target language, and converting the translated text into speech. At the core of the system is a continuous loop that listens for speech input through the microphone, processes the audio to recognize the spoken words, and then translates the recognized text into the desired language. The translated text is not only displayed but also spoken aloud using the text-to-speech functionality. The interface is user-friendly, with clear indications of the current status (listening, processing, recognized text, translated text) and handles errors gracefully, providing feedback in case of issues. This project demonstrates a practical application of speech recognition and translation technologies, offering a seamless and interactive multilingual communication tool

Divergence-Guided Simultaneous Speech Translation

To achieve high-quality translation with low latency, a Simultaneous Speech Translation (SimulST) system relies on a policy module to decide whether to translate immediately or wait for additional streaming input, along with a translation model capable of effectively handling partial speech input. Prior research has tackled these components separately, either using ``wait-k'' policies based on fixed-length segments or detected word boundaries, or dynamic policies based on different strategies (e.g., meaningful units), while employing offline models for prefix-to-prefix translation. In this paper, we propose Divergence-Guided Simultaneous Speech Translation (DiG-SST), a tightly integrated approach focusing on both translation quality and latency for streaming input. Specifically, we introduce a simple yet effective prefix-based strategy for training translation models with partial speech input, and develop an adaptive policy that makes read/write decisions for the translation model based on the expected divergence in translation distributions resulting from future input. Our experiments on multiple translation directions of the MuST-C benchmark demonstrate that our approach achieves a better trade-off between translation quality and latency compared to existing methods.

Speaker voice normalization for end-to-end speech translation

Speaker voices exhibit acoustic variation. Our preliminary experiments reveal that normalized voice can significantly improve end-to-end speech translation. To mitigate the negative impact of acoustic voice variation across speakers on speech translation, we propose SVN-ST, a Speaker-Voice-Normalized end-to-end Speech Translation framework. In SVN-ST, we use synthetic speech inputs generated from a Text-to-Speech system to complement raw speech inputs. In order to explore synthetic speech inputs, we introduce two essential components for SVN-ST: an alignment adapter at the encoder side and a normalized speech knowledge distillation module at the decoder side. The former forces the representations of raw speech inputs to be close to those of synthetic (normalized) speech inputs while the latter attempts to guide the translations of raw speech inputs with those yielded from synthetic speech inputs. Two additional losses are also defined to equip with the two components. Experimental results on the MuST-C benchmark dataset demonstrate that SVN-ST outperforms previous state-of-the-art end-to-end non-normalized speech translation systems by 0.4 BLEU and cascaded speech translation systems by 2.3 BLEU. On the Covost 2 testset, SVN-ST also outperforms other normalized speech methods on robustness. Further analyses suggest that our model effectively aligns speech representations from different speakers, enhances robustness, and significantly improves sentence-level translation quality.

Design of Translation Error Correction System Based on Improved Seq2Seq

Speech translation technology is an important booster for promoting social communication and advancing human civilization. With the solid advancement of theories and technologies such as speech processing and machine translation, as well as the continuous deepening and development of computer science, the computing power and storage capacity have been further improved. Speech translation systems widely used in English, French, English and Chinese have successively reached a commercial level. However, the development of speech translation systems is limited by corpus resources and a lack of bilingual language research. The experiments and applications of speech translation in some languages are still in its early stages. In addition, existing research mostly adopts a cascading voice translation system as the basis, breaking through the key issues individually, and less adopts direct voice translation methods without relying on intermediate text representation, which brings problems such as cumbersome research content, complex translation models, and long translation delays. This article mainly focuses on the design of a translation error correction system based on improved Seq2Seq. Firstly, relevant research is summarized, relevant application methods are proposed, and the results are discussed. I hope to bring some inspiration to relevant researchers and provide assistance for the development of related fields.

University English Translation Teaching in the Context of Artificial Intelligence: Problems and Countermeasures

Abstract With the rapid development of artificial intelligence, university English translation teaching faces many challenges and opportunities. This article studies the problems in teaching in this context and proposes effective countermeasures. Diagnostic analysis and factor analysis are used in the article to conduct an in-depth study of the three levels of the teaching process: knowledge skills, autonomy, and emotional development. The students achieved a 70% mastery rate in knowledge and skills, while their independence and emotional development performance was 60% and 50%, respectively. This paper proposes an English speech translation system based on artificial intelligence to enhance teaching effectiveness. After a 14-week teaching experiment, the average grade of the students in the experimental class improved by about 6.82 points compared to the traditional teaching class, proving the system’s effectiveness. The study diagnosed the current problems in university English translation teaching, verified the potential of AI technology in improving teaching quality through empirical research, and combined with AI technology can effectively improve the quality and efficiency of university English translation teaching.

Multi-user and multi-language collaborative speech translation system and method in the era of big data

In the rapid development of information technology and network technology, the cultural knowledge of human society blends with each other, and people have more and more ways to obtain information. Faced with the continuous growth and various types of information composed of different languages, researchers have proposed to use machine translation technology to build auxiliary translation system, which can play an important role in daily learning and life work. After understanding the research and application status of speech translation system in the era of big data, this paper mainly explores the architecture of multi-user and multi-language collaborative speech translation system according to the application characteristics of multi-language assisted translation system in the new era, and puts forward practical application methods from the perspective of long-term development of the industry, so as to let the world economy and culture communicate with each other. Fully demonstrate the translation efficiency and application quality of the design system.

Retracted: Design of Intelligent Speech Translation System Based on Deep Learning

Retracted: Design of Intelligent Speech Translation System Based on Deep Learning

Cascade Speech Translation for the Kazakh Language

Speech translation systems have become indispensable in facilitating seamless communication across language barriers. This paper presents a cascade speech translation system tailored specifically for translating speech from the Kazakh language to Russian. The system aims to enable effective cross-lingual communication between Kazakh and Russian speakers, addressing the unique challenges posed by these languages. To develop the cascade speech translation system, we first created a dedicated speech translation dataset ST-kk-ru based on the ISSAI Corpus. The ST-kk-ru dataset comprises a large collection of Kazakh speech recordings along with their corresponding Russian translations. The automatic speech recognition (ASR) module of the system utilizes deep learning techniques to convert spoken Kazakh input into text. The machine translation (MT) module employs state-of-the-art neural machine translation methods, leveraging the parallel Kazakh-Russian translations available in the dataset to generate accurate translations. By conducting extensive experiments and evaluations, we have thoroughly assessed the performance of the cascade speech translation system on the ST-kk-ru dataset. The outcomes of our evaluation highlight the effectiveness of incorporating additional datasets for both the ASR and MT modules. This augmentation leads to a significant improvement in the performance of the cascade speech translation system, increasing the BLEU score by approximately 2 points when translating from Kazakh to Russian. These findings underscore the importance of leveraging supplementary data to enhance the capabilities of speech translation systems.

Improving End-to-End Speech Translation by Leveraging Auxiliary Speech and Text Data

We present a method for introducing a text encoder into pre-trained end-to-end speech translation systems. It enhances the ability of adapting one modality (i.e., source-language speech) to another (i.e., source-language text). Thus, the speech translation model can learn from both unlabeled and labeled data, especially when the source-language text data is abundant. Beyond this, we present a denoising method to build a robust text encoder that can deal with both normal and noisy text data. Our system sets new state-of-the-arts on the MuST-C En-De, En-Fr, and LibriSpeech En-Fr tasks.

Design of Intelligent Speech Translation System Based on Deep Learning

In order to solve the problem of low translation accuracy caused by complex sentence parameters in traditional machine translation systems, a method based on deep learning was proposed. First, MCU SPCE061A is used to study the problem of complex digital signal. The training data in the synchronous translation server support the translation services of a large number of users, and the translation results were displayed through the session interface of the user terminal. The PMDL model is used to detect the keyword signal, record the PCM audio data, and slice the collected pulse code modulation signal, so as to wake up the artificial intelligence voice service. Then, this study establishes a speech recognition process that accurately outputs the speech-related semantics. In this paper, a manual interactive synchronous translation program is designed with the input text as the search criterion, and the set is trimmed to obtain the best translation effect. The experimental results show that the sentence translation accuracy of the system is 0.9 ∼ 1.0. It is proved that the method based on deep learning solves the problem of low accuracy of the traditional translation system.

Armenian Speech Recognition System: Acoustic and Language Models

Nowadays automatic speech recognition (ASR) is an important task for machines. Several applications such as speech translation, virtual assistants and voice bot systems use ASR to understand human speech. Most of the research and available models are for widely used languages, such as English, German, French, Chinese and Spanish. This paper presents the Armenian speech recognition system. As a result of this research developed acoustic and language models for the Armenian language (modern ASR systems combine acoustic and language models to achieve higher accuracy). RNN-based Baidu’s Deep Speech deep neural network was used to train the acoustic model, and the KenLM toolkit was used to train the probabilistic language model. The acoustic model was trained and validated on ArmSpeech Armenian native speech corpus using transfer-learning and data augmentation techniques and tested on the Common Voice Armenian database. The language model was built based on the texts scraped from Armenian news websites. Final models are small in size and can be run and do real-time speech-to-text tasks on IoT devices. Testing on the Common Voice Armenian database the model gave 0.902565 WER and 0.305321 CER without the language model, and 0.552975 WER and 0.285904 CER with the language model. The paper aims to describe environment setup, data collection, acoustic and language models training processes, as well as final results and benchmarks.

TriECCC: Trilingual Corpus of the Extraordinary Chambers in the Courts of Cambodia for Speech Recognition and Translation Studies

This paper presents an extended work on the trilingual spoken language translation corpus of the Extraordinary Chambers in the Courts of Cambodia (ECCC), namely TriECCC. TriECCC is a simultaneously spoken language translation corpus with parallel resources of speech and text in three languages: Khmer, English, and French. This corpus has approximately [Formula: see text] thousand utterances, approximately [Formula: see text], [Formula: see text], and [Formula: see text] h in length of speech, and [Formula: see text], [Formula: see text] and [Formula: see text] million words in text, in Khmer, English, and French, respectively. We first report the baseline results of machine translation (MT), and speech translation (ST) systems, which show reasonable performance. We then investigate the use of the ROVER method to combine multiple MT outputs and fine-tune the pre-trained English–French MT models to enhance the Khmer MT systems. Experimental results show that the ROVER is effective for combining English-to-Khmer and French-to-Khmer systems. Fine-tuning from both single and multiple parents shows the effective improvement on the BLEU scores for Khmer-to-English/French and English/French-to-Khmer MT systems.

UWSpeech: Speech to Speech Translation for Unwritten Languages

Existing speech to speech translation systems heavily rely on the text of target language: they usually translate source language either to target text and then synthesize target speech from text, or directly to target speech with target text for auxiliary training. However, those methods cannot be applied to unwritten target languages, which have no written text or phoneme available. In this paper, we develop a translation system for unwritten languages, named as UWSpeech, which converts target unwritten speech into discrete tokens with a converter, and then translates source-language speech into target discrete tokens with a translator, and finally synthesizes target speech from target discrete tokens with an inverter. We propose a method called XL-VAE, which enhances vector quantized variational autoencoder (VQ-VAE) with cross-lingual (XL) speech recognition, to train the converter and inverter of UWSpeech jointly. Experiments on Fisher Spanish-English conversation translation dataset show that UWSpeech outperforms direct translation and VQ-VAE baseline by about 16 and 10 BLEU points respectively, which demonstrate the advantages and potentials of UWSpeech.

Accuracy Improvement of Speech Translation System Using Intentional Reference Data

In the fourth industrial revolution, speech recognizers are widely used to operate machines, such as artificial intelligence secretaries, phones, and factory machines. Hence, the accuracy of speech recognition is crucial. In previous studies, methods for improving the accuracy of speech recognition have been proposed. These methods may have led to improved accuracy of speech recognition and speech translation. However, it is still not entirely accurate to use speech translation to interpret sentences in another language because both results from the speech recognizers and translators have problems of misunderstanding the language. In this study, a method for improving the accuracy of speech translation from Korean to English using intentional reference data was developed. Korean sentences, which are usually used in software engineering classes, were used to evaluate the proposed method. From the results, the translated sentences were more fluently understandable than those that were not modified using the intentional reference data. It is expected that the proposed method will be useful for human communication, particularly in professional settings, such as conferences, meetings, or major classes in colleges.

Finite State Automata Approach for Text to Speech Translation System in Indonesian-Madurese Language

Madurese language is one of the regional languages in Indonesia. This laguage used by the Madurese people. Preservation of Madurese language now is minimal. Many migrants come from outside Madura, so communication between the surrounding communities in the Madura region often uses the national language, Indonesian. The use of Madurese as a language of communication began to decrease. This research is an effort to preserve Madurese language by utilizing the translator system technology. Madurese language is a regional language that is difficult to learn, because there are many differences found between writing and pronunciation. To overcome this problem this reasearch develop text to speech module in the Indonesian-Madurase language translation system. There are 3 versions of Madurese language level: Enja’-iyyeh, engghi-enten, and enggi-bunten. The conversion of text into sound is used with the help of syllable recording data created by the author. The process of chopping words into syllables is done using the two-level Finite State Automata (FSA) method. The output of the first level FSA becomes input for the second level. The application of FSA in Text to Speech applications is effectively used with accuracy value of 90%. The resulting sound output is in accordance with the results of syllables, but the pronunciation of some translated sentences does not have the correct intonation. The accuracy results of the intonation pattern in pronunciation of the system is 85%.

An Albanian Text-to-Speech System for the BabelDr Medical Speech Translator.

In this paper we present work on creating and evaluating a Text-to-Speech system for the Albanian language to be used in the BabelDr medical speech translation system. Its quality was assessed by twelve native speakers who provided feedback on 60 prompts generated by the synthesizer and on 60 real human recordings across three dimensions, namely comprehensibility, naturalness and likeability. The results suggest that the newly created voice can be incorporated in the content creation pipeline of the BabelDr platform.

Machine Learning for Classification of Emotion in Speech

The naturalness of the speech in any human being comes from his or her emotions. All human beings deliver and construe the messages with heavy use of emotions. So there is a need to develop a speech interface through which emotions embedded in the speech signal can be analyzed and processed. There are many speech translation systems developed with intent to interpret the inherent emotions in the speech signals but lack in processing the embedded emotions in the speech as because there is a lacuna in their modeling and depiction. The main objective of any speech processing system is to retrieve interesting information from speech like features, models so that the retrieved knowledge of interest can be further used in various speech processing applications. The scope of the present paper is to travel around the attributes of speech and its respective models with a goal to distinguish emotions by imprisoning precise information about emotion. This paper also studied various sources like source of excitement, vocal track system’s silhouettes and its sequence, attributes of supra- segment to obtain a rich source of emotional information of a speech. The paper end with a final conclusion saying that source of excitation and its characteristics may be be single handedly enough for efficient acknowledgment of emotions.

Implementation of Embedded Technology-Based English Speech Identification and Translation System

Due to the increase in globalization, communication between different countries has become more and more frequent. Language barriers are the most important issues in communication. Machine translation is limited to texts, and cannot be an adequate substitute for oral communication. In this study, a speech recognition and translation system based on embedded technology was developed for the purpose of English speech recognition and translation. The system adopted the Hidden Markov Model (HMM) and Windows CE operating system. Experiments involving English speech recognition and EnglishChinese translation found that the accuracy of the system in identifying English speech was about 88%, and the accuracy rate of the system in translating English to Chinese was over 85%. The embedded technology-based English speech recognition and translation system demonstrated a level of high accuracy in speech identification and translation, demonstrating its value as a practical application. Therefore, it merits further research and development.

Microcontroller Based Speech to Text Translation System

For the purpose of an effective communication, speech becomes a convenient conduit to convey messages as an important activity in human life. The need for better reception of voice messages between humans and the environment they interact with becomes a trend and an area of interest that needs to be studied for improvements. A speech to text translation system is an embedded based design that convert analogue signals particularly voice from an input into digital signals that a computer or any electronic device can understand and perform a required task or display the equivalent digital signal in text on a screen. Speech translation systems mitigates the bottlenecks to an efficient communication caused by other varieties of communication methods. Even though speech translation and recognition designs haven’t been well explored for electronic integration due to complexity and variation of sound signals from sources, this low cost, simple and portable project was incorporated to serve as a substratum and alternative in speech to text translation designs using microcontroller, in other to bridge the gaps in the world of human communications. This research paper addresses design methodology, limitations, recommendations and applications of the implemented speech to text translation system for improved communication reception.