Real-time Speech Translation Research Articles

Real-time Transcription Language Translation using Webrtc

This literature survey paper titled "Real-Time Transcription Language Translation Using WebRTC" explores the current landscape of real-time speech translation technologies. We summarize and analyze various research efforts in the domain, highlighting their methodologies, findings, and limitations. Additionally, we propose a novel approach that leverages WebRTC to enhance real-time transcription and translation efficiency. By identifying gaps in existing studies, we outline the scope for future research, aiming to improve user experience in multilingual communication. This survey serves as a comprehensive resource for researchers and practitioners interested in advancing real-time language translation technologies

Multilingual sound sopt synthesis systems

Localized sound zone synthesis, which can generate acoustically bright and dark zones using loudspeakers, is gaining attention as one of important acoustic communication techniques. By superposing multiple localized sound zones, multiple sound spot synthesis can simultaneously deriver different sound signals at different zones using a loudspeaker array. Additionally, it can be implemented as a user interface for real-time speech translation by combining multilingual speech synthesis technology. This paper introduces multilingual sound spot synthesis systems based on spatial Fourier transform implemented by using compact circular array of 16 loudspeakers and linear array of 64 loudspeakers combined with multilingual neural speech synthesis.

Performance Analysis of WebRTC and SIP-based Audio and Video Communication Systems

Video and audio communications have become an integral part of all spheres of life. Two commonly used real-time communication protocols for IP-based video and audio communications are the session initiation protocol (SIP) and web real-time communications (WebRTC). These two protocols have been widely used in softphone and video conferencing applications. The main aim of this paper is to make a comparative analysis of the performance of client server applications for video and audio communications developed by both SIP and WebRTC. The SIP system comprises of a desktop client developed in C#, a mobile client developed in Android studio and a FreeSwitch server. For the WebRTC application, the client was developed in JavaScript and the server in Node.js. The WebRTC application also included a speech translation API for real-time speech translation and employed two different codecs namely VP8 and VP9 via a modification of the Session Description Protocol (SDP) header. Results showed that WebRTC peer-to-peer audio communication provides a much lower quality in terms of SSNR than the SIP audio calls. However, for video communications, WebRTC provides better quality in terms of PSNR than SIP. The performance of the WebRTC application was also evaluated with the different codecs in terms of PSNR as well as the impact of the real-time speech translation delay.

Adapting Translation Models for Transcript Disfluency Detection

Transcript disfluency detection (TDD) is an important component of the real-time speech translation system, which arouses more and more interests in recent years. This paper presents our study on adapting neural machine translation (NMT) models for TDD. We propose a general training framework for adapting NMT models to TDD task rapidly. In this framework, the main structure of the model is implemented similar to the NMT model. Additionally, several extended modules and training techniques which are independent of the NMT model are proposed to improve the performance, such as the constrained decoding, denoising autoencoder initialization and a TDD-specific training object. With the proposed training framework, we achieve significant improvement. However, it is too slow in decoding to be practical. To build a feasible and production-ready solution for TDD, we propose a fast non-autoregressive TDD model following the non-autoregressive NMT model emerged recently. Even we do not assume the specific architecture of the NMT model, we build our TDD model on the basis of Transformer, which is the state-of-the-art NMT model. We conduct extensive experiments on the publicly available set, Switchboard, and in-house Chinese set. Experimental results show that the proposed model significantly outperforms previous state-ofthe-art models.

Statistical speech translation system based on voice recognition optimization using multimodal sources of knowledge and characteristics vectors

Synergic combination of different sources of knowledge is a key issue for the development of modern statistical translators. In this work, a speech translation statistical system that adds additional other-than-voice information in a voice translation system is presented. The additional information serves as a base for the log-linear combination of several statistical models. We describe the theoretical framework of the problem, summarize the overall architecture of the system, and show how the system is enhanced with the additional information. Our real prototype implements a real-time speech translation system from Spanish to English that is adapted to specific teaching-related environments.

Interlingua-based English–Korean Two-way Speech Translation of Doctor–Patient Dialogues with CCLINC

Development of a robust two-way real-time speech translation system exposes researchers and system developers to various challenges of machine translation (MT) and spoken language dialogues. The need for communicating in at least two different languages poses problems not present for a monolingual spoken language dialogue system, where no MT engine is embedded within the process flow. Integration of various component modules for real-time operation poses challenges not present for text translation. In this paper, we present the CCLINC (Common Coalition Language System at Lincoln Laboratory) English–Korean two-way speech translation system prototype trained on doctor–patient dialogues, which integrates various techniques to tackle the challenges of automatic real-time speech translation. Key features of the system include (i) language–independent meaning representation which preserves the hierarchical predicate–argument structure of an input utterance, providing a powerful mechanism for discourse understanding of utterances originating from different languages, word-sense disambiguation and generation of various word orders of many languages, (ii) adoption of the DARPA Communicator architecture, a plug-and-play distributed system architecture which facilitates integration of component modules and system operation in real time, and (iii) automatic acquisition of grammar rules and lexicons for easy porting of the system to different languages and domains. We describe these features in detail and present experimental results.

CMU wearable computers for real-time speech translation

Carnegie Mellon's Wearable Computers Laboratory has built four generations of real-time speech translation wearable computers, culminating in the Speech Translator Smart Module. Smart Modules use a family of interoperable modules supporting real-time speech recognition, language translation, and speech synthesis. We examine the effect of various design factors on performance with emphasis on modularity and scalability. A system-level approach to power/performance optimization is described that improved the metric of (performance/(weight*volume*power)) by over a factor of 300 through the four generations.