Multimodal Machine Translation Research Articles

Speech recognition and intelligent translation under multimodal human–computer interaction system

Abstract The traditional translation robot is limited to the translation of single-mode text images and text videos, which has the problem of low translation accuracy. Therefore, speech recognition and intelligent translation in multimodal human–computer interaction (HCI) system are proposed. First, the network structure of speech recognition model in multi-channel HCI system is established, and the multi-head self-attention mechanism is constructed. Then, the artificial intelligence voice wake-up function is designed, and a multimodal machine translation model is constructed. On this basis, selective attention is added to obtain visual recognition of perceived text, and the decoder is used for multimodal gating fusion to realize the output of encoder translation results. Experimental results show that this method has high BLUE value and high translation accuracy.

Multimodal Machine Translation Based on Enhanced Knowledge Distillation and Feature Fusion

Existing research on multimodal machine translation (MMT) has typically enhanced bilingual translation by introducing additional alignment visual information. However, picture form requirements in multimodal datasets pose important constraints on the development of MMT because this requires a form of alignment between image, source text, and target text. This limitation is especially compounded by the fact that the inference phase, when aligning images, is not directly available in a conventional neural machine translation (NMT) setup. Therefore, we propose an innovative MMT framework called the DSKP-MMT model, which supports machine translation by enhancing knowledge distillation and feature refinement methods in the absence of images. Our model first generates multimodal features from the source text. Then, the purified features are obtained through the multimodal feature generator and knowledge distillation module. The features generated through image feature enhancement are subsequently further purified. Finally, the image–text fusion features are generated as input in the transformer-based machine translation reasoning task. In the Multi30K dataset test, the DSKP-MMT model has achieved a BLEU of 40.42 and a METEOR of 58.15, showing its ability to improve translation effectiveness and facilitating utterance communication.

Encoder–Decoder Calibration for Multimodal Machine Translation

Encoder–Decoder Calibration for Multimodal Machine Translation

CLIP-enhanced multimodal machine translation: integrating visual and label features with transformer fusion

CLIP-enhanced multimodal machine translation: integrating visual and label features with transformer fusion

Multimodal Machine Translation Approaches for Indian Languages: A Comprehensive Survey

Multimodal machine translation (MMT) is a challenging task in the linguistically diverse Indian landscape. Machine translation refers to the task of automatically converting content from one language to another without human involvement. Within the realm of natural language processing, a significant challenge arises from the inherent ambiguity present in human language. Translation ambiguity is a cross-lingual phenomenon that can manifest itself for various reasons, including lexical ambiguity, the occasional need to impute missing words, the presence of gen-der ambiguity, and word-sense ambiguities. These factors can lead to a decrease in translation accuracy. The integration of multiple modalities, such as images, videos, and audio, in addition to text, plays a pivotal role in improving the robustness and precision of translation systems. Over the past five years, extensive research has been dedicated to incorporating secondary modalities alongside text to improve language translation and comprehension. In this comprehensive study, our objective was to identify and explore promising MMT approaches, available corpora, eval-uation metrics, research challenges, and the future direction of research specifically for Indian languages. We evaluated 81 papers, including MMT models, MMT dataset in Indian languages, survey on MMT approach, and the effects of multiple modalities in machine translation. The performance of the different proposed approaches has also been briefly analyzed on the basis of the claimed results and comparative evaluations. Finally, the challenges associated with the MMT task for India and some possible directions for future research in this domain are highlighted.

Sanskrit to Hindi language translation using multimodal neural machine translation

Machine translation (MT) is a subfield of computer features that focuses on the automatic translation from one natural language into another without any human involvement. Due to native people interacting in a variety of languages, there is a great need for translating information between languages to send and communicate thoughts. However, they disregard the significance of semantic data encoded in the text features. In this paper, multimodal neural machine translation (MNMT) is proposed for Sanskrit-Hindi translation. The main goal of the proposed method is to fully utilize semantic text features on NMT architecture and to minimize testing and training time. The MNMT is validated on two different NMT architectures: recurrent neural network (RNN) and self-attention network (SAN). The MNMT method’s efficacy is demonstrated by employing the dataset of Sanskrit-Hindi Corpora. Extensive experimental outcomes represent the proposed method’s enhancement over baselines on both architectures. The existing methods, namely, English-to-Indian MT system, Sanskrit-Hindi MT system, and hybrid MT system are used to justify the efficacy of the MNMT method. When compared to the above-mentioned existing methods, RA-RNN respectively achieves a superior BLEU and METEOR of 80.5% and 75.3%, while the RA-SAN respectively achieves a superior BLEU and METEOR of 78.2% and 77.1%.

Dose multimodal machine translation can improve translation performance?

Dose multimodal machine translation can improve translation performance?

Unsupervised Multimodal Machine Translation for Low-resource Distant Language Pairs

Unsupervised machine translation (UMT) has recently attracted more attention from researchers, enabling models to translate when languages lack parallel corpora. However, the current works mainly consider close language pairs (e.g., English-German and English-French), and the effectiveness of visual content for distant language pairs has yet to be investigated. This article proposes an unsupervised multimodal machine translation model for low-resource distant language pairs. Specifically, we first employ adequate measures such as transliteration and re-ordering to bring distant language pairs closer together. We then use visual content to extend masked language modeling and generate visual masked language modeling for UMT. Finally, empirical experiments are conducted on our distant language pair dataset and the public Multi30k dataset. Experimental results demonstrate the superior performance of our model, with BLEU score improvements of 2.5 and 2.6 on translation for distant language pairs English-Uyghur and Chinese-Uyghur. Moreover, our model also brings remarkable results for close language pairs, improving 2.3 BLEU compared with the existing models in English-German.

Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models

Chain-of-thought (CoT) reasoning has exhibited impressive performance in language models for solving complex tasks and answering questions. However, many real-world questions require multi-modal information, such as text and images. Previous research on multi-modal CoT has primarily focused on extracting fixed image features from off-the-shelf vision models and then fusing them with text using attention mechanisms. This approach has limitations because these vision models were not designed for complex reasoning tasks and do not align well with language thoughts. To overcome this limitation, we introduce a novel approach for multi-modal CoT reasoning that utilizes latent space learning via diffusion processes to generate effective image features that align with language thoughts. Our method fuses image features and text representations at a deep level and improves the complex reasoning ability of multi-modal CoT. We demonstrate the efficacy of our proposed method on multi-modal ScienceQA and machine translation benchmarks, achieving state-of-the-art performance on ScienceQA. Overall, our approach offers a more robust and effective solution for multi-modal reasoning in language models, enhancing their ability to tackle complex real-world problems.

Progressive modality-complement aggregative multitransformer for domain multi-modal neural machine translation

Domain-specific Multi-modal Neural Machine Translation (DMNMT) aims to translate domain-specific sentences from a source language to a target language by incorporating text-related visual information. Generally, domain-specific text-image data often complement each other and have the potential to collaboratively enhance the representation of domain-specific information. Unfortunately, there is a considerable modality gap between image and text in data format and semantic expression, which leads to distinctive challenges in domain-text translation tasks. Narrowing the modality gap and improving domain-aware representation are two critical challenges in DMNMT. To this end, this paper proposes a progressive modality-complement aggregative MultiTransformer, which aims to simultaneously narrow the modality gap and capture domain-specific multi-modal representation. We first adopt a bidirectional progressive cross-modal interactive strategy to effectively narrow the text-to-text, text-to-visual, and visual-to-text semantics in the multi-modal representation space by integrating visual and text information layer-by-layer. Subsequently, we introduce a modality-complement MultiTransformer based on progressive cross-modal interaction to extract the domain-related multi-modal representation, thereby enhancing machine translation performance. Experiment results on the Fashion-MMT and Multi-30k datasets are conducted, and the results show that the proposed approach outperforms the compared state-of-the-art (SOTA) methods on the En-Zh task in E-commerce domain, En-De, En-Fr and En-Cs tasks of Multi-30k in general domain. The in-depth analysis confirms the validity of the proposed modality-complement MultiTransformer and bidirectional progressive cross-modal interactive strategy for DMNMT.

Multimodal Machine Translation

In recent years, neural network machine translation, especially in the field of multimodality, has developed rapidly. It has been widely used in natural languages processing tasks such as event detection and sentiment classification. The existing multimodal neural network machine translation is mostly based on the autoencoder framework of the attention mechanism, which further integrates spatial-visual features. However, due to the ubiquitous lack of corpus and the semantic interaction between multimodalities, the quality of machine translation is difficult to guarantee. Therefore, this paper proposes a multi-modal machine translation model that integrates external linguistic knowledge. Specifically, on the encoder side, we adopt the pre-trained Bert model to be used as an additional encoder to integrate with the original text encoder and picture encoder. Under the cooperation of the three encoders, a better text representation and picture representation at the source end is generated. Besides, the decoder decodes and generates a translation based on the image and text representation of the source. To sum up, this paper studies the visual-text semantic interaction on the encoder side and the visual-text semantic interaction on the decoder side, and further improves the quality of translation by introducing external linguistic knowledge. We compared the performance of the multimodal neural network machine translation model with pre-trained Bert and other baseline models in English German translation tasks on the multi30k data sets. The results show that the model can significantly improve the quality of multimodal neural network machine translation, which also verifies the importance of integrating external knowledge and visual text semantic interaction.

Multi-modal graph contrastive encoding for neural machine translation

As an important extension of conventional text-only neural machine translation (NMT), multi-modal neural machine translation (MNMT) aims to translate input source sentences paired with images into the target language. Although a lot of MNMT models have been proposed to perform multi-modal semantic fusion, they do not consider fine-grained semantic correspondences between semantic units of different modalities (i.e., words and visual objects), which can be exploited to refine multi-modal representation learning via fine-grained semantic interactions. To address this issue, we propose a graph-based multi-modal fusion encoder for NMT. Concretely, we first employ a unified multi-modal graph to represent the input sentence and image, in which the multi-modal semantic units are considered as the nodes in the graph, connected by two kinds of edges with different semantic relationships. Then, we stack multiple graph-based multi-modal fusion layers that iteratively conduct intra- and inter-modal interactions to learn node representations. Finally, via an attention mechanism, we induce a multi-modal context from the top node representations for the decoder. Particularly, we introduce a progressive contrastive learning strategy based on the multi-modal graph to refine the training of our proposed model, where hard negative samples are introduced gradually. To evaluate our model, we conduct experiments on commonly-used datasets. Experimental results and analysis show that our MNMT model obtains significant improvements over competitive baselines, achieving state-of-the-art performance on the Multi30K dataset.

Contrastive Adversarial Training for Multi-Modal Machine Translation

The multi-modal machine translation task is to improve translation quality with the help of additional visual input. It is expected to disambiguate or complement semantics while there are ambiguous words or incomplete expressions in the sentences. Existing methods have tried many ways to fuse visual information into text representations. However, only a minority of sentences need extra visual information as complementary. Without guidance, models tend to learn text-only translation from the major well-aligned translation pairs. In this article, we propose a contrastive adversarial training approach to enhance visual participation in semantic representation learning. By contrasting multi-modal input with the adversarial samples, the model learns to identify the most informed sample that is coupled with a congruent image and several visual objects extracted from it. This approach can prevent the visual information from being ignored and further fuse cross-modal information. We examine our method in three multi-modal language pairs. Experimental results show that our model is capable of improving translation accuracy. Further analysis shows that our model is more sensitive to visual information.

Morphology & word sense disambiguation embedded multimodal neural machine translation system between Sanskrit and Malayalam

Machine translation is the process of conversion from one language to another using computers. In spite of the existence of several traditional machine translation systems like rule-based, statistical etc., the development of neural machine translation has significantly improved translation quality. However, the morphological richness and agglutinative nature of Indian languages hampers the usage of neural machine translation for translation among Indian languages despite its potential to overcome the difficulties posed by conventional translation systems. Hence the combination of neural machine translation with morphology, part of speech tagger and word sense disambiguation is proposed to address these challenges and adapt it for bidirectional translation between Sanskrit and Malayalam. Conventional neural machine translations are unimodal systems that utilize textual data for translation. Translation quality of conventional neural machine translation is enhanced with a novel multimodal bidirectional neural machine translation system with text & speech as the two modalities based on parallel text corpus and speech database. The best performance score is obtained for two level fusion, the feature vectors extracted from speech signals using different transforms (Wavelet transform, 1-D sequency mapped real transform & 1-D GCD based mapped real transform) are fused and which is further fused with a context vector from the text modality. Automatic and manual review methods are employed to assess quality of translation. Findings from the experiments reveal that addition of speech modality improved the overall quality of translation. BLEU scores of 43.89 and 42.72 are obtained for Sanskrit to Malayalam and Malayalam to Sanskrit multimodal translation respectively.

Do cues in a video help in handling rare words in a machine translation system under a low-resource setting?

Video modality is an emerging research area among the numerous modalities utilized for multimodal machine translation. Multimodal machine translation uses multiple modalities to improve the machine-translated target language from the source language. However, the currently available multimodal dataset is focused on a few well-studied languages. In this paper, we propose a video-guided multimodal machine translation (VMMT) model under a low-resource setting by building a synthetic multimodal dataset of the English-Hindi language pair, the first one of its kind for this language pair. The VMMT system employs spatio-temporal video context as an additional input modality along with the source text. The spatio-temporal video context is extracted using a pre-trained 3D convolutional neural network. We report how well the VMMT systems outperform the text-only neural machine translation (NMT) system using automatic evaluation metrics and human evaluation on two test datasets: one in-domain and another out-domain. Our results indicate that the use of video context as an additional input modality enhances the performance of the MT system in resolving the various MT challenges, such as handling rare words, ambiguity, etc., in both English→Hindi and Hindi→English translations. Our experimental results show a significant improvement of up to +4.2 BLEU and +0.07 chrF scores in English→Hindi and +5.4 BLEU and +0.07 chrF scores in Hindi→English with our VMMT system over unimodal NMT system. Our findings highlight the potential of visual cues as an additional modality for improving machine translation systems especially in low-resource settings and emphasize the importance of synthetic multimodal datasets in addressing the scarcity of diverse data for less-studied language pairs.

Learning to decode to future success for multi-modal neural machine translation

Existing only-text NMT (neural machine translation) systems can benefit from explicitly modelling target future contexts as recurrent states. However, the modelled target future context is implicit in the conventional only-text NMT as the target is invisible in inference. For the Multi-modal Neural Machine Translation (MNMT), the visual annotation presents the content described in the bilingual parallel sentence pair, so-called multi-modal consistency. This consistency provides an advantage that future target context can be simulated in visual features. This paper proposes a novel translation model that allows MNMT to encode the future target context from the visual annotation in auto-regressive decoding. Our model uses visual-target consistency to enhance the target generation. Moreover, we use the multi-modal consistency that fully uses the visual annotation to encourage the semantic agreement between bilingual parallel sentences and the pivoted visual annotation. Empirical results on several recent multi-model translation datasets demonstrated the MNMT model which we proposed significantly improved translation performance on a strong baseline, especially achieving new state-of-the-art results on all three language pairs with visual annotations. Our code will be available after acceptance.

Multi-modal simultaneous machine translation fusion of image information

Simultaneous translation is to translate a sentence before people finish it, to understand the speaker's intention in real-time. At present, simultaneous machine translation still relies on text-to-text data resources. However, the output information from the encoder side is used for the decoder as the input data recourse in the pure text translation system. This information is only derived from the text content, and the input information is single, causing a shortage of decoding information at the decoder and the vocabulary is missed in translation. The translator will also visually capture the information of the surrounding scenes to assist himself in the translation work, based on this feature, we propose a multi-modal simultaneous machine translation of fusion image information. We extract information from the image, add the information to the decoder side of the translation system, increase the input data resource of the decoder, and help the system improve the translation quality. We use the Multi30K dataset for experimental verification. Compared with the translation system of plain text, the method we propose can translate more complete sentences, richer content, and better translation results.

Adding visual attention into encoder-decoder model for multi-modal machine translation

Multi-modal neural machine translation (MNMT) aims to integrate visual and textual information to translate source sentences into target and attracts a lot of attentions. Existing methods contribute a lot for capturing interactions between visual and textual features to improve the performance of neural machine translation (NMT). However, most of them don't consider the multi-modal consistency for MNMT. In fact, the image provides the semantic global consistency between different languages. We believe that adding bilingual-visual agreement into the encoder and decoder simultaneously can obtain bilingual representations and is useful for NMT. In this paper, we propose to simultaneously integrate visual information in encoder and decoder to learn the interactions between visual and textual features, in this paper called the model VMNMT. As the visual information provide global context, the encoder and decoder can learn the bilingual representations. Besides, we introduce a new bilingual-visual agreement decoder to learn to better representations of corresponding image-sentence pairs. In the experiment, the improvement was 2.02 BLEU on the English-German16 dataset and 1.9 BLEU on the English-German17 dataset. The results show that our method can outperform baselines on several widely-used datasets in terms of various metrics.

Hindi to English Multimodal Machine Translation on News Dataset in Low Resource Setting

This work proposes a multimodal Hindi-to-English machine translation on a news corpus by integrating multiple input modalities. The experimental dataset comprises of an image from a news article and its caption in the English-Hindi language. The parallel English-Hindi dataset is constructed by translating the image caption from English-language news articles into Hindi. Unlike standard images used for NLP tasks, images in news articles frequently depict significant events, locations, or individuals. The caption provides a more extensive description of the image, resulting in the inclusion of multiple identified entities. Our methodology consists of encoders for each input modality, allowing for simultaneous consideration of both text and image. We undertake two tests to examine how effectively the image content improves MT systems when 1) the entire image is considered first and 2) the largest object recognized in the image is considered. We employ Byte Pair Encoding (BPE) to represent the text, and the visual attributes are extracted from the image with VGG-19, a pre-trained CNN model. In terms of BLEU and chrF, our Multimodal Machine Translation (MMT) systems are superior than the baseline unimodal NMT system. Our MMT system outperforms the NMT model by +1.8 BLEU and +0.03 chrF.

Layer-Level Progressive Transformer With Modality Difference Awareness for Multi-Modal Neural Machine Translation

Layer-Level Progressive Transformer With Modality Difference Awareness for Multi-Modal Neural Machine Translation