Multimodal Machine Translation Research Articles

Sanskrit to Hindi language translation using multimodal neural machine translation

Machine translation (MT) is a subfield of computer features that focuses on the automatic translation from one natural language into another without any human involvement. Due to native people interacting in a variety of languages, there is a great need for translating information between languages to send and communicate thoughts. However, they disregard the significance of semantic data encoded in the text features. In this paper, multimodal neural machine translation (MNMT) is proposed for Sanskrit-Hindi translation. The main goal of the proposed method is to fully utilize semantic text features on NMT architecture and to minimize testing and training time. The MNMT is validated on two different NMT architectures: recurrent neural network (RNN) and self-attention network (SAN). The MNMT method’s efficacy is demonstrated by employing the dataset of Sanskrit-Hindi Corpora. Extensive experimental outcomes represent the proposed method’s enhancement over baselines on both architectures. The existing methods, namely, English-to-Indian MT system, Sanskrit-Hindi MT system, and hybrid MT system are used to justify the efficacy of the MNMT method. When compared to the above-mentioned existing methods, RA-RNN respectively achieves a superior BLEU and METEOR of 80.5% and 75.3%, while the RA-SAN respectively achieves a superior BLEU and METEOR of 78.2% and 77.1%.

Dose multimodal machine translation can improve translation performance?

Dose multimodal machine translation can improve translation performance?

Unsupervised Multimodal Machine Translation for Low-resource Distant Language Pairs

Unsupervised machine translation (UMT) has recently attracted more attention from researchers, enabling models to translate when languages lack parallel corpora. However, the current works mainly consider close language pairs (e.g., English-German and English-French), and the effectiveness of visual content for distant language pairs has yet to be investigated. This article proposes an unsupervised multimodal machine translation model for low-resource distant language pairs. Specifically, we first employ adequate measures such as transliteration and re-ordering to bring distant language pairs closer together. We then use visual content to extend masked language modeling and generate visual masked language modeling for UMT. Finally, empirical experiments are conducted on our distant language pair dataset and the public Multi30k dataset. Experimental results demonstrate the superior performance of our model, with BLEU score improvements of 2.5 and 2.6 on translation for distant language pairs English-Uyghur and Chinese-Uyghur. Moreover, our model also brings remarkable results for close language pairs, improving 2.3 BLEU compared with the existing models in English-German.

Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models

Chain-of-thought (CoT) reasoning has exhibited impressive performance in language models for solving complex tasks and answering questions. However, many real-world questions require multi-modal information, such as text and images. Previous research on multi-modal CoT has primarily focused on extracting fixed image features from off-the-shelf vision models and then fusing them with text using attention mechanisms. This approach has limitations because these vision models were not designed for complex reasoning tasks and do not align well with language thoughts. To overcome this limitation, we introduce a novel approach for multi-modal CoT reasoning that utilizes latent space learning via diffusion processes to generate effective image features that align with language thoughts. Our method fuses image features and text representations at a deep level and improves the complex reasoning ability of multi-modal CoT. We demonstrate the efficacy of our proposed method on multi-modal ScienceQA and machine translation benchmarks, achieving state-of-the-art performance on ScienceQA. Overall, our approach offers a more robust and effective solution for multi-modal reasoning in language models, enhancing their ability to tackle complex real-world problems.

Progressive modality-complement aggregative multitransformer for domain multi-modal neural machine translation

Domain-specific Multi-modal Neural Machine Translation (DMNMT) aims to translate domain-specific sentences from a source language to a target language by incorporating text-related visual information. Generally, domain-specific text-image data often complement each other and have the potential to collaboratively enhance the representation of domain-specific information. Unfortunately, there is a considerable modality gap between image and text in data format and semantic expression, which leads to distinctive challenges in domain-text translation tasks. Narrowing the modality gap and improving domain-aware representation are two critical challenges in DMNMT. To this end, this paper proposes a progressive modality-complement aggregative MultiTransformer, which aims to simultaneously narrow the modality gap and capture domain-specific multi-modal representation. We first adopt a bidirectional progressive cross-modal interactive strategy to effectively narrow the text-to-text, text-to-visual, and visual-to-text semantics in the multi-modal representation space by integrating visual and text information layer-by-layer. Subsequently, we introduce a modality-complement MultiTransformer based on progressive cross-modal interaction to extract the domain-related multi-modal representation, thereby enhancing machine translation performance. Experiment results on the Fashion-MMT and Multi-30k datasets are conducted, and the results show that the proposed approach outperforms the compared state-of-the-art (SOTA) methods on the En-Zh task in E-commerce domain, En-De, En-Fr and En-Cs tasks of Multi-30k in general domain. The in-depth analysis confirms the validity of the proposed modality-complement MultiTransformer and bidirectional progressive cross-modal interactive strategy for DMNMT.

Multimodal Machine Translation

In recent years, neural network machine translation, especially in the field of multimodality, has developed rapidly. It has been widely used in natural languages processing tasks such as event detection and sentiment classification. The existing multimodal neural network machine translation is mostly based on the autoencoder framework of the attention mechanism, which further integrates spatial-visual features. However, due to the ubiquitous lack of corpus and the semantic interaction between multimodalities, the quality of machine translation is difficult to guarantee. Therefore, this paper proposes a multi-modal machine translation model that integrates external linguistic knowledge. Specifically, on the encoder side, we adopt the pre-trained Bert model to be used as an additional encoder to integrate with the original text encoder and picture encoder. Under the cooperation of the three encoders, a better text representation and picture representation at the source end is generated. Besides, the decoder decodes and generates a translation based on the image and text representation of the source. To sum up, this paper studies the visual-text semantic interaction on the encoder side and the visual-text semantic interaction on the decoder side, and further improves the quality of translation by introducing external linguistic knowledge. We compared the performance of the multimodal neural network machine translation model with pre-trained Bert and other baseline models in English German translation tasks on the multi30k data sets. The results show that the model can significantly improve the quality of multimodal neural network machine translation, which also verifies the importance of integrating external knowledge and visual text semantic interaction.

Multi-modal graph contrastive encoding for neural machine translation

As an important extension of conventional text-only neural machine translation (NMT), multi-modal neural machine translation (MNMT) aims to translate input source sentences paired with images into the target language. Although a lot of MNMT models have been proposed to perform multi-modal semantic fusion, they do not consider fine-grained semantic correspondences between semantic units of different modalities (i.e., words and visual objects), which can be exploited to refine multi-modal representation learning via fine-grained semantic interactions. To address this issue, we propose a graph-based multi-modal fusion encoder for NMT. Concretely, we first employ a unified multi-modal graph to represent the input sentence and image, in which the multi-modal semantic units are considered as the nodes in the graph, connected by two kinds of edges with different semantic relationships. Then, we stack multiple graph-based multi-modal fusion layers that iteratively conduct intra- and inter-modal interactions to learn node representations. Finally, via an attention mechanism, we induce a multi-modal context from the top node representations for the decoder. Particularly, we introduce a progressive contrastive learning strategy based on the multi-modal graph to refine the training of our proposed model, where hard negative samples are introduced gradually. To evaluate our model, we conduct experiments on commonly-used datasets. Experimental results and analysis show that our MNMT model obtains significant improvements over competitive baselines, achieving state-of-the-art performance on the Multi30K dataset.

Contrastive Adversarial Training for Multi-Modal Machine Translation

The multi-modal machine translation task is to improve translation quality with the help of additional visual input. It is expected to disambiguate or complement semantics while there are ambiguous words or incomplete expressions in the sentences. Existing methods have tried many ways to fuse visual information into text representations. However, only a minority of sentences need extra visual information as complementary. Without guidance, models tend to learn text-only translation from the major well-aligned translation pairs. In this article, we propose a contrastive adversarial training approach to enhance visual participation in semantic representation learning. By contrasting multi-modal input with the adversarial samples, the model learns to identify the most informed sample that is coupled with a congruent image and several visual objects extracted from it. This approach can prevent the visual information from being ignored and further fuse cross-modal information. We examine our method in three multi-modal language pairs. Experimental results show that our model is capable of improving translation accuracy. Further analysis shows that our model is more sensitive to visual information.

Morphology & word sense disambiguation embedded multimodal neural machine translation system between Sanskrit and Malayalam

Machine translation is the process of conversion from one language to another using computers. In spite of the existence of several traditional machine translation systems like rule-based, statistical etc., the development of neural machine translation has significantly improved translation quality. However, the morphological richness and agglutinative nature of Indian languages hampers the usage of neural machine translation for translation among Indian languages despite its potential to overcome the difficulties posed by conventional translation systems. Hence the combination of neural machine translation with morphology, part of speech tagger and word sense disambiguation is proposed to address these challenges and adapt it for bidirectional translation between Sanskrit and Malayalam. Conventional neural machine translations are unimodal systems that utilize textual data for translation. Translation quality of conventional neural machine translation is enhanced with a novel multimodal bidirectional neural machine translation system with text & speech as the two modalities based on parallel text corpus and speech database. The best performance score is obtained for two level fusion, the feature vectors extracted from speech signals using different transforms (Wavelet transform, 1-D sequency mapped real transform & 1-D GCD based mapped real transform) are fused and which is further fused with a context vector from the text modality. Automatic and manual review methods are employed to assess quality of translation. Findings from the experiments reveal that addition of speech modality improved the overall quality of translation. BLEU scores of 43.89 and 42.72 are obtained for Sanskrit to Malayalam and Malayalam to Sanskrit multimodal translation respectively.

Do cues in a video help in handling rare words in a machine translation system under a low-resource setting?

Video modality is an emerging research area among the numerous modalities utilized for multimodal machine translation. Multimodal machine translation uses multiple modalities to improve the machine-translated target language from the source language. However, the currently available multimodal dataset is focused on a few well-studied languages. In this paper, we propose a video-guided multimodal machine translation (VMMT) model under a low-resource setting by building a synthetic multimodal dataset of the English-Hindi language pair, the first one of its kind for this language pair. The VMMT system employs spatio-temporal video context as an additional input modality along with the source text. The spatio-temporal video context is extracted using a pre-trained 3D convolutional neural network. We report how well the VMMT systems outperform the text-only neural machine translation (NMT) system using automatic evaluation metrics and human evaluation on two test datasets: one in-domain and another out-domain. Our results indicate that the use of video context as an additional input modality enhances the performance of the MT system in resolving the various MT challenges, such as handling rare words, ambiguity, etc., in both English→Hindi and Hindi→English translations. Our experimental results show a significant improvement of up to +4.2 BLEU and +0.07 chrF scores in English→Hindi and +5.4 BLEU and +0.07 chrF scores in Hindi→English with our VMMT system over unimodal NMT system. Our findings highlight the potential of visual cues as an additional modality for improving machine translation systems especially in low-resource settings and emphasize the importance of synthetic multimodal datasets in addressing the scarcity of diverse data for less-studied language pairs.

Learning to decode to future success for multi-modal neural machine translation

Existing only-text NMT (neural machine translation) systems can benefit from explicitly modelling target future contexts as recurrent states. However, the modelled target future context is implicit in the conventional only-text NMT as the target is invisible in inference. For the Multi-modal Neural Machine Translation (MNMT), the visual annotation presents the content described in the bilingual parallel sentence pair, so-called multi-modal consistency. This consistency provides an advantage that future target context can be simulated in visual features. This paper proposes a novel translation model that allows MNMT to encode the future target context from the visual annotation in auto-regressive decoding. Our model uses visual-target consistency to enhance the target generation. Moreover, we use the multi-modal consistency that fully uses the visual annotation to encourage the semantic agreement between bilingual parallel sentences and the pivoted visual annotation. Empirical results on several recent multi-model translation datasets demonstrated the MNMT model which we proposed significantly improved translation performance on a strong baseline, especially achieving new state-of-the-art results on all three language pairs with visual annotations. Our code will be available after acceptance.

Multi-modal simultaneous machine translation fusion of image information

Simultaneous translation is to translate a sentence before people finish it, to understand the speaker's intention in real-time. At present, simultaneous machine translation still relies on text-to-text data resources. However, the output information from the encoder side is used for the decoder as the input data recourse in the pure text translation system. This information is only derived from the text content, and the input information is single, causing a shortage of decoding information at the decoder and the vocabulary is missed in translation. The translator will also visually capture the information of the surrounding scenes to assist himself in the translation work, based on this feature, we propose a multi-modal simultaneous machine translation of fusion image information. We extract information from the image, add the information to the decoder side of the translation system, increase the input data resource of the decoder, and help the system improve the translation quality. We use the Multi30K dataset for experimental verification. Compared with the translation system of plain text, the method we propose can translate more complete sentences, richer content, and better translation results.

Adding visual attention into encoder-decoder model for multi-modal machine translation

Multi-modal neural machine translation (MNMT) aims to integrate visual and textual information to translate source sentences into target and attracts a lot of attentions. Existing methods contribute a lot for capturing interactions between visual and textual features to improve the performance of neural machine translation (NMT). However, most of them don't consider the multi-modal consistency for MNMT. In fact, the image provides the semantic global consistency between different languages. We believe that adding bilingual-visual agreement into the encoder and decoder simultaneously can obtain bilingual representations and is useful for NMT. In this paper, we propose to simultaneously integrate visual information in encoder and decoder to learn the interactions between visual and textual features, in this paper called the model VMNMT. As the visual information provide global context, the encoder and decoder can learn the bilingual representations. Besides, we introduce a new bilingual-visual agreement decoder to learn to better representations of corresponding image-sentence pairs. In the experiment, the improvement was 2.02 BLEU on the English-German16 dataset and 1.9 BLEU on the English-German17 dataset. The results show that our method can outperform baselines on several widely-used datasets in terms of various metrics.

Hindi to English Multimodal Machine Translation on News Dataset in Low Resource Setting

This work proposes a multimodal Hindi-to-English machine translation on a news corpus by integrating multiple input modalities. The experimental dataset comprises of an image from a news article and its caption in the English-Hindi language. The parallel English-Hindi dataset is constructed by translating the image caption from English-language news articles into Hindi. Unlike standard images used for NLP tasks, images in news articles frequently depict significant events, locations, or individuals. The caption provides a more extensive description of the image, resulting in the inclusion of multiple identified entities. Our methodology consists of encoders for each input modality, allowing for simultaneous consideration of both text and image. We undertake two tests to examine how effectively the image content improves MT systems when 1) the entire image is considered first and 2) the largest object recognized in the image is considered. We employ Byte Pair Encoding (BPE) to represent the text, and the visual attributes are extracted from the image with VGG-19, a pre-trained CNN model. In terms of BLEU and chrF, our Multimodal Machine Translation (MMT) systems are superior than the baseline unimodal NMT system. Our MMT system outperforms the NMT model by +1.8 BLEU and +0.03 chrF.

Layer-Level Progressive Transformer With Modality Difference Awareness for Multi-Modal Neural Machine Translation

Layer-Level Progressive Transformer With Modality Difference Awareness for Multi-Modal Neural Machine Translation

Spatial Hierarchical Attention Network Based Video-guided Machine Translation

Video-guided machine translation, as one type of multimodal machine translation, aims to engage video contents as auxiliary information to address the word sense ambiguity problem in machine translation. Previous studies only use features from pre-trained action detection models as motion representations of the video to solve the verb sense ambiguity and neglect the noun sense ambiguity problem. To address this, we propose a video-guided machine translation system using both spatial and motion representations. For the spatial part, we propose a hierarchical attention network to model the spatial information from object-level to video-level. We investigate and discuss spatial features extracted from objects with pre-trained convolutional neural network models and spatial concept features extracted from object labels and attributes with pre-trained language models. We further investigate spatial feature filtering by referring to corresponding source sentences. Experiments on the VATEX dataset show that our system achieves a 35.86 BLEU-4 score, which is 0.51 score higher than the single model of the SOTA method. Experiments on the How2 dataset further verify the generalization ability of our proposed system.

Adding Visual Information to Improve Multimodal Machine Translation for Low-Resource Language

Machine translation makes it easy for people to communicate across languages. Multimodal machine translation is also one of the important directions of research in machine translation, which uses feature information such as images and audio to assist translation models in obtaining higher quality target languages. However, in the vast majority of current research work has been conducted on the basis of commonly used corpora such as English, French, German, less research has been done on low-resource languages, and this has left the translation of low-resource languages relatively behind. This paper selects the English-Hindi and English-Hausa corpus, researched on low-resource language translation. The different models we use for image feature information extraction are fusion of image features with text information in the text encoding process of translation, using image features to provide additional information, and assisting the translation model for translation. Compared with text-only machine translation, the experimental results show that our method improves 3 BLEU in the English-Hindi dataset and improves 0.47 BLEU in the English-Hausa dataset. In addition, we also analyze the effect of image feature information extracted by different feature extraction models on the translation results. Different models pay different attention to each region of the image, and ResNet model is able to extract more feature information compared to VGG model, which is more effective for translation.

Supervised Visual Attention for Simultaneous Multimodal Machine Translation

There has been a surge in research in multimodal machine translation (MMT), where additional modalities such as images are used to improve translation quality of textual systems. A particular use for such multimodal systems is the task of simultaneous machine translation, where visual context has been shown to complement the partial information provided by the source sentence, especially in the early phases of translation. In this paper, we propose the first Transformer-based simultaneous MMT architecture, which has not been previously explored in simultaneous translation. Additionally, we extend this model with an auxiliary supervision signal that guides the visual attention mechanism using labelled phrase-region alignments. We perform comprehensive experiments on three language directions and conduct thorough quantitative and qualitative analyses using both automatic metrics and manual inspection. Our results show that (i) supervised visual attention consistently improves the translation quality of the simultaneous MMT models, and (ii) fine-tuning the MMT with supervision loss enabled leads to better performance than training the MMT from scratch. Compared to the state-of-the-art, our proposed model achieves improvements of up to 2.3 BLEU and 3.5 METEOR points.

Image Visual Attention Mechanism-based Global and Local Semantic Information Fusion for Multi-modal English Machine Translation

<p>Machine translation is a hot research topic at present. Traditional machine translation methods are not effective because they require a large number of training samples. Image visual semantic information can improve the effect of the text machine translation model. Most of the existing works fuse the whole image visual semantic information into the translation model, but the image may contain different semantic objects. These different local semantic objects have different effects on the words prediction of the decoder. Therefore, this paper proposes a multi-modal machine translation model based on the image visual attention mechanism via global and local semantic information fusion. The global semantic information in the image and the local semantic information are fused into the text attention weight as the image attention. Thus, the alignment information between the hidden state of the decoder and the text of the source language is further enhanced. Experimental results on the English-German translation pair and the Indonesian-Chinese translation pair on the Multi30K dataset show that the proposed model has a better performance than the state-of-the-art multi-modal machine translation models, the BLEU values of English-German translation results and Indonesian-Chinese translation results exceed 43% and 29%, which proves the effectiveness of the proposed model.</p> <p> </p>

Dual-level interactive multimodal-mixup encoder for multi-modal neural machine translation

Dual-level interactive multimodal-mixup encoder for multi-modal neural machine translation