Flickr30K Datasets Research Articles

VITR: Augmenting Vision Transformers with Relation-Focused Learning for Cross-Modal Information Retrieval

The relations expressed in user queries are vital for cross-modal information retrieval. Relation-focused cross-modal retrieval aims to retrieve information that corresponds to these relations, enabling effective retrieval across different modalities. Pre-trained networks, such as Contrastive Language-Image Pre-training networks, have gained significant attention and acclaim for their exceptional performance in various cross-modal learning tasks. However, the Vision Transformer (ViT) used in these networks is limited in its ability to focus on image region relations. Specifically, ViT is trained to match images with relevant descriptions at the global level, without considering the alignment between image regions and descriptions. This article introduces VITR, a novel network that enhances ViT by extracting and reasoning about image region relations based on a local encoder. VITR is comprised of two key components. Firstly, it extends the capabilities of ViT-based cross-modal networks by enabling them to extract and reason with region relations present in images. Secondly, VITR incorporates a fusion module that combines the reasoned results with global knowledge to predict similarity scores between images and descriptions. The proposed VITR network was evaluated through experiments on the tasks of relation-focused cross-modal information retrieval. The results derived from the analysis of the Flickr30K, MS-COCO, RefCOCOg, and CLEVR datasets demonstrated that the proposed VITR network consistently outperforms state-of-the-art networks in image-to-text and text-to-image retrieval.

Novel concept-based image captioning models using LSTM and multi-encoder transformer architecture

Captioning an image involves using a combination of vision and language models to describe the image in an expressive and concise sentence. Successful captioning task requires extracting as much information as possible from the corresponding image. One of these key pieces of information is the topic to which the image belongs. The state-of-the-art methods used topic modeling depending only on caption text in order to extract these topics. The problem with extracting the topics using topic modeling only on caption text is that it lacks the consideration of the image’s semantic information. Instead, concept modeling extracts the concepts directly from the images in addition to considering the corresponding caption text. Concept modeling can be used in image captioning to extremely capture the image contexts and benefit from it to produce more accurate descriptions. In this paper, novel image captioning models are proposed by utilizing the concept modeling technique. The first concept-based model is proposed by utilizing LSTM as a decoder while the second model is proposed in association with new multi-encoder transformer architecture. Standard metrics have been used to evaluate the proposed models using Microsoft COCO and Flickr30K datasets. The proposed models outperformed the related work methods with reduced computational complexity.

A method for image–text matching based on semantic filtering and adaptive adjustment

As image–text matching (a critical task in the field of computer vision) links cross-modal data, it has captured extensive attention. Most of the existing methods intended for matching images and texts explore the local similarity levels between images and sentences to align images with texts. Even though this fine-grained approach has remarkable gains, how to further mine the deep semantics between data pairs and focus on the essential semantics in data remains to be quested. In this work, a new semantic filtering and adaptive approach (FAAR) was proposed to ease the above problem. To be specific, the filtered attention (FA) module selectively focuses on typical alignments with the interference of meaningless comparisons eliminated. Next, the adaptive regulator (AR) further adjusts the attention weights of key segments for filtered regions and words. The superiority of our proposed method was validated by a number of qualitative experiments and analyses on the Flickr30K and MSCOCO data sets.

Zero-Shot Content-Based Crossmodal Recommendation System

Information Recommendation (IR) systems are conventionally designed to operate within a single modality at a time, such as Text2Text or Image2Image. However, the concept of cross-modality aims to facilitate a versatile recommendation experience across different modalities, such as Text2Image. In recent years, significant strides have been made in developing neural recommender models that are multimodal and capable of generalizing across a broad spectrum of domains in a zero-shot manner, thanks to the robust representation capabilities of neural networks. These architectures enable the generation of embeddings for assets (i.e., content uploaded on a platform by users), presenting a concise representation of their semantics and allowing for comparisons through similarity ranking. In this paper, we present ZCCR, a Zero-shot Content-based Crossmodal Recommendation System that leverages knowledge from large-scale pretrained Vision-Language Models (VLMs) such as CLIP and ALBEF to redefine the recommendation task as a zero-shot retrieval task, eliminating the need for labeled data or prior knowledge about the recommended content. Furthermore, ZCCR performs crossmodal similarity search on an optimized index, such as FAISS, to improve the speed of recommendation. The goal is to recommend to the user assets that are similar to those they have previously uploaded, commonly referred to as their user profile. Within the user profile, we identify “areas of interest”—groups of assets associated with specific user interests, such as cooking, sports, or cars. To identify these areas of interest and construct the search query for the retrieval operation, ZCCR employs an innovative use of Agglomerative Clustering. This technique groups user past assets by similarity without requiring prior knowledge of the number of clusters. Once the areas of interest, or clusters, are identified, the centroid is utilized as the search seed to find similar assets. Experimental results demonstrate the efficiency of the selected components in terms of search time and retrieval performance on modified MSCOCO and FLICKR30k datasets tailored for the recommendation task. Furthermore, ZCCR outperforms both a baseline tagging system (BT) and a more advanced tag system which utilizes a Large Language Model (LLM) to extract embeddings from tags. The results show that, even compared with the latter, embeddings directly extracted from raw assets yield superior outcomes compared to relying on intermediate tags generated by other tools. The code implementation to reproduce all experiments and results shown in this paper is provided at the following link: ZCCR-experiments.

A transformer-based Urdu image caption generation

Image caption generation has emerged as a remarkable development that bridges the gap between Natural Language Processing (NLP) and Computer Vision (CV). It lies at the intersection of these fields and presents unique challenges, particularly when dealing with low-resource languages such as Urdu. Limited research on basic Urdu language understanding necessitates further exploration in this domain. In this study, we propose three Seq2Seq-based architectures specifically tailored for Urdu image caption generation. Our approach involves leveraging transformer models to generate captions in Urdu, a significantly more challenging task than English. To facilitate the training and evaluation of our models, we created an Urdu-translated subset of the flickr8k dataset, which contains images featuring dogs in action accompanied by corresponding Urdu captions. Our designed models encompassed a deep learning-based approach, utilizing three different architectures: Convolutional Neural Network (CNN) + Long Short-term Memory (LSTM) with Soft attention employing word2Vec embeddings, CNN+Transformer, and Vit+Roberta models. Experimental results demonstrate that our proposed model outperforms existing state-of-the-art approaches, achieving 86 BLEU-1 and 90 BERT-F1 scores. The generated Urdu image captions exhibit syntactic, contextual, and semantic correctness. Our study highlights the inherent challenges associated with retraining models on low-resource languages. Our findings highlight the potential of pre-trained models for facilitating the development of NLP and CV applications in low-resource language settings.

Multi-level Symmetric Semantic Alignment Network for image–text matching

Image–text matching has attracted much attention as one of the visual-linguistic tasks. Most of the existing methods tend to concentrate on single-level semantic similarity by global embeddings or local alignment, which fail to distinguish the ambiguous image–text instances with highly similar contexts due to insufficient interactions of cross-modal features. To mitigate this problem, in this paper, we propose a novel Multi-level Symmetric Semantic Alignment Network (MSSAN), which fully exploits the multi-level semantic features to capture the complicated correlations between images and text. Specifically, global–global-level alignment is first performed based on visual–textual global representations. Then, considering the local semantic consistency, local–local-level alignment is carried out through inter-modal bidirectional cross-attention module to model fine-grained region–word relations. Moreover, a Multi-granularity Feature Fusion Module (MFFM) is constructed to learn intra-modal distribution of significance and incorporate global concepts into local features to obtain more comprehensive semantic representations, thus achieving global–local-level alignment. Finally, in order to achieve a strong separation of semantically similar samples, we develop a novel generic triplet ranking loss with adaptively updatable margins to train our model. Extensive experimental results on Flickr8K, MSCOCO and Flickr30K datasets demonstrate that our proposed MSSAN is superior to other state-of-the-art methods by a considerable margin.

RagBERT: Relationship-aligned and grammar-wise BERT model for image captioning

Image captioning has become one of the most popular research problems in the field of artificial intelligence. Although many studies have achieved excellent results, there still are some challenges, for example, cross-modal feature alignment lacks explicit guidance, and model-generated sentences contain grammatical errors. In this paper, we propose a relationship-aligned and grammar-wise BERT model, which integrates a relationship exploration module and a grammar enhancement module into the BERT-based model. Specifically, in the relationship exploration module, to explore relationship tags as anchors to guide semantic alignment, we design a network to calculate the cosine similarity between visual features and word vector information. We construct the grammar enhancement module similarly to the BERT. That means we use two BERT modules in our framework. The first is the main frame for generating captions, and the second is the auxiliary model to determine whether the syntax of the generated caption is correct. To validate the performance of our proposed model, we conduct abundant experiments on the MSCOCO dataset, Flickr30k dataset, and Flickr8k dataset. Experimental results show that our proposed method performs better than state-of-the-art approaches.

SEMScene: Semantic-Consistency Enhanced Multi-Level Scene Graph Matching for Image-Text Retrieval

Image-text retrieval, a fundamental cross-modal task, performs similarity reasoning for images and texts. The primary challenge for image-text retrieval is cross-modal semantic heterogeneity, where the semantic features of visual and textual modalities are rich but distinct. Scene graph is an effective representation for images and texts as it explicitly models objects and their relations. Existing scene graph based methods have not fully taken the features regarding various granularities implicit in scene graph into consideration (e.g. triplets), the inadequate feature matching incurs the absence of non-trivial semantic information (e.g. inner relations among triplets). Therefore, we propose a S emantic-Consistency E nhanced M ulti-Level Scene Graph Matching (SEMScene) network, which exploits the semantic relevance between visual and textual scene graphs from fine-grained to coarse-grained. Firstly, under the scene graph representation, we perform feature matching including low-level node matching, mid-level semantic triplet matching, and high-level holistic scene graph matching. Secondly, to enhance the semantic-consistency for object-fused triplets carrying key correlation information, we propose a dual-step constraint mechanism in mid-level matching. Thirdly, to guide the model to learn the semantic-consistency of matched image-text pairs, we devise effective loss functions for each stage of the dual-step constraint. Comprehensive experiments on Flickr30K and MS-COCO datasets demonstrate that SEMScene achieves state-of-the-art performances with significant improvements.

CAST: Cross-Modal Retrieval and Visual Conditioning for image captioning

Image captioning requires not only accurate recognition of objects and corresponding relationships, but also full comprehension of the scene information. However, existing models suffer from partial understanding and object hallucination. In this paper, a Cross-modal retrievAl and viSual condiTioning model (CAST) is proposed to address the above issues for image captioning with three key modules: an image–text retriever, an image & memory comprehender and a dual attention decoder. Aiming at a comprehensive understanding, we propose to exploit cross-modal retrieval to mimic human cognition, i.e., to trigger retrieval of contextual information (called episodic memory) about a specific event. Specifically, the image–text retriever searches the top n relevant sentences which serve as episodic memory for each input image. Then the image & memory comprehender encodes an input image and enriches episodic memory by self-attention and relevance attention respectively, which can encourage CAST to comprehend the scene thoroughly and support decoding more effectively. Finally, such image representation and memory are integrated into our dual attention decoder, which performs visual conditioning by re-weighting image and text features to alleviate object hallucination. Extensive experiments are conducted on MS COCO and Flickr30k datasets, which demonstrate that our CAST achieves state-of-the-art performance. Our model also has a promising performance even in low-resource scenarios (i.e. 0.1%, 0.5% and 1% of MS COCO training set).

The Role of Attention Mechanism in Generating Image Captions: An Innovative Approach with Neural Network-Based Seq2seq Model

Image-to-text generation contributes significantly across various domains such as entertainment, communication, commerce, security, and education by establishing a connection between visual and textual content through the creation of explanations. This process aims to transform image data into meaningful text, enhancing content accessibility, comprehensibility, and processability. Hence, advancements and studies in this field hold paramount importance. This study focuses on how the fusion of the Sequence-to-Sequence (Seq2seq) model and attention mechanism enhances the generation of more meaningful captions from images. Experiments conducted on the Flickr8k dataset highlight the Seq2seq model's capacity to produce captions in alignment with reference sentences. Leveraging the dynamic focus of the attention mechanism, the model effectively captures detailed aspects of images.

IMAGE CAPTION GENERATOR USING DEEP LEARNING

Image Captioning is a task where each image must be understood properly and are able generate suitable caption with proper grammatical structure.Here it is a hybrid system which uses multilayer CNN (Convolutional Neural Network) for generating keywords which narrates given input images and Long Short Term Memory(LSTM) for precisely constructing the significant captions utilizing the obtained words .Convolution Neural Network (CNN) proven to be so effective that there is a way to get to any kind of estimating problem that includes image data as input. LSTM was developed to avoid the poor predictive problem which occurred while using traditional approaches. We used an encoder-decoder based model that is capable of generating grammatically correct captions for images. This model makes use of VGG16(Visual Geometry Group) as an encoder and LSTM as a decoder. The model will be trained like when an image is given model produces captions that almost describe the image. The efficiency is demonstrated for the given model using Flickr8K data sets which contains 8000 images and captions for each image but we use CNN and LSTM to capture dependencies and tell both the spatial relationships of images and contextual information of captions and generate contextually relevant captions. Keywords—CNN(Convolutional Neural Network),LSTM(Long Short Term Memory),VGG16(Visual Geometry Group),Deep Learning,Encoder-Decoder.

Automatic Image Caption Generation with Deep Learning

Artificial intelligence research has long focused on the task of automatically defining what is included in a picture or image. This paper presents the CNN and RNN-LSTM models used in the Automatic Caption Generator implementation. It takes into account recent developments in image processing and performing research on translation through automation. The dataset utilized was Flickr8k. We have employed BLEU scores to assess the efficiency of the system that has been described. Based on the scores, one could classify the resulting captions as excellent or awful. The primary uses of this paradigm are virtual assistants, picture indexing, social media, assistive technology for the blind, altering app suggestions, and many more areas. Key Words: Automatic Caption Generator, CNN, RNN-LSTM, Computer vision, Machine translation, Flickr8k dataset, BLEU scores, Performance evaluation, Virtual assistants, Picture indexing, Social media, Assistive technology for the blind, App suggestions.

Towards bridging the semantic gap between image and text: An empirical approach

Vlogs, Recordings, news, sport coverages are huge sources of multimodal information that do not just limit to text but extend to audio, images and videos. Applications such as summary generation, image/video captioning, multimodal sentiment analysis, cross modal retrieval requires Computer Vision along with Natural Language Processing techniques to extract relevant information. Information from different modalities must be leveraged in order to extract quality content. Hence, reducing the gap between different modalities is of utmost importance. Image to text conversion is an emerging field and employs the use of encoder decoder architecture. Deep CNNs extract the feature of images and sequence to sequence models are used to generate text description. This paper is a contribution towards the growing body of research in multimodal information retrieval. In order to generate the textual description of images, we have performed 5 experiments using the benchmark Flickr8k dataset. In these experiments we have utilized different architectures - simple sequence to sequence model, attention mechanism, transformer-based architecture to name a few. The results have been evaluated using BLEAU score. Results show that the best descriptions are attained by making use of transformer architecture. We have also compared our results with the pretrained visual model vit-gpt2 that incorporates visual transformer.

Cascade & allocate: A cross-structure adversarial attack against models fusing vision and language

The data fusion systems between multiple modals have attracted a wide range of concerns about their adversarial robustness. Recent image captioning attacks lack cross-structure transferability against models with diverse structures. Therefore, the attackers demonstrate satisfactory performance only when they gain full or partial knowledge of target image captioning models. So far, the cross-structure transferability of multi-modal adversarial examples has not been thoroughly investigated. In this paper, a theorem is proposed to analyze the upper bound captioning adversarial transferability. We design a new transfer-based adversarial attack method (Cascade & Allocate) against image captioning models. Our method can be divided into two steps. The first step randomly selects a set of candidate models with diverse structures, and we use a momentum-like strategy to generate perturbations. This ‘model cascade’ step narrows the gap of the gradient directions between different image captioning models, therefore enhancing the cross-model transferability and generating model-agnostic adversarial perturbations. The second step utilizes the upper bound transferability theorem to allocate the weight of perturbations per model. This ‘weight allocate’ step enhances the weight of model with the most consistent gradient in all candidate models, which generates more transferable adversarial examples. In the experiments, we compare our approach with other captioning and ensemble-based attacks against five black-box models with different structures on MS COCO and Flickr-30k datasets. The adversarial examples exhibit state-of-the-art adversarial transferability against five black-box models with different structures. The results demonstrate that our approach is practical and generalizes well to a wide range of captioning models.

Heterogeneous Graph Fusion Network for cross-modal image-text retrieval

Exploring the semantic correspondence of image-text pairs is significant as it bridges vision and language. Most prior works focus on global semantic alignment or local semantic alignment, by developing a fine neural network that facilitates the corresponding alignment but neglects the semantic information and relative position information between image regions, or text words, which will lead to a non-meaningful alignment. To this end, a Heterogeneous Graph Fusion Network (HGFN) is proposed to explore the correlation score of vision-language for improving the accuracy of cross-modal image-text retrieval in this paper. Specifically, we first construct an undirected fully-connected graph based on the semantic or relative position information for each image, as well as a textual graph with neighborhood information of the text. Then, we present a graph fusion module to integrate the features of heterogeneous graphs into a unified hybrid representation, in which the graph convolutional network is utilized to gather neighborhood information to alleviate potentially non-meaningful alignment. In addition, we also propose a novel “Dynamic top-K negative” strategy for the selection of negative examples in the training process. Experimental results demonstrate that HGFN achieves comparable performance with state-of-the-art approaches on the Flickr30K and MSCOCO datasets.

TSIC-CLIP: Traffic Scene Image Captioning Model Based on Clip

Image captioning in traffic scenes presents several challenges, including imprecise caption generation, lack of personalization, and an unwieldy number of model parameters. We propose a new image captioning model for traffic scenes to address these issues. The model incorporates an adapter-based fine-tuned feature extraction part to enhance personalization and a caption generation module using global weighted attention pooling to reduce model parameters and improve accuracy. The proposed model consists of four main stages. In the first stage, the Image-Encoder extracts the global features of the input image and divides it into nine sub-regions, encoding each sub-region separately. In the second stage, the Text-Encoder encodes the text dataset to obtain text features. It then calculates the similarity between the image sub-region features and encoded text features, selecting the text features with the highest similarity. Subsequently, the pre-trained Faster RCNN model extracts local image features. The model then splices together the text features, global image features, and local image features to fuse the multimodal information. In the final stage, the extracted features are fed into the Captioning model, which effectively fuses the different features using a novel global weighted attention pooling layer. The Captioning model then generates natural language image captions. The proposed model is evaluated on the MS-COCO dataset, Flickr 30K dataset, and BUUISE-Image dataset, using mainstream evaluation metrics. Experiments demonstrate significant improvements across all evaluation metrics on the public datasets and strong performance on the BUUISE-Image traffic scene dataset.

Adaptive Semantic-Enhanced Transformer for Image Captioning.

In the research on image captioning, rich semantic information is very important for generating critical caption words as guiding information. However, semantic information from offline object detectors involves many semantic objects that do not appear in the caption, thereby bringing noise into the decoding process. To produce more accurate semantic guiding information and further optimize the decoding process, we propose an end-to-end adaptive semantic-enhanced transformer (AS-Transformer) model for image captioning. For semantic enhancement information extraction, we propose a constrained weaklysupervised learning (CWSL) module, which reconstructs the semantic object's probability distribution detected by the multiple instances learning (MIL) through a joint loss function. These strengthened semantic objects from the reconstructed probability distribution can better depict the semantic meaning of images. Also, for semantic enhancement decoding, we propose an adaptive gated mechanism (AGM) module to adjust the attention between visual and semantic information adaptively for the more accurate generation of caption words. Through the joint control of the CWSL module and AGM module, our proposed model constructs a complete adaptive enhancement mechanism from encoding to decoding and obtains visual context that is more suitable for captions. Experiments on the public Microsoft Common Objects in COntext (MSCOCO) and Flickr30K datasets illustrate that our proposed AS-Transformer can adaptively obtain effective semantic information and adjust the attention weights between semantic and visual information automatically, which achieves more accurate captions compared with semantic enhancement methods and outperforms state-of-the-art methods.

Ar-CM-ViMETA: Arabic Image Captioning based on Concept Model and Vision-based Multi-Encoder Transformer Architecture

Image captioning is a major artificial intelligence research field that involves visual interpretation and linguistic description of a corresponding image. Successful image captioning relies on acquiring as much information as feasible from the original image. One of these essential bits of knowledge is the topic or the concept that the image is associated with. Recently, concept modeling technique has been utilized in English image captioning for completely capturing the image contexts and make use of these contexts to produce more accurate image descriptions. In this paper, a concept-based model is proposed for Arabic Image Captioning (AIC). A novel Vision-based Multi-Encoder Transformer Architecture (ViMETA) is proposed for handling the multi-outputs result from the concept modeling technique while producing the image caption. BiLingual Evaluation Understudy (BLEU) and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) standard metrics have been used to evaluate the proposed model using the Flickr8K dataset with Arabic captions. Furthermore, qualitative analysis has been conducted to compare the produced captions of the proposed model with the ground truth descriptions. Based on the experimental results, the proposed model outperformed the related works both quantitatively, using BLEU and ROUGE metrics, and qualitatively

USER: Unified Semantic Enhancement With Momentum Contrast for Image-Text Retrieval.

As a fundamental and challenging task in bridging language and vision domains, Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality, and its key challenge is to measure the semantic similarity across different modalities. Although significant progress has been achieved, existing approaches typically suffer from two major limitations: (1) It hurts the accuracy of the representation by directly exploiting the bottom-up attention based region-level features where each region is equally treated. (2) It limits the scale of negative sample pairs by employing the mini-batch based end-to-end training mechanism. To address these limitations, we propose a Unified Semantic Enhancement Momentum Contrastive Learning (USER) method for ITR. Specifically, we delicately design two simple but effective Global representation based Semantic Enhancement (GSE) modules. One learns the global representation via the self-attention algorithm, noted as Self-Guided Enhancement (SGE) module. The other module benefits from the pre-trained CLIP module, which provides a novel scheme to exploit and transfer the knowledge from an off-the-shelf model, noted as CLIP-Guided Enhancement (CGE) module. Moreover, we incorporate the training mechanism of MoCo into ITR, in which two dynamic queues are employed to enrich and enlarge the scale of negative sample pairs. Meanwhile, a Unified Training Objective (UTO) is developed to learn from mini-batch based and dynamic queue based samples. Extensive experiments on the benchmark MSCOCO and Flickr30K datasets demonstrate the superiority of both retrieval accuracy and inference efficiency. For instance, compared with the existing best method NAAF, the metric R@1 of our USER on the MSCOCO 5K Testing set is improved by 5% and 2.4% on caption retrieval and image retrieval without any external knowledge or pre-trained model while enjoying over 60 times faster inference speed. Our source code will be released at https://github.com/zhangy0822/USER.

Synthesis of Vision and Language: Multifaceted Image Captioning Application

The rapid advancement in image captioning has been a pivotal area of research, aiming to mimic human-like understanding of visual content. This paper presents an innovative approach that integrates attention mechanisms and object features into an image captioning model. Leveraging the Flickr8k dataset, this research explores the fusion of these components to enhance image comprehension and caption generation. Furthermore, the study showcases the implementation of this model in a user-friendly application using FASTAPI and ReactJS, offering text-to-speech translation in multiple languages. The findings underscore the efficacy of this approach in advancing image captioning technology. This tutorial outlines the construction of an image caption generator, employing Convolutional Neural Network (CNN) for image feature extraction and Long Short-Term Memory Network (LSTM) for Natural Language Processing (NLP). Keywords—Convolutional Neural Networks, Long Short Term Memory, Attention Mechanism, Transformer Architecture, Vision Transformers, Transfer Learning, Multimodal fusion, Deep Learning Models, Pre-Trained Models, Image Processing Techniques