Image Captioning Research Articles

A Survey on Image Caption Generation in Various Languages

The image caption is the process of adding an explicit, coherent description to the contents of the image. This is done by using the latest deep learning techniques, which include computer vision and natural language processing, to understand the contents of the image and give it an appropriate caption. Multiple datasets suitable for many applications have been proposed. The biggest challenge for researchers with natural language processing is that the datasets are incompatible with all languages. The researchers worked on translating the most famous English data sets with Google Translate to understand the content of the images in their mother tongue. In this paper, the proposed review aims to enhance the understanding of image captioning strategies and to survey previous research related to image captioning while examining the most popular databases in different languages, mostly English, translating into other languages using the latest models for describing images, summarizing evaluation measures, and comparing them.

Enhancing Traffic Scene and Understanding Through Image Captioning and Audio

Enhancing Traffic Scene and Understanding Through Image Captioning and Audio

Generating Descriptive Text From Images - Image Caption Generator

In the modern era, image captioning has become one of the most widely required tools. Moreover, there are inbuilt applications that generate and provide a caption for a certain image, all these things are done with the help of deep neural network models. The process of generating a description of an image is called image captioning. It requires recognizing the important objects, their attributes, and the relationships among the objects in an image. It generates syntactically and semantically correct sentences. In this paper, we present a deep learning model to describe images and generate captions using computer vision and machine translation. This paper aims to detect different objects found in an image, recognize the relationships between those objects and generate captions. The dataset used is Flickr8k and the programming language used was Python3, and an ML technique called Transfer Learning will be implemented with the help of the caption model, to demonstrate the proposed experiment. This paper will also elaborate on the functions and structure of the various Neural networks involved. Generating image captions is an important aspect of Computer Vision and Natural language processing. Image caption generators can find applications in Image segmentation as used by Facebook and Google Photos, and even more so, its use can be extended to video frames. They will easily automate the job of a person who has to interpret images. Not to mention it has immense scope in helping visually impaired people

Cross-region feature fusion with geometrical relationship for OCR-based image captioning

Automatically generating a readable sentence that describes the text-contained image is a challenging task. Compared to traditional image captioning algorithms, OCR-based image captioning focuses on reading OCR tokens in images and understanding them with the image content to generate descriptions. However, existing research mainly concentrate on improving the quantity and quality of obtaining OCR tokens and exploring their spatial relationships while lacking investigation into how to effectively join OCR tokens with image content. This paper proposes a cross-region feature fusion with a geometrical relationship Transformer(CFGR-Transformer) for OCR-based image captioning. The network first establishes the associations between the OCR and object regions of the image by constructing relative geometric relationships, including width/height difference, distance, IOU(Intersection over Union), inclusion relationships, and angles offset, and then incorporates intra-region and cross-region features to aggregate entities from different modalities by a multi-head attention mechanism based on relative relationships. Benefiting from the guidance of the relative relationship, visual entities like OCR tokens and object regions can consider multiple relative relationships as the attention weight for feature fusion within each subspace. Extensive experiments conducted on the TextCaps dataset demonstrate the effectiveness of the proposed CFGR-Transformer method. In particular, our results on the online testing of TextCaps achieve an improvement in CIDEr score from 93.0% to 98.2%.

A novel image captioning model with visual-semantic similarities and visual representations re-weighting

Image captioning, the task of generating descriptive sentences for images, has seen significant advancements by incorporating semantic information. However, previous studies employed semantic attribute detectors to extract predetermined attributes consistently applied at every time step, resulting in the use of irrelevant attributes to the linguistic context during words’ generation. Furthermore, the integration between semantic attributes and visual representations in previous works is considered superficial and ineffective, leading to the neglection of the rich visual-semantic connections affecting the captioning model’s performance. To address the limitations of previous models, we introduced a novel framework that adapts attribute usage based on contextual relevance and effectively utilizes the similarities between visual features and semantic attributes. Our framework includes an Attribute Detection Component (ADC) that predicts relevant attributes using visual features and attribute embeddings. The Attribute Prediction and Visual Weighting module (APVW) then dynamically adjusts these attributes and generates weights to refine the visual context vector, enhancing semantic alignment. Our approach demonstrated an average improvement of 3.30% in BLEU@1 and 5.24% in CIDEr on MS-COCO, and 6.55% in BLEU@1 and 25.72% in CIDEr on Flickr30K, during CIDEr optimization phase.

Mining core information by evaluating semantic importance for unpaired image captioning

Recently, exciting progress has been made in the research of supervised image captioning. However, manually annotated image-annotation pair data is difficult and expensive to obtain. Therefore, unpaired image captioning becomes an emerging challenge. This paper proposes a method called Mining Core Information by Evaluating Semantic Importance (MCIESI) for Unpaired Image Captioning, which is a method for image captioning using unpaired images and sentences. The main difference from the existing methods is that MCIESI focuses on mining the information that should be described in the image and embodies them in the generated natural language that conforms to human thinking. To achieve this goal, we use scene graphs to represent the semantics of images and evaluates the importance of objects and interaction relationships to mine core information in images, which are then encouraged to be embodied in generated sentences through semantic constraint. Combined with grammatical constraint using adversarial training with real sentence corpus and relative constraint using a triplet loss, the generator is trained to generate semantically plausible and grammatically correct sentences. Extensive experiments verify the effectiveness of MCIESI.

Automatic Audio and Image Caption Generation with Deep Learning

A novel approach to image caption generation tailored specifically for visually impaired individuals. The proposed system employs advanced computer vision algorithms to analyze images and generate descriptive textual captions. Furthermore, it integrates seamless text-to-speech conversion functionality, allowing for the automatic transformation of these captions into spoken audio, thereby enabling access to visual content for individuals with visual impairments. The goal of this project is to generate descriptive captions for a given photograph or image. We achieve this by employing Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) models, both of which are advanced deep learning techniques. Using computer vision, the system identifies the content of the image and generates a relevant caption. This caption is then converted into audio using Natural Language Processing (NLP).

A transformer-based Urdu image caption generation

Image caption generation has emerged as a remarkable development that bridges the gap between Natural Language Processing (NLP) and Computer Vision (CV). It lies at the intersection of these fields and presents unique challenges, particularly when dealing with low-resource languages such as Urdu. Limited research on basic Urdu language understanding necessitates further exploration in this domain. In this study, we propose three Seq2Seq-based architectures specifically tailored for Urdu image caption generation. Our approach involves leveraging transformer models to generate captions in Urdu, a significantly more challenging task than English. To facilitate the training and evaluation of our models, we created an Urdu-translated subset of the flickr8k dataset, which contains images featuring dogs in action accompanied by corresponding Urdu captions. Our designed models encompassed a deep learning-based approach, utilizing three different architectures: Convolutional Neural Network (CNN) + Long Short-term Memory (LSTM) with Soft attention employing word2Vec embeddings, CNN+Transformer, and Vit+Roberta models. Experimental results demonstrate that our proposed model outperforms existing state-of-the-art approaches, achieving 86 BLEU-1 and 90 BERT-F1 scores. The generated Urdu image captions exhibit syntactic, contextual, and semantic correctness. Our study highlights the inherent challenges associated with retraining models on low-resource languages. Our findings highlight the potential of pre-trained models for facilitating the development of NLP and CV applications in low-resource language settings.

Enhancing Image Captioning Using Deep Convolutional Generative Adversarial Networks

Introduction:: Introduction: Image caption generation has long been a fundamental challenge in the area of computer vision (CV) and natural language processing (NLP). In this research, we present an innovative approach that harnesses the power of Deep Convolutional Generative Adversarial Networks (DCGAN) and adversarial training to revolutionize the generation of natural and contextually relevant image captions. Method:: Our method significantly improves the fluency, coherence, and contextual relevance of generated captions and showcases the effectiveness of RL reward-based fine-tuning. Through a comprehensive evaluation of COCO datasets, our model demonstrates superior performance over baseline and state-of-the-art methods. On the COCO dataset, our model outperforms current state-of-the-art (SOTA) models across all metrics, achieving BLEU-4 (0.327), METEOR (0.249), Rough (0.525) and CIDEr (1.155) scores. Result:: The integration of DCGAN and adversarial training opens new possibilities in image captioning, with applications spanning from automated content generation to enhanced accessibility solutions. Conclusion:: This research paves the way for more intelligent and context-aware image understanding systems, promising exciting future exploration and innovation prospects.

Every Problem, Every Step, All in Focus: Learning to Solve Vision-Language Problems With Integrated Attention.

Integrating information from vision and language modalities has sparked interesting applications in the fields of computer vision and natural language processing. Existing methods, though promising in tasks like image captioning and visual question answering, face challenges in understanding real-life issues and offering step-by-step solutions. In particular, they typically limit their scope to solutions with a sequential structure, thus ignoring complex inter-step dependencies. To bridge this gap, we propose a graph-based approach to vision-language problem solving. It leverages a novel integrated attention mechanism that jointly considers the importance of features within each step as well as across multiple steps. Together with a graph neural network method, this attention mechanism can be progressively learned to predict sequential and non-sequential solution graphs depending on the characterization of the problem-solving process. To tightly couple attention with the problem-solving procedure, we further design new learning objectives with attention metrics that quantify this integrated attention, which better aligns visual and language information within steps, and more accurately captures information flow between steps. Experimental results on VisualHow, a comprehensive dataset of varying solution structures, show significant improvements in predicting steps and dependencies, demonstrating the effectiveness of our approach in tackling various vision-language problems.

Clustering-based mask recovery for image captioning

Region features play a huge role in image captioning. However, obtaining region features requires pre-training an object detector by using a large number of object detection datasets. It may be impossible for the end-to-end training. And if there is a large distribution difference between the object detection datasets and the image captioning datasets, the object detector may not be able to extract accurate region features. This makes it limited in application. In this paper, we propose a clustering-based mask recovery for image captioning. In the encoder, the pseudo-region features are obtained by clustering the grid features, which are extracted using Swin Transformer. Then we input the grid features together with the pseudo-region features into the decoder, and make the model to dynamically learns the weights of the two features in the decoding process to minimize the effect of errors caused by clustering. By using a clustering method to generate pseudo-region features for images, not only does the training process become end-to-end, but there is no need to introduce additional object detection datasets to train the object detector. In addition, the Transformer decoder has a misplaced problem in the decoding process. This means that the positional information used by the model when generating a word is not the same as the positional information used when it continues to use the word to reason. This may have some negative impact on the position encoding of the model. Therefore, we changed the original decoding method to mask recovery. Furthermore, a masked multi-head attention module with relative position is proposed in the decoder to integrate the information in the fusion features, and reconstruct the relative position relationship between words. We conduct experiments on MSCOCO 2014 dataset. The experiment results show that our model obtains 144.3% (single model) and 147.0% (ensemble of 4 models) CIDEr scores on ‘Karpathy’ offline test split, and 143.2% (c40) CIDEr scores on the official online test server.

Language-based machine perception: linguistic perspectives on the compilation of captioning datasets

Abstract Over the last decade, a plethora of training datasets have been compiled for use in language-based machine perception and in human-centered AI, alongside research regarding their compilation methods. From a primarily linguistic perspective, we add to these studies in two ways. First, we provide an overview of sixty-six training datasets used in automatic image, video, and audio captioning, examining their compilation methods with a metadata analysis. Second, we delve into the annotation process of crowdsourced datasets with an interest in understanding the linguistic factors that affect the form and content of the captions, such as contextualization and perspectivation. With a qualitative content analysis, we examine annotator instructions with a selection of eleven datasets. Drawing from various theoretical frameworks that help assess the effectiveness of the instructions, we discuss the visual and textual presentation of the instructions, as well as the perspective-guidance that is an essential part of the language instructions. While our analysis indicates that some standards in the formulation of instructions seem to have formed in the field, we also identified various reoccurring issues potentially hindering readability and comprehensibility of the instructions, and therefore, caption quality. To enhance readability, we emphasize the importance of text structure, organization of the information, consistent use of typographical cues, and clarity of language use. Last, engaging with previous research, we assess the compilation of both web-sourced and crowdsourced captioning datasets from various perspectives, discussing factors affecting the diversity of the datasets.

Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis.

Conventional approaches to dietary assessment are primarily grounded in self-reporting methods or structured interviews conducted under the supervision of dietitians. These methods, however, are often subjective, potentially inaccurate, and time-intensive. Although artificial intelligence (AI)-based solutions have been devised to automate the dietary assessment process, prior AI methodologies tackle dietary assessment in a fragmented landscape (e.g., merely recognizing food types or estimating portion size), and encounter challenges in their ability to generalize across a diverse range of food categories, dietary behaviors, and cultural contexts. Recently, the emergence of multimodal foundation models, such as GPT-4V, has exhibited transformative potential across a wide range of tasks (e.g., scene understanding and image captioning) in various research domains. These models have demonstrated remarkable generalist intelligence and accuracy, owing to their large-scale pre-training on broad datasets and substantially scaled model size. In this study, we explore the application of GPT-4V powering multimodal ChatGPT for dietary assessment, along with prompt engineering and passive monitoring techniques. We evaluated the proposed pipeline using a self-collected, semi free-living dietary intake dataset comprising 16 real-life eating episodes, captured through wearable cameras. Our findings reveal that GPT-4V excels in food detection under challenging conditions without any fine-tuning or adaptation using food-specific datasets. By guiding the model with specific language prompts (e.g., African cuisine), it shifts from recognizing common staples like rice and bread to accurately identifying regional dishes like banku and ugali. Another GPT-4V's standout feature is its contextual awareness. GPT-4V can leverage surrounding objects as scale references to deduce the portion sizes of food items, further facilitating the process of dietary assessment.

A Systematic Survey of Automatic Image Description Generation Systems

Automatic image description generation is an active research area in Computer Vision and Natural Language Processing. Objects, their attributes, actions, and spatial relationships are identified in the image description generation system. Earlier, these systems used classical machine learning approaches. Later majority of these works follow deep learning strategies. The essential goal of these systems is to produce syntactically and semantically correct sentences. This review aims to synthesize the studies conducted from 2010 to 2023 to get a deeper view of various image description generation systems and their applications. The prominent contribution of this review is that it covers the different aspects of image captioning systems, such as the methods used, the various applied domains, evaluation measures, and the datasets used. A single synthesized study directs scholars regarding developing image captioning systems to date utilizing machine learning approaches. It also offers suggestions for researchers in this sector for the future. Image captioning is applied in many fields like natural images, medical images, remote sensing images, videos, etc. This review paper reviews the various taxonomies of image description generation systems. We also analyzed multiple methods used in the architecture of image captioning systems. The datasets and the evaluation metrics used in these systems are discussed. We studied the performance of the system under various circumstances.

An Enhanced Feature Extraction Framework for Cross-Modal Image–Text Retrieval

In general, remote sensing images depict intricate scenes. In cross-modal retrieval tasks involving remote sensing images, the accompanying text includes numerus information with an emphasis on mainly large objects due to higher attention, and the features from small targets are often omitted naturally. While the conventional vision transformer (ViT) method adeptly captures information regarding large global targets, its capability to extract features of small targets is limited. This limitation stems from the constrained receptive field in ViT’s self-attention layer, which hinders the extraction of information pertaining to small targets due to interference from large targets. To address this concern, this study introduces a patch classification framework based on feature similarity, which establishes distinct receptive fields in the feature space to mitigate interference from large targets on small ones, thereby enhancing the ability of traditional ViT to extract features from small targets. We conducted evaluation experiments on two popular datasets—the Remote Sensing Image–Text Match Dataset (RSITMD) and the Remote Sensing Image Captioning Dataset (RSICD)—resulting in mR indices of 35.6% and 19.47%, respectively. The proposed approach contributes to improving the detection accuracy of small targets and can be applied to more complex image–text retrieval tasks involving multi-scale ground objects.

Explicit Image Caption Reasoning: Generating Accurate and Informative Captions for Complex Scenes with LMM.

The rapid advancement of sensor technologies and deep learning has significantly advanced the field of image captioning, especially for complex scenes. Traditional image captioning methods are often unable to handle the intricacies and detailed relationships within complex scenes. To overcome these limitations, this paper introduces Explicit Image Caption Reasoning (ECR), a novel approach that generates accurate and informative captions for complex scenes captured by advanced sensors. ECR employs an enhanced inference chain to analyze sensor-derived images, examining object relationships and interactions to achieve deeper semantic understanding. We implement ECR using the optimized ICICD dataset, a subset of the sensor-oriented Flickr30K-EE dataset containing comprehensive inference chain information. This dataset enhances training efficiency and caption quality by leveraging rich sensor data. We create the Explicit Image Caption Reasoning Multimodal Model (ECRMM) by fine-tuning TinyLLaVA with the ICICD dataset. Experiments demonstrate ECR's effectiveness and robustness in processing sensor data, outperforming traditional methods.

Optimizing image captioning: The effectiveness of vision transformers and VGG networks for remote sensing

This study presents a comprehensive evaluation of two prominent deep learning models, Vision Transformer (ViT) and VGG16, within the domain of image captioning for remote sensing data. By leveraging the BLEU score, a widely accepted metric for assessing the quality of text generated by machine learning models against a set of reference captions, this research aims to dissect and understand the capabilities and performance nuances of these models across various sample sizes: 25, 50, 75, and 100 samples. Our findings reveal that the Vision Transformer model generally outperforms the VGG16 model across all evaluated sample sizes, achieving its peak performance at 50 samples with a BLEU score of 0.5507. This performance shows that ViT benefits from its ability to capture global dependencies within the data, providing a more nuanced understanding of the images. However, the performance slightly decreases as the sample size increases beyond 50, indicating potential challenges in scalability or overfitting to the training data. Conversely, the VGG16 model shows a different performance trajectory, starting with a lower BLEU score for smaller sample sizes but demonstrating a consistent improvement as the sample size increases, culminating in its highest BLEU score of 0.4783 for 100 samples. This pattern suggests that VGG16 may require a larger dataset to adequately learn and generalize from the data, although it achieves a more modest performance ceiling compared to ViT. Through a detailed analysis of these findings, the study underscores the strengths and limitations of each model in the context of image captioning. The Vision Transformer's superior performance highlights its potential for applications requiring high accuracy in text generation from images. In contrast, the gradual improvement exhibited by VGG16 suggests its utility in scenarios where large datasets are available, and scalability is a priority. This study contributes to the ongoing discourse in the AI community regarding the selection and optimization of deep learning models for complex tasks such as image captioning, offering insights that could guide future research and application development in this field.

AI Assistant for Visually Impaired

This paper presents an advanced assistive technol- ogy system aimed at improving accessibility and independence for visually impaired individuals. Utilizing artificial intelligence (AI) and computer vision techniques, the system provides real- time auditory feedback to users about their surroundings. The core components of the system include image captioning, face recognition, and depth estimation, all integrated to offer a comprehensive understanding of the environment. The system captures live video feed through a webcam, processes the images using pre-trained models like CLIPSeg for image segmentation and DPT for depth estimation, and generates textual descriptions of detected objects and their spatial distances. These descriptions are translated from English to Kannada using Google’s transla- tion services and converted into speech with the gTTS library, ensuring accessibility for Kannada-speaking users. Additionally, the system employs face recognition to identify known individuals in the vicinity, providing personalized auditory notifications. The combination of these technologies enables the system to offer context-aware assistance, helping visually impaired users to navigate and interact with their surroundings more effectively. Experimental results demonstrate the system’s capability to deliver accurate, real-time feedback, highlighting its potential to significantly enhance the quality of life for visually impaired individuals.

Chinese image captioning with fusion encoder and visual keyword search

AbstractAutomatic generation of image captions is essentially a cross‐modal conversion from image to text. Owing to the differences in linguistic characteristics between Chinese and English, quite a few Chinese image captioning methods have recently been proposed. Nevertheless, the existing Chinese image captioning models usually lack attention to local details of images or tend to produce general descriptions. To address these challenges, a Chinese image captioning method is proposed that incorporates fusion encoder, visual keyword search, and reinforcement learning. The fusion encoder can simultaneously extract local and global features of the input image to enrich the semantic information in the decoding stage, visual keyword search can pursue potential visual words associated with the image content, and the reinforcement learning mechanism can optimize the evaluation metric CIDEr at sentence level to promote the lexical diversity of image description. The results of extensive experiments demonstrate that the proposed model outperforms the state‐of‐the‐art models and delivers expressive and informative Chinese image captions.

SegLD: Achieving universal, zero-shot and open-vocabulary segmentation through multimodal fusion via latent diffusion processes

Open-vocabulary learning can identify categories marked during training (seen categories) and generalize to categories not annotated in the training set (unseen categories). It could theoretically extend segmentation systems to more universal applications. However, current open-vocabulary segmentation frameworks are primarily suited for specific tasks or require retraining according to the task, and they significantly underperform in inferring seen categories compared to fully supervised frameworks. Therefore, we introduce a universal open-vocabulary segmentation framework based on the latent diffusion process (SegLD), which requires only a single training session on a panoptic dataset to achieve inference across all open-vocabulary segmentation tasks, and reaches SOTA segmentation performance for both seen and unseen categories in every task. Specifically, SegLD comprises two stages: in the first stage, we deploy two parallel latent diffusion processes to deeply fuse the text (image caption or category labels) and image information, further aggregating the multi-scale features output from both latent diffusion processes on a scale basis. In the second stage, we introduce text queries, text list queries, and task queries, facilitating the learning of inter-category and inter-task differences through the computation of contrastive losses between them. Text queries are then further fed into a Transformer Decoder to obtain category-agnostic segmentation masks. Then we establish classification loss functions for the type of text input during training, whether image captions or category labels, to help assign a category label from the open vocabulary to each predicted binary mask. Experimental results show that, with just a single training session, SegLD significantly outperforms other contemporary SOTA fully supervised segmentation frameworks and open-vocabulary segmentation frameworks across almost all evaluation metrics for both known and unknown categories on the ADE20K, Cityscapes, and COCO datasets. This highlights SegLD’s capability as a universal segmentation framework, with the potential to replace other segmentation frameworks and adapt to various segmentation domains. The project link for SegLD is https://zht-segld.github.io/.