Image Captioning Research Articles

Understanding remote sensing imagery like reading a text document: What can remote sensing image captioning offer?

Remote sensing imagery offers intricate and nuanced data, emphasizing the need for a profound understanding of the relationships among varied geographical elements and events. In this study, we explore the transitions from the image domain to the text domain by employing four state-of-the-art image captioning algorithms, i.e., BLIP, mPLUG, OFA, and X-VLM. Specifically, we investigate (1) the stability of these image captioning algorithms for remote sensing image captioning, (2) the preservation of similarity between images and their corresponding captions, and (3) the characteristics of their caption embedding spaces. The results suggest a moderate consistency across generated captions from different image captioning models, with observable variations contingent upon the urban entities presented. In addition, a dynamic relationship emerges between image space and the corresponding caption space, evidenced by their fluctuated correlation coefficient. Most importantly, patterns within the caption embedding space align with the observed land cover and land use in the image patches, reaffirming the potential of our pilot work as an impactful analytical approach in future remote sensing analytics. We advocate that integrating image captioning techniques with remote sensing imagery paves the way for an innovative data extraction and interpretation approach with diverse applications.

An Analysis on Recent Approaches for Image Captioning

Image captioning is an interdisciplinary area that uses techniques from computer vision and natural language processing to provide a textual description of a picture. The Image captioning task is the process of understanding the scene present in the image by identifying objects and associated actions present to create a meaningful human-like caption which can be used for wide range of applications, including image retrieval, video indexing, assistive technology for the visually impaired, content-based image search, biomedicine, and autonomous cars. Formerly, Machine Learning was utilized for this purpose which will be extensive use of hand-crafted features such as Scale-Invariant Feature Transform (SIFT), Local Binary Patterns (LBP), the Histogram of Oriented Gradients (HOG), and combinations of these features. Extracting handmade characteristics from huge datasets is not straightforward or viable. Many deep learning-based techniques were later proposed. Deep Learning retrieval and template-based approaches were presented; however, both had drawbacks such as losing crucial objects. Recent breakthroughs in deep learning and natural language processing have resulted in considerable increases in image captioning system performance which involves adopting attention mechanisms, transformer-based architectures, multi modal connections, Object-Detection based encoder-decoder and many others. In this survey will explore some of the most recent techniques for image captioning, the datasets and evaluation measures that have been employed in deep learning-based automatic image captioning. The ultimate intention of this study is to act as a guide for researchers by emphasizing future directions for research work. Index Terms: image captioning, computer vision, deep learning, Textual description, natural language processing.

Visual Analytics for Efficient Image Exploration and User-Guided Image Captioning.

Recent advancements in pre-trained language-image models have ushered in a new era of visual comprehension. Leveraging the power of these models, this article tackles two issues within the realm of visual analytics: (1) the efficient exploration of large-scale image datasets and identification of data biases within them; (2) the evaluation of image captions and steering of their generation process. On the one hand, by visually examining the captions generated from language-image models for an image dataset, we gain deeper insights into the visual contents, unearthing data biases that may be entrenched within the dataset. On the other hand, by depicting the association between visual features and textual captions, we expose the weaknesses of pre-trained language-image models in their captioning capability and propose an interactive interface to steer caption generation. The two parts have been coalesced into a coordinated visual analytics system, fostering the mutual enrichment of visual and textual contents. We validate the effectiveness of the system with domain practitioners through concrete case studies with large-scale image datasets.

A multimodal machine learning approach to generate news articles from geo-tagged images

Classical machine learning algorithms typically operate on unimodal data and hence it can analyze and make predictions based on data from a single source (modality). Whereas multimodal machine learning algorithm, learns from information across multiple modalities, such as text, images, audio, and sensor data. The paper leverages the functionalities of multimodal machine learning (ML) application for generating text from images. The proposed work presents an innovative multimodal algorithm that automates the creation of news articles from geo-tagged images by leveraging cutting-edge developments in machine learning, image captioning, and advanced text generation technologies. Employing a multimodal approach that integrates machine learning and transformer algorithms, such as visual geometry group network16 (VGGNet16), convolutional neural network (CNN) and a long short-term memory (LSTM) based system, the algorithm initiates by extracting the location from exchangeable image file format (Exif) data from the image. The features are extracted from the image and corresponding news headline is generated. The headlines are used for generating a comprehensive article with contemporary large language model (LLM). Further, the algorithm generates the news article big-science large open-science open-access multilingual language model (BLOOM). The algorithm was tested on real time photographs as well as images from the internet. In both the cases the news articles generated were validated with ROUGE and BULE score. The proposed work is found to be successful attempt in journalism field.

Image Caption Prediction Using Deep Learning

Abstract: In the era of rapidly growing digital content, the automatic generation of image captions has gain significant attention in the field of deep learning and computer vision. Our proposed model leverages deep learning techniques, including convolutional neural networks (CNNs) for image feature extraction and recurrent neural networks (RNNs) with attention mechanisms for caption generation. To evaluate the performance of our model, we employ a comprehensive dataset containing a diverse range of images and corresponding human-generated captions. In conclusion, our deep learning-based image captioning project offers a promising solution to the challenge of automatically generating meaningful and contextually accurate descriptions for images. The potential applications of the technology are vast, and we anticipate that it will continue to various fields, including artificial intelligence, accessibility, content creation, content management and publishing across various social media platform.

Image Caption Generator

An Image Caption Generator is a sophisticated AI system that combines computer vision and natural language processing to automatically create descriptive textual captions for images. This technology utilizes deep learning, particularly Convolutional Neural Networks (CNNs), to analyze and extract meaningful visual features from the input image. These features capture details about the objects, scenes, and elements within the image. Subsequently, a natural language processing model, often built on Recurrent Neural Networks (RNNs) or Transformers, processes these visual features and generates coherent, contextually relevant captions. Post-processing steps may be applied to enhance the quality of the generated text. The primary aim of Image Caption Generators is to facilitate image understanding, improve accessibility, and enhance content search ability by providing human-readable descriptions for visual content. This technology is instrumental in various fields, including content tagging, accessibility tools for the visually impaired, and enhancing user experiences in multimedia content management systems, ultimately bridging the gap between visual and textual information for a more comprehensive and humanlike interpretation of images

Research on image caption generation method based on multi-modal pre-training model and text mixup optimization

Research on image caption generation method based on multi-modal pre-training model and text mixup optimization

Attribute guided fusion network for obtaining fine-grained image captions

Attribute guided fusion network for obtaining fine-grained image captions

Image Caption Bot for Assistive Vision

It's challenging to automatically produce brief descriptions of an image's meaning because it can have diverse connotations in different languages. However, due to the vast amount of information packed into a single image, it is challenging to parse out the necessary context to use it to build sentences. It's a great way for the visually impaired to get around independently. This type of system can be built using the emerging programming technique of deep learning. This paper presents the development of an Image Caption Bot designed to aid individuals with visual impairments. We achieve enhanced accuracy in caption generation by modeling on the MSCOCO dataset using a Transformer encoder and Inception v3 for image processing. Image captioning, which entails generating textual descriptions for images, is the primary focus of our research. We achieve enhanced accuracy in caption generation by utilizing a Transformer encoder during training. The MSCOCO dataset serves as a valuable The results of the model are translated into speech for the benefit of the visually handicapped. Keywords—CNN, Google Text To Speech, MS-COCO, Inspection v3.

Clustering swap prediction for image-text pre-training

It is essential to delve into the strategy of multimodal model pre-training, which is an obvious impact on downstream tasks. Currently, clustering learning has achieved noteworthy benefits in multiple methods. However, due to the availability of open image-text pairs, it is challenging for multimodal with clustering learning. In this paper, we propose an approach that utilizes clustering swap prediction strategy to learn image-text clustering embedding space by interaction prediction between image and text features. Unlike existing models with clustering learning, our method (Clus) allows for an open number of clusters for web-scale alt-text data. Furthermore, in order to train the image and text encoders efficiently, we introduce distillation learning approach and evaluate the performance of the image-encoder in downstream visual tasks. In addition, Clus is pre-trained end-to-end by using large-scale image-text pairs. Specifically, both text and image serve as ground truth for swap prediction, enabling effective representation learning. Concurrently, extensive experiments demonstrate that Clus achieves state-of-the-art performance on multiple downstream fine-tuning and zero-shot tasks (i.e., Image-Text Retrieval, VQA, NLVR2, Image Captioning, Object Detection, and Semantic Segmentation).

TSFE: Two-Stage Feature Enhancement for Remote Sensing Image Captioning

In the field of remote sensing image captioning (RSIC), mainstream methods typically adopt an encoder–decoder framework. Methods based on this framework often use only simple feature fusion strategies, failing to fully mine the fine-grained features of the remote sensing image. Moreover, the lack of context information introduction in the decoder results in less accurate generated sentences. To address these problems, we propose a two-stage feature enhancement model (TSFE) for remote sensing image captioning. In the first stage, we adopt an adaptive feature fusion strategy to acquire multi-scale features. In the second stage, we further mine fine-grained features based on multi-scale features by establishing associations between different regions of the image. In addition, we introduce global features with scene information in the decoder to help generate descriptions. Experimental results on the RSICD, UCM-Captions, and Sydney-Captions datasets demonstrate that the proposed method outperforms existing state-of-the-art approaches.

Unified deep learning model for multitask representation and transfer learning: image classification, object detection, and image captioning

Unified deep learning model for multitask representation and transfer learning: image classification, object detection, and image captioning

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

Bridging Languages through Images: A Multilingual Text-to-Image Synthesis Approach

This research investigates the challenges posed by the predominant focus on English language text-to-image generation (TTI) because of the lack of annotated image caption data in other languages. The resulting inequitable access to TTI technology in non-English-speaking regions motivates the research of multilingual TTI (mTTI) and the potential of neural machine translation (NMT) to facilitate its development. The study presents two main contributions. Firstly, a systematic empirical study employing a multilingual multi-modal encoder evaluates standard cross-lingual NLP methods applied to mTTI, including TRANSLATE TRAIN, TRANSLATE TEST, and ZERO-SHOT TRANSFER. Secondly, a novel parameter-efficient approach called Ensemble Adapter (ENSAD) is introduced, leveraging multilingual text knowledge within the mTTI framework to avoid the language gap and enhance mTTI performance. Additionally, the research addresses challenges associated with transformer-based TTI models, such as slow generation and complexity for high-resolution images. It proposes hierarchical transformers and local parallel autoregressive generation techniques to overcome these limitations. A 6B-parameter transformer pretrained with a cross-modal general language model (CogLM) and fine-tuned for fast super-resolution results in a new text-to-image system, denoted as It, which demonstrates competitive performance compared to the state-of-the-art DALL-E-2. Furthermore, It supports interactive text-guided editing on images, offering a versatile and efficient solution for text-to-image generation.. Keywords: Text-to-image generation, Multilingual TTI (mTTI), Neural machine translation (NMT), Cross-lingual NLP, Ensemble Adapter (ENSAD), Hierarchical transformers, Super- resolution, Transformer-based models, Cross-modal general language model (CogLM).

Automatic Image Captioning using Deep Learning

Image captioning aims to detect this information by describing the image content through image and text processing techniques. Now-a-days image captioning is one of the recent and growing research problem. Day by day various solutions are being introduced for solving the problem. Even though, many solutions are already available, a lot of attention is still required for getting better and precise results. So, we came up with the idea of developing a image captioning model using different combinations of Convolutional Neural Network architecture along with Long Short Term Memory to get better results. We have used three combination of CNN and LSTM for developing the model. The proposed model is trained with three Convolutional Neural Network architecture such as Inception-v3,Xception, ResNet50 for feature extraction and Long ShortTerm Memory for generating the relevant captions. Among these models, the best combination based on the accuracy of the model. It is trained using the Flicker8k dataset

SAMT-generator: A second-attention for image captioning based on multi-stage transformer network

In recent years, Transformer has been widely used in the crossing task of computer vision (CV) and natural language processing (NLP), e.g., image captioning. The prior works of image captioning based on Transformer have achieved remarkable progress. In order to further improve the model’s ability to describe detail features and generate high-quality sentences, in the paper, we first construct a Multi-stage Transformer Feature Enhancement Network (MT-FEN), which obtains more semantics by fusing features at different scales. Furthermore, we crucially propose a novel second-attention (SA) that can focus more on valuable features and filter out noises in MT-FEN. Besides that, to generate more sensible sentences, we use multiple decoding layers to choose the best-described word via the Max-out module. The overall model is dubbed as SAMT-Generator. After conducting extensive experiments on two well-known datasets: MS COCO and Flickr8k, the results demonstrate the effectiveness of our proposed model, and it is comparable with SOTA methods.

A Novel Pretrained General-purpose Vision Language Model for the Vietnamese Language

Lying in the cross-section of computer vision and natural language processing, vision language models are capable of processing images and text at once. These models are helpful in various tasks: text generation from image and vice versa, image-text retrieval, or visual navigation. Besides building a model trained on a dataset for a task, people also study general-purpose models to utilize many datasets for multitasks. Their two primary applications are image captioning and visual question answering. For English, large datasets and foundation models are already abundant. However, for Vietnamese, they are still limited. To expand the language range, this work proposes a pretrained general-purpose image-text model named VisualRoBERTa. A dataset of 600k images with captions (translated MS COCO 2017 from English to Vietnamese) is introduced to pretrain VisualRoBERTa. The model’s architecture is built using Convolutional Neural Network and Transformer blocks. Fine-tuning VisualRoBERTa shows promising results on the ViVQA dataset with 34.49% accuracy, 0.4173 BLEU 4, and 0.4390 RougeL (in visual question answering task), and best outcomes on the sViIC dataset with 0.6685 BLEU 4, 0.6320 RougeL (in image captioning task).

Paradoxes of Global History

Do we know what global history is? Since it is fair to say that we cannot agree on a definition, why do we institutionalise global history and argue over its pros and cons? This piece lays out and pries open some of the paradoxes that grip academic debates concerning global history today, at least in most of Western Europe and North America. It suggests that one way out of our current predicaments is to downgrade global history from the status of 'methodology' or even 'discipline' (as some have called it) to a perspective that has the potential of raising new questions. Doing so will allow us to focus on how we go about answering those questions, stressing the connections but also the differences between subject-matter and method in historical writing. Image caption: Giovanni Battista Piranesi, The Staircase with Trophies, from ‘Carceri d’invenzione’ (Imaginary Prisons) (ca. 1749–1750), New York, Metropolitan Museum of Art, Harris Brisbane Dick Fund, 1938, accession number 37.45.3(24), public domain, via https://www.metmuseum.org/art/collection/search/362676

MISTRA: Misogyny Detection through Text–Image Fusion and Representation Analysis

Detecting misogynous memes poses a significant challenge due to the presence of multiple modalities (image + text). The inherent complexity arises from the lack of direct correspondence between the textual and visual elements, where an image and overlaid text often convey disparate meanings. Additionally, memes conveying messages of hatred or taunting, particularly targeted towards women, present additional comprehension difficulties. This article introduces the MISTRA framework, which leverages variational autoencoders for dimensionality reduction of the large-sized image features before fusing multimodal features. The framework also harnesses the capabilities of large language models through transfer learning to develop fusion embeddings by extracting and concatenating features from different modalities (image, text, and image-generated caption text) for the misogynous classification task. The components of the framework include state-of-the-art models such as the Vision Transformer model (ViT), textual model (DistilBERT), CLIP (Contrastive Language–Image Pre-training), and BLIP (Bootstrapping Language–Image Pre-training for Unified Vision-Language Understanding and Generation) models. Our experiments are conducted on the SemEval-2022 Task 5 MAMI dataset. To establish a baseline model, we perform separate experiments using the Naive Bayes machine learning classifier on meme texts and ViT on meme images. We evaluate the performance on six different bootstrap samples and report evaluation metrics such as precision, recall, and Macro-F1 score for each bootstrap sample. Additionally, we compute the confidence interval on our evaluation scores and conduct paired t-tests to understand whether our best-performing model has significant differences from the other experiments or not. The experimental results demonstrate that the dimensionality reduction approach on multimodal features with a multilayer perceptron classifier achieved the highest performance with a Macro–F1 score of 71.5 percent, outperforming the baseline approaches in individual modalities.

The Role of Attention Mechanism in Generating Image Captions: An Innovative Approach with Neural Network-Based Seq2seq Model

Image-to-text generation contributes significantly across various domains such as entertainment, communication, commerce, security, and education by establishing a connection between visual and textual content through the creation of explanations. This process aims to transform image data into meaningful text, enhancing content accessibility, comprehensibility, and processability. Hence, advancements and studies in this field hold paramount importance. This study focuses on how the fusion of the Sequence-to-Sequence (Seq2seq) model and attention mechanism enhances the generation of more meaningful captions from images. Experiments conducted on the Flickr8k dataset highlight the Seq2seq model's capacity to produce captions in alignment with reference sentences. Leveraging the dynamic focus of the attention mechanism, the model effectively captures detailed aspects of images.