Image Captioning Research Articles

Detection and Caption Generation of Image Using Deep Learning

In this research paper, we employ a combination of CNN and LSTM to tackle image caption generation, a task at the intersection of natural language processing and computer vision. We meticulously discuss key concepts in photograph captioning and its methodologies, leveraging resources like the Keras library, numpy, and Jupyter notebooks. Our research also delves into the flickr_dataset and CNN for photo classification, aiming to shed light on the intricate processes underlying image understanding and caption generation. Keywords- CNN ,LSTM, Image captioning, Deep Learning.

Extracting Audio from Image Using Machine Learning

This study introduces a new method for extracting sound from pictures by utilizing machine learning. Lately, there has been a lot of excitement around multi-modal learning because of its ability to reveal valuable information from various sources, like images and sound. Our research is centered on using the unique qualities of visual and auditory signals to predict sound content from pictures. This opens up possibilities for enhancing accessibility, creating content, and providing immersive user experiences. We start by exploring previous research in multi-modal learning, audio-visual processing, and tasks like image captioning and sound source localization. Based on this background, we introduce an approach that merges convolutional neural networks (CNNs) for image analysis with recurrent neural networks (RNNs) or transformers for sequence interpretation. The system is educated on a collection of matched images and associated audio tracks, allowing it to grasp the intricate connections between visual and auditory data. In our study, we carefully assessed the performance of our proposed method by using well-known metrics. We measure how well our method works by comparing it to other methods and showing that it can accurately and quickly extract audio from images. We also show through qualitative analysis that our model can create clear audio representations from a variety of visual inputs. After a thorough discussion, we analyze the findings, pointing out both the advantages and drawbacks of our method. We pinpoint potential areas for further study, such as delving into more advanced structures and incorporating semantic data to enhance audio extraction. To sum up, this study adds to the expanding field of multi-modal learning by introducing a promising model for extracting audio from images through machine learning. Our results emphasize the potential of this technology to improve accessibility, inspire creativity, and increase user engagement in different fields. Key Words: Audio Extraction, Machine Learning, Computer Vision, Deep Learning, Convolutional Neural Networks

DeepLens: Integrating Deep Learning for Image Captioning and Hashtag Generation

In this research, we introduce a groundbreaking methodology that leverages deep learning techniques to revolutionize the process of generating descriptive captions and hashtags for images, effectively bridging the gap between computer vision and natural language understanding. Traditional approaches to image caption generation have often relied on rudimentary techniques such as handcrafted features and rule-based systems, which inherently struggle to capture the intricate semantics of images and adapt to diverse datasets. Recognizing these limitations, our novel framework integrates state-of-the-art convolutional neural networks (CNNs) for precise image feature extraction and recurrent neural networks (RNNs), specifically employing the ResNet-50 architecture, for seamless sequence generation. Furthermore, in addition to traditional evaluation metrics such as BLEU scores and human assessments, our system, DeepLens, introduces a groundbreaking feature: automatic hashtag generation. By meticulously analyzing both the content and context of images, DeepLens autonomously generates hashtags that enrich social media content sharing and engagement, presenting a novel paradigm in the realm of image captioning. In addition to its advancements in captioning accuracy and user experience enhancement, DeepLens offers scalability and adaptability to various domains. Its robust architecture allows for seamless integration with different datasets and environments, making it versatile for a wide range of applications. Through this comprehensive approach, we aim to not only enhance the accuracy and relevance of generated captions but also elevate the overall user experience in navigating and interacting with visual content across various platforms and applications.

RS-LLaVA: A Large Vision-Language Model for Joint Captioning and Question Answering in Remote Sensing Imagery

In this paper, we delve into the innovative application of large language models (LLMs) and their extension, large vision-language models (LVLMs), in the field of remote sensing (RS) image analysis. We particularly emphasize their multi-tasking potential with a focus on image captioning and visual question answering (VQA). In particular, we introduce an improved version of the Large Language and Vision Assistant Model (LLaVA), specifically adapted for RS imagery through a low-rank adaptation approach. To evaluate the model performance, we create the RS-instructions dataset, a comprehensive benchmark dataset that integrates four diverse single-task datasets related to captioning and VQA. The experimental results confirm the model’s effectiveness, marking a step forward toward the development of efficient multi-task models for RS image analysis.

Optimized Image Captioning: Hybrid Transformers Vision Transformers and Convolutional Neural Networks: Enhanced with Beam Search

Deep learning has improved image captioning. Transformer, a neural network architecture built for natural language processing, excels at image captioning and other computer vision applications. This paper reviews Transformer-based image captioning methods in detail. Convolutional neural networks (CNNs) extracted image features and RNNs or LSTM networks generated captions in traditional image captioning. This method often has information bottlenecks and trouble capturing long-range dependencies. Transformer architecture revolutionized natural language processing with its attention strategy and parallel processing. Researchers used Transformers' language success to solve image captioning problems. Transformer-based image captioning systems outperform previous methods in accuracy and efficiency by integrating visual and textual information into a single model. This paper discusses how the Transformer architecture's self-attention mechanisms and positional encodings are adapted for image captioning. Vision Transformers (ViTs) and CNN-Transformer hybrid models are discussed. We also discuss pre-training, fine-tuning, and reinforcement learning to improve caption quality. Transformer-based image captioning difficulties, trends, and future approaches are also examined. Multimodal fusion, visual-text alignment, and caption interpretability are challenges. We expect research to address these issues and apply Transformer-based image captioning to medical imaging and distant sensing. This paper covers how Transformer-based approaches have changed image captioning and their potential to revolutionize multimodal interpretation and generation, advancing artificial intelligence and human-computer interactions.

PhrasIS: Phrase Inference and Similarity benchmark

Abstract We present PhrasIS, a benchmark dataset composed of natural occurring Phrase pairs with Inference and Similarity annotations for the evaluation of semantic representations. The described dataset fills the gap between word and sentence-level datasets, allowing to evaluate compositional models at a finer granularity than sentences. Contrary to other datasets, the phrase pairs are extracted from naturally occurring text in image captions and news headlines. All the text fragments have been annotated by experts following a rigorous process also described in the manuscript achieving high inter annotator agreement. In this work we analyse the dataset, showing the relation between inference labels and similarity scores. With 10K phrase pairs split in development and test, the dataset is an excellent benchmark for testing meaning representation systems.

Audio Based Object Detection System: A Comprehensive Survey

This survey addresses visual impairment by leveraging advanced technology to empower independent navigation for visually impaired individuals. It emphasizes the importance of tailored systems and highlights successful studies implementing innovative solutions. Incorporating diverse technologies like object detection, NLP, and information retrieval, the project explores deep learning algorithms such as CNN, RNN, and YOLO, alongside audio-based input/output integration. It recommends utilizing APIs like Google Text-to-Speech and Python libraries for efficient implementation, aiming to enhance system functionality and accessibility. Ultimately, the survey aims to aid in selecting appropriate tools and methodologies for developing audio-based object detection systems, contributing to ongoing efforts in supporting visually impaired individuals. Key Words: Object Detection, Natural Language Processing, Audio Data, YOLO, Real-Time identification, Visually Impaired, Image Captioning.

Recurrent Neural Networks for Image Captioning: A Case Study with LSTM

This research investigates the viability of Long Short-Term Memory (LSTM) systems, a subtype of Recurrent Neural Networks (RNNs), for picture captioning. Leveraging the MS COCO dataset, the study compares the execution of LSTM-based RNNs with Vanilla RNN, Gated Recurrent Unit (GRU), consideration components, and transformer-based models. Experimental comes about to illustrate that the LSTM-based RNN shows competitive execution, accomplishing a BLEU-4 score of 0.72, a METEOR score of 0.68, and a CIDEr score of 2.1. The comparative investigation uncovers its prevalence over Vanilla RNN and GRU, highlighting its capability to capture long-range conditions inside successive picture information. Moreover, the study investigates the effect of consideration instruments and transformer designs, exhibiting their potential improvements in the context-aware caption era. The transformer-based show outflanks all other models, accomplishing a BLEU-4 score of 0.78, a METEOR score of 0.72, and a CIDEr score of 2.5. The findings give important bits of knowledge toward the creating scene of picture captioning strategy, which makes LSTM-based RNNs solid and productive approaches for capturing worldly groupings in visual substance. In achieving these, the study provides a framework for future developments in hybrid models and manufacturing processes that push boundaries of smart image perception and understanding

Self-Enhanced Attention for Image Captioning

Image captioning, which involves automatically generating textual descriptions based on the content of images, has garnered increasing attention from researchers. Recently, Transformers have emerged as the preferred choice for the language model in image captioning models. Transformers leverage self-attention mechanisms to address gradient accumulation issues and eliminate the risk of gradient explosion commonly associated with RNN networks. However, a challenge arises when the input features of the self-attention mechanism belong to different categories, as it may result in ineffective highlighting of important features. To address this issue, our paper proposes a novel attention mechanism called Self-Enhanced Attention (SEA), which replaces the self-attention mechanism in the decoder part of the Transformer model. In our proposed SEA, after generating the attention weight matrix, it further adjusts the matrix based on its own distribution to effectively highlight important features. To evaluate the effectiveness of SEA, we conducted experiments on the COCO dataset, comparing the results with different visual models and training strategies. The experimental results demonstrate that when using SEA, the CIDEr score is significantly higher compared to the scores obtained without using SEA. This indicates the successful addressing of the challenge of effectively highlighting important features with our proposed mechanism.

Exploring a Spectrum of Deep Learning Models for Automated Image Captioning: A Comprehensive Survey

Automatic caption generation from images has emerged as a fundamental and challenging problem at the intersection of computer vision and natural language processing. This paper presents a comprehensive survey of the techniques, methodologies, and advancements in the field of automatic caption generation from images. The primary objective is to provide an extensive review of the state-of-the-art models, evaluation metrics, datasets, and applications associated with this domain. The survey begins by elucidating the underlying principles of image feature extraction and caption generation. Various neural network architectures, including Convolutional Neural Networks (CNNs) and recurrent models such as Long Short-Term Memory (LSTM) networks, are discussed in detail. Additionally, the paper explores the integration of attention mechanisms and reinforcement learning strategies to enhance the quality and relevance of generated captions. A thorough examination of evaluation metrics, encompassing both automated and human-centric approaches, is presented to evaluate the generated captions quantitatively and qualitatively. The survey also highlights prominent datasets that have significantly contributed to the advancement of research in this field, facilitating a deeper understanding of challenges and trends. Furthermore, the paper discusses practical applications and real-world use cases where automatic caption generation plays a pivotal role, including accessibility, multimedia indexing, and assistive technologies. The discussion concludes by outlining open challenges and future directions, aiming to inspire further research and innovation in automatic caption generation from images. The aim of this paper is to examine and contrast diverse end-to-end learning frameworks for image captioning, employing established evaluation metrics to comprehend their applicability across different research domains. In addition to the comparative analysis, the paper addresses future challenges in this domain.

Graph neural networks in vision-language image understanding: a survey

Abstract2D image understanding is a complex problem within computer vision, but it holds the key to providing human-level scene comprehension. It goes further than identifying the objects in an image, and instead, it attempts to understand the scene. Solutions to this problem form the underpinning of a range of tasks, including image captioning, visual question answering (VQA), and image retrieval. Graphs provide a natural way to represent the relational arrangement between objects in an image, and thus, in recent years graph neural networks (GNNs) have become a standard component of many 2D image understanding pipelines, becoming a core architectural component, especially in the VQA group of tasks. In this survey, we review this rapidly evolving field and we provide a taxonomy of graph types used in 2D image understanding approaches, a comprehensive list of the GNN models used in this domain, and a roadmap of future potential developments. To the best of our knowledge, this is the first comprehensive survey that covers image captioning, visual question answering, and image retrieval techniques that focus on using GNNs as the main part of their architecture.

Application of Multimodal Transformer Model in Intelligent Agricultural Disease Detection and Question-Answering Systems.

In this study, an innovative approach based on multimodal data and the transformer model was proposed to address challenges in agricultural disease detection and question-answering systems. This method effectively integrates image, text, and sensor data, utilizing deep learning technologies to profoundly analyze and process complex agriculture-related issues. The study achieved technical breakthroughs and provides new perspectives and tools for the development of intelligent agriculture. In the task of agricultural disease detection, the proposed method demonstrated outstanding performance, achieving a precision, recall, and accuracy of 0.95, 0.92, and 0.94, respectively, significantly outperforming the other conventional deep learning models. These results indicate the method's effectiveness in identifying and accurately classifying various agricultural diseases, particularly excelling in handling subtle features and complex data. In the task of generating descriptive text from agricultural images, the method also exhibited impressive performance, with a precision, recall, and accuracy of 0.92, 0.88, and 0.91, respectively. This demonstrates that the method can not only deeply understand the content of agricultural images but also generate accurate and rich descriptive texts. The object detection experiment further validated the effectiveness of our approach, where the method achieved a precision, recall, and accuracy of 0.96, 0.91, and 0.94. This achievement highlights the method's capability for accurately locating and identifying agricultural targets, especially in complex environments. Overall, the approach in this study not only demonstrated exceptional performance in multiple tasks such as agricultural disease detection, image captioning, and object detection but also showcased the immense potential of multimodal data and deep learning technologies in the application of intelligent agriculture.

ImageCaptioner2: Image Captioner for Image Captioning Bias Amplification Assessment

Most pre-trained learning systems are known to suffer from bias, which typically emerges from the data, the model, or both. Measuring and quantifying bias and its sources is a challenging task and has been extensively studied in image captioning. Despite the significant effort in this direction, we observed that existing metrics lack consistency in the inclusion of the visual signal. In this paper, we introduce a new bias assessment metric, dubbed ImageCaptioner2, for image captioning. Instead of measuring the absolute bias in the model or the data, ImageCaptioner2pay more attention to the bias introduced by the model w.r.t the data bias, termed bias amplification. Unlike the existing methods, which only evaluate the image captioning algorithms based on the generated captions only, ImageCaptioner2incorporates the image while measuring the bias. In addition, we design a formulation for measuring the bias of generated captions as prompt-based image captioning instead of using language classifiers. Finally, we apply our ImageCaptioner2metric across 11 different image captioning architectures on three different datasets, i.e., MS-COCO caption dataset, Artemis V1, and Artemis V2, and on three different protected attributes, i.e., gender, race, and emotions. Consequently, we verify the effectiveness of our ImageCaptioner2metric by proposing Anonymous-Bench, which is a novel human evaluation paradigm for bias metrics. Our metric shows significant superiority over the recent bias metric; LIC, in terms of human alignment, where the correlation scores are 80% and 54% for our metric and LIC, respectively. The code and more details are available at https://eslambakr.github.io/imagecaptioner2.github.io/.

Cycle-Consistency Learning for Captioning and Grounding

We present that visual grounding and image captioning, which perform as two mutually inverse processes, can be bridged together for collaborative training by careful designs. By consolidating this idea, we introduce CyCo, a cyclic-consistent learning framework to ameliorate the independent training pipelines of visual grounding and image captioning. The proposed framework (1) allows the semi-weakly supervised training of visual grounding; (2) improves the performance of fully supervised visual grounding; (3) yields a general captioning model that can describe arbitrary image regions. Extensive experiments show that our fully supervised grounding model achieves state-of-the-art performance, and the semi-weakly supervised one also exhibits competitive performance compared to the fully supervised counterparts. Our image captioning model has the capability to freely describe image regions and meanwhile shows impressive performance on prevalent captioning benchmarks.

Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning

Although image captioning models have made significant advancements in recent years, the majority of them heavily depend on high-quality datasets containing paired images and texts which are costly to acquire. Previous works leverage the CLIP's cross-modal association ability for image captioning, relying solely on textual information under unsupervised settings. However, not only does a modality gap exist between CLIP text and image features, but a discrepancy also arises between training and inference due to the unavailability of real-world images, which hinders the cross-modal alignment in text-only captioning. This paper proposes a novel method to address these issues by incorporating synthetic image-text pairs. A pre-trained text-to-image model is deployed to obtain images that correspond to textual data, and the pseudo features of generated images are optimized toward the real ones in the CLIP embedding space. Furthermore, textual information is gathered to represent image features, resulting in the image features with various semantics and the bridged modality gap. To unify training and inference, synthetic image features would serve as the training prefix for the language decoder, while real images are used for inference. Additionally, salient objects in images are detected as assistance to enhance the learning of modality alignment. Experimental results demonstrate that our method obtains the state-of-the-art performance on benchmark datasets.

Relational Distant Supervision for Image Captioning without Image-Text Pairs

Unsupervised image captioning aims to generate descriptions of images without relying on any image-sentence pairs for training. Most existing works use detected visual objects or concepts as bridge to connect images and texts. Considering that the relationship between objects carries more information, we use the object relationship as a more accurate connection between images and texts. In this paper, we adapt the idea of distant supervision that extracts the knowledge about object relationships from an external corpus and imparts them to images to facilitate inferring visual object relationships, without introducing any extra pre-trained relationship detectors. Based on these learned informative relationships, we construct pseudo image-sentence pairs for captioning model training. Specifically, our method consists of three modules: (1) a relationship learning module that learns to infer relationships from images under the distant supervision; (2) a relationship-to-sentence module that transforms the inferred relationships into sentences to generate pseudo image-sentence pairs; (3) an image captioning module that is trained by using the generated image-sentence pairs. Promising results on three datasets show that our method outperforms the state-of-the-art methods of unsupervised image captioning.

Noise-Aware Image Captioning with Progressively Exploring Mismatched Words

Image captioning aims to automatically generate captions for images by learning a cross-modal generator from vision to language. The large amount of image-text pairs required for training is usually sourced from the internet due to the manual cost, which brings the noise with mismatched relevance that affects the learning process. Unlike traditional noisy label learning, the key challenge in processing noisy image-text pairs is to finely identify the mismatched words to make the most use of trustworthy information in the text, rather than coarsely weighing the entire examples. To tackle this challenge, we propose a Noise-aware Image Captioning method (NIC) to adaptively mitigate the erroneous guidance from noise by progressively exploring mismatched words. Specifically, NIC first identifies mismatched words by quantifying word-label reliability from two aspects: 1) inter-modal representativeness, which measures the significance of the current word by assessing cross-modal correlation via prediction certainty; 2) intra-modal informativeness, which amplifies the effect of current prediction by combining the quality of subsequent word generation. During optimization, NIC constructs the pseudo-word-labels considering the reliability of the origin word-labels and model convergence to periodically coordinate mismatched words. As a result, NIC can effectively exploit both clean and noisy image-text pairs to learn a more robust mapping function. Extensive experiments conducted on the MS-COCO and Conceptual Caption datasets validate the effectiveness of our method in various noisy scenarios.

BARET: Balanced Attention Based Real Image Editing Driven by Target-Text Inversion

Image editing approaches with diffusion models have been rapidly developed, yet their applicability are subject to requirements such as specific editing types (e.g., foreground or background object editing, style transfer), multiple conditions (e.g., mask, sketch, caption), and time consuming fine-tuning of diffusion models. For alleviating these limitations and realizing efficient real image editing, we propose a novel editing technique that only requires an input image and target text for various editing types including non-rigid edits without fine-tuning diffusion model. Our method contains three novelties: (I) Target-text Inversion Schedule (TTIS) is designed to fine-tune the input target text embedding to achieve fast image reconstruction without image caption and acceleration of convergence. (II) Progressive Transition Scheme applies progressive linear interpolation between target text embedding and its fine-tuned version to generate transition embedding for maintaining non-rigid editing capability. (III) Balanced Attention Module (BAM) balances the tradeoff between textual description and image semantics. By the means of combining self-attention map from reconstruction process and cross-attention map from transition process, the guidance of target text embeddings in diffusion process is optimized. In order to demonstrate editing capability, effectiveness and efficiency of the proposed BARET, we have conducted extensive qualitative and quantitative experiments. Moreover, results derived from user study and ablation study further prove the superiority over other methods.

P-Laplacian Adaptation for Generative Pre-trained Vision-Language Models

Vision-Language models (VLMs) pre-trained on large corpora have demonstrated notable success across a range of downstream tasks. In light of the rapidly increasing size of pre-trained VLMs, parameter-efficient transfer learning (PETL) has garnered attention as a viable alternative to full fine-tuning. One such approach is the adapter, which introduces a few trainable parameters into the pre-trained models while preserving the original parameters during adaptation. In this paper, we present a novel modeling framework that recasts adapter tuning after attention as a graph message passing process on attention graphs, where the projected query and value features and attention matrix constitute the node features and the graph adjacency matrix, respectively. Within this framework, tuning adapters in VLMs necessitates handling heterophilic graphs, owing to the disparity between the projected query and value space. To address this challenge, we propose a new adapter architecture, p-adapter, which employs p-Laplacian message passing in Graph Neural Networks (GNNs). Specifically, the attention weights are re-normalized based on the features, and the features are then aggregated using the calibrated attention matrix, enabling the dynamic exploitation of information with varying frequencies in the heterophilic attention graphs. We conduct extensive experiments on different pre-trained VLMs and multi-modal tasks, including visual question answering, visual entailment, and image captioning. The experimental results validate our method's significant superiority over other PETL methods. Our code is available at https://github.com/wuhy68/p-Adapter/.

Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classification

Existing image augmentation methods consist of two categories: perturbation-based methods and generative methods. Perturbation-based methods apply pre-defined perturbations to augment an original image, but only locally vary the image, thus lacking image diversity. In contrast, generative methods bring more image diversity in the augmented images but may not preserve semantic consistency, thus may incorrectly change the essential semantics of the original image. To balance image diversity and semantic consistency in augmented images, we propose SGID, a Semantic-guided Generative Image augmentation method with Diffusion models for image classification. Specifically, SGID employs diffusion models to generate augmented images with good image diversity. More importantly, SGID takes image labels and captions as guidance to maintain semantic consistency between the augmented and original images. Experimental results show that SGID outperforms the best augmentation baseline by 1.72% on ResNet-50 (from scratch), 0.33% on ViT (ImageNet-21k), and 0.14% on CLIP-ViT (LAION-2B). Moreover, SGID can be combined with other image augmentation baselines and further improves the overall performance. We demonstrate the semantic consistency and image diversity of SGID through quantitative human and automated evaluations, as well as qualitative case studies.