CIDEr Score Research Articles

Transformer with token attention and attribute prediction for image captioning

Recently, Vision Transformers (ViTs) have become the mainstream models in image captioning tasks. ViTs take all image tokens as inputs to extract visual features, which may cause concerns about worthless tokens, and meanwhile lead to a huge amount of computation. This paper proposes a novel token reduction module to remedy this drawback. Specifically, the module employs ViTs to embed the input tokens, and adaptively learns informative visual tokens in way of token attention on the channel-spatial granularity. Furthermore, an attribute prediction module is designed to strengthen the relationship between vision and language. Technically, the attribute prediction is achieved via a classifier in form of Multi-Layer Perceptron (MLP). Both the visual representations and attribute representations are obtained by Transformers, which are then combined as the input of the Transformer decoder for caption generation. All of the modules are constructed in an encoder–decoder framework and support the end-to-end learning. Experiment results have shown that our approach can effectively reduce the computational cost of ViTs while maintaining comparable performance on the MS COCO and NoCaps datasets. For example, by pruning more than 70% of the input tokens, our approach greatly reduces GFLOPs by 41% ∼ 47%, while preserving its accuracy of a 142.1 CIDEr score on the MS COCO dataset.

NeuSyRE: Neuro-symbolic visual understanding and reasoning framework based on scene graph enrichment

Exploring the potential of neuro-symbolic hybrid approaches offers promising avenues for seamless high-level understanding and reasoning about visual scenes. Scene Graph Generation (SGG) is a symbolic image representation approach based on deep neural networks (DNN) that involves predicting objects, their attributes, and pairwise visual relationships in images to create scene graphs, which are utilized in downstream visual reasoning. The crowdsourced training datasets used in SGG are highly imbalanced, which results in biased SGG results. The vast number of possible triplets makes it challenging to collect sufficient training samples for every visual concept or relationship. To address these challenges, we propose augmenting the typical data-driven SGG approach with common sense knowledge to enhance the expressiveness and autonomy of visual understanding and reasoning. We present a loosely-coupled neuro-symbolic visual understanding and reasoning framework that employs a DNN-based pipeline for object detection and multi-modal pairwise relationship prediction for scene graph generation and leverages common sense knowledge in heterogenous knowledge graphs to enrich scene graphs for improved downstream reasoning. A comprehensive evaluation is performed on multiple standard datasets, including Visual Genome and Microsoft COCO, in which the proposed approach outperformed the state-of-the-art SGG methods in terms of relationship recall scores, i.e. Recall@K and mean Recall@K, as well as the state-of-the-art scene graph-based image captioning methods in terms of SPICE and CIDEr scores with comparable BLEU, ROGUE and METEOR scores. As a result of enrichment, the qualitative results showed improved expressiveness of scene graphs, resulting in more intuitive and meaningful caption generation using scene graphs. Our results validate the effectiveness of enriching scene graphs with common sense knowledge using heterogeneous knowledge graphs. This work provides a baseline for future research in knowledge-enhanced visual understanding and reasoning. The source code is available at https://github.com/jaleedkhan/neusire.

Generative adversarial network for semi-supervised image captioning

Traditional supervised image captioning methods usually rely on a large number of images and paired captions for training. However, the creation of such datasets necessitates considerable temporal and human resources. Therefore, we propose a new semi-supervised image captioning algorithm to solve this problem. The proposed method uses a generative adversarial network to generate images that match captions, and uses these generated images and captions as new training data. This avoids the error accumulation problem when generating pseudo captions with autoregressive method and the network can directly perform backpropagation. At the same time, in order to ensure the correlation between the generated images and captions, we introduced the CLIP model for constraints. The CLIP model has been pre-trained on a large amount of image–text data, so it shows excellent performance in semantic alignment of images and text. To verify the effectiveness of our method, we validate on MSCOCO offline “Karpathy” test split. Experiment results show that our method can significantly improve the performance of the model when using 1% paired data, with the CIDEr score increasing from 69.5% to 77.7%. This shows that our method can effectively utilize unlabeled data for image caption tasks.

Discrete diffusion models with Refined Language-Image Pre-trained representations for remote sensing image captioning

RS image captioning (RSIC) utilizes natural language to provide a description of image content, assisting in the comprehension of object properties and relationships. Nonetheless, RS images are characterized by variations in object scales, distributions, and quantities, which make it challenging to obtain global semantic information and object connections. To enhance the accuracy of captions produced from RS images, this paper proposes a novel method referred to as Discrete Diffusion Models with Refined Language-Image Pre-trained representations (DDM-RLIP), leveraging an advanced discrete diffusion model (DDM) for nosing and denoising text tokens. DDM-RLIP is based on an advanced DDM-based method designed for natural pictures. The primary approach for refining image representations involves fine-tuning a CLIP image encoder on RS images, followed by adapting the transformer with an additional attention module to focus on crucial image regions and relevant words. Furthermore, experiments were conducted on three datasets, Sydney-Captions, UCM-Captions, and NWPU-Captions, and the results demonstrated the superior performance of the proposed method compared to conventional autoregressive models. On the NWPU-Captions dataset, the CIDEr score improved from 116.4 to 197.7, further validating the efficacy and potential of DDM-RLIP. The implementation codes for our approach DDM-RLIP are available at https://github.com/Leng-bingo/DDM-RLIP.

M[formula omitted]ixup: A multi-modal data augmentation approach for image captioning

Despite the great success, most models in image captioning (IC) are still stuck in the dilemma of generating simple and non-discriminative captions. In this paper, we study this problem from the perspective of data augmentation and propose a novel method called Multi-modal Mixup (M3ixup). Compared with the original Mixup strategy designed for image classification, the proposed M3ixup has three novel designs to mix IC samples from the aspects of visual features, sentence embeddings and loss values, respectively. In practice, M3ixup can not only enrich the diversity of IC training data, but also enforce the model to focus more on visual information for captioning, thereby alleviating the negative effect of dataset bias and addressing the issue of simple captioning. To validate M3ixup, we apply it to three baseline models and conduct extensive experiments on MS COCO. The experimental results demonstrate that our proposed M3ixup can not only improve the discriminability and quality of generated captions, but also help the baseline models obtain obvious performance gains, i.e., improving the CIDEr scores of the state-of-the-art model from 133.8 to 135.3 on off-line testing and 135.4 to 137.1 on online testing.

Cross-region feature fusion with geometrical relationship for OCR-based image captioning

Automatically generating a readable sentence that describes the text-contained image is a challenging task. Compared to traditional image captioning algorithms, OCR-based image captioning focuses on reading OCR tokens in images and understanding them with the image content to generate descriptions. However, existing research mainly concentrate on improving the quantity and quality of obtaining OCR tokens and exploring their spatial relationships while lacking investigation into how to effectively join OCR tokens with image content. This paper proposes a cross-region feature fusion with a geometrical relationship Transformer(CFGR-Transformer) for OCR-based image captioning. The network first establishes the associations between the OCR and object regions of the image by constructing relative geometric relationships, including width/height difference, distance, IOU(Intersection over Union), inclusion relationships, and angles offset, and then incorporates intra-region and cross-region features to aggregate entities from different modalities by a multi-head attention mechanism based on relative relationships. Benefiting from the guidance of the relative relationship, visual entities like OCR tokens and object regions can consider multiple relative relationships as the attention weight for feature fusion within each subspace. Extensive experiments conducted on the TextCaps dataset demonstrate the effectiveness of the proposed CFGR-Transformer method. In particular, our results on the online testing of TextCaps achieve an improvement in CIDEr score from 93.0% to 98.2%.

Enhancing Image Captioning Using Deep Convolutional Generative Adversarial Networks

Introduction:: Introduction: Image caption generation has long been a fundamental challenge in the area of computer vision (CV) and natural language processing (NLP). In this research, we present an innovative approach that harnesses the power of Deep Convolutional Generative Adversarial Networks (DCGAN) and adversarial training to revolutionize the generation of natural and contextually relevant image captions. Method:: Our method significantly improves the fluency, coherence, and contextual relevance of generated captions and showcases the effectiveness of RL reward-based fine-tuning. Through a comprehensive evaluation of COCO datasets, our model demonstrates superior performance over baseline and state-of-the-art methods. On the COCO dataset, our model outperforms current state-of-the-art (SOTA) models across all metrics, achieving BLEU-4 (0.327), METEOR (0.249), Rough (0.525) and CIDEr (1.155) scores. Result:: The integration of DCGAN and adversarial training opens new possibilities in image captioning, with applications spanning from automated content generation to enhanced accessibility solutions. Conclusion:: This research paves the way for more intelligent and context-aware image understanding systems, promising exciting future exploration and innovation prospects.

Clustering-based mask recovery for image captioning

Region features play a huge role in image captioning. However, obtaining region features requires pre-training an object detector by using a large number of object detection datasets. It may be impossible for the end-to-end training. And if there is a large distribution difference between the object detection datasets and the image captioning datasets, the object detector may not be able to extract accurate region features. This makes it limited in application. In this paper, we propose a clustering-based mask recovery for image captioning. In the encoder, the pseudo-region features are obtained by clustering the grid features, which are extracted using Swin Transformer. Then we input the grid features together with the pseudo-region features into the decoder, and make the model to dynamically learns the weights of the two features in the decoding process to minimize the effect of errors caused by clustering. By using a clustering method to generate pseudo-region features for images, not only does the training process become end-to-end, but there is no need to introduce additional object detection datasets to train the object detector. In addition, the Transformer decoder has a misplaced problem in the decoding process. This means that the positional information used by the model when generating a word is not the same as the positional information used when it continues to use the word to reason. This may have some negative impact on the position encoding of the model. Therefore, we changed the original decoding method to mask recovery. Furthermore, a masked multi-head attention module with relative position is proposed in the decoder to integrate the information in the fusion features, and reconstruct the relative position relationship between words. We conduct experiments on MSCOCO 2014 dataset. The experiment results show that our model obtains 144.3% (single model) and 147.0% (ensemble of 4 models) CIDEr scores on ‘Karpathy’ offline test split, and 143.2% (c40) CIDEr scores on the official online test server.

Investigation of Visual Language Landscape of Tourist Attractions from Multimodal Perspective

With the development of economic globalization, the tourism industry has been welcomed by the public. The visual language landscape of tourist attractions can not only assist tourists to play and watch the project, but if it is properly planned, the language landscape can also become a major feature and highlight of the scenic spot. Therefore, how to set up and construct the visual language landscape of tourist attractions is a problem that needs to be considered in each region. In response to the above problems, on the basis of understanding the concept types of the visual language landscape of tourist attractions, this article conducts in-depth research and investigation on the visual language landscape of tourist attractions, combining the evaluation dataset in the multimodal perspective and the Convolutional Neural Network (CNN) –Recurrent Neural Network (RNN) model based on semantic regularization. This article conducted a comparative experiment on each model on the NUS-WIDE dataset and the MS-COCO dataset. The experimental results showed that it was crucial to give full play to the expressive power of the CNN. Compared to the NUS-WIDE dataset, the MS-COCO dataset brought less additional boost by leveraging social media tags. The CIDEr score of the CNN-RNN model based on semantic regularization was improved by 11.4%, which placed the foundation for the investigation and analysis of the linguistic landscape of tourist attractions.

Image description software generation based on data mining and computer vision

In order to solve the problems of low quality of image description, insufficient use of image features, and single level of recurrent neural network in image description generation, this paper proposes an image description generation method based on multi-scale features and computer vision. The algorithm uses the pre trained target detection network to extract the image features on different layers of the convolution neural network, inputs the image features into the multi attention structure layer by layer, connects the multi attention structure with the multi-layer recurrent neural network in turn, and constructs a multi-level image description generation network model. Adding residual connections to multilayer recurrent neural networks can improve the network performance, and can effectively avoid the network degradation caused by network deepening. The experimental results show that in the mscoco test set, the bleu-1 and cider scores of the proposed algorithm can reach 0.804 and 1.167 respectively, which is significantly better than the top-down image description generation algorithm based on a single attention structure. Conclusionthe image description generated by the proposed algorithm can show better image details.

Recurrent Neural Networks for Image Captioning: A Case Study with LSTM

This research investigates the viability of Long Short-Term Memory (LSTM) systems, a subtype of Recurrent Neural Networks (RNNs), for picture captioning. Leveraging the MS COCO dataset, the study compares the execution of LSTM-based RNNs with Vanilla RNN, Gated Recurrent Unit (GRU), consideration components, and transformer-based models. Experimental comes about to illustrate that the LSTM-based RNN shows competitive execution, accomplishing a BLEU-4 score of 0.72, a METEOR score of 0.68, and a CIDEr score of 2.1. The comparative investigation uncovers its prevalence over Vanilla RNN and GRU, highlighting its capability to capture long-range conditions inside successive picture information. Moreover, the study investigates the effect of consideration instruments and transformer designs, exhibiting their potential improvements in the context-aware caption era. The transformer-based show outflanks all other models, accomplishing a BLEU-4 score of 0.78, a METEOR score of 0.72, and a CIDEr score of 2.5. The findings give important bits of knowledge toward the creating scene of picture captioning strategy, which makes LSTM-based RNNs solid and productive approaches for capturing worldly groupings in visual substance. In achieving these, the study provides a framework for future developments in hybrid models and manufacturing processes that push boundaries of smart image perception and understanding

Self-Enhanced Attention for Image Captioning

Image captioning, which involves automatically generating textual descriptions based on the content of images, has garnered increasing attention from researchers. Recently, Transformers have emerged as the preferred choice for the language model in image captioning models. Transformers leverage self-attention mechanisms to address gradient accumulation issues and eliminate the risk of gradient explosion commonly associated with RNN networks. However, a challenge arises when the input features of the self-attention mechanism belong to different categories, as it may result in ineffective highlighting of important features. To address this issue, our paper proposes a novel attention mechanism called Self-Enhanced Attention (SEA), which replaces the self-attention mechanism in the decoder part of the Transformer model. In our proposed SEA, after generating the attention weight matrix, it further adjusts the matrix based on its own distribution to effectively highlight important features. To evaluate the effectiveness of SEA, we conducted experiments on the COCO dataset, comparing the results with different visual models and training strategies. The experimental results demonstrate that when using SEA, the CIDEr score is significantly higher compared to the scores obtained without using SEA. This indicates the successful addressing of the challenge of effectively highlighting important features with our proposed mechanism.

Tag‐inferring and tag‐guided Transformer for image captioning

AbstractImage captioning is an important task for understanding images. Recently, many studies have used tags to build alignments between image information and language information. However, existing methods ignore the problem that simple semantic tags have difficulty expressing the detailed semantics for different image contents. Therefore, the authors propose a tag‐inferring and tag‐guided Transformer for image captioning to generate fine‐grained captions. First, a tag‐inferring encoder is proposed, which uses the tags extracted by the scene graph model to infer tags with deeper semantic information. Then, with the obtained deep tag information, a tag‐guided decoder that includes short‐term attention to improve the features of words in the sentence and gated cross‐modal attention to combine image features, tag features and language features to produce informative semantic features is proposed. Finally, the word probability distribution of all positions in the sequence is calculated to generate descriptions for the image. The experiments demonstrate that the authors’ method can combine tags to obtain precise captions and that it achieves competitive performance with a 40.6% BLEU‐4 score and 135.3% CIDEr score on the MSCOCO data set.

A Unified Visual and Linguistic Semantics Method for Enhanced Image Captioning

Image captioning, also recognized as the challenge of transforming visual data into coherent natural language descriptions, has persisted as a complex problem. Traditional approaches often suffer from semantic gaps, wherein the generated textual descriptions lack depth, context, or the nuanced relationships contained within the images. In an effort to overcome these limitations, we introduce a novel encoder–decoder framework called A Unified Visual and Linguistic Semantics Method. Our method comprises three key components: an encoder, a mapping network, and a decoder. The encoder employs a fusion of CLIP (Contrastive Language–Image Pre-training) and SegmentCLIP to process and extract salient image features. SegmentCLIP builds upon CLIP’s foundational architecture by employing a clustering mechanism, thereby enhancing the semantic relationships between textual and visual elements in the image. The extracted features are then transformed by a mapping network into a fixed-length prefix. A GPT-2-based decoder subsequently generates a corresponding Chinese language description for the image. This framework aims to harmonize feature extraction and semantic enrichment, thereby producing more contextually accurate and comprehensive image descriptions. Our quantitative assessment reveals that our model exhibits notable enhancements across the intricate AIC-ICC, Flickr8k-CN, and COCO-CN datasets, evidenced by a 2% improvement in BLEU@4 and a 10% uplift in CIDEr scores. Additionally, it demonstrates acceptable efficiency in terms of simplicity, speed, and reduction in computational burden.

EgoCap and EgoFormer: First-person image captioning with context fusion

First-person captioning is significant because it provides veracious descriptions of egocentric scenes in a unique perspective. Also, there is a need to caption the scene, a.k.a. life-logging, for patients, travellers, and emergency responders in an egocentric narrative. Ego-captioning is indeed non-trivial since (1) Ego-images can be noisy due to motion and angles; (2) Describing a scene in a first-person narrative involves drastically different semantics; (3) Empirical implications have to be made on top of visual appearance because the cameraperson is often outside the field of view. We note we humans make good sense out of casual footage thanks to our contextual awareness in judging when and where the event unfolds, and whom the cameraperson is interacting with. This inspires the infusion of such “contexts” for situation-aware captioning. We create EgoCap which contains 2.1K ego-images, over 10K ego-captions, and 6.3K contextual labels, to close the gap of lacking ego-captioning datasets. We propose EgoFormer, a dual-encoder transformer-based network which fuses both contextual and visual features. The context encoder is pre-trained on ImageNet before fine tuning with context classification tasks. Similar to visual attention, we exploit stacked multi-head attention layers in the captioning decoder to reinforce attention to the context features. The EgoFormer has realized state-of-the-art performance on EgoCap achieving a CIDEr score of 125.52. The EgoCap dataset and EgoFormer are publicly available at https://github.com/zdai257/EgoCap-EgoFormer.

Real-time Arabic Video Captioning Using CNN and Transformer Networks Based on Parallel Implementation

Video captioning techniques have practical applications in fields like video surveillance and robotic vision, particularly in real-time scenarios. However, most of the current approaches still exhibit certain limitations when applied to live video, and research has predominantly focused on English language captioning. In this paper, we introduced a novel approach for live real-time Arabic video captioning using deep neural networks with a parallel architecture implementation. The proposed model primarily relied on the encoder-decoder architecture trained end-to-end on Arabic text. Video Swin Transformer and deep convolutional network are employed for video understanding, while the standard Transformer architecture is utilized for both video feature encoding and caption decoding. Results from experiments conducted on the translated MSVD and MSR-VTT datasets demonstrate that utilizing an end-to-end Arabic model yielded better performance than methods involving the translation of generated English captions to Arabic. Our approach demonstrates notable advancements over compared methods, yielding a CIDEr score of 78.3 and 36.3 for the MSVD and MSRVTT datasets, respectively. In the context of inference speed, our model achieved a latency of approximately 95 ms using an RTX 3090 GPU for a temporal video segment with 16 frames captured online from a camera device.

Style-Enhanced Transformer for Image Captioning in Construction Scenes.

Image captioning is important for improving the intelligence of construction projects and assisting managers in mastering construction site activities. However, there are few image-captioning models for construction scenes at present, and the existing methods do not perform well in complex construction scenes. According to the characteristics of construction scenes, we label a text description dataset based on the MOCS dataset and propose a style-enhanced Transformer for image captioning in construction scenes, simply called SETCAP. Specifically, we extract the grid features using the Swin Transformer. Then, to enhance the style information, we not only use the grid features as the initial detail semantic features but also extract style information by style encoder. In addition, in the decoder, we integrate the style information into the text features. The interaction between the image semantic information and the text features is carried out to generate content-appropriate sentences word by word. Finally, we add the sentence style loss into the total loss function to make the style of generated sentences closer to the training set. The experimental results show that the proposed method achieves encouraging results on both the MSCOCO and the MOCS datasets. In particular, SETCAP outperforms state-of-the-art methods by 4.2% CIDEr scores on the MOCS dataset and 3.9% CIDEr scores on the MSCOCO dataset, respectively.

Exploring EfficientNet Variants for Image Encoding in Auxiliary Signal Guided Knowledge Encoder-Decoder Framework

The growing number of medical images has led to radiologist burnout, which seriously impacts the radiologist's performance. To address the previously mentioned issue, an Auxiliary Signal Guided Knowledge (ASGK) multimodal encoder-decoder framework was designed to automatically generate the medical report based on the proposed medical graph and natural language decoder. It utilizes DenseNet-121 as the image encoder. With DenseNet-121 lack of computational and memory efficiency, this study aims to explore the potential of EfficientNetB0 to EfficientNetB4 as an ASGK image encoder substitute. The framework is trained with IU X-Ray dataset for 30 epochs, with Adam optimizer, a learning rate of 0.01 with 0.8 decay rate, binary cross entropy loss for the medical tags, and cross-entropy loss for the generated medical captions. During the framework training process with each image encoder, the parameter that achieves the highest CIDEr score on the validation set is considered the best image encoder parameter and will be used on the test set. On the test set, EfficientNetB3 as an ASGK image encoder has been shown to increase the CIDEr score to 0.35, a significant increase from the 0.28 CIDEr score obtained by the ASGK using DenseNet-121. This score is only a 1% decrease from the best validation score. It suggests that not only EfficientNetB3 increases the framework's performance, it is also less prone to overfitting. This study has demonstrated that EfficientNetB3 is a potential image encoder substitute for DenseNet-121 in the ASGK framework.

Efficient Image Annotation and Caption System Using Deep Convolutional Neural Networks

This study aims to develop an annotation and image annotation system using the Fashion MNIST dataset, which consists of 70,000 grayscale images of ten clothing categories. The system uses a long short-term memory (LSTM) network to generate captions and a convolutional neural network (CNN) to extract image features. Performance evaluation metrics such as Precision, Recall, F1 score, BLEU score, METEOR score, CIDEr score, and ROUGE-L score are used where the accuracy of each clothing category is calculated to evaluate the performance of the model across different categories. Visual analysis of the generated captions is performed to gain insight into the effectiveness of the model and potential areas for improvement. The results indicate the model's success in classifying clothing items, as evidenced by its high accuracy on the test set. The qualitative study reveals the model's ability to identify different types of clothing by providing relevant captions, where the feature representation layer (normalization) plays a crucial role in transforming the detected features. to a flattened row which is then passed to a fully connected layer to learn the relationships and make final decisions with the output layer using a softmax activation function to assign probabilities to each image class, with the class with the highest probability selected as the predicted image class.

External knowledge-assisted Transformer for image captioning

Internal relationship exploring based on the descriptive region features of image objects and grid features have contributed significantly to the development of image captioning, especially when combined with Transformer architecture. However, when conducting self-attention calculation, most of these methods only consider the relationship intra-objects and ignore the connection between entities and background. Besides, the way of exploring the relation information inside the image can also be extended. In this paper, we introduce a novel Mixed Knowledge Relation Transformer (MKRT) to explore the relationship between objects from both internal attribute relationship and external the object-verb-subject relationship. Furthermore we embed the important image background information into the relation module. In MKRT, the semantic relation obtained from the external knowledge is incorporated into the relation modeling in the novel Mixed Knowledge Relation Attention (MKRA). To validate the effectiveness of our model, we conduct extensive experiments on the most popular MSCOCO dataset, and achieve 134.5 CIDEr score on the offline test split and 133.5 CIDEr (c40) score on the official online testing server.