Flickr30K Datasets Research Articles

Image Caption Generator for Sinhala Using Deep Learning

In this study, for the image caption generation in the Sinhala language, we have implemented a Recurrent Neural Network based model consisting of an InceptionV3 model as an image feature extraction model and a Long Short Term Memory network for the language model by referring to the literature. The different variations of Sinhala versions of the Flickr8K and MS COCO datasets have been constructed and used to train experimental models. Evaluation of the generated captions has been done using both automated and manual approaches. The model trained on the MS COCO dataset with Google translated Sinhala captions has achieved the highest BLEU score of 0.592 and the highest METEOR score of 0.281. After doing the manual caption analysis, it was observed that there could be generated captions which could provide a good idea to the reader while having lower BLEU and METEOR scores.

Adequate alignment and interaction for cross-modal retrieval

Cross-modal retrieval has attracted widespread attention in many cross-media similarity search applications, especially image-text retrieval in the fields of computer vision and natural language processing. Recently, visual and semantic embedding (VSE) learning has shown promising improvements on image-text retrieval tasks. Most existing VSE models employ two unrelated encoders to extract features, then use complex methods to contextualize and aggregate those features into holistic embeddings. Despite recent advances, existing approaches still suffer from two limitations: 1) without considering intermediate interaction and adequate alignment between different modalities, these models cannot guarantee the discriminative ability of representations; 2) existing feature aggregators are susceptible to certain noisy regions, which may lead to unreasonable pooling coefficients and affect the quality of the final aggregated features. To address these challenges, we propose a novel cross-modal retrieval model containing a well-designed alignment module and a novel multimodal fusion encoder, which aims to learn adequate alignment and interaction on aggregated features for effectively bridging the modality gap. Experiments on Microsoft COCO and Flickr30k datasets demonstrates the superiority of our model over the state-of-the-art methods.

Piksellerden Paragraflara: Inception v3 ve Dikkat Mekanizmalarını Kullanarak Gelişmiş Görüntüden Metin Üretimi Keşfetme

Processing visual data and converting it into text plays a crucial role in fields like information retrieval and data analysis in the digital world. At this juncture, the "image-to-text" transformation, which bridges the gap between visual and textual data, has garnered significant interest from researchers and industry experts. This article presents a study on generating text from images. The study aims to measure the contribution of adding an attention mechanism to the encoder-decoder-based Inception v3 deep learning architecture for image-to-text generation. In the model, the Inception v3 model is trained on the Flickr8k dataset to extract image features. The encoder-decoder structure with an attention mechanism is employed for next-word prediction, and the model is trained on the train images of the Flickr8k dataset for performance evaluation. Experimental results demonstrate the model's satisfactory ability to accurately perceive objects in images.

Global-local fusion based on adversarial sample generation for image-text matching

In the increasingly popular era of adversarial machine learning (AML), developing more robust and generalized algorithms has become a key research topic. Image-text matching as the foundation of tasks such as video Q&A and text-image generation also faces various attacks in AML. Current image-text matching based on the similarity of matching fragments only focuses on the local matching results, which does not establish a comprehensive cognition of content in text and image, resulting in mismatching of the abstract scene when facing complex attacks. Meanwhile, existing methods are not sensitive enough to identify the internal relationship between objects in different local areas, which also confuse matching. Therefore, aiming at the above problems, a global similarity matching module is proposed, which is dynamically fused with local similarity to measure the matching results flexibly and improve the understanding of abstract scenes. Furthermore, a global-local cognition fusion training mechanism based on relationship adversarial sample generation is proposedto enhance understanding of internal relationships between objects in different local area through adversarial sample generation. Global loss is introduced to train the overall model, and adjust the proportion of global-local loss in the training process to better identified the relationships between objects in different local areas, and avoided confusion and matching caused by the similarity of matching objects. Experimental results show that the proposed method is 7.4 % (rSum) better than the SOTA method on the Flickr30K dataset, and 4.0 % (rSum using the 1K test set) better on the MS-COCO dataset. The proposed global-local fusion (GLF) based on adversarial sample generation for image-text matching algorithm improves the accuracy and robustness of image-text matching performs well in facing some security challenges, promoting the development of visual and linguistic modal fusion.

RelNet-MAM: Relation Network with Multilevel Attention Mechanism for Image Captioning

Text present in an image contains rich semantic information which is crucial for the understanding of an image. For example, a signboard having the text “deep water” conveys the danger involved in the image. The current image captioning models do not effectively utilize this useful semantic information due to their limited representation capabilities of scene-text tokens. Our work presents a novel image captioning model called RelNet-MAM, which utilizes a multilevel attention mechanism and relation network. To improve the appearance feature representation, RelNet-MAM uses multilevel attention which consists of spatial attention, channel-wise attention, and semantic attention. For representing the scene-text token effectively, RelNet-MAM uses appearance, FastText, location, and PHOC features for each token. Further, the proposed RelNet-MAM uses the relation network to establish the relationships between the objects and scene-text tokens. Finally, the transformer model together with dynamic pointer networks is used as a decoder in the caption generation process. The proposed RelNet-MAM model outperforms the state-of-the-art models on TextCaps, Flickr30k, and MS COCO datasets. TextCaps requires models to read and reason about the texts in an image for caption generation. MSCOCO and Flickr30k contain diverse images; persons, animals, automobiles, indoor and outdoor scenes. Remarkably, the proposed RelNet-MAM model outperforms the current best model by 2.3% on B-4, 1.8% on METEOR, 2.2% on ROUGE-L, 2.0% on CIDEr-D and 3.0% on SPICE metric scores on TextCaps dataset.

A performance analysis of transformer-based deep learning models for Arabic image captioning

Image captioning has become a fundamental operation that allows the automatic generation of text descriptions of images. However, most existing work focused on performing the image captioning task in English, and only a few proposals exist that address the image captioning task in Arabic. This paper focuses on understanding the factors that affect the performance of machine learning models performing Arabic image captioning (AIC). In particular, we focus on transformer-based models for AIC and study the impact of various text-preprocessing methods: CAMeL Tools, ArabertPreprocessor, and Stanza. Our study shows that using CAMeL Tools to preprocess text labels improves the AIC performance by up to 34–92% in the BLEU-4 score. In addition, we study the impact of image recognition models. Our results show that ResNet152 is better than EfficientNet-B0 and can improve BLEU scores performance by 9–11%. Furthermore, we investigate the impact of different datasets on the overall AIC performance and build an extended version of the Arabic Flickr8k dataset. Using the extended version improves the BLEU-4 score of the AIC model by up to 148%. Finally, utilizing our results, we build a model that significantly outperforms the state-of-the-art proposals in AIC by up to 196–379% in the BLUE-4 score.

Attention Mechanism Model Combined with Adversarial Learning for E-commerce User Behavior Classification and Personality Recommendation

In traditional e-commerce websites, consumers’ evaluation of products will affect new customers’ decisions on whether to buy the products. Some fraudulent merchants manipulate consumers’ online comments for their interests, and multitudes of fake comments abuse consumers’ rights and interests and the development of traditional e-commerce. The purpose of the present work is to detect and identify fake comments through user behavior classification. A series of innovative research works are carried out around the user behavior recognition task from four aspects: extraction and description of low-level behavior features, spatial representation of high-level user behavior, design of behavior classification model, and user behavior detection in unsegmented text. A feature extraction model based on the super-complete independent component analysis algorithm and a behavior classification model via attention mechanism are proposed. Moreover, a feature source discriminator is designed, and adversarial learning is used to optimize discriminator loss and generator loss. Finally, an experiment is implemented to test the effects of attentional mechanism and adversarial learning on the text retrieval model and visualize the results. In this experiment, the text retrieval algorithm based on a stacked cross-attention mechanism and adversarial learning retrieves the Microsoft Common Objects in Context (MS-COCO) and Flickr30K data sets on mainstream transmedia. The experimental results demonstrate that the stacked cross-attention mechanism has an excellent matching ability of fine-grained hierarchical features; the average accuracy of the algorithm after improvement increases from 81.23% to 83.11%. Besides, the prediction accuracy coverage is above 95%, which can significantly improve the predicted effect of text characteristics and image features, thus enhancing the accuracy of text retrieval and classification. The research has a certain experimental reference value for the classification and discrimination of business users’ behavior.

Evaluating Performances of Attention-Based Merge Architecture Models for Image Captioning in Indian Languages

Image captioning is a growing topic of research in which numerous advancements have been made in the past few years. Deep learning methods have been used extensively for generating textual descriptions of image data. In addition, attention-based image captioning mechanisms have also been proposed, which give state-ofthe- art results in image captioning. However, many applications and analyses of these methodologies have not been made in the case of languages from the Indian subcontinent. This paper presents attention-based merge architecture models to achieve accurate captions of images in four Indian languages- Marathi, Kannada, Malayalam, and Tamil. The widely known Flickr8K dataset was used for this project. Pre-trained Convolutional Neural Network (CNN) models and language decoder attention models were implemented, which serve as the components of the mergearchitecture proposed here. Finally, the accuracy of the generated captions was compared against the gold captions using Bilingual Evaluation Understudy (BLEU) as an evaluation metric. It was observed that the merge architectures consisting of InceptionV3 give the best results for the languages we test on, the scores discussed in the paper. Highest BLEU-1 scores obtained for each language were: 0.4939 for Marathi, 0.4557 for Kannada, 0.5082 for Malayalam, and 0.5201 for Tamil. Our proposed architectures gave much higher scores than other architectures implemented for these languages.

Semantic-Enhanced Cross-Modal Fusion for Improved Unsupervised Image Captioning

Unsupervised image captioning often grapples with challenges such as image–text mismatches and modality gaps, resulting in suboptimal captions. This paper introduces a semantic-enhanced cross-modal fusion model (SCFM) to address these issues. The SCFM integrates three innovative components: a text semantic enhancement network (TSE-Net) for nuanced semantic representation; contrast learning for optimizing similarity measures between text and images; and enhanced visual selection decoding (EVSD) for precise captioning. Unlike existing methods that struggle with capturing accurate semantic relationships and flexibility across scenarios, the proposed model provides a robust solution for unbiased and diverse captioning. Through experimental evaluations on the MS COCO and Flickr30k datasets, SCFM demonstrates significant improvements over the benchmark model, enhancing the CIDEr and BLEU-4 metrics by 3.6% and 3.2%, respectively. Visualization analysis further reveals the model’s superiority in increasing variability between hidden features and its potential in cross-domain and stylized image captioning. The findings not only contribute to the advancement of image captioning techniques but also open avenues for future research. Further investigations will explore SCFM’s adaptability to other multimodal tasks and refine it for more intricate image–text relationships.

Optimization of urban and rural ecological spatial planning based on deep learning under the concept of sustainable development

At present, the speed of urbanization in China is constantly accelerating. At the same time, due to the severe situation of tight resource constraints, severe environmental pollution, and ecosystem degradation, vigorously promoting the construction of ecological civilization has become a key planning direction. However, traditional urban and rural ecological spatial planning is influenced by factors such as region, terrain, and spatial scale, which cannot adapt to the current spatial planning requirements. To achieve sustainable urban and rural ecological spatial planning, we propose a method that uses the optimized remote sensing images and convolutional neural networks to achieve spatial planning. In the analysis of the application effect of the usage method, the experimental results show that increasing the amount of data such as image size can improve the execution performance of the computer when the computer is not fully utilizing its resources and its computational volume fails to saturate the computational capacity. The parallel configuration designed in this experiment can accelerate the performance of the computer better, and the acceleration effect becomes more obvious as the difficulty of the algorithm increases. The Faster RCNN algorithm proposed in this experiment has the highest retrieval accuracy in the Flickr30K dataset and MS-COCO dataset compared with other algorithms. In Flickr30k data set, compared with other models in the table, the model used in this paper has the highest retrieval accuracy. The retrieval accuracy of R@1, R@5, R@10 increased by 23.1%, 8.1% and 5.3%, respectively. In MS-COCO data set, the retrieval accuracy increased by 19.2%, 13.1% and 8.3% respectively. The above results confirm that the combination of remote sensing images and convolutional neural network technology can perform simple ecological planning of a city's urban and rural areas, which proves that the method proposed in this experiment has practicality.

Mask-guided network for image captioning

Attention mechanisms have been widely adopted for image captioning because of their powerful performance. In this paper, we propose a Mask Captioning Network (MaC) consisting of an object layer and a background layer to capture the objects and scenes of an image to generate a sentence. To this end, we leverage the Mask RCNN to detect salient regions at the pixel level instead of a bounding box in the object layer. Meanwhile, in the background layer, a CNN model is used to encode the scene features. In addition, MaC is implemented in both LSTM-based and Transformer-based image captioning architectures. We introduce a mask-guided transformer encoder with additional features to enhance the model. Experimental results show that our model significantly outperforms (with a much richer sentence) baseline models and achieves comparable results with state-of-the-art methods on MSCOCO and Flickr30k datasets.

Maintain a Better Balance between Performance and Cost for Image Captioning by a Size-Adjustable Convolutional Module

Image captioning is a challenging AI problem that connects computer vision and natural language processing. Many deep learning (DL) models have been proposed in the literature for solving this problem. So far, the primary concern of image captioning has been focused on increasing the accuracy of generating human-style sentences for describing given images. As a result, state-of-the-art (SOTA) models are often too expensive to be implemented in computationally weak devices. In contrast, the primary concern of this paper is to maintain a balance between performance and cost. For this purpose, we propose using a DL model pre-trained for object detection to encode the given image so that features of various objects can be extracted simultaneously. We also propose adding a size-adjustable convolutional module (SACM) before decoding the features into sentences. The experimental results show that the model with the properly adjusted SACM could reach a BLEU-1 score of 82.3 and a BLEU-4 score of 43.9 on the Flickr 8K dataset, and a BLEU-1 score of 83.1 and a BLEU-4 score of 44.3 on the MS COCO dataset. With the SACM, the number of parameters is decreased to 108M, which is about 1/4 of the original YOLOv3-LSTM model with 430M parameters. Specifically, compared with mPLUG with 510M parameters, which is one of the SOTA methods, the proposed method can achieve almost the same BLEU-4 scores, but the number of parameters is 78% less than the mPLUG.

Image captioning using transformer-based double attention network

Image captioning generates a human-like description for a query image, which has attracted considerable attention recently. The most broadly utilized model for image description is an encoder–decoder structure, where the encoder extracts the visual information of the image, and the decoder generates textual descriptions of the image. Transformers have significantly enhanced the performance of image description models. However, a single attention structure in transformers cannot consider more complex relationships between key and query vectors. Furthermore, attention weights are assigned to entire candidate vectors based on the assumption that entire vectors are related. In this paper, a new double-attention framework is presented, which improves the encoder–decoder structure to consider image captioning problems. Hence, a local generator module and a global generator module are designed to predict textual descriptions collaboratively. The proposed approach improves Self-Attention (SA) from two aspects to enhance the performance of image description. First, a Masked Self-Attention module is presented to attend on the most relevant information. Second, to evade a single shallow attention distribution and make deeper internal relations, a Hybrid Weight Distribution (HWD) module is proposed, that develops SA to use the relations between key and query vectors efficiently. Experiments over the Flickr30k and MS-COCO datasets prove that the proposed approach achieves desirable performance on different evaluation measures compared to the state-of-the-art frameworks.

Dynamic Pruning of Regions for Image–Sentence Matching

Image–sentence matching is becoming increasingly essential in the integrated understanding of vision and language. Prior approaches apply a pre-trained detection model to extract region features and explore fine-grained relationships between image and sentence by aggregating the similarities of all region–word pairs. However, all images are represented by the same number of regions, regardless of their respective semantic complexity, which results in a large number of redundant regions interfering with semantic inference and bringing additional computational burden. To address the lack of flexibility in image representation and information redundancy, a novel method named Dynamic Pruning of Regions for Image–Sentence Matching (DPRM) is proposed to efficiently capture relationships between text and image. In particular, a dynamic region pruning module is presented to dynamically select the appropriate number of regions according to the semantic complexity of each image, thus pruning redundant regions and reducing superfluous computations. Moreover, an inter-modality refinement module is designed to refine the fine-grained relationships of region–word pairs by retaining meaningful interaction features and suppressing interference from redundant alignments, which learns the more accurate semantic correspondences. Extensive experiments on MSCOCO and Flickr30K datasets prove the superiority of DPRM compared with previous approaches.

Bi-LS-AttM: A Bidirectional LSTM and Attention Mechanism Model for Improving Image Captioning

The discipline of automatic image captioning represents an integration of two pivotal branches of artificial intelligence, namely computer vision (CV) and natural language processing (NLP). The principal functionality of this technology lies in transmuting the extracted visual features into semantic information of a higher order. The bidirectional long short-term memory (Bi-LSTM) has garnered wide acceptance in executing image captioning tasks. Of late, scholarly attention has been focused on modifying suitable models for innovative and precise subtitle captions, although tuning the parameters of the model does not invariably yield optimal outcomes. Given this, the current research proposes a model that effectively employs the bidirectional LSTM and attention mechanism (Bi-LS-AttM) for image captioning endeavors. This model exploits the contextual comprehension from both anterior and posterior aspects of the input data, synergistically with the attention mechanism, thereby augmenting the precision of visual language interpretation. The distinctiveness of this research is embodied in its incorporation of Bi-LSTM and the attention mechanism to engender sentences that are both structurally innovative and accurately reflective of the image content. To enhance temporal efficiency and accuracy, this study substitutes convolutional neural networks (CNNs) with fast region-based convolutional networks (Fast RCNNs). Additionally, it refines the process of generation and evaluation of common space, thus fostering improved efficiency. Our model was tested for its performance on Flickr30k and MSCOCO datasets (80 object categories). Comparative analyses of performance metrics reveal that our model, leveraging the Bi-LS-AttM, surpasses unidirectional and Bi-LSTM models. When applied to caption generation and image-sentence retrieval tasks, our model manifests time economies of approximately 36.5% and 26.3% vis-a-vis the Bi-LSTM model and the deep Bi-LSTM model, respectively.

Improved Arabic image captioning model using feature concatenation with pre-trained word embedding

Automatic captioning of images contributes to identifying features of multimedia content and helps in the detection of interesting patterns, trends, and occurrences. English image captioning has recently made incredible progress; however, Arabic image captioning is still lagging. In the field of machine learning, Arabic image-caption generation is generally a very difficult problem. This paper presents a more accurate model for Arabic image captioning by using transformer models in both the encoder and decoder phases as feature extractors from images in the encoder phase and a pre-trained word embedding model in the decoder phase. The models are demonstrated, and all of them are implemented, trained, and tested on Arabic Flickr8k datasets. For the image feature extraction subsystem, we compared using three different individual vision models (SWIN, XCIT, and ConvNexT) with concatenation to get among them the most expressive extracted feature vector of the image, and for the caption generation lingual subsystem, which is tested by four different pre-trained language embedding models: (ARABERT, ARAELECTRA, MARBERTv2, and CamelBERT), to select from them the most accurate pre-trained language embedding model. Our experiments showed that building an Arabic image captioning system that uses a concatenation of the three transformer-based models ConvNexT combined with SWIN and XCIT as an image feature extractor, combined with the CamelBERT language embedding model produces the best results among the other combinations, having scores of 0.5980 with BLEU-1 and with ConvNexT combined with SWIN the araelectra language embedding model having a score of 0.1664 with BLEU-4 which are higher than the previously reported values of 0.443 and 0.157.

Improving scene text image captioning using transformer-based multilevel attention

Many existing image captioning methods only focus on image objects and their relationships for generating image captions, ignoring the text present in an image. Scene text (ST) contains crucial information to understand an image and facilitating reasoning. The existing methods fail to establish strong correlations between optical character recognition (OCR) tokens, as they have limited OCR representation power. Further, these methods have not efficiently used the positional information of the text. In this work, we have proposed an ST-based image captioning model (Trans-MAtt) based on a multilevel attention mechanism and relation network. We have used relation networks to enhance the connections between ST tokens. We have employed a multi-level attention method, which comprises of spatial, semantic, and appearance attention modules that precisely define the image. To represent context-enriched ST tokens, we use a combination of appearance, location, FastText, and PHOC features. We predict the ST location in the image, which is further integrated with the generated word embeddings for final caption generation. Experiments on the TextCaps dataset demonstrate the effectiveness of the proposed Trans-MAtt model, where it outperforms the current best model by 3.4% on B-4, 2.9% on METEOR, 3.3% on ROUGE-L, 3.1% on CIDEr-D, and 4.1% on SPICE metric scores. Our experiments on the Flickr30k and MSCOCO datasets demonstrated the superiority of our proposed model over existing methods.

Pre-Trained CNN Architecture Analysis for Transformer-Based Indonesian Image Caption Generation Model

Classification and object recognition in image processing has significantly improved computer vision tasks. The method is often used for visual problems, especially in picture classification utilizing the Convolutional Neural Network (CNN). In the popular state-of-the-art (SOTA) task of generating a caption on an image, the implementation is often used for feature extraction of an image as an encoder. Instead of performing direct classification, these extracted features are sent from the encoder to the decoder section to generate the sequence. So, some CNN layers related to the classification task are not required. This study aims to determine which CNN pre-trained architecture or model performs best in extracting image features using a state-of-the-art Transformer model as its decoder. Unlike the original Transformer’s architecture, we implemented a vector-to-sequence way instead of sequence-to-sequence for the model. Indonesian Flickr8k and Flick30k datasets were used in this research. Evaluations were carried out using several pre-trained architectures, including ResNet18, ResNet34, ResNet50, ResNet101, VGG16, Efficientnet_b0, Efficientnet_b1, and Googlenet. The qualitative model inference results and quantitative evaluation scores were analyzed in this study. The test results show that the ResNet50 architecture can produce stable sequence generation with the highest accuracy value. With some experimentation, finetuning the encoder can significantly increase the model evaluation score. As for future work, further exploration with larger datasets like Flickr30k, MS COCO 14, MS COCO 17, and other image captioning datasets in Indonesian also implementing a new Transformers-based method can be used to get a better Indonesian automatic image captioning model.Â

Globally Guided Confidence Enhancement Network for Image-Text Matching

Image-text matching is a crucial aspect of multi-modal intelligence. The main challenge in this area is accurately measuring the relevance between the image and text, using evidence obtained through matching. Previous studies either concentrated on obtaining a well-represented global feature to measure similarity directly or on investigating complex matching patterns at a local level before aggregating them, with little attention paid to combining them. We propose a Globally Guided Confidence Enhancement Network that combines both approaches by obtaining a good global representation to guide fine-grained local interactions. In this process, content that better matches the text from a global perspective is enhanced and represented with confidence scores. Extensive experiments demonstrate that the approach we have employed achieves superior performance on Flickr30K and MSCOCO datasets.

A Multiview Text Imagination Network Based on Latent Alignment for Image-Text Matching

In image-text matching field, one of the keys to improving performance is to extract features with more semantic information. Existing works demonstrate that semantic enrichment through knowledge expansion can improve the performance. Most of them expand image features. However, the shortage of semantic information in text modality and the unilateral character of the view are often bottlenecks that limit the performance of image-text matching models. To solve the two problems, we aggregate knowledge from multiple views and propose a Word Imagination Graph (WIG). WIG can be used to expand textual semantic information by imagination based on input images. Then, utilizing WIG, we construct a novel Multi-View Text Imagination Network (MTIN). MTIN enables latent alignment of images and texts on tags which can assist matching on a semantic level. Results on Flickr30K and MS-COCO datasets can demonstrate the effectiveness of our method.