Image Caption Generation Research Articles

Review on Image Caption Generation

With the rapid development of Deep learning, AI along with Computer Vision and Natural Language processing Image caption has become an interesting and complex task. Image caption generation is the process of generating textual description of the given image and it is a challenging task because it consists of apprehension of objects. If the machine will be programmed to accurately describe an image or environment like human vision, it will be highly beneficial for robotic vision, business and many more. In order to generate an effective description of the image, the machine needs to detect, recognize objects as well as understand the scene type or location, object properties, their relationships and their interactions with each other. In this paper, we focus on advanced image captioning techniques such as CNN (Convolutional Neural Network)-LSTM(Long Short Term Memory) to generate meaningful captions. and the advantages and limitations of each method are discussed.

Automatic Image Caption Generation Based on Some Machine Learning Algorithms

This paper is dedicated to machine learning, the branches of machine learning, which include the methods for solving this issue, and the practical implementation of the solution to the automatic image description generation. Automatic image caption generation is one of the frequent goals of computer vision. Image description generation models must solve a larger number of complex problems to have this task successfully solved. The objects in the image must be detected and recognized, after which a logical and syntactically correct textual description is generated. For that reason, description generation is a complex problem. It is an extremely important challenge for machine learning algorithms because it represents an impersonation of a complicated human ability to encapsulate huge amounts of highlighted visual pieces of information in descriptive language. The results of the generated descriptions are compared depending on the used pretrained convolutional networks. The BLEU metrics are used to calculate the quality of the image description. Although the solution to the problem of image description automatic generation does provide us with good results, there is yet room for improvement since there are images that are not adequately described.

Learning cross-modality features for image caption generation

Learning cross-modality features for image caption generation

Variational joint self‐attention for image captioning

The image captioning task has attracted great attention from many researchers, and significant progress has been made in the past few years. Existing image captioning models, which mainly apply attention-based encoder-decoder architecture, achieve great developments image captioning. These attention-based models, however, are limited in the caption generation due to the potential errors resulting from the inaccurate detection of objects and incorrect attention to the objects. To alleviate the limitation, a Variational Joint Self-Attention model (VJSA) is proposed to learn a latent semantic alignment between the given image and its label description for guiding better image captioning. Unlike the existing image captioning models, VJSA first uses a self-attention module to encode the effective relationship information of intra-sequence and inter-sequences relationships. And then the variational neural inference module learns a distribution over the latent semantic alignment between the image and its corresponding description. In the decoding, the learned semantic alignment guides the decoder to generate the higher quality image caption. The results of the experiments reveal that the VJSA outperforms the compared models, and the performances of various metrics show that the proposed model is effective and feasible in image caption generation.

Diverse and styled image captioning using singular value decomposition‐based mixture of recurrent experts

AbstractWith significant advances in vision and natural language processing, the generation of image captions becomes a need. Mathews, Xie, and He extended a new model to generate styled captions by separating semantics and style. In continuation of their work, here, a new captioning model is developed, including an image encoder to extract the features, a mixture of recurrent networks to embed the set of extracted features to a group of words, and a sentence generator that combines the obtained words as a stylized sentence. This Mixture of Recurrent Experts (MoRE) system uses a new training algorithm that derives singular value decomposition from weighting matrices of Recurrent Neural Networks (RNNs) to increase the diversity of captions. Each decomposition step depends on a distinctive factor based on the number of RNNs in MoRE. The used sentence generator gives a stylized language corpus without paired images. Besides, the styled and diverse captions are extracted without training on a densely labeled or styled dataset. MoRE on the COCO dataset generated diverse and stylized image captions without the necessity of extra‐labeling and improved descriptions in terms of content accuracy.

A novel framework for automatic caption and audio generation

In recent times, with the advancements of Artificial Intelligence, multidomain problems of Computer vision (CV) and Natural language processing (NLP) such as Image caption generation, audio generation has piqued the attention of researchers all across the globe due to its application in medical, business and technology. Image caption generating entails automatically generating text according to the contents available in the image. In this paper we present a novel caption generation and audio generation framework. We use Deep Neural Networks like Convolutional Neural Network (CNN), Long short-term memory (LSTM) and transfer learning techniques to perform this task. The model has two stages: 1] Generate captions for any given image 2] Then gTTS (google text to speech) generator is used to generate audio for the generated captions. This framework is extremely beneficial to visually impaired people since it allows them to comprehend visuals. The Flickr8K dataset was used to train and test the model. A total of 6000 photos were utilised to train the model, with an additional 1000 images used for validation and testing.

CapNet: An Encoder-Decoder based Neural Network Model for Automatic Bangla Image Caption Generation

Automatic caption generation from images has be-come an active research topic in the field of Computer Vision (CV) and Natural Language Processing (NLP). Machine generated image caption plays a vital role for the visually impaired people by converting the caption to speech to have a better understanding of their surrounding. Though significant amount of research has been conducted for automatic caption generation in other languages, far too little effort has been devoted to Bangla image caption generation. In this paper, we propose an encoder-decoder based model which takes an image as input and generates the corresponding Bangla caption as output. The encoder network consists of a pretrained image feature extractor called ResNet-50, while the decoder network consists of Bidirectional LSTMs for caption generation. The model has been trained and evaluated using a Bangla image captioning dataset named BanglaLekhaIm-ageCaptions. The proposed model achieved a training accuracy of 91% and BLEU-1, BLEU-2, BLEU-3, BLEU-4 scores of 0.81, 0.67, 0.57, and 0.51 respectively. Moreover, a comparative study for different pretrained feature extractors such as VGG-16 and Xception is presented. Finally, the proposed model has been deployed on an embedded device for analysing the inference time and power consumption.

Image Caption Generation Method Based on Knowledge Graph Guidance and Self-Attention Mechanism

Image Caption Generation Method Based on Knowledge Graph Guidance and Self-Attention Mechanism

ATTENTION BASED IMAGE CAPTIONING USING DEEP LEARNING

The process of deriving written descriptions from a picture based on the items and behaviours portrayed is known as image caption generation. The resulting description will represent what is in the image and the interactions among the objects. However, like with any other image processing challenge, recreating this characteristic in an artificial system is a difficult feat, and so the adoption Deep Learning procedures can help to tackle the problem. The main goal of this project is to generate an image captioning model that can provide a concise and relatable description of the picture by incorporating an attention mechanism. Ideally, the model can be separated into two components – one is called Encoder and the other is a Decoder. The encoder used here is a Google InceptionV3 pre-trained model. And the decoder is a language-based model, Gated Recurrent Unit (GRU) to translate the characteristics and objects provided by the encoder into natural sentences. Furthermore, the model's effectiveness is aided by the attention mechanism. To conclude, the model is trained on a subset of the MS COCO dataset, which contains about 80,000 pictures with at least five descriptions each and evaluated based on the BLEU score. To reduce the loss, the model weights are tuned using the categorical cross-entropy loss function. The results of the model are promising and competitive.

Multilevel Attention Networks and Policy Reinforcement Learning for Image Caption Generation.

The analysis of large-scale multimodal data has become very popular recently. Image captioning, whose goal is to describe the content of image with natural language automatically, is an essential and challenging task in artificial intelligence. Commonly, most existing image caption methods utilize the mixture of Convolutional Neural Network and Recurrent Neural Network framework. These methods either pay attention to global representation at the image level or only focus on the specific concepts, such as regions and objects. To make the most of characteristics about a given image, in this study, we present a novel model named Multilevel Attention Networks and Policy Reinforcement Learning for image caption generation. Specifically, our model is composed of a multilevel attention network module and a policy reinforcement learning module. In the multilevel attention network, the object-attention network aims to capture global and local details about objects, whereas the region-attention network obtains global and local features about regions. After that, a policy reinforcement learning algorithm is adopted to overcome the exposure bias problem in the training phase and solve the loss-evaluation mismatching problem at the caption generation stage. With the attention network and policy algorithm, our model can automatically generate accurate and natural sentences for any particular image. We carry out extensive experiments on the MSCOCO and Flickr30k data sets, demonstrating that our model is superior to other competitive methods.

Image Caption Generator Using Deep Learning

Abstract: When humans see an image, their brain can easily tell what the image is about, but a computer cannot do it easily. Computer vision researchers worked on this a lot and they considered it impossible until now! With the advancement in Deep learning techniques, availability of huge datasets and computer power, we can build models that can generate captions for an image. Image Caption Generator is a popular research area of Deep Learning that deals with image understanding and a language description for that image. Generating well-formed sentences requires both syntactic and semantic understanding of the language. Being able to describe the content of an image using accurately formed sentences is a very challenging task, but it could also have a great impact, by helping visually impaired people better understand the content of images. The biggest challenge is most definitely being able to create a description that must capture not only the objects contained in an image, but also express how these objects relate to each other. This paper uses Flickr_8K dataset and Flickr8k_text folder that contains Flickr8k.token which is the main file of our dataset that contains image name and their respective caption separated by newline(“\n”). CNN is used for extracting features from the image. We will use the pre-trained model Xception. LSTM will use the information from CNN to help generate a description of the image. In our Flickr8k_text folder, we have Flickr_8k.trainImages.txt file that contains a list of 6000 images names that we will use for training. After CNN-LSTM model is defined we give an image file as parameter through command prompt for testing image caption generator and it generates the caption of an image and its accuracy is observed by calculating bleu score for generated and reference captions. Keywords: Image Caption Generator, Convolutional Neural Network, Long Short-Term Memory, Bleu score, Flickr_8K

Analysis of the Fuzziness of Image Caption Generation Models due to Data Augmentation Techniques

Automatic Image Caption Generation is one of the core problems in the field of Deep Learning. Data Augmentation is a technique which helps in increasing the amount of data at hand and this is done by augmenting the training data using various techniques like flipping, rotating, Zooming, Brightening, etc. In this work, we create an Image Captioning model and check its robustness on all the major types of Image Augmentation techniques. The results show the fuzziness of the model while working with the same image but a different augmentation technique and because of this, a different caption is produced every time a different data augmentation technique is employed. We also show the change in the performance of the model after applying these augmentation techniques. Flickr8k dataset is used for this study along with BLEU score as the evaluation metric for the image captioning model.

A Rapid Review of Image Captioning

Image captioning is an automatic process for generating text based on the content observed in an image. We do review, create framework, and build application model. We review image captioning into 4 categories based on input model, process model, output model, and lingual image caption. Input model is based on criteria caption, method, and dataset. Process model is based on type of learning, encoder-decoder, image extractor, and metric evaluation. Output model based on architecture, features extraction, feature aping, model, and number of caption. Lingual image caption based on language model with 2 groups: bilingual image caption and cross-language image caption. We also design framework with 3 framework model. Furthermore, we also build application with 3 application models. We also provide research opinions on trends and future research that can be developed with image caption generation. Image captioning can be further developed on computer vision versus human vision.

Automatic Image Caption Generation Using CNN, RNN and LSTM

Automatic Image Caption Generation Using CNN, RNN and LSTM

Two-Tier LSTM Model for Image Caption Generation

Two-Tier LSTM Model for Image Caption Generation

Image Captioning using CNN and LSTM

Abstract: In the model we proposed, we examine the deep neural networks-based image caption generation technique. We give image as input to the model, the technique give output in three different forms i.e., sentence in three different languages describing the image, mp3 audio file and an image file is also generated. In this model, we use the techniques of both computer vision and natural language processing. We are aiming to develop a model using the techniques of Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) to build a model to generate a Caption. Target image is compared with the training images, we have a large dataset containing the training images, this is done by convolutional neural network. This model generates a decent description utilizing the trained data. To extract features from images we need encoder, we use CNN as encoder. To decode the description of image generated we use LSTM. To evaluate the accuracy of generated caption we use BLEU metric algorithm. It grades the quality of content generated. Performance is calculated by the standard calculation matrices. Keywords: CNN, RNN, LSTM, BLEU score, encoder, decoder, captions, image description.

Detection and Recognition of Object for Image Captioning

As one of the most intelligent beings on the planet, we are equipped with the most powerful visual and language system as it is easy for us to extract visual information from a given image and transform it into proper linguistic description. Image Caption Generator deals with generating captions for a given image. The capturing mechanism involves a tiring task that collaborates both computer vision and image processing. The mechanism must detect and establish relationships between objects, people, and animals. The aim of this paper is to detect, recognize and generate worthwhile captions for a given image. We use transfer learning CNN on sentences, and extract image representation with Neural Networks.

Image Caption Generation and Comprehensive Comparison of Image Encoders

Image Caption Generation and Comprehensive Comparison of Image Encoders

Natural language guided object retrieval in images

The ability to understand the surrounding environment and being able to communicate with interacting humans are important functionalities for many automated systems where visual input (e.g., images, video) and natural language input (speech or text) have to be related to each other. Possible applications are automatic image caption generation, interactive surveillance systems, or human robot interaction. In this paper, we propose algorithms for automatic responses to natural language queries about an image. Our approach uses a predefined neural net for detection of bounding boxes and objects in images, spatial relations between bounding boxes are modeled with a neural net, the queries are analyzed with a syntactic parser, and algorithms to map natural language to properties in the images are introduced. The algorithms make use of semantic similarity and antonyms. We evaluate the performance of our approach with test users assessing the quality of our system’s generated answers.

An encoder-decoder based framework for hindi image caption generation

An encoder-decoder based framework for hindi image caption generation