Image Description Generation Research Articles

A Systematic Survey of Automatic Image Description Generation Systems

Automatic image description generation is an active research area in Computer Vision and Natural Language Processing. Objects, their attributes, actions, and spatial relationships are identified in the image description generation system. Earlier, these systems used classical machine learning approaches. Later majority of these works follow deep learning strategies. The essential goal of these systems is to produce syntactically and semantically correct sentences. This review aims to synthesize the studies conducted from 2010 to 2023 to get a deeper view of various image description generation systems and their applications. The prominent contribution of this review is that it covers the different aspects of image captioning systems, such as the methods used, the various applied domains, evaluation measures, and the datasets used. A single synthesized study directs scholars regarding developing image captioning systems to date utilizing machine learning approaches. It also offers suggestions for researchers in this sector for the future. Image captioning is applied in many fields like natural images, medical images, remote sensing images, videos, etc. This review paper reviews the various taxonomies of image description generation systems. We also analyzed multiple methods used in the architecture of image captioning systems. The datasets and the evaluation metrics used in these systems are discussed. We studied the performance of the system under various circumstances.

Image description software generation based on data mining and computer vision

In order to solve the problems of low quality of image description, insufficient use of image features, and single level of recurrent neural network in image description generation, this paper proposes an image description generation method based on multi-scale features and computer vision. The algorithm uses the pre trained target detection network to extract the image features on different layers of the convolution neural network, inputs the image features into the multi attention structure layer by layer, connects the multi attention structure with the multi-layer recurrent neural network in turn, and constructs a multi-level image description generation network model. Adding residual connections to multilayer recurrent neural networks can improve the network performance, and can effectively avoid the network degradation caused by network deepening. The experimental results show that in the mscoco test set, the bleu-1 and cider scores of the proposed algorithm can reach 0.804 and 1.167 respectively, which is significantly better than the top-down image description generation algorithm based on a single attention structure. Conclusionthe image description generated by the proposed algorithm can show better image details.

DIC-Transformer: interpretation of plant disease classification results using image caption generation technology

Disease image classification systems play a crucial role in identifying disease categories in the field of agricultural diseases. However, current plant disease image classification methods can only predict the disease category and do not offer explanations for the characteristics of the predicted disease images. Due to the current situation, this paper employed image description generation technology to produce distinct descriptions for different plant disease categories. A two-stage model called DIC-Transformer, which encompasses three tasks (detection, interpretation, and classification), was proposed. In the first stage, Faster R-CNN was utilized to detect the diseased area and generate the feature vector of the diseased image, with the Swin Transformer as the backbone. In the second stage, the model utilized the Transformer to generate image captions. It then generated the image feature vector, which is weighted by text features, to improve the performance of image classification in the subsequent classification decoder. Additionally, a dataset containing text and visualizations for agricultural diseases (ADCG-18) was compiled. The dataset contains images of 18 diseases and descriptive information about their characteristics. Then, using the ADCG-18, the DIC-Transformer was compared to 11 existing classical caption generation methods and 10 image classification models. The evaluation indicators for captions include Bleu1–4, CiderD, and Rouge. The values of BLEU-1, CIDEr-D, and ROUGE were 0.756, 450.51, and 0.721. The results of DIC-Transformer were 0.01, 29.55, and 0.014 higher than those of the highest-performing comparison model, Fc. The classification evaluation metrics include accuracy, recall, and F1 score, with accuracy at 0.854, recall at 0.854, and F1 score at 0.853. The results of DIC-Transformer were 0.024, 0.078, and 0.075 higher than those of the highest-performing comparison model, MobileNetV2. The results indicate that the DIC-Transformer outperforms other comparison models in classification and caption generation.

Object Recognition to Content Based Image Retrieval: A Study of the Developments and Applications of Computer Vision

Natural Language Processing (NLP) and Computer Vision (CV) are interconnected fields within the domain of Artificial Intelligence (AI). CV is tasked with the process of engaging with computer systems to effectively interpret and recognize visual data, while NLP is responsible for comprehending and processing the human voice. The two fields have practical applicability in various tasks such as image description generation, object recognition, and question-based answering after a visual input. Deep learning algorithms such as word input are typically employed in enhancing the performance of Content-Based Image Processing (CBIR) techniques. Generally, NLP and CV play a vital role in enhancing computer comprehension and engagements with both visual and written information. This paper seeks to review various major elements of computer vision, such as CBIR, visual effects, image documentation, video documentation, visual learning, and inquiry to explore various databases, techniques, and methods employed in this field. The authors focus on the challenges and progress in each area and offer new strategies for improving the performance of CV systems.

Medical Image Super-Resolution Based on Semantic Perception Transfer Learning.

Medical images are an important basis for doctors to diagnose diseases, but some medical images have low resolution due to hardware technology and cost constraints. Super-resolution technology can reconstruct low-resolution medical images into high-resolution images and enhance the quality of low-resolution images, thus assisting doctors in diagnosing diseases. However, traditional super-resolution methods mainly learn the mapping relationships among modal pixels from low resolution to high resolution, lacking the learning of high-level semantic features, resulting in a lack of understanding and utilization of semantic information, such as reconstructed objects, object attributes, and spatial relationships between two objects. In this paper, we propose a medical image super-resolution method based on semantic perception transfer learning. First, we propose a novel semantic perception super-resolution method that empowers super-resolution models to perceive high-level semantics by transferring features of the image description generation network in natural language processing. Second, we construct a semantic feature extraction network and an image description generation network and comprehensively utilized image and text modal data to learn transferable, high-level semantic features. Third, we train an end-to-end, semantic perception super-resolution model by fusing dynamic perceptual convolution, a semantic extraction network, and distillation polarization self-attention. Experiments show that semantic perception transfer learning can effectively improve the quality of super-resolution reconstruction.

Visual Description Augmented Integration Network for Multimodal Entity and Relation Extraction

Multimodal Named Entity Recognition (MNER) and multimodal Relationship Extraction (MRE) play an important role in processing multimodal data and understanding entity relationships across textual and visual domains. However, irrelevant image information may introduce noise that misleads the recognition of information. Additionally, visual and semantic features originate from different modalities, and modal disparity hinders semantic alignment. Therefore, this paper proposes the Visual Description Augmentation Integration Network (VDAIN), which introduces an image description generation technique that allows semantic features generated from image descriptions to be presented in the same modality as the semantic features of textual information. This not only reduces the modal gap but also captures more accurately the high-level semantic information and underlying visual structure in the images. To filter out the modal noise, we use VDAIN to adaptively fuse visual features, semantic features of image descriptions, and textual information, thus eliminating irrelevant modal noise. The F1 score of the proposed model in this paper reaches 75.8% and 87.78% for the MNER task and 82.54% for the MRE task on the three public data sets, respectively, which are significantly better than the baseline model. The experimental results demonstrate the effectiveness of the proposed method in solving the modal noise and modal gap problems.

A Novel Bi-LSTM Based Automatic Image Description Generation

A Novel Bi-LSTM Based Automatic Image Description Generation

Automatic Image Caption Generation Based on Some Machine Learning Algorithms

This paper is dedicated to machine learning, the branches of machine learning, which include the methods for solving this issue, and the practical implementation of the solution to the automatic image description generation. Automatic image caption generation is one of the frequent goals of computer vision. Image description generation models must solve a larger number of complex problems to have this task successfully solved. The objects in the image must be detected and recognized, after which a logical and syntactically correct textual description is generated. For that reason, description generation is a complex problem. It is an extremely important challenge for machine learning algorithms because it represents an impersonation of a complicated human ability to encapsulate huge amounts of highlighted visual pieces of information in descriptive language. The results of the generated descriptions are compared depending on the used pretrained convolutional networks. The BLEU metrics are used to calculate the quality of the image description. Although the solution to the problem of image description automatic generation does provide us with good results, there is yet room for improvement since there are images that are not adequately described.

A Unified Visual Saliency Model for Automatic Image Description Generation for General and Medical Images

A Unified Visual Saliency Model for Automatic Image Description Generation for General and Medical Images

Research on the Application of Generative Adversarial Networks in the Generation of Stock Market Forecast Trend Images

Investors make capital investment by buying stocks and expect to get a certain income from the stock market. When buying stocks, they need to draw up investment plans based on various information such as stock market historical transaction data and related news data of listed companies and collect and analyze these data. The data are relatively cumbersome and require a lot of time and effort. If you only rely on subjective analysis, the reference factors are often not comprehensive enough. At the same time, Internet social media, such as the speech in stock forums, also affect the judgment and behavior of investors, and investor sentiment will have a positive or negative effect on the stock market. This has an impact on the trend of stock prices. Therefore, this article proposes a stock market prediction model that uses data preprocessing technology based on past stock market transaction data to establish a stock market prediction model, and secondly, an image description generation model based on a generative confrontation network is designed. The model includes a generator and a discriminator. A time-varying preattention mechanism is proposed in the generator. This mechanism allows each image feature to pay attention to the image features of other stock markets to predict stock market trends so that the decoder can better understand the relational information in the image. The discriminator is based on the recurrent neural network and considers the degree of matching between the input sentence and the 4 reference sentences and the image features. Experiments show that the accuracy of the model is higher than that of the stock pretrend forecast model based on historical data, which proves the effectiveness of the data used in this paper in the stock price trend forecast.

Multi Spatial Relation Detection in Images

ABSTRACT Detecting spatial relationships between objects depicted in an image is an important sub-task in vision and language understanding. Its practical use lies in visual discourse when referring to objects by their relationship in context of others and finds application in higher level tasks such as visual question answering and image description generation. Presumably, the selection of spatial prepositions grounded in an image is straightforward. However, in general, human beings either do not always agree or are not consistent when choosing spatial prepositions. This could be due to various reasons, such as near synonyms, overlapping terms and different frames of reference. For these reasons, the automatic detection of spatial relations is a non-trivial multi-label problem. This paper addresses the automatic multi-selection of prepositions. The study is based on the development of a number of machine learning models, namely Nearest Neighbor (NN), k-Means Clustering (kM-C), Agglomerative Hierarchical Clustering (A-HC) and Multi-label Neural Network (ML-NN). The model performances are compared quantitatively using multi-label metrics as well as human evaluations that are independent of the ground truth labels. Additionally, the classification results are used as a basis to carry out an error and qualitative analysis that sheds light on the relative merits of how each model deals with synonymous and overlapping relations, and groups common errors to inform future directions. Furthermore, to gain insight into the merits of multi-label models, a single-label Random Forest (RF) classifier is developed and its results are included in the analysis. Of all multi-label models, the ML-NN exhibits the best overall performance when evaluated on both the dataset ground truth and the independent human evaluations. It, however, suffers from under-generating prepositions, while the rest of the models often generate more prepositions at the expense of precision. The clustering-based methods are also not quite consistent, although they do better than the other models in less frequent spatial configurations that other models struggle with. The results from the single-label RF classifier highlight the usefulness of having a multi-label model. Finally, the error analysis indicates that the majority of errors is due to lack of features that give cues on object position and orientation (object pose), the fixed frame of reference, and the failure to resolve depth in perspective view.

Research on HR management based on RNN expectation theory

RNN convolutional neural network is an important part of deep learning algorithm. Compared with fully connected neural network, the biggest difference is that the hidden units are not independent of each other. Not only the hidden layer neurons are irrelated to each other, but also the state of the current hidden layer cells is affected by the historical input data. This feature enables it to extract the temporal relationship of temporal data structure. One of the current outputs of one sequence is also related to the previous output. The specific expression is that the network will memorize the information before and apply it to the calculation of current output, the nodes between hidden layers are no longer connected but connected. Moreover, the input of hidden layer includes not only the output of input layer but also the output of hidden layer at the last time. RNN neural network is widely used in natural language processing (NLP): machine translation, speech recognition, image description generation, text similarity calculation. The DAFDC-RNN model in this paper is verified by the financial and real estate industry data from 2019 to 2020, which shows that the model has excellent stability and sensitivity. Compared with CNN model, it increases by 34.3% and 24.9%. The advantages of data processing as follow: it can process a large number of time series data, the processing capacity of data set is large, it can process 1000 + data, and it has more data base for time series prediction. The disadvantage as follow: Compared with CNN model, the working time of CNN model is increased by 48%, but the model has strong stability and adaptability. Taking the data from 2019 to 2020 as the input of DAFDC-RNN time series forecasting model, the salary forecast of real estate and financial industry in 2021 is obtained. Based on this data, the industry expectation of Finance and real estate industry under human resource management under expectation theory is reduced.

Image Description Generation based on Deep Learning

The paper firstly explains the importance and research significance of image description generation and makes a brief research summary on the generation of various algorithms for image description; it further introduces the principle and the algorithm process of the LSTM deep learning algorithm in details, specifically describes establishing steps and the model parameter selection process of the LSTM deep learning algorithm of the Microsoft COCO dataset, implements the deep learning models by employing Python, and finally applies the trained LSTM deep learning model to image description generation by using the Microsoft COCO dataset as an instance.

Remote sensing image description based on word embedding and end-to-end deep learning

This study proposes an end-to-end image description generation model based on word embedding technology to realise the classification and identification of Populus euphratica and Tamarix in complex remote sensing images by providing descriptions in precise and concise natural sentences. First, category ambiguity over large-scale regions in remote sensing images is addressed by introducing the co-occurrence matrix and global vectors for word representation to generate the word vector features of the object to be identified. Second, a new multi-level end-to-end model is employed to further describe the content of remote sensing images and to better advance the description tasks for P. euphratica and Tamarix in remote sensing images. Experimental results reveal that the natural language sentences generated using this method can better describe P. euphratica and Tamarix in remote sensing images compared with conventional deep learning methods.

Chinese Image Captioning via Fuzzy Attention-based DenseNet-BiLSTM

Chinese image description generation tasks usually have some challenges, such as single-feature extraction, lack of global information, and lack of detailed description of the image content. To address these limitations, we propose a fuzzy attention-based DenseNet-BiLSTM Chinese image captioning method in this article. In the proposed method, we first improve the densely connected network to extract features of the image at different scales and to enhance the model’s ability to capture the weak features. At the same time, a bidirectional LSTM is used as the decoder to enhance the use of context information. The introduction of an improved fuzzy attention mechanism effectively improves the problem of correspondence between image features and contextual information. We conduct experiments on the AI Challenger dataset to evaluate the performance of the model. The results show that compared with other models, our proposed model achieves higher scores in objective quantitative evaluation indicators, including BLEU , BLEU , METEOR, ROUGEl, and CIDEr. The generated description sentence can accurately express the image content.

Automatically Generating Natural Language Descriptions of Images by a Deep Hierarchical Framework.

Automatically generating an accurate and meaningful description of an image is very challenging. However, the recent scheme of generating an image caption by maximizing the likelihood of target sentences lacks the capacity of recognizing the human-object interaction (HOI) and semantic relationship between HOIs and scenes, which are the essential parts of an image caption. This article proposes a novel two-phase framework to generate an image caption by addressing the above challenges: 1) a hybrid deep learning and 2) an image description generation. In the hybrid deep-learning phase, a novel factored three-way interaction machine was proposed to learn the relational features of the human-object pairs hierarchically. In this way, the image recognition problem is transformed into a latent structured labeling task. In the image description generation phase, a lexicalized probabilistic context-free tree growing scheme is innovatively integrated with a description generator to transform the descriptions generation task into a syntactic-tree generation process. Extensively comparing state-of-the-art image captioning methods on benchmark datasets, we demonstrated that our proposed framework outperformed the existing captioning methods in different ways, such as significantly improving the performance of the HOI and relationships between HOIs and scenes (RHIS) predictions, and quality of generated image captions in a semantically and structurally coherent manner.

Chinese description generation of dual attention images based on multi-modal fusion

In view of the lack of attention to image features in most of the current research on image Chinese description generation, the low quality of image details, and the low accuracy of generated language description, a multi-modal fusion dual attention image description generation model is proposed. The model uses the pre-trained VGG19 network to extract image features, LSTM extracts key information features of text, and combines image features and text features through attention stitching to obtain the attention weight of each spatial position of the image, and then input it into the LSTM language model Generate descriptive sentences. In the AI Challenger 2017 test set, the BLEU-1 and CIDEr scores of this algorithm can reach 71.2 and 106.1 respectively, which are better than baseline models based on a single attention structure, such as NIC, and perform manual observation and comparison of the effects of several models. The model can generate natural language descriptions that can better represent the details of the picture.

基于关键词指导的图像中文描述生成

基于关键词指导的图像中文描述生成

Application of Dual Attention Mechanism in Chinese Image Captioning

Objective: The Chinese description of images combines the two directions of computer vision and natural language processing. It is a typical representative of multi-mode and cross-domain problems with artificial intelligence algorithms. The image Chinese description model needs to output a Chinese description for each given test picture, describe the sentence requirements to conform to the natural language habits, and point out the important information in the image, covering the main characters, scenes, actions and other content. Since the current open source datasets are mostly in English, the research on the direction of image description is mainly in English. Chinese descriptions usually have greater flexibility in syntax and lexicalization, and the challenges of algorithm implementation are also large. Therefore, only a few people have studied image descriptions, especially Chinese descriptions. Methods: This study attempts to derive a model of image description generation from the Flickr8k-cn and Flickr30k-cn datasets. At each time period of the description, the model can decide whether to rely more on images or text information. The model captures more important information from the image to improve the richness and accuracy of the Chinese description of the image. The image description data set of this study is mainly composed of Chinese description sentences. The method consists of an encoder and a decoder. The encoder is based on a convolutional neural network. The decoder is based on a long-short memory network and is composed of a multi-modal summary generation network. Results: Experiments on Flickr8k-cn and Flickr30k-cn Chinese datasets show that the proposed method is superior to the existing Chinese abstract generation model. Conclusion: The method proposed in this paper is effective, and the performance has been greatly improved on the basis of the benchmark model. Compared with the existing Chinese abstract generation model, its performance is also superior. In the next step, more visual prior information will be incorporated into the model, such as the action category, the relationship between the object and the object, etc., to further improve the quality of the description sentence, and achieve the effect of “seeing the picture writing”.

Dense Model for Automatic Image Description Generation with Game Theoretic Optimization

Due to the rapid growth of deep learning technologies, automatic image description generation is an interesting problem in computer vision and natural language generation. It helps to improve access to photo collections on social media and gives guidance for visually impaired people. Currently, deep neural networks play a vital role in computer vision and natural language processing tasks. The main objective of the work is to generate the grammatically correct description of the image using the semantics of the trained captions. An encoder-decoder framework using the deep neural system is used to implement an image description generation task. The encoder is an image parsing module, and the decoder is a surface realization module. The framework uses Densely connected convolutional neural networks (Densenet) for image encoding and Bidirectional Long Short Term Memory (BLSTM) for language modeling, and the outputs are given to bidirectional LSTM in the caption generator, which is trained to optimize the log-likelihood of the target description of the image. Most of the existing image captioning works use RNN and LSTM for language modeling. RNNs are computationally expensive with limited memory. LSTM checks the inputs in one direction. BLSTM is used in practice, which avoids the problem of RNN and LSTM. In this work, the selection of the best combination of words in caption generation is made using beam search and game theoretic search. The results show the game theoretic search outperforms beam search. The model was evaluated with the standard benchmark dataset Flickr8k. The Bilingual Evaluation Understudy (BLEU) score is taken as the evaluation measure of the system. A new evaluation measure called GCorrectwas used to check the grammatical correctness of the description. The performance of the proposed model achieves greater improvements over previous methods on the Flickr8k dataset. The proposed model produces grammatically correct sentences for images with a GCorrect of 0.040625 and a BLEU score of 69.96%