A Novel Hybrid Deep Learning Framework for Image Captioning Using Combined Semantic and Object-Level Features

Image captioning in traffic scenes presents several challenges, including imprecise caption generation, lack of personalization, and an unwieldy number of model parameters. We propose a new image captioning model for traffic scenes to address these issues. The model incorporates an adapter-based fine-tuned feature extraction part to enhance personalization and a caption generation module using global weighted attention pooling to reduce model parameters and improve accuracy. The proposed model consists of four main stages. In the first stage, the Image-Encoder extracts the global features of the input image and divides it into nine sub-regions, encoding each sub-region separately. In the second stage, the Text-Encoder encodes the text dataset to obtain text features. It then calculates the similarity between the image sub-region features and encoded text features, selecting the text features with the highest similarity. Subsequently, the pre-trained Faster RCNN model extracts local image features. The model then splices together the text features, global image features, and local image features to fuse the multimodal information. In the final stage, the extracted features are fed into the Captioning model, which effectively fuses the different features using a novel global weighted attention pooling layer. The Captioning model then generates natural language image captions. The proposed model is evaluated on the MS-COCO dataset, Flickr 30K dataset, and BUUISE-Image dataset, using mainstream evaluation metrics. Experiments demonstrate significant improvements across all evaluation metrics on the public datasets and strong performance on the BUUISE-Image traffic scene dataset.

Automatic description of image has attracted many researchers in the field of computer vision for captioning the image in artificial intelligence which connects with Natural Language Processing. Exact generation of captions to image is necessary but it lacks due to Gradient Diminishing problem, LSTM can overcome this problem by fusing local and global characteristics of image and text that generates sequenced word prediction for accurate image captioning. We consider Flickr 8k data-set which consists of text as descriptions of images. The use of GLoVe embedding helps for the word representation to consider the global and local features of images which finds distance with Euclidean to understand the relationship between words in vector space. Inception V3 architecture which is pretrained on ImageNet used to extract image features of different objects in scenes. We propose Linear Sub-Structure that helps to generate sequenced order of words for captioning by understanding relationship between words. For extracting image features considers co-variance shift which mainly concentrates on moving parts of the image to generate accurate description of the image to maintain a semantic visual grammar relationship between the predicted text for image as the caption, the proposed model evaluated with the help of BLEU score which achieves state of art model in our work while compared with others that has greater than 81% of accuracy.

Vision-based robot localization in outdoor environments is difficult because of changing illumination conditions. Another problem is the rough and cluttered environment which makes it hard to use visual features that are not rotation invariant. A popular method that is rotation invariant and relatively robust to changing illumination is the Scale Invariant Feature Transform (SIFT). However, due to the computationally intensive feature extraction and image matching, localization using SIFT is slow. On the other hand, techniques which use global image features are in general less robust and exact than SIFT, but are often much faster due to fast image matching. In this paper, we present a hybrid localization approach that switches between local and global image features. For most images, the hybrid approach uses fast global features. Only in difficult situations, e.g. containing strong illumination changes, the hybrid approach switches to local features. To decide which features to use for an image, we analyze the particle cloud of the particle filter that we use for position estimation. Experiments on outdoor images taken under varying illumination conditions show that the position estimates of the hybrid approach are about as exact as the estimates of SIFT alone. However, the average localization time using the hybrid approach is more than 3.5 times faster than using SIFT.

Faster-RCNN is commonly used to extract image features in image-to-text generative models since the development of deep learning, although the extraction procedure is time-consuming. Existing approaches extract fixed-size grid features and then use language models to generate image captions, but they only focus on the grid spatial location features without considering grid feature interaction and image global features. An image caption method based on graph attention network with global context is proposed to generate higher-quality image captions. By building a grid feature interaction graph, a multi-layer convolutional neural network is utilized for visual encoding, and the grid features and entire image features of a given image are retrieved. Then, using the graph attention network, which includes a global node and many local nodes, the feature extraction problem is changed into a node classification problem, and the global and local features can be completely utilized after updating and optimization. Finally, the Transformer-based decoding module makes use of the enhanced visual features to provide image captions. The Microsoft COCO dataset is used for experiments evaluation. The experimental results demonstrate that the image caption method based on graph attention network with global context successfully captures the global and local features of the image and achieves 133.1% CIDEr, significantly improving quality of image caption.

This study investigates the effectiveness of an image captioning model utilizing VGG16 and LSTM architectures on the Flickr8K dataset. Through meticulous experimentation and evaluation, valuable insights into the model's capabilities and limitations in generating descriptive captions for images were gained. The findings contribute to the broader understanding of image captioning techniques and offer guidance for future advancements in the field. The exploration of VGG16 and LSTM architecture involved data preprocessing, model training, and evaluation. The Flickr8K dataset, comprising 8,000 images paired with textual descriptions, served as the foundation. Data preprocessing, feature extraction using VGG16, and LSTM training were conducted. Optimization of model parameters and hyperparameters was performed to achieve optimal performance. Evaluation metrics including BLEU score, Semantic Similarity score, and ROUGE scores were utilized. While moderate overlap with reference captions was observed according to the BLEU score, the model demonstrated a high degree of semantic similarity. However, challenges in maintaining coherence and capturing higher-order linguistic structures were revealed by the analysis of ROUGE scores. Implications of this research extend to domains such as computer vision, natural language processing, and human-computer interaction. By bridging the semantic gap between visual content and textual descriptions, image captioning models can enhance accessibility, improve image understanding, and facilitate human-machine communication. Despite promising performance in capturing semantic content, opportunities for improvement exist, including refining model architecture, integrating attention mechanisms, and leveraging larger datasets. Continued innovation in image captioning promises advanced systems with widespread applications across industries and disciplines.

The issue of generating a natural language explanation of images to define their visual content has garnered significant attention in computer vision (CV) and natural language processing (NLP). It is driven by applications such as image virtual assistants, indexing and retrieval, image perception, and assistance for visually challenged people. While this kind of person utilizes other senses, such as hearing and touch, for identifying events and objects, their quality of life is reduced to a typical level. Automated Image captioning generates captions that will be spoken aloud to individuals with disabilities, thereby recognizing objects and events happening nearby them. With the aid of image captioning techniques and artificial intelligence (AI) speech recognition methods, visually impaired individuals can quickly understand the content of an image, as these methods can automatically generate text captions that accurately describe the image’s content. Therefore, this study presents a novel Fusion of Deep Transfer Learning Models and the Gannet Optimisation Algorithm for an Advanced Image Captioning System for Visual Disabilities (FDTLGO-AICSVD) model. The aim is to present a robust and efficient image captioning framework specifically designed to assist visually impaired persons through precise and descriptive image-to-text conversion. Initially, the FDTLGO-AICSVD approach comprises two distinct types of image preprocessing: noise removal and contrast enhancement, aimed at improving the clarity of visual features. Text preprocessing involves distinct steps to standardize and prepare the textual data for analysis. Furthermore, DenseNet121, VGG19, and MobileNetV2 models are utilized for extracting features from image data, whereas Term Frequency Inverse Document Frequency (TF-IDF) is applied for extracting features from text data. To achieve optimal performance, the Gannet optimization algorithm (GOA) model is employed for hyperparameter tuning, enabling the method to generate precise and context-aware captions. A wide range of experimentation of the FDTLGO-AICSVD method is performed under the Flickr8k and Flickr30k datasets. The comparison study of the FDTLGO-AICSVD method portrayed a superior BLEU-4 score of 45.11% over the Flickr8K dataset and 58.91% over the Flickr30K dataset, along with a significantly higher CIDEr score of 63.17 on Flickr8K and 69.81 on Flickr30K, demonstrating the enhanced descriptive accuracy and language generation capability of the model across both datasets.

3G structure for image caption generation

Image captioning involves generating a sentence that describes an image. More recently, it has been driven by encoderdecoder approaches where the encoder such as convolutional neural network (CNN) can extract the visual features of an image. The extracted visual features are passed to a decoder such as a long short-term memory (LSTM) network in order to generate a sentence that describes the image. One major challenge with this approach is to precisely include the scene of an image in the generated sentences. To resolve this challenge, visual scene features have been used with unidirectional LSTM decoders. However, for long sentences, this limits the precision of the generated text. This research proposes a novel approach to generate sentences using visual scene information with a bidirectional LSTM decoder. The encoder is based on Inception v3 to extract the object features and Places365 to extract the scene features. The decoder uses a bidirectional LSTM to generate a sentence. The encoder-decoder model is trained using the Flickr8k dataset. Results show improved performance for generating longer sentences with a 9% increase in BLEU-3 and a 12% increase in BLEU-4 scores compared to compared to other encoder-decoder methods that are limited to only using global image features. Visually impaired people that use screen readers would benefit from this research as they would get an enhanced description of an image that includes the background scene thereby creating a wholesome picture in the mind of the reader.

In the proposed chapter, a novel, effective, and efficient approach to face recognition is presented. It is a fusion of both global and local features of images, which significantly achieves higher recognition. Initially, the global features of images are determined using polar cosine transforms (PCTs), which exhibit very less computation complexity as compared to other global feature extractors. For local features, the rotation invariant local ternary patterns are used rather than using the existing ones, which help improving the recognition rate and are in alignment with the rotation invariant property of PCTs. The fusion of both acquired global and local features is performed by mapping their features into a common domain. Finally, the proposed hybrid approach provides a robust feature set for face recognition. The experiments are performed on benchmark face databases, representing various expressions of facial images. The results of extensive set of experiments reveal the supremacy of the proposed method over other approaches in terms of efficiency and recognition results.

Human-computer interaction for virtual-real fusion

Transformer-based image captioning models have recently achieved remarkable performance by using new fully attentive paradigms. However, existing models generally follow the conventional language model of predicting the next word conditioned on the visual features and partially generated words. They treat the predictions of visual and nonvisual words equally and usually tend to produce generic captions. To address these issues, we propose a novel part-of-speech-guided transformer (PoS-Transformer) framework for image captioning. Specifically, a self-attention part-of-speech prediction network is first presented to model the part-of-speech tag sequences for the corresponding image captions. Then, different attention mechanisms are constructed for the decoder to guide the caption generation by using the part-of-speech information. Benefiting from the part-of-speech guiding mechanisms, the proposed framework not only adaptively adjusts the weights between visual features and language signals for the word prediction, but also facilitates the generation of more fine-grained and grounded captions. Finally, a multitask learning is introduced to train the whole PoS-Transformer network in an end-to-end manner. Our model was trained and tested on the MSCOCO and Flickr30k datasets with the experimental evaluation standard CIDEr scores of 1.299 and 0.612, respectively. The qualitative experimental results indicated that the captions generated by our method conformed to the grammatical rules better.

During acquisition, storage, and transmission, the quality of digital videos degrades significantly. Low-quality videos lead to the failure of many computer vision applications, such as object tracking or detection, intelligent surveillance, etc. Over the years, many different features have been developed to resolve the problem of no-reference video quality assessment (NR-VQA). In this paper, we propose a novel NR-VQA algorithm that integrates the fusion of temporal statistics of local and global image features with an ensemble learning framework in a single architecture. Namely, the temporal statistics of global features reflect all parts of the video frames, while the temporal statistics of local features reflect the details. Specifically, we apply a broad spectrum of statistics of local and global features to characterize the variety of possible video distortions. In order to study the effectiveness of the method introduced in this paper, we conducted experiments on two large benchmark databases, i.e., KoNViD-1k and LIVE VQC, which contain authentic distortions, and we compared it to 14 other well-known NR-VQA algorithms. The experimental results show that the proposed method is able to achieve greatly improved results on the considered benchmark datasets. Namely, the proposed method exhibits significant progress in performance over other recent NR-VQA approaches.

Image captioning is used to describe an image based on the features and actions that are present in that image. The existing image captioning primarily uses an encoding and decoding structure, with the encoder which extracts image features using CNN model as well as the decoder uses LSTM model. The current encoding and decoding scheme makes extensive use of the attention mechanism. Gradient explosion is a problem with the current image caption models, which are based on recurrent and convolutional neural networks, and therefore are not particularly good at extracting valuable information from images. To overcome this problem, the YOLOv5 and Bidirectional LSTM model is proposed. YOLOv5 is used to identify the objects in a given image and a bidirectional LSTM (Bi-LSTM) layer is used to extract the features of the given image. This proposed algorithm gives the optimized result with good accuracy. The Flickr8k benchmark dataset is used to test this approach. From the results it is known that the trained model performs better than alternative encoder-decoder methods that depend solely on global image features. The metric which has been used to evaluate the model is BLEU which is mainly used for machine translated text evaluation. This model gave a 0.7 BLEU score.

Image Retrieval is a fundamental task of obtaining images similar to the query one from a database. A common image retrieval practice is to firstly retrieve candidate images via similarity search using global image features and then re-rank the candidates by leveraging their local features. Previous learning-based studies mainly focus on either global or local image representation learning to tackle the retrieval task. In this paper, we abandon the two-stage paradigm and seek to design an effective single-stage solution by integrating local and global information inside images into compact image representations. Specifically, we propose a Deep Orthogonal Local and Global (DOLG) information fusion framework for end-to-end image retrieval. It attentively extracts representative local information with multi-atrous convolutions and self-attention at first. Components orthogonal to the global image representation are then extracted from the local information. At last, the orthogonal components are concatenated with the global representation as a complementary, and then aggregation is performed to generate the final representation. The whole framework is end-to-end differentiable and can be trained with image-level labels. Extensive experimental results validate the effectiveness of our solution and show that our model achieves state-of-the-art image retrieval performances on Revisited Oxford and Paris datasets. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

A longstanding objective in the field of multi-modal research uniting computer vision and natural language processing is to develop models that can comprehend the intricate relationship between vision and language. In recent years, we have witnessed notable developments directed towards this objective, enabling computers to interpret visual information and articulate it through captions. Although significant advances have been made, aligning complex visual scenes and language for image captioning tasks using deep learning approaches remains challenging due to the distinct characteristics between the modalities. In this thesis, we propose four new methods that effectively align vision and language for image captioning-based tasks through deep learning techniques. These methods include integrating visual and semantic attributes for controllable image captioning, aligning visual and object semantics for grounded image captioning, learning the relationship between multi-scale visual features with language for image change captioning, and bridging visual and global temporally caption knowledge for temporal sentence grounding. The major contributions of this thesis are summarized as follows: A new Attribute Controlled Image Captioning (ACC) method has been proposed that seamlessly integrates semantic attributes with visual content, enabling automatic modification of the generated captions in the fashion domain. Our approach utilizes semantic attributes as a control signal, giving users the ability to specify particular fashion attributes and styles to incorporate while generating captions. Furthermore, we clean, filter, and assemble a new fashion image caption dataset to facilitate learning and enable us to investigate the effectiveness of our method. A new one-stage Weakly Supervised Grounded Image Captioner (WS-GIC) is proposed that aligns visual and word representation to perform captioning and grounding at the top-down image level. We introduce a Recurrent Grounding Module (RGM) within the decoder to compute Visual Language Attention Maps (VLAMs) for grounding, where VLAMs indicate the spatial region and location of the generated groundable object words in the caption. In addition, we explicitly inject a relation module into our one-stage framework to encourage the relation understanding. The relation semantics aid the prediction of relation words in the caption. A new Interactive Change-aware Transformer Network (ICT-Net) is proposed to extract and incorporate the most critical change of interest in the image, enhancing the generation of change descriptions for complex remote sensing bitemporal scenes. The proposed framework comprises an Interactive Change-aware Encoder (ICE) to capture the crucial difference between bitemporal image features, an Adaptive Fusion Module (AFM) to adaptively aggregate the relevant change-aware features in the encoder layers while minimizing the impact of irrelevant visual features, and a Cross Gated-Attention (CGA) module in the change decoder that enhances the modeling of essential relationships between multi-scale features with word representation, thereby improving change captioning generation. A new Temporal Sentence Grounding (TSG) method is proposed to bridge the domain gap between multi-modal features by leveraging extensive temporally global caption knowledge that is sourced from the relevant video and temporally localized text queries. We introduce the Pseudo-query Intermediary Network (PIN) to contrastively align visual features with temporally global textual knowledge to enhance the similarity between visual and language features. Furthermore, we leverage the pseudo queries prompt to propagate the knowledge, enhancing the learning of feature alignment within the multi-modal fusion module for better temporal grounding. Throughout the thesis, we illustrate how each of the proposed methods aligns vision and language for image captioning-based tasks. Experimental results on public datasets indicate that the proposed methods can achieve better performance. These approaches contribute to enhancing vision and language multi-modalities understanding.