Automatic Image Annotation Research Articles

Enhancing Image Annotation Technique of Fruit Classification Using a Deep Learning Approach

An accurate image retrieval technique is required due to the rapidly increasing number of images. It is important to implement image annotation techniques that are fast, simple, and, most importantly, automatically annotate. Image annotation has recently received much attention due to the massive rise in image data volume. Focusing on the agriculture field, this study implements automatic image annotation, namely, a repetitive annotation task technique, to classify the ripeness of oil palm fruit and recognize a variety of fruits. This approach assists farmers to enhance the classification of fruit methods and increase their production. This study proposes simple and effective models using a deep learning approach with You Only Look Once (YOLO) versions. The models were developed through transfer learning where the dataset was trained with 100 images of oil fruit palm and 400 images of a variety of fruit in RGB images. Model performance and accuracy of automatically annotating the images with 3500 fruits were examined. The results show that the annotation technique successfully annotated a large number of images accurately. The mAP result achieved for oil palm fruit was 98.7% and the variety of fruit was 99.5%.

Automated Image Annotation With Novel Features Based on Deep ResNet50-SLT

Due to their vast size, the growing number of digital images found in personal archives and on websites has become unmanageable, making it challenging to accurately retrieve images from these large databases. While these collections are popular due to their convenience, they are often not equipped with proper indexing information, making it difficult for users to find what they need. One of the most significant challenges in the field of computer vision and multimedia is image annotation, which involves labeling images with descriptive keywords. However, computers do not possess the capability to understand the essence of images in the same way that humans do, and people can only identify images based on their visual attributes, not their deeper semantic meaning. Therefore, image annotation requires the use of keywords to effectively communicate the contents of an image to a computer system. However, raw pixels in an image do not provide enough information to generate semantic concepts, making image annotation a complex task. Unlike text annotation, where the dictionary linking words to semantics is well established, image annotation lacks a clear definition of "words" or "sentences" that can be associated with the meaning of the image, known as the semantic gap. To address this challenge, this study aimed to characterize image content meaningfully to make information retrieval easier. An improved automatic image annotation (AIA) system was proposed to bridge the semantic gap between low-level computer features and human interpretation of images by assigning one or multiple labels to images. The proposed AIA system can convert raw image pixels into semantic-level concepts, providing a clearer representation of the image content. The study combined the ResNet50 and slantlet transform with word2vec and principal component analysis with t-distributed stochastic neighbor embedding to balance precision and recall. This allowed the researchers to determine the optimal model for the proposed ResNet50-SLT AIA framework. A Word2vec model with ResNet50-SLT was used with principal component analysis and t-distributed stochastic neighbor embedding to improve IA prediction accuracy. The distributed representation approach involved encoding and storing information about image features. The proposed AIA system utilized seq2seq to generate sentences depending on feature vectors. The system was implemented on the most popular datasets (Flickr8k, Corel-5k, ESP-Game). The results showed that the newly developed AIA scheme overcame the computational time complexity associated with most existing image annotation models during the training phase for large datasets. The performance evaluation of the AIA scheme showed its excellent flexibility of annotation, improved accuracy, and reduced computational costs, thus outperforming the existing state-of-the-art methods. In conclusion, this AIA framework can provide immense benefits in accurately selecting and extracting image features and easily retrieving images from large databases. The extracted features can effectively be used to represent the image, thus accelerating the annotation process and minimizing the computational complexity.

Robust Deep Transfer Learning Based Object Detection and Tracking Approach

At present days, object detection and tracking concepts have gained more importance among researchers and business people. Presently, deep learning (DL) approaches have been used for object tracking as it increases the performance and speed of the tracking process. This paper presents a novel robust DL based object detection and tracking algorithm using Automated Image Annotation with ResNet based Faster regional convolutional neural network (R-CNN) named (AIA-FRCNN) model. The AIA-RFRCNN method performs image annotation using a Discriminative Correlation Filter (DCF) with Channel and Spatial Reliability tracker (CSR) called DCF-CSRT model. The AIA-RFRCNN model makes use of Faster RCNN as an object detector and tracker, which involves region proposal network (RPN) and Fast R-CNN. The RPN is a full convolution network that concurrently predicts the bounding box and score of different objects. The RPN is a trained model used for the generation of the high-quality region proposals, which are utilized by Fast R-CNN for detection process. Besides, Residual Network (ResNet 101) model is used as a shared convolutional neural network (CNN) for the generation of feature maps. The performance of the ResNet 101 model is further improved by the use of Adam optimizer, which tunes the hyperparameters namely learning rate, batch size, momentum, and weight decay. Finally, softmax layer is applied to classify the images. The performance of the AIA-RFRCNN method has been assessed using a benchmark dataset and a detailed comparative analysis of the results takes place. The outcome of the experiments indicated the superior characteristics of the AIA-RFRCNN model under diverse aspects.

Automatic Image Annotation Using Adaptive Convolutional Deep Learning Model

Every day, websites and personal archives create more and more photos. The size of these archives is immeasurable. The comfort of use of these huge digital image gatherings donates to their admiration. However, not all of these folders deliver relevant indexing information. From the outcomes, it is difficult to discover data that the user can be absorbed in. Therefore, in order to determine the significance of the data, it is important to identify the contents in an informative manner. Image annotation can be one of the greatest problematic domains in multimedia research and computer vision. Hence, in this paper, Adaptive Convolutional Deep Learning Model (ACDLM) is developed for automatic image annotation. Initially, the databases are collected from the open-source system which consists of some labelled images (for training phase) and some unlabeled images {Corel 5 K, MSRC v2}. After that, the images are sent to the pre-processing step such as colour space quantization and texture color class map. The pre-processed images are sent to the segmentation approach for efficient labelling technique using J-image segmentation (JSEG). The final step is an automatic annotation using ACDLM which is a combination of Convolutional Neural Network (CNN) and Honey Badger Algorithm (HBA). Based on the proposed classifier, the unlabeled images are labelled. The proposed methodology is implemented in MATLAB and performance is evaluated by performance metrics such as accuracy, precision, recall and F1_Measure. With the assistance of the proposed methodology, the unlabeled images are labelled.

Error-Driven Learning: Development of Vocabulary Learning Support System that Utilizes Suggestibility of Error by Image Generation

Vocabulary learning has long been taught as a basis for L2 English language learning. The use of visual clues is a widely recognized method in vocabulary learning. Several studies have applied automatic image caption generation, invented with computer vision and natural language processing, to English vocabulary learning in recent years. However, these vocabulary learning systems mainly use correct answers and their corresponding images. Another study focuses on the effectiveness of making errors, utilizing simulation to visualize the learner’s error, and its suggestibility contributes to the learning effectiveness of the physics task. We focus on the effectiveness of error with image generation implying suggestibility and hypothesize that errors encourage the students to memorize the vocabulary effectively. We propose a vocabulary learning support system with image generation corresponding to learner’s error for L2 English language learners. To accomplish the development, we defined three tasks and researched its possibility.

Generating Image Captions Using Bahdanau Attention Mechanism and Transfer Learning

Automatic image caption prediction is a challenging task in natural language processing. Most of the researchers have used the convolutional neural network as an encoder and decoder. However, an accurate image caption prediction requires a model to understand the semantic relationship that exists between the various objects present in an image. The attention mechanism performs a linear combination of encoder and decoder states. It emphasizes the semantic information present in the caption with the visual information present in an image. In this paper, we incorporated the Bahdanau attention mechanism with two pre-trained convolutional neural networks—Vector Geometry Group and InceptionV3—to predict the captions of a given image. The two pre-trained models are used as encoders and the Recurrent neural network is used as a decoder. With the help of the attention mechanism, the two encoders are able to provide semantic context information to the decoder and achieve a bilingual evaluation understudy score of 62.5. Our main goal is to compare the performance of the two pre-trained models incorporated with the Bahdanau attention mechanism on the same dataset.

VLSP 2021 - VieCap4H Challenge: Automatic Image Caption Generation for Healthcare Domain in Vietnamese

This paper presents VieCap4H, a grand data challenge on automatic image caption generation for the healthcare domain in Vietnamese. VieCap4H is held as part of the eighth annual workshop on VietnameseLanguage and Speech Processing (VLSP 2021). The task is considered as an image captioning task. Given a static image, mostly about healthcare-related scenarios, participants are asked to design machine learning methods to generate natural language captions in Vietnamese to describe the visual content of the image. We introduce VieCap4H, a novel human-annotated image captioning dataset in Vietnamese that contains over 10,000 image-caption pairs collected from real-world scenarios in the healthcare domain. All the models proposed by the challenge participants are evaluated using BLEU scores against groundtruths. The challenge was run on AIHUB.VN platform. Within less than two months, the challenge has attracted over 90 individual participants and recorded more than 900 valid submissions.

VieCap4H Challenge 2021: A transformer-based method for Healthcare Image Captioning in Vietnamese

The automatic image caption generation is attractive to both Computer Vision and Natural Language Processing research community because it lies in the gap between these two fields. Within the vieCap4H contest organized by VLSP 2021, we participate and present a Transformer-based solution for image captioning in the healthcare domain. In detail, we use grid features as visual presentation and pre-training a BERT-based language model from PhoBERT-base pre-trained model to obtain language presentation used in the Adaptive Decoder module in the RSTNet model. Besides, we indicate a suitable schedule with the self-critical training sequence (SCST) technique to achieve the best results. Through experiments, we achieve an average of 30.3% BLEU score on the public-test round and 28.9% on the private-test round, which ranks 3rd and 4th, respectively. Source code is available at https://github.com/caodoanh2001/uit-vlsp-viecap4h-solution.

Exponential Sailfish Optimizer-based generative adversarial network for image annotation on natural scene images

Generally, automatic image annotation can offer semantic graphics for recognizing image contents, and it creates a base for devising various techniques, which can search images in a huge dataset. Although most existing techniques mainly focus on resolving annotation issues through sculpting tag semantic information and visual image content, it ignores additional information, like picture positions and descriptions. The established Exponential Sailfish Optimizer-based Generative Adversarial Networks are therefore used to provide an efficient approach for image annotation (ESFO-based GAN). By combining Exponentially Weighted Moving Average (EWMA) and Sailfish Optimizer (SFO), the ESFO is a newly created design that is used to train the GAN classifier. Additionally, the Grabcut is presented to successfully do image annotation by extracting the background and foreground images. Additionally, DeepJoint segmentation is used to divide apart the images based on the background image that was extracted. Finally, image annotation is successfully accomplished with the aid of GAN. As a result, image annotation uses the produced ESFO-based GAN's subsequent results. The developed approach exhibited enhanced outcomes with maximum F-Measure of 98.37%, maximum precision of 97.02%, and maximal recall of 96.64%, respectively, using the flicker dataset.

A Novel Deep Learning-Based Relabeling Architecture for Space Objects Detection from Partially Annotated Astronomical Images

Space Surveillance and Tracking is a task that requires the development of systems that can accurately discriminate between natural and man-made objects that orbit around Earth. To manage the discrimination between these objects, it is required to analyze a large amount of partially annotated astronomical images collected using a network of on-ground and potentially space-based optical telescopes. Thus, the main objective of this article is to propose a novel architecture that improves the automatic annotation of astronomical images. To achieve this objective, we present a new method for automatic detection and classification of space objects (point-like and streaks) in a supervised manner, given real-world partially annotated images in the FITS (Flexible Image Transport System) format. Results are strongly dependent on the preprocessing techniques applied to the images. Therefore, different techniques were tested including our method for object filtering and bounding box extraction. Based on our relabeling pipeline, we can easily follow how the number of detected objects is gradually increasing after each iteration, achieving a mean average precision of 98%.

Caption Generation Based on Emotions Using CSPDenseNet and BiLSTM with Self-Attention

Automatic image caption generation is an intricate task of describing an image in natural language by gaining insights present in an image. Featuring facial expressions in the conventional image captioning system brings out new prospects to generate pertinent descriptions, revealing the emotional aspects of the image. The proposed work encapsulates the facial emotional features to produce more expressive captions similar to human-annotated ones with the help of Cross Stage Partial Dense Network (CSPDenseNet) and Self-attentive Bidirectional Long Short-Term Memory (BiLSTM) network. The encoding unit captures the facial expressions and dense image features using a Facial Expression Recognition (FER) model and CSPDense neural network, respectively. Further, the word embedding vectors of the ground truth image captions are created and learned using the Word2Vec embedding technique. Then, the extracted image feature vectors and word vectors are fused to form an encoding vector representing the rich image content. The decoding unit employs a self-attention mechanism encompassed with BiLSTM to create more descriptive and relevant captions in natural language. The Flickr11k dataset, a subset of the Flickr30k dataset is used to train, test, and evaluate the present model based on five benchmark image captioning metrics. They are BiLingual Evaluation Understudy (BLEU), Metric for Evaluation of Translation with Explicit Ordering (METEOR), Recall-Oriented Understudy for Gisting Evaluation (ROGUE), Consensus-based Image Description Evaluation (CIDEr), and Semantic Propositional Image Caption Evaluation (SPICE). The experimental analysis indicates that the proposed model enhances the quality of captions with 0.6012(BLEU-1), 0.3992(BLEU-2), 0.2703(BLEU-3), 0.1921(BLEU-4), 0.1932(METEOR), 0.2617(CIDEr), 0.4793(ROUGE-L), and 0.1260(SPICE) scores, respectively, using additive emotional characteristics and behavioral components of the objects present in the image.

An Enhanced Triadic Color Scheme for Content-Based Image Retrieval

The complexity of multimedia content, particularly images, has risen dramatically in recent years, and millions of images are shared on social media every day. Finding or retrieving an appropriate image is becoming more difficult due to the increase in the volume of shared and archived multimedia data. Any image retrieval model must, at a bare minimum, locate and classify images that are visually related to the user’s query. The vast majority of Internet search engines employ text algorithms that fetch images using captions as input. Even though there is a lot of study being done to increase the effectiveness of automatic image annotation, retrieval errors can occur due to differences in visual perception. Content-based image retrieval (CBIR) addresses the aforementioned issue because visual analysis of the content is included in the query image. On the other hand, feature extraction is significantly challenging because of semantic gap. This work proposes a strategy for effective retrieval in similarity images using the triadic color scheme RGB, YCbCr, and L ∗ a ∗ b ∗ based on reranking. We want to increase image similarity and encourage more relevant reranking. As a result of the findings, it can be concluded that a triadic color scheme improves precision by 5% more dramatically than existing schemes and also efficiently improves retrieved results while reducing user effort.

An accurate generation of image captions for blind people using extended convolutional atom neural network.

Recently, the progress on image understanding and AIC (Automatic Image Captioning) has attracted lots of researchers to make use of AI (Artificial Intelligence) models to assist the blind people. AIC integrates the principle of both computer vision and NLP (Natural Language Processing) to generate automatic language descriptions in relation to the image observed. This work presents a new assistive technology based on deep learning which helps the blind people to distinguish the food items in online grocery shopping. The proposed AIC model involves the following steps such as Data Collection, Non-captioned image selection, Extraction of appearance, texture features and Generation of automatic image captions. Initially, the data is collected from two public sources and the selection of non-captioned images are done using the ARO (Adaptive Rain Optimization). Next, the appearance feature is extracted using SDM (Spatial Derivative and Multi-scale) approach and WPLBP (Weighted Patch Local Binary Pattern) is used in the extraction of texture features. Finally, the captions are automatically generated using ECANN (Extended Convolutional Atom Neural Network). ECANN model combines the CNN (Convolutional Neural Network) and LSTM (Long Short-Term Memory) architectures to perform the caption reusable system to select the most accurate caption. The loss in the ECANN architecture is minimized using AAS (Adaptive Atom Search) Optimization algorithm. The implementation tool used is PYTHON and the dataset used for the analysis are Grocery datasets (Freiburg Groceries and Grocery Store Dataset). The proposed ECANN model acquired accuracy (99.46%) on Grocery Store Dataset and (99.32%) accuracy on Freiburg Groceries dataset. Thus, the performance of the proposed ECANN model is compared with other existing models to verify the supremacy of the proposed work over the other existing works.

IMAGE CAPTIONING

Image captioning using artificial intelligence is a growing trend. This is formally known as automatic image annotation- a process by which a computer system will generate an automatic textual description to a corresponding image by making use of NLP (Natural Language Processing) and Computer Vision. In today’s day and age where visual data is abundant, Image Captioning plays a significant role in various applications and in recent years, it has gained a lot of study attention in the field of cognitive computing. This research paper contains the analysis and comparison of a few common and efficient image caption generation methods.

Instagram hashtags as image metadata for Automatic Image Annotation

Instagram hashtags as image metadata for Automatic Image Annotation

Tracker/Camera Calibration for Accurate Automatic Gaze Annotation of Images and Videos.

Modern appearance-based gaze tracking algorithms require vast amounts of training data, with images of a viewer annotated with "ground truth" gaze direction. The standard approach to obtain gaze annotations is to ask subjects to fixate at specific known locations, then use a head model to determine the location of "origin of gaze". We propose using an IR gaze tracker to generate gaze annotations in natural settings that do not require the fixation of target points. This requires prior geometric calibration of the IR gaze tracker with the camera, such that the data produced by the IR tracker can be expressed in the camera's reference frame. This contribution introduces a simple tracker/camera calibration procedure based on the PnP algorithm and demonstrates its use to obtain a full characterization of gaze direction that can be used for ground truth annotation.

Automatic image caption generation using deep learning and multimodal attention

AbstractWe present an improved image caption generation model that incorporating multimodal attention mechanism. We use ResNet‐101 to extract image features while incorporating channel attention mechanism and spatial attention mechanism. We use Faster R‐CNN for object detection and use a multi‐head attention structure consisting of spatial attention and self‐attention. This allows our algorithm to improve the model's capability to learn and use the internal grammatical features of natural sentences. Moreover, we use GPU parallel computing to accelerate the entire model training. We apply our model and algorithm to early education scenarios: show and tell for kids. We compare our algorithm with the state‐of‐the‐art deep learning algorithms. Our experimental results show that our model improves the captioning accuracy in terms of standard automatic evaluation metrics.

Social Image Annotation Based on Image Captioning

With the popularity of new social media, automatic image annotation (AIA) has been an active research topic due to its great importance in image retrieval, understanding, and management. Despite their relative success, most of annotation models suffer from the low-level visual representation and semantic gap. To address the above shortcomings, we propose a novel annotation method utilizing textual feature generated by image captioning, in contrast to all previous methods that use visual feature as image feature. In our method, each image is regarded as a label-vector of k userprovided textual tags rather than a visual vector. We summarize our method as follows. First, the image visual features are extracted by combining the deep residual network and the object detection model, which are encoded and decoded by the mesh-connected Transformer network model. Then, the textual modal feature vector of the image is constructed by removing stop-words and retaining high-frequency tags. Finally, the textual feature vector of the image is applied to the propagation annotation model to generate a high-quality image annotation labels. Experimental results conducted on standard MS-COCO datasets demonstrate that the proposed method significantly outperforms existing classical models, mainly benefiting from the proposed textual feature generated by image captioning technology.

Image retrieval based on weighted nearest neighbor tag prediction

Abstract With the development of communication and computer technology, the application of big data technology has become increasingly widespread. Reasonable, effective, and fast retrieval methods for querying information from massive data have become an important content of current research. This article provides an image retrieval method based on the weighted nearest neighbor label prediction for the problem of automatic image annotation and keyword image retrieval. In order to improve the performance of the test method, scientific experimental verification was implemented. The nearest neighbor weights are determined by maximizing the training image annotation, and experiments are carried out from multiple angles based on the Mahalanobis metric learning integration model. The experimental results show that the proposed tag correlation prediction propagation model has obvious improvements in accuracy, recall rate, break-even point, and overall average accuracy performance compared with other widely used algorithm models.

Video object segmentation for automatic image annotation of ethernet connectors with environment mapping and 3D projection

Video object segmentation for automatic image annotation of ethernet connectors with environment mapping and 3D projection