Automatic Image Annotation System Research Articles

Automatic image annotation system using deep learning method to analyse ambiguous images

Automatic image annotation system using deep learning method to analyse ambiguous images

Automated Image Annotation With Novel Features Based on Deep ResNet50-SLT

Due to their vast size, the growing number of digital images found in personal archives and on websites has become unmanageable, making it challenging to accurately retrieve images from these large databases. While these collections are popular due to their convenience, they are often not equipped with proper indexing information, making it difficult for users to find what they need. One of the most significant challenges in the field of computer vision and multimedia is image annotation, which involves labeling images with descriptive keywords. However, computers do not possess the capability to understand the essence of images in the same way that humans do, and people can only identify images based on their visual attributes, not their deeper semantic meaning. Therefore, image annotation requires the use of keywords to effectively communicate the contents of an image to a computer system. However, raw pixels in an image do not provide enough information to generate semantic concepts, making image annotation a complex task. Unlike text annotation, where the dictionary linking words to semantics is well established, image annotation lacks a clear definition of "words" or "sentences" that can be associated with the meaning of the image, known as the semantic gap. To address this challenge, this study aimed to characterize image content meaningfully to make information retrieval easier. An improved automatic image annotation (AIA) system was proposed to bridge the semantic gap between low-level computer features and human interpretation of images by assigning one or multiple labels to images. The proposed AIA system can convert raw image pixels into semantic-level concepts, providing a clearer representation of the image content. The study combined the ResNet50 and slantlet transform with word2vec and principal component analysis with t-distributed stochastic neighbor embedding to balance precision and recall. This allowed the researchers to determine the optimal model for the proposed ResNet50-SLT AIA framework. A Word2vec model with ResNet50-SLT was used with principal component analysis and t-distributed stochastic neighbor embedding to improve IA prediction accuracy. The distributed representation approach involved encoding and storing information about image features. The proposed AIA system utilized seq2seq to generate sentences depending on feature vectors. The system was implemented on the most popular datasets (Flickr8k, Corel-5k, ESP-Game). The results showed that the newly developed AIA scheme overcame the computational time complexity associated with most existing image annotation models during the training phase for large datasets. The performance evaluation of the AIA scheme showed its excellent flexibility of annotation, improved accuracy, and reduced computational costs, thus outperforming the existing state-of-the-art methods. In conclusion, this AIA framework can provide immense benefits in accurately selecting and extracting image features and easily retrieving images from large databases. The extracted features can effectively be used to represent the image, thus accelerating the annotation process and minimizing the computational complexity.

Racing Bib Number Recognition Method using Deep Learning

Mass running event has gained popularity ever since recreational running becomes more common as they often held annually by various organizers. As image documentation took a huge part to showcase the event, many thousands of images were generated during the event. Along with thousands of images that were generated, the participant is unlikely to found an image of themselves. To solve this problem, image annotation could be performed to address image with specific tags such as participant attribute like racing bib number (RBN). Manually annotate thousands of images would result in inefficiency of time and hard-labor. As a work to tackle this problem, this paper proposed an automatic image annotation system using the YOLOv3 algorithm based RBN recognition method. The experiment result shows 83.0% precision, 81.5% recall, and 82.2% F1 score as a result of our proposed method on running event dataset. Therefore, this implemented method will promote efficiency to solve the image annotation problem because it doesn't require manual annotation over thousand of running event images

Architecture to improve the accuracy of automatic image annotation systems

Automatic image annotation (AIA) is an image retrieval mechanism to extract relative semantic tags from visual content. So far, the improvement of accuracy in newly developed such methods have been about 1 or 2% in the F1‐score and the architectures seem to have room for improvement. Therefore, the authors designed a more detailed architecture for AIA and suggested new algorithms for its main parts. The proposed architecture has three main parts: feature extraction, learning, and annotation. They designed a novel learning method using machine learning and probability bases. In the annotation part, they suggest a novel method that gains the maximum benefit from the learning part. The combination of the proposed architecture, algorithms, and novel ideas resulted in new accuracy milestones in F1‐score on most commonly used datasets. In their architecture, N+ measure which shows the number of tags with non‐zero recalls showed that they could recall all tags for IAPRTC‐12 and ESP‐Games datasets.

Detection Method of Data Integrity in Network Storage Based on Symmetrical Difference

In order to enhance the recall and the precision performance of data integrity detection, a method to detect the network storage data integrity based on symmetric difference was proposed. Through the complete automatic image annotation system, the crawler technology was used to capture the image and related text information. According to the automatic word segmentation, pos tagging and Chinese word segmentation, the feature analysis of text data was achieved. Based on the symmetrical difference algorithm and the background subtraction, the feature extraction of image data was realized. On the basis of data collection and feature extraction, the sentry data segment was introduced, and then the sentry data segment was randomly selected to detect the data integrity. Combined with the accountability scheme of data security of the trusted third party, the trusted third party was taken as the core. The online state judgment was made for each user operation. Meanwhile, credentials that cannot be denied by both parties were generated, and thus to prevent the verifier from providing false validation results. Experimental results prove that the proposed method has high precision rate, high recall rate, and strong reliability.

Filtering Instagram Hashtags Through Crowdtagging and the HITS Algorithm

Instagram is a rich source for mining descriptive tags for images and multimedia in general. The tags–image pairs can be used to train automatic image annotation (AIA) systems in accordance with the learning by example paradigm. In previous studies, we had concluded that, on average, 20% of the Instagram hashtags are related to the actual visual content of the image they accompany, i.e., they are descriptive hashtags, while there are many irrelevant hashtags, i.e., stop-hashtags, that are used across totally different images just for gathering clicks and for searchability enhancement. In this paper, we present a novel methodology, based on the principles of collective intelligence that helps in locating those hashtags. In particular, we show that the application of a modified version of the well-known hyperlink-induced topic search (HITS) algorithm, in a crowdtagging context, provides an effective and consistent way for finding pairs of Instagram images and hashtags, which lead to representative and noise-free training sets for content-based image retrieval. As a proof of concept, we used the crowdsourcing platform Figure-eight to allow collective intelligence to be gathered in the form of tag selection (crowdtagging) for Instagram hashtags. The crowdtagging data of Figure-eight are used to form bipartite graphs in which the first type of nodes corresponds to the annotators and the second type to the hashtags they selected. The HITS algorithm is first used to rank the annotators in terms of their effectiveness in the crowdtagging task and then to identify the right hashtags per image.

Automatic Image Annotation via Combining Low-level Colour Feature with Features Learned from Convolutional Neural Networks

In this paper, a feature combination approach to annotate and retrieve images is proposed. In addition to using low-level colour features from original images, we extract features learned from convolutional neural networks (CNNs). We find these two sets are complementary to each other in conducting automatic image annotation (AIA). For both single-label CIFAR-10 and multi-label COREL-5K AIA tasks, the CNN-learned features perform slightly better than the low-level image features. Finally, when combining the two feature sets as inputs into the deep neural network-based AIA systems, we obtain the best performance in both cases.

A multi-view-group non-negative matrix factorization approach for automatic image annotation

In automatic image annotation (AIA) different features describe images from different aspects or views. Part of information embedded in some views is common for all views, while other parts are individual and specific. In this paper, we present the Mvg-NMF approach, a multi-view-group non-negative matrix factorization (NMF) method for an AIA system which considers both common and individual factors. The NMF framework discovers a latent space by decomposing data into a set of non-negative basis vectors and coefficients. The views divided into homogeneous groups and latent spaces are extracted for each group. After mapping the test images into these spaces, a unified distance matrix is computed from the distance between images in all spaces. Then a search-based method is used to propagate tags from the nearest neighbors to test images. The evaluation on three datasets commonly used for image annotation showed that the Mvg-NMF is highly competitive with the recent state-of-the-art works.

Image annotation using multi-view non-negative matrix factorization with different number of basis vectors

Automatic Image Annotation (AIA) helps image retrieval systems by predicting tags for images. In this paper, we propose an AIA system using Non-negative Matrix Factorization (NMF) framework. The NMF framework discovers a latent space, by factorizing data into a set of non-negative basis and coefficients. To model the images, multiple features are extracted, each one represents images from a specific view. We use multi-view graph regularization NMF and allow NMF to choose a different number of basis vectors for each view. For tag prediction, each test image is mapped onto the multiple latent spaces. The distances of images in these spaces are used to form a unified distance matrix. The weights of distances are learned automatically. Then a search-based method is used to predict tags based on tags of nearest neighbors’. We evaluate our method on three datasets and show that it is competitive with the current state-of-the-art methods.

Image Retrieval using Fused Features and Annotation Techniques

Due to the development of digital technology large amount of images are stored and shared in internet and personal database. Lot of research has been done in this area. Traditional image retrieval focus on text and content based. It has some drawback while receiving the required informations. AIA is the new interesting research area. It automatically assigns the keywords to an image. In this work fusion of texture feature and shape features are used. Proposed technique used to learns the relationship between image features and keywords. This proposed work focus automatic image retrieval and annotation system with multi class SVM

Automatic image annotation by a loosely joint non‐negative matrix factorisation

Nowadays, the number of digital images has increased so that the management of this volume of data needs an efficient system for browsing, categorising and searching. Automatic image annotation is designed for assigning tags to images for more accurate retrieval. Non‐negative matrix factorisation (NMF) is a traditional machine learning technique for decomposing a matrix into a set of basis and coefficients under the non‐negative constraints. In this study, the authors propose a two‐step algorithm for designing an automatic image annotation system that employs the NMF framework for its first step and a variant of K‐nearest neighbourhood as its second step. In the first step, a new multimodal NMF algorithm is proposed to extract the latent factors which reflect the content of images. This is done by jointly factorising the visual and textual data feature matrices so that they have close representation, although not necessarily the same. In the second step, after mapping images to the latent factors space a few tags are predicted for the new images based on a weighted average of similar data. They evaluated the performance of the proposed method and compared it to the state‐of‐the‐art literature. Comparison results demonstrate the effectiveness and potential of the proposed method in image annotation applications.

Semantic, Automatic Image Annotation Based on Multi-Layered Active Contours and Decision Trees

In this paper, we propose a new approach for automatic image annotation (AIA) in order to automatically and efficiently assign linguistic concepts to visual data such as digital images, based on both numeric and semantic features. The presented method first computes multi-layered active contours. The first-layer active contour corresponds to the main object or foreground, while the next-layers active contours delineate the object’s subparts. Then, visual features are extracted within the regions segmented by these active contours and are mapped into semantic notions. Next, decision trees are trained based on these attributes, and the image is semantically annotated using the resulting decision rules. Experiments carried out on several standards datasets have demonstrated the reliability and the computational effectiveness of our AIA system.

Effective and Efficient Multi-Facet Web Image Annotation

The vast amount of images available on the Web request for an effective and efficient search service to help users find relevant images. The prevalent way is to provide a keyword interface for users to submit queries. However, the amount of images without any tags or annotations are beyond the reach of manual efforts. To overcome this, automatic image annotation techniques emerge, which are generally a process of selecting a suitable set of tags for a given image without user intervention. However, there are three main challenges with respect to Web-scale image annotation: scalability, noise-resistance and diversity. Scalability has a twofold meaning: first an automatic image annotation system should be scalable with respect to billions of images on the Web; second it should be able to automatically identify several relevant tags among a huge tag set for a given image within seconds or even faster. Noise-resistance means that the system should be robust enough against typos and ambiguous terms used in tags. Diversity represents that image content may include both scenes and objects, which are further described by multiple different image features constituting different facets in annotation. In this paper, we propose a unified framework to tackle the above three challenges for automatic Web image annotation. It mainly involves two components: tag candidate retrieval and multi-facet annotation. In the former content-based indexing and concept-based codebook are leveraged to solve scalability and noise-resistance issues. In the latter the joint feature map has been designed to describe different facets of tags in annotations and the relations between these facets. Tag graph is adopted to represent tags in the entire annotation and the structured learning technique is employed to construct a learning model on top of the tag graph based on the generated joint feature map. Millions of images from Flickr are used in our evaluation. Experimental results show that we have achieved 33% performance improvements compared with those single facet approaches in terms of three metrics: precision, recall and F1 score.

A Semi-Supervised Algorithm for Auto-Annotation and Unknown Structures Discovery in Satellite Image Databases

The increasing number and resolution of earth observation (EO) imaging sensors has had a significant impact on both the acquired image data volume and the information content in images. There is consequently a strong need for highly efficient search tools for EO image databases and for search methods to automatically identify and recognize structures within EO images. Content Based Image Retrieval (CBIR) and automatic image annotation systems have been designed to tackle the problem of image retrieval in large image databases. These two systems achieve a common goal, that is to learn the mapping function between low-level visual features and high-level image semantics. A setup, which has hardly been explored in annotating systems and which is the rule rather than the exception, is the case when the training database used to learn the mapping function is not exhaustive regarding semantic classes present in the images. This means that there exists unknown image classes for which there is no training examples in the training database. In this paper, we propose a semi-supervised method for auto-annotating satellite image databases and discovering unknown semantic image classes in these databases. The idea is to incorporate into the learning process the unannotated data which by definition contain the unknown image classes. The latter are considered to be latent structures in the data that appear when we train a hierarchical latent variable model with both the labeled and unlabeled data. We also show that, in our case, the use of unlabeled data leads to more reliable estimates regarding the model parameters. We present experimental results on a synthetic dataset, making a comparison of our algorithm with a semi-supervised Support Vector Machine (S3VM) on this dataset. We also demonstrate the effectiveness of our unknown image classes discovery procedure on a database of SPOT5 satellite images. We show that the results obtained on this database are rather positive since the new structures detected correspond to semantic classes which are not represented in the training database.

Context Dependent Automatic Textile Image Annotation Using Networked Knowledge

This paper presents a new method of an automatic image annotation system that estimates keywords from an image. Typical automatic image annotation systems extract features from an image and recognize keywords. However this method has two problems. One is that it treats features statically. Features should change depending on what keywords are attached so keywords should not be treated equally. Another is that it does not consider the level of keywords. Visual keywords, such as color or texture, can be recognized easily from image features, while high-level semantics such as context are hard to recognize from the features. To solve these problems, our approach is to recognize context by using networked specialist knowledge and to recognize keywords by changing feature values dynamically depending on the context. To evaluate our system, we conducted two experiments of applying it to textile images. As a result, we obtained improved accuracy and confirmed the effectiveness of using networked knowledge.

Incorporating multiple SVMs for automatic image annotation

In this paper, a novel automatic image annotation system is proposed, which integrates two sets of support vector machines (SVMs), namely the multiple instance learning (MIL)-based and global-feature-based SVMs, for annotation. The MIL-based bag features are obtained by applying MIL on the image blocks, where the enhanced diversity density (DD) algorithm and a faster searching algorithm are applied to improve the efficiency and accuracy. They are further input to a set of SVMs for finding the optimum hyperplanes to annotate training images. Similarly, global color and texture features, including color histogram and modified edge histogram, are fed into another set of SVMs for categorizing training images. Consequently, two sets of image features are constructed for each test image and are, respectively, sent to the two sets of SVMs, whose outputs are incorporated by an automatic weight estimation method to obtain the final annotation results. Our proposed annotation approach demonstrates a promising performance for an image database of 12 000 general-purpose images from COREL, as compared with some current peer systems in the literature.