Automatic Image Annotation Research Articles

ACM ICMR 2014 best papers in image retrieval

The ACM International Conference onMultimedia Retrieval (ICMR) is the top multimedia retrieval conference worldwide. This year it was held in Glasgow, Scotland by the general chairs: Joemon Jose and Keith van Rijsbergen. It is my honor to present in this issue the extended versions of the best image retrieval papers which were selected on the basis of the evaluations from the program committee (special thanks to the program chairs: Mohan Kankanhalli, Stefan Rueger and R. Manmatha). One of the grand challenges for multimedia information retrieval is automatic image annotation because it can provide a direct bridge over the semantic gap from low-level features to human level tags and annotations. The paper “A Sparse Kernel RelevanceModel for Automatic Image Annotation” by SeanMoran andVictor Lavrenko introduces a new form of the Continuous RelevanceModel which can perform adaptive kernel selection per feature type. Also, they describe two novel data-adaptive kernels which improve the modeling of image feature distributions. Their findings show that the adaptive kernel assignment can yield significantly higher annotation accuracy. After the initial results have been found from an image retrieval query, a fundamental problem is to re-rank the images toward providing the user withmore accurate and relevant results. The paper “Image re-ranking system based on frequent closed patterns” by Winn Voravuthikunchai, Bruno

Augmenting Image Descriptions Using <newline/>Structured Prediction Output

The need for richer descriptions of images arises in a wide spectrum of applications ranging from image understanding to image retrieval. While the Automatic Image Annotation (AIA) has been extensively studied, image descriptions with the output labels lack sufficient information. This paper proposes to augment image descriptions using structured prediction output. We define a hierarchical tree-structured semantic unit to describe images, from which we can obtain not only the class and subclass one image belongs to, but also the attributes one image has. After defining a new feature map function of structured SVM, we decompose the loss function into every node of the hierarchical tree-structured semantic unit and then predict the tree-structured semantic unit for testing images. In the experiments, we evaluate the performance of the proposed method on two open benchmark datasets and compare with the state-of-the-art methods. Experimental results show the better prediction performance of the proposed method and demonstrate the strength of augmenting image descriptions.

Improving the quality of K-NN graphs through vector sparsification: application to image databases

$$K$$ -nearest neighbor (K-NN) graphs are an essential component of many established methods for content-based image retrieval and automated image annotation. The performance of such methods relies heavily on the semantic quality of the graphs, which can be measured as the proportion of neighbors sharing the same class label as their query images. Due to the noise in image features, the K-NN graphs produced by existing methods may suffer from low semantic quality. In this article, we propose NNF-Descent for the efficient construction of K-NN graphs based on nearest-neighbor and feature descent, in which selective sparsification of feature vectors is interleaved with neighborhood refinement operations in an effort to improve the semantic quality of the result. A variant of the Laplacian Score is proposed for the identification of noisy features local to individual images, whose values are then set to $$0$$ (the global mean value after standardization). We show through extensive experiments on several datasets that NNF-Descent is able to increase the proportion of semantically-related images over unrelated images within the neighbor sets, and that the proposed method generalizes well for other types of data which are represented by high-dimensional feature vectors.

A sparse kernel relevance model for automatic image annotation

In this paper, we introduce a new form of the continuous relevance model (CRM), dubbed the SKL-CRM, that adaptively selects the best performing kernel per feature type for automatic image annotation. Previous image annotation models apply a standard selection of kernels to model the distribution of image features. Popular examples include a Gaussian kernel for modelling GIST features or a Laplacian kernel for global colour histograms. In this work, we demonstrate that this standard assignment of kernels to feature types is sub-optimal and a substantially higher image annotation accuracy can be attained by adapting the kernel-feature assignment. We formulate an efficient greedy algorithm to find the best kernel-feature alignment and show that it is able to rapidly find a sparse subset of features that maximises annotation \(F_{1}\) score. In a second contribution, we introduce two data-adaptive kernels for image annotation—the generalised Gaussian and multinomial kernels—which we demonstrate can better model the distribution of image features as compared to standard kernels. Evaluation is conducted on three standard image datasets across a selection of different feature representations. The proposed SKL-CRM model is found to attain performance that is competitive to a suite of state-of-the-art image annotation models.

Semi-supervised learning for refining image annotation based on random walk model

Automatic image annotation has been an active research topic in recent years due to its potential impact on both image understanding and semantic based image retrieval. In this paper, we present a novel two-stage refining image annotation scheme based on Gaussian mixture model (GMM) and random walk method. To begin with, GMM is applied to estimate the posterior probabilities of each annotation keyword for the image, during which a semi-supervised learning, i.e. transductive support vector machine (TSVM), is employed to enhance the quality of training data. Next, a label similarity graph is constructed by a weighted linear combination of label similarity and visual similarity of images associated with the corresponding labels. In this way, it can seamlessly integrate the information from image low-level visual features and high-level semantic concepts. Followed by a random walk process over the constructed label graph is implemented to further mine the correlation of the candidate annotations so as to capture the refining results, which plays a crucial role in semantic based image retrieval. Finally, extensive experiments carried out on two publicly available image datasets bear out that this approach can achieve marked improvement in annotation performance over several state-of-the-art methods.

Automatic Image Annotation using Image clustering in Multi – Agent Society

The rapid growth of the internet provides tremendous resource for information in different domains (text, image, voice, and many others). This growth introduces new challenge to hit an exact match due to huge number of document returned by search engines where millions of items can be returned for certain subject. Images have been important resources for information, and billions of images are searched to fulfill user demands, which face the mentioned challenge. Automatic image annotation is a promising methodology for image retrieval. However most current annotation models are not yet sophisticated enough to produce high quality annotations. This thesis presents online intelligent indexing for image repositories based on their contents, although content based indexing and retrieving systems have been introduced, this thesis is adding an intelligent technique to re-index images upon better understanding for its composed concepts. Collaborative Agent scheme has been developed to promote objects of an image to concepts and re-index it according to domain specifications. Also this thesis presents automatic annotation system based on the interaction between intelligent agents. Agent interaction is synonym to socialization behavior dominating Agent society. The presented system is exploiting knowledge evolution revenue due to the socialization to charge up the annotation process.

Automatic Image Annotation Based on Scene Analysis

Automatic image annotation is an important and challenging job for image analysis and understanding such as content-based image retrieval (CBIR). The relationship between the keywords and visual features is too complicated due to the semantic gap. We present an approach of automatic image annotation based on scene analysis. With the constrain of scene semantics, the correlation between keywords and visual features becomes simpler and clearer. Our model has two stages of process. The first stage is training process which groups training image data set into semantic scenes using the extracted semantic feature and visual scenes constructed from the calculation distances of visual features for every pairs of training images by using Earth mover's distance (EMD). Then, combine a pair of semantic and visual scene together and apply Gaussian mixture model (GMM) for all scenes. The second stage is to test and annotate keywords for test image data set. Using the visual features provided by Duygulu, experimental results show that our model outperforms probabilistic latent semantic analysis (PLSA) & GMM (PLSA&GMM) model on Corel5K database.

Recent Improvements in Image Retrieval

The target of this paper is to introduce the improvement of the technique of image retrieval. At first, it comes up with the concept of image retrieval and shows the importance of this technique. As the techniques of multimedia and Internet are developing rapidly, the resources of images that users obtain are also extended. And then this paper gives the problem about the image retrieval, namely the information of images are disordering. As the result, it is significant to do the effective organization, management and retrieval based on the increasingly extensive image information storage. After that, this paper presents the concept of TBIR and CBIR and gives the definitions of them. It proposes an issue that CBIR is the improvement of TBIR. Based on CBIR, there are also some disadvantages that need to be improved. In terms of the main point of CBIR, the paper raises that the annotation is one of the most difficult techniques that need to be promoted. Then it describes some algorithms about the technique of automatic image annotation. After these algorithms, the paper shows the challenges and developing direction of the technique of image retrieval. At last, it presented the conclusion to emphasize the main points of this paper.

Semi-supervised Learning for Automatic Image Annotation Based on Bayesian Framework

In this paper, we present a new method for automatic image annotation by applying semisupervised learning based on the Bayesian framework. On the one hand, we employ the semisupervised learning, i.e., transductive support vector machine (TSVM) to enhance the quality of training image data, which is a promising way to find out the underlying relevant data from the unlabeled ones. On the other hand, a simple yet very efficient Bayesian model is built to implement image annotation by the maximum a posteriori (MAP) criterion. The novelty of our method mainly lies in two aspects: exploiting TSVM to improve the quality of training image dataset and utilizing the Bayesian model to predict the candidate annotations for the unseen images. Experimental results on the standard Corel dataset demonstrate that the proposed method is superior or highly competitive to several state-of-the-art approaches.

Geo-location driven image tagging via cross-domain learning

With the rapid development of location-based social network, more and more multimedia data are uploaded by users. These data always include large-scale of independent information with both textual and visual contents. To bridge the semantic gap in between, we propose a novel cross-domain learning method for automatic image annotation with geo-location information. First, we propose the topic model-based method for popular concept extraction to adaptively construct cross-domain datasets. Then these concepts are utilized to collect the visual correlation information from Flickr. Finally, we leverage cross-domain learning method for model learning. The comparison experiments on cross-domain datasets are conducted to demonstrate the superiority of the proposed method.

Cross-domain semantic transfer from large-scale social media

With the rapid development of location-based social networks (LBSNs), more and more media data are unceasingly uploaded by users. The asynchrony between the visual and textual information has made it extremely difficult to manage the multimodal information for manual annotation-free retrieval and personalized recommendation. Consequently the automated image semantic discovery of multimedia location-related user-generated contents (UGCs) for user experience has become mandatory. Most of the literatures leverage single-modality data or correlated multimedia data for image semantic detection. However, the intrinsically heterogeneous UGCs in LBSNs are usually independent and uncorrelated. It is hard to build correlation between textual information and visual information. In this paper, we propose a cross-domain semantic modeling method for automatic image annotation for visual information from social network platforms. First, we extract a set of hot topics from the collected textual information for image dataset preparation. Then the proposed noisy sample filtering is implemented to remove low-relevance photos. Finally, we leverage cross-domain datasets to discover the common knowledge of each semantic concept from UGCs and boost the performance of semantic annotation by semantic transfer. The comparison experiments on cross-domain datasets were conducted to demonstrate the superiority of the proposed method.

Bag of words for semantic automatic medical image annotation

A medical report contains many elements such as medical images accompanied by text descriptions. We present in this paper a new approach for semantic automatic annotation of medical images. The proposed approach uses the bag of words model to represent the visual content of the medical image combined with text descriptors based on term frequency–inverse document frequency technique and reduced by latent semantic to extract the co-occurrence between text and visual terms. In a first phase, we are interested in indexing texts and extracting all relevant terms using a thesaurus containing medical subject headings and concepts. In a second phase, medical images are indexed while recovering areas of interest which are invariant to change in scale such as light and tilt. To annotate a new medical image, we use the bag of words model to recover the feature vector. Indeed, we use the vector space model to retrieve similar medical images from the training database. The computation of the relevance value of an image according to a query image is based on the cosine function. To evaluate the performance of our proposed approach, we present an experiment carried out on five types of radiological imaging. The results showed that our approach works efficiently, especially with more images taken from the radiology of the skull.

Improving image annotation via ranking‐oriented neighbor search and learning‐based keyword propagation

Automatic image annotation plays a critical role in modern keyword‐based image retrieval systems. For this task, the nearest‐neighbor–based scheme works in two phases: first, it finds the most similar neighbors of a new image from the set of labeled images; then, it propagates the keywords associated with the neighbors to the new image. In this article, we propose a novel approach for image annotation, which simultaneously improves both phases of the nearest‐neighbor–based scheme. In the phase of neighbor search, different from existing work discovering the nearest neighbors with the predicted distance, we introduce a ranking‐oriented neighbor search mechanism (RNSM), where the ordering of labeled images is optimized directly without going through the intermediate step of distance prediction. In the phase of keyword propagation, different from existing work using simple heuristic rules to select the propagated keywords, we present a learning‐based keyword propagation strategy (LKPS), where a scoring function is learned to evaluate the relevance of keywords based on their multiple relations with the nearest neighbors. Extensive experiments on the Corel 5K data set and the MIR Flickr data set demonstrate the effectiveness of our approach.

Automatic image annotation using affective vocabularies: Attribute-based learning approach

To improve image search results, understanding and exploiting the subjective aspects of an image is critical. However, how to effectively extract these subjective aspects (e.g. feeling, emotion, and so on) from an image is a challenging problem. In this paper, we propose a novel approach for predicting affective aspects, one of the most interesting subjective aspects, of concepts in images by learning the semantic attributes of the concept and mining the association between the attributes and affective aspects. The main idea of the proposed approach comes from the assumption that semantic attributes of a concept will influence the user’s affect towards the concept (e.g. an animal with the semantic attributes ‘ small’, ‘ furry’, ‘ white’ can be associated with the affective term ‘ cute’). Based on this assumption, we build a multi-layer affect learning framework that consists of (1) an attribute learning layer that predicts semantic attributes of a concept and (2) an affect learning layer that exploits the outputs from the attribute learning layer for predicting the affective aspects of the concept. Through the experimental results on the Animals with Attributes dataset, we show that the proposed approach outperforms traditional approaches by up to 25% in terms of precision and successfully predicts the affect of concepts in images according to different user preferences.

Automatic Image Annotation Based on Co-Training

Automatic image annotation is a critical and challenging problem in pattern recognition and image understanding areas. There are some problems in existing automatic image annotation areas. For example, the size of unlabeled data is much larger than the labeled data. Besides, most image annotation models can only use one kind of image segmentation strategy and certain image description method. According to the above problems, an automatic image annotation model based on Co-training is proposed. In this model, four independent feature properties are constructed and then four corresponding sub-classifiers are built. In this way, different image segmentation strategies and feature representation methods can be integrated into a unified framework. An adaptive algorithm based on vote and consistency is proposed to extend the training dataset. The proposed method use Co-training algorithm and mass unlabeled data to improve the performance of automatic image annotation. Experiments conducted on Corel 5 K dataset verify the effectiveness of proposed method.

A Novel Semantic Statistical Model for Automatic Image Annotation Using the Relationship between the Regions Based on Multi-Criteria Decision Making

Automatic image annotation has emerged as an important research topic due to the existence of the semantic gap and in addition to its potential application on image retrieval and management. In this paper we present an approach which combines regional contexts and visual topics to automatic image annotation. Regional contexts model the relationship between the regions, whereas visual topics provide the global distribution of topics over an image. Conventional image annotation methods neglected the relationship between the regions in an image, while these regions are exactly explanation of the image semantics, therefore considering the relationship between them are helpful to annotate the images. The proposed model extracts regional contexts and visual topics from the image, and incorporates them by MCDM (Multi Criteria Decision Making) approach based on TOPSIS (Technique for Order Preference by Similarity to the Ideal Solution) method. Regional contexts and visual topics are learned by PLSA (Probability Latent Semantic Analysis) from the training data. The experiments on 5k Corel images show that integrating these two kinds of information is beneficial to image annotation. DOI: http://dx.doi.org/10.11591/ijece.v4i1.4598

The Method of Web Image Annotation Classification Automatic

It has been heavy work that to find the related pictures form Internet without annotation. Therefore, the automatic image annotation was extremely important in image retrieval. The traditional method were translated image visual feature into keywords simply, but it ignored the image similarity problem between the low-level visual features and high-level semantic. That is image "gap" problem, so image annotation was very lower. This paper puts forward a classification of web based image content automatic tagging mixing technology, the first it will map visual feature of image to one or more rough images, then we will preprocess the web page text information, finally we select some keywords similarity as image annotation by using similar semantic processing module. So it realizes the image and text combining the automatic annotation and it achieve high precision of image annotation.

Multiple Features Image Fusion Based on Visual Dictionary

It has been heavy work that to find the related pictures form Internet without annotation. Therefore, the automatic image annotation was extremely important in image retrieval. The traditional method were translated image visual feature into keywords simply, but it ignored the image similarity problem between the low-level visual features and high-level semantic. That is image "gap" problem, so image annotation was very lower. This paper puts forward a classification of web based image content automatic tagging mixing technology, the first it will map visual feature of image to one or more rough images, then we will preprocess the web page text information, finally we select some keywords similarity as image annotation by using similar semantic processing module. So it realizes the image and text combining the automatic annotation and it achieve high precision of image annotation.

The SUN Attribute Database: Beyond Categories for Deeper Scene Understanding

In this paper we present the first large-scale scene attribute database. First, we perform crowdsourced human studies to find a taxonomy of 102 discriminative attributes. We discover attributes related to materials, surface properties, lighting, affordances, and spatial layout. Next, we build the “SUN attribute database” on top of the diverse SUN categorical database. We use crowdsourcing to annotate attributes for 14,340 images from 707 scene categories. We perform numerous experiments to study the interplay between scene attributes and scene categories. We train and evaluate attribute classifiers and then study the feasibility of attributes as an intermediate scene representation for scene classification, zero shot learning, automatic image captioning, semantic image search, and parsing natural images. We show that when used as features for these tasks, low dimensional scene attributes can compete with or improve on the state of the art performance. The experiments suggest that scene attributes are an effective low-dimensional feature for capturing high-level context and semantics in scenes.

Multilabel image annotation based on double-layer PLSA model.

Due to the semantic gap between visual features and semantic concepts, automatic image annotation has become a difficult issue in computer vision recently. We propose a new image multilabel annotation method based on double-layer probabilistic latent semantic analysis (PLSA) in this paper. The new double-layer PLSA model is constructed to bridge the low-level visual features and high-level semantic concepts of images for effective image understanding. The low-level features of images are represented as visual words by Bag-of-Words model; latent semantic topics are obtained by the first layer PLSA from two aspects of visual and texture, respectively. Furthermore, we adopt the second layer PLSA to fuse the visual and texture latent semantic topics and achieve a top-layer latent semantic topic. By the double-layer PLSA, the relationships between visual features and semantic concepts of images are established, and we can predict the labels of new images by their low-level features. Experimental results demonstrate that our automatic image annotation model based on double-layer PLSA can achieve promising performance for labeling and outperform previous methods on standard Corel dataset.