Abstract Automatic Image Annotation Research Articles

Image Annotation by a Hierarchical and Iterative Combination of Recognition and Segmentation

Automatic image annotation and image segmentation are two prominent research fields of Computer Vision, that are getting higher attention these days to accomplish image analysis and scene understanding. In this work, we present an annotation algorithm based on a hierarchical image partition, that makes use of Markov Random Fields (MRFs) to model spatial and hierarchical relations among regions in the image. In this way, we can capture local, global and contextual information. Also, we combine the processes of annotation and segmentation in an iterative way so that each process benefits from the other. Furthermore, we investigate the selection of the starting segmentation level for the hierarchical annotation process, to show its relevance for the final results. We experimentally validate our approach in three well-known datasets: CorelA, Stanford Background and MSRC-21 datasets. In these datasets, we achieved better or comparable results to other state-of-the-art algorithms, improving our base classifier in all cases.

Automatic Image Annotation based on Dense Weighted Regional Graph

Automatic image annotation refers to create text labels in accordance with images' context automatically. Although, numerous studies have been conducted in this area for the past decade, existence of multiple labels and semantic gap between these labels and visual low-level features reduced its performance accuracy. In this paper, we suggested an annotation method, based on dense weighted regional graph. In this method, clustering areas was done by forming a dense regional graph of area classification based on strong fuzzy feature vector in images with great precision, as by weighting edges in the graph, less important areas are removed over time and thus semantic gap between low-level features of image and human interpretation of high-level concepts reduces much more. To evaluate the proposed method, COREL database, with 5,000 samples have been used. The results of the images in this database, show acceptable performance of the proposed method in comparison to other methods.

Using manifold structure for automatic image annotation by fusion of multiple feature spaces

Automatic image annotation has been an active research topic in recent years. Low level features like as color, texture, shape as well as object spatial relations are extracted to represent images in general. These syntaxes are further used to retrieve images from large image data sets. However, the similarity of images could not be found correctly by similarity measures such as Euclidean distance in many situations. On the other hand, graph models have been shown powerful in solving many machine learning problems in recent years. In this paper, we propose a graph-based learning approach, named Conceptual Manifold Structure (CMS), based on transition from conceptual to observation space. In the proposed method, a graph including both the trained and tested samples is constructed by fusion of multiple feature spaces. Conceptual transition in graph structure is found by altering the edge values in an innovative manner. This is caused to learn the manifold structure where the samples dissimilarity is closer to the conceptual distance. Furthermore, the continuity between the instances of a semantic in the conceptual space is kept in feature space. Keeping the continuity in manifold structure is the main idea to decrease the semantic gap in this study. The experiments on different image data sets indicated that the geometrical distances between the samples on the manifold space are closer to their conceptual distance. The proposed method has been compared to other well-known approaches. The results confirmed the effectiveness and validity of the proposed method.

Multi-scale salient region and relevant visual keywords based model for automatic image annotation

Automatic image annotation is a vital and challenging problem in pattern recognition and image understanding areas. The existing models directly extract visual features from segmented image regions. Since segmented image regions may still have multi-objects, the extractive visual features may not effectively describe corresponding regions. In addition, existing models did not consider the visual representations of corresponding keywords, which would lead to appearing plenty of irrelevant annotations in final annotation results, and these annotations did not relate to any part of images considering visual contents. In order to overcome the above problems, an image annotation model based on multi-scale salient region and relevant visual keywords is proposed. In this model, each image is segmented by using multi-scale grid segmentation method and the global contrast based method is used to extract the saliency maps from each image region. Visual features are extracted from each salient region. In addition, each keyword is divided into two categories: abstract words or non-abstract words. Visual seeds of each non-abstract word are established, and then a new method is proposed to extract visual keyword collections by using corresponding seeds. According to the traits of abstract words, an algorithm based on subtraction regions is proposed to extract visual seeds and corresponding visual keyword collections of each abstract word. Adaptive parameter method and a fast solution algorithm are proposed to determine the similarity thresholds of each keyword. Finally, multi-scale visual features and the combinations of the above methods are used to improve the annotation performance. Our model can improve the object descriptions of images and image regions. Experimental results verify the effectiveness of the proposed model.

Semi-supervised learning for refining image annotation based on random walk model

Automatic image annotation has been an active research topic in recent years due to its potential impact on both image understanding and semantic based image retrieval. In this paper, we present a novel two-stage refining image annotation scheme based on Gaussian mixture model (GMM) and random walk method. To begin with, GMM is applied to estimate the posterior probabilities of each annotation keyword for the image, during which a semi-supervised learning, i.e. transductive support vector machine (TSVM), is employed to enhance the quality of training data. Next, a label similarity graph is constructed by a weighted linear combination of label similarity and visual similarity of images associated with the corresponding labels. In this way, it can seamlessly integrate the information from image low-level visual features and high-level semantic concepts. Followed by a random walk process over the constructed label graph is implemented to further mine the correlation of the candidate annotations so as to capture the refining results, which plays a crucial role in semantic based image retrieval. Finally, extensive experiments carried out on two publicly available image datasets bear out that this approach can achieve marked improvement in annotation performance over several state-of-the-art methods.

Automatic Image Annotation Based on Scene Analysis

Automatic image annotation is an important and challenging job for image analysis and understanding such as content-based image retrieval (CBIR). The relationship between the keywords and visual features is too complicated due to the semantic gap. We present an approach of automatic image annotation based on scene analysis. With the constrain of scene semantics, the correlation between keywords and visual features becomes simpler and clearer. Our model has two stages of process. The first stage is training process which groups training image data set into semantic scenes using the extracted semantic feature and visual scenes constructed from the calculation distances of visual features for every pairs of training images by using Earth mover's distance (EMD). Then, combine a pair of semantic and visual scene together and apply Gaussian mixture model (GMM) for all scenes. The second stage is to test and annotate keywords for test image data set. Using the visual features provided by Duygulu, experimental results show that our model outperforms probabilistic latent semantic analysis (PLSA) & GMM (PLSA&GMM) model on Corel5K database.

A Novel Semantic Statistical Model for Automatic Image Annotation Using the Relationship between the Regions Based on Multi-Criteria Decision Making

Automatic image annotation has emerged as an important research topic due to the existence of the semantic gap and in addition to its potential application on image retrieval and management. In this paper we present an approach which combines regional contexts and visual topics to automatic image annotation. Regional contexts model the relationship between the regions, whereas visual topics provide the global distribution of topics over an image. Conventional image annotation methods neglected the relationship between the regions in an image, while these regions are exactly explanation of the image semantics, therefore considering the relationship between them are helpful to annotate the images. The proposed model extracts regional contexts and visual topics from the image, and incorporates them by MCDM (Multi Criteria Decision Making) approach based on TOPSIS (Technique for Order Preference by Similarity to the Ideal Solution) method. Regional contexts and visual topics are learned by PLSA (Probability Latent Semantic Analysis) from the training data. The experiments on 5k Corel images show that integrating these two kinds of information is beneficial to image annotation. DOI: http://dx.doi.org/10.11591/ijece.v4i1.4598

Collaborative Similarity Metric Learning for Semantic Image Annotation and Retrieval

Automatic image annotation has become an increasingly important research topic owing to its key role in image retrieval. Simultaneously, it is highly challenging when facing to large-scale dataset with large variance. Practical approaches generally rely on similarity measures defined over images and multi-label prediction methods. More specifically, those approaches usually 1) leverage similarity measures predefined or learned by optimizing for ranking or annotation, which might be not adaptive enough to datasets; and 2) predict labels separately without taking the correlation of labels into account. In this paper, we propose a method for image annotation through collaborative similarity metric learning from dataset and modeling the label correlation of the dataset. The similarity metric is learned by simultaneously optimizing the 1) image ranking using structural SVM (SSVM), and 2) image annotation using correlated label propagation, with respect to the similarity metric. The learned similarity metric, fully exploiting the available information of datasets, would improve the two collaborative components, ranking and annotation, and sequentially the retrieval system itself. We evaluated the proposed method on Corel5k, Corel30k and EspGame databases. The results for annotation and retrieval show the competitive performance of the proposed method.

Objective-Guided Image Annotation

Automatic image annotation, which is usually formulated as a multi-label classification problem, is one of the major tools used to enhance the semantic understanding of web images. Many multimedia applications (e.g., tag-based image retrieval) can greatly benefit from image annotation. However, the insufficient performance of image annotation methods prevents these applications from being practical. On the other hand, specific measures are usually designed to evaluate how well one annotation method performs for a specific objective or application, but most image annotation methods do not consider optimization of these measures, so that they are inevitably trapped into suboptimal performance of these objective-specific measures. To address this issue, we first summarize a variety of objective-guided performance measures under a unified representation. Our analysis reveals that macro-averaging measures are very sensitive to infrequent keywords, and hamming measure is easily affected by skewed distributions. We then propose a unified multi-label learning framework, which directly optimizes a variety of objective-specific measures of multi-label learning tasks. Specifically, we first present a multilayer hierarchical structure of learning hypotheses for multi-label problems based on which a variety of loss functions with respect to objective-guided measures are defined. And then, we formulate these loss functions as relaxed surrogate functions and optimize them by structural SVMs. According to the analysis of various measures and the high time complexity of optimizing micro-averaging measures, in this paper, we focus on example-based measures that are tailor-made for image annotation tasks but are seldom explored in the literature. Experiments show consistency with the formal analysis on two widely used multi-label datasets, and demonstrate the superior performance of our proposed method over state-of-the-art baseline methods in terms of example-based measures on four image annotation datasets.

Supervised Semantic Image Annotation Using Region Relevance

Abstract Automatic image annotation is the key to semantic-based image retrieval. In this paper we formulate image annotation as a multi-class classification problem, which deals with the weak annotation problem and works with image-level ground truth training data. The relationship between low-level visual features and semantic concepts is found by supervised Bayesian learning. For each region in the test image, a posterior probability for each concept is calculated from class densities estimated from the training set and then the probability is modified using relevance with the other regions in the image. The image-level posterior probabilities are obtained by combining the regional posterior probabilities and keywords are selected according to their ranks. The proposed algorithm achieves good annotation performance on the Corel5K benchmark dataset.

Automatic image annotation method based on Gaussian mixture model

Automatic image annotation already becomes a feasible way to reduce "semantic gap". In order to improve the performance of automatic image annotation, Gaussian Mixture Model (GMM) based automatic image annotation method was proposed in this paper. GMM was built for each keyword to accurately characterize its semantic content. Simultaneously, the precise results can be obtained to promote the performance of image annotation. At last, based on the COREL database, the proposed method is verified to be effective in average precision and average recall.

A Hierarchical Image Annotation Method Based on SVM and Semi-supervised EM

Automatic image annotation, which aims at automatically identifying and then assigning semantic Keywords to the meaningful objects in a digital image, is not a very difficult task for human but has been regarded as a difficult and challenging problem to machines. In this paper, we present a hierarchical annotation scheme considering that generally human s visual identification to a scenery object is a rough-to-fine hierarchical process. First, the input image is segmented into multiple regions and each segmented region is roughly labeled with a general keyword using the multi-classification support vector machine. Since the results of rough annotation affect fine annotation directly, we construct the statistical contextual relationship to revise the improper labels and improve the accuracy of rough annotation. To obtain reasonable fine annotation for those roughly classified regions, we propose an active semi-supervised expectation-maximization algorithm, which can not only find the representative pattern of each fine class but also classify the roughly labeled regions into corresponded fine classes. Finally, the contextual relationship is applied again to revise the improper fine labels. To illustrate the effectiveness of the presented approaches, a prototype image annotation system is developed, the preliminary results of which showed that the hierarchical annotation scheme is effective.

Semi-Supervised Learning Model Based Efficient Image Annotation

Automatic image annotation is a promising way to achieve more effective image management and retrieval. However, system performances of the existing state-of-the-art keyword annotation schemes are often not so satisfactory. Therefore, image annotation refinement is crucial to improve the imprecise annotation results. In this paper, a novel approach is developed to automatically annotate image content by a semi-supervised learning model. With perceptual visual characteristics, the candidate annotations of unlabelled images are first obtained based on a progressive model. Then, a transducitive model, random walk with restart algorithm is used to refine these candidate annotations and the top ones are reserved as the final annotations. Experiments conducted on the typical Corel dataset show the effectiveness of the proposed approach.

Improved Resulted Word Counts Optimizer for Automatic Image Annotation Problem

Automatic Image Annotation is an important research topic in pattern recognition area. There are many different approaches to Automatic Image Annotation. In many of these approaches a key problem is determining correct word distributions in the generated annotations. Incorrect word distributions (word frequencies) results in large reduction of annotation quality. The paper presents a generic approach to find correct word frequencies. The approach may be used with many automatic image annotators, based on various machine learning paradigms. Proposed method is an improved version of our already presented method, called Greedy Resulted Word Counts Optimization. The key differences and novelties in the paper are: the concept of border support values and a method of optimal step selection in the optimization routine. Optimal step selection allows to reduce the number of computations during the optimization procedure, comparing to old Greedy Resulted Word Counts Optimization method.