Automatic Image Annotation Research Articles

An Enhanced Approach for Solving Class Imbalance Problem in Automatic Image Annotation

An Enhanced Approach for Solving Class Imbalance Problem in Automatic Image Annotation

Automatic face image annotation based on a single template with constrained warping deformation

In this study, an automatic face image annotation method is proposed by aligning faces with different expressions to an annotated neutral face. This work is useful in reducing tedious manual work for labelling image data in large databases. However, it is challenging because of the appearance variations caused by non‐rigid face deformations under various expressions. Unlike some conventional approaches acquiring sufficient image templates to model the query appearance, only a single given template is necessary for the proposed method. The authors address the problem through dense image alignment. Specifically, image warping in the alignment process is constrained by prior knowledge about facial shape deformation. The proposed method is independent of the appearance model, and is available for unseen faces. In addition, to initialise warping parameters, the authors present a robust patch‐based estimation method. Context information for feature points is carefully modelled to propagate the searching path for local patch matching. The face annotation experiments are performed on some large expressions, with noisy image qualities and in low image resolutions. Comparison results with conventional methods demonstrate the proposed method's superiority on both accuracy and robustness.

Mining User Queries with Markov Chains: Application to Online Image Retrieval

We propose a novel method for automatic annotation, indexing and annotation-based retrieval of images. The new method, that we call Markovian Semantic Indexing (MSI), is presented in the context of an online image retrieval system. Assuming such a system, the users' queries are used to construct an Aggregate Markov Chain (AMC) through which the relevance between the keywords seen by the system is defined. The users' queries are also used to automatically annotate the images. A stochastic distance between images, based on their annotation and the keyword relevance captured in the AMC, is then introduced. Geometric interpretations of the proposed distance are provided and its relation to a clustering in the keyword space is investigated. By means of a new measure of Markovian state similarity, the mean first cross passage time (CPT), optimality properties of the proposed distance are proved. Images are modeled as points in a vector space and their similarity is measured with MSI. The new method is shown to possess certain theoretical advantages and also to achieve better Precision versus Recall results when compared to Latent Semantic Indexing (LSI) and probabilistic Latent Semantic Indexing (pLSI) methods in Annotation-Based Image Retrieval (ABIR) tasks.

FILTERING TERMS FROM THE WEB FOR IMAGE ANNOTATIONS

In this paper, we propose a novel automatic image annotation model by mining the web. In our approach, the terms or words appearing in the associated text are extracted and filtered as labels or annotations for the corresponding web images. Sure, much noise exists in those selected labels. In order to reduce the influence caused by the noisy labels, for each label or potential word, we improve web image-word relationships using Mixture Gaussian Distribution Model. By doing so, the relationships between words and images are re-weighted both in terms of sematic relevance and in terms of visual feature similarity. In fact, all the words associated to an image are not semantically independent. We use co-occurrences between two words to describe their semantic relevance. Thus, we further use a method, called Word Promotion, to co-enhance the weights of all the words associated to a given image based on their co-occurrences. Our experiments are conducted in several ways and the results show that our annotation method can achieve a satisfactory performance in respects of system scalability and sematic evolution.

Semantic, Automatic Image Annotation Based on Multi-Layered Active Contours and Decision Trees

In this paper, we propose a new approach for automatic image annotation (AIA) in order to automatically and efficiently assign linguistic concepts to visual data such as digital images, based on both numeric and semantic features. The presented method first computes multi-layered active contours. The first-layer active contour corresponds to the main object or foreground, while the next-layers active contours delineate the object’s subparts. Then, visual features are extracted within the regions segmented by these active contours and are mapped into semantic notions. Next, decision trees are trained based on these attributes, and the image is semantically annotated using the resulting decision rules. Experiments carried out on several standards datasets have demonstrated the reliability and the computational effectiveness of our AIA system.

Automatic semantic annotation by using fuzzy theory for natural images

Nowadays, more and more digital images are available. How to find a required image quickly for a user has become harder and harder. Acquiring semantic information of images and realising automatic image annotation is an effective technology to improve the performance of image retrieval. This paper presents a method of sentiment annotation of natural images based on fuzzy theory. The method describes image emotional level by computing fuzzy membership degree, uses BP neural network to implement it and solves semantic ambiguity on automatic image annotation. Using 967 natural images downloaded by Baidu photo channel to train and test, experiments achieved good effect compared with manual annotation results. The proposed method can lay a good foundation for automatic semantic annotation of more types of images.

Color, texture and shape descriptor fusion with Bayesian network classifier for automatic image annotation

Due to the large amounts of multimedia data prevalent on the Web, Some images presents textural motifs while others may be recognized with colors or shapes of their content. The use of descriptors based on one’s features extraction method, such as color or texture or shape, for automatic image annotation are not efficient in some situations or in absence of the chosen type. The proposed approach is to use a fusion of some efficient color, texture and shape descriptors with Bayesian networks classifier to allow automatic annotation of different image types. This document provides an automatic image annotation that merges some descriptors in a parallel manner to have a vector that represents the various types of image characteristics. This allows increasing the rate and accuracy of the annotation system. The Texture, color histograms, and Legendre moments, are used and merged respectively together in parallel as color, texture and shape features extraction methods, with Bayesian network classifier, to annotate the image content with the appropriate keywords. The accuracy of the proposed approach is supported by the good experimental results obtained from ETH-80 databases.

User Perspectives on Spatial Relationships in Describing Images

Image retrieval is stimulating activities and new approach is essential to improve the inference of semantic relationship of image annotation for low-level feature, enhance semantic retrieval and bridging the Semantic Gap. Current research on automatic image annotation pays little attention to the spatial relationships between objects in images; thus fail to provide and extract the relation information. Spatial relationships could enrich the semantic description of images and enhances the power and precision of queries in retrieval. This paper discusses work done in order to identify and select a set of specific spatial relationships terms to be used for further experiments in automate the spatial extractions in images. The spatial relationships are considered fuzzy and usually depend on human interpretation. A preliminary study: an online Image Description Survey is developed consisting ten Corel Dataset images to discover and perceive human aspects for describing images using spatial terms. The survey is implemented using PHP and published online for public to respond. Analysis of the result found that there are 45 spatial terms used by the users in describing the relations of objects in the images, and 28 terms occurred more than once. Further analysis and discussion focus on these 28 terms. The most commonly used is spatial term above for relative relationships with frequency occurrence of 120, followed by spatial term bottom for absolute relationships with frequency occurrence of 57. The terms frequency of each spatial relationships is also discussed in a form of correlations matrix for detecting significant relations of spatial terms used between images. There are a number of different ways how each of the users used spatial terminology in describing an image and analyzing these responses is quite challenging. The study show a set of frequently used spatial terms. These terms and their reciprocal shall be considered for next stage of algorithms development in computing and extracting the spatial terms automatically from images to enhance the capability of the retrieval system as well as for meeting the needs and requirements of users in the future.

Graph-based semi-supervised learning with multi-modality propagation for large-scale image datasets

Semi-supervised learning (SSL) is widely-used to explore the vast amount of unlabeled data in the world. Over the decade, graph-based SSL becomes popular in automatic image annotation due to its power of learning globally based on local similarity. However, recent studies have shown that the emergence of large-scale datasets challenges traditional methods. On the other hand, most previous works have concentrated on single-label annotation, which may not describe image contents well. To remedy the deficiencies, this paper proposes a new graph-based SSL technique with multi-label propagation, leveraging the distributed computing power of the MapReduce programming model. For high learning performance, the paper further presents both a multi-layer learning structure and a tag refinement approach, where the former unifies both visual and textual information of image data during learning, while the latter simultaneously suppresses noisy tags and emphasizes the other tags after learning. Experimental results based on a medium-scale and a large-scale image datasets show the effectiveness of the proposed methods.

Objective-Guided Image Annotation

Automatic image annotation, which is usually formulated as a multi-label classification problem, is one of the major tools used to enhance the semantic understanding of web images. Many multimedia applications (e.g., tag-based image retrieval) can greatly benefit from image annotation. However, the insufficient performance of image annotation methods prevents these applications from being practical. On the other hand, specific measures are usually designed to evaluate how well one annotation method performs for a specific objective or application, but most image annotation methods do not consider optimization of these measures, so that they are inevitably trapped into suboptimal performance of these objective-specific measures. To address this issue, we first summarize a variety of objective-guided performance measures under a unified representation. Our analysis reveals that macro-averaging measures are very sensitive to infrequent keywords, and hamming measure is easily affected by skewed distributions. We then propose a unified multi-label learning framework, which directly optimizes a variety of objective-specific measures of multi-label learning tasks. Specifically, we first present a multilayer hierarchical structure of learning hypotheses for multi-label problems based on which a variety of loss functions with respect to objective-guided measures are defined. And then, we formulate these loss functions as relaxed surrogate functions and optimize them by structural SVMs. According to the analysis of various measures and the high time complexity of optimizing micro-averaging measures, in this paper, we focus on example-based measures that are tailor-made for image annotation tasks but are seldom explored in the literature. Experiments show consistency with the formal analysis on two widely used multi-label datasets, and demonstrate the superior performance of our proposed method over state-of-the-art baseline methods in terms of example-based measures on four image annotation datasets.

Intuitive Image Database Browsing to Support Image Annotation

Image annotation is typically performed manually since automatic image annotation approaches have not matured yet to be used in practice. Consequently, image annotation is a labour intensive and time consuming task. In this paper, we show how an image browsing system can be employed to support efficient and effective (manual) annotation of image databases. In contrast to other approaches, which typically present images in a linear fashion, we employ a visualisation where images are arranged by mutual visual similarity. Since in this arrangement similar images are close to each other, they can easily be selected and annotated together. Organisation on a grid layout prevents image overlap and thus contributes to a clear presentation. Large image databases are handled through a hierarchical data structure where each image in the visualisation can correspond to a cluster of images that can be expanded by the user. Experimental results indicate that annotation can be performed faster on our proposed system.

Training inter-related classifiers for automatic image classification and annotation

A structural learning algorithm is developed in this paper to achieve more effective training of large numbers of inter-related classifiers for supporting large-scale image classification and annotation. A visual concept network is constructed for characterizing the inter-concept visual correlations intuitively and determining the inter-related learning tasks automatically in the visual feature space rather than in the label space. By partitioning large numbers of object classes and image concepts into a set of groups according to their inter-concept visual correlations, the object classes and image concepts in the same group will share similar visual properties and their classifiers are strongly inter-related while the object classes and image concepts in different groups will contain various visual properties and their classifiers can be trained independently. By leveraging the inter-concept visual correlations for inter-related classifier training, our structural learning algorithm can train the inter-related classifiers jointly rather than independently, which can enhance their discrimination power significantly. Our experiments have also provided very positive results on large-scale image classification and annotation.

Real web community based automatic image annotation

Automatic image annotation is a challenging problem in pattern recognition. Large annotation databases are difficult to build, and the existing models can only label small scale of image sets. In order to solve the problems with the annotation of large databases, a web community based image annotation model is proposed. Firstly, an algorithm for effectively selecting training image is proposed to delete noise images from training dataset. Secondly, a method based on weighted KNN is proposed to assign weights on each image in nearest neighbor collections, which taking into account the impacts of different images in nearest neighbor collections. By this way, training images that are more similar to unlabeled image will have higher confidences in the annotation propagation process. Thirdly, an annotation refinement method based on WordNet level is proposed to improve the annotation results of non-abstract words. Our model is appropriate for annotating images from large-scale real web community. Experiments conducted on large-scale datasets verify the effectiveness of the proposed model.

Automatic image annotation and semantic based image retrieval for medical domain

Automatic image annotation is the process of assigning meaningful words to an image taking into account its content. This process is of great interest as it allows indexing, retrieving, and understanding of large collections of image data. This paper presents a system used in the medical domain for three distinct tasks: image annotation, semantic based image retrieval and content based image retrieval. An original image segmentation algorithm based on a hexagonal structure was used to perform the segmentation of medical images. Image's regions are described using a vocabulary of blobs generated from image features using the K-means clustering algorithm. The annotation and semantic based retrieval task is evaluated for two annotation models: Cross Media Relevance Model and Continuous-space Relevance Model. Semantic based image retrieval is performed using the methods provided by the annotation models. The ontology used by the annotation process was created in an original manner starting from the information content provided by the Medical Subject Headings (MeSH). The experiments were made using a database containing color images retrieved from medical domain using an endoscope and related to digestive diseases.

Automated Image Annotation for Semantic Indexing and Retrieval of Medical Images

Medical image retrieval to search for clinically relevant and visually similar images depicting suspecious lesions have been attracting research interest. Content-based image retrieval (CBIR) is an important alternate and complement to traditional text-based retrieval using keywords. We have implemented CBIR system based on effective use of texture information within the images obtained by statistical cooccurrence matrix method. Also, the method is improved by bridging the semantic gap between low-level visual features and the high-level semantic concepts using automated image annotations. In this paper, we have proposed a classification-based multi-class multi-label semantic model and the corresponding learning procedure to address the problem of automatic image annotation using J48 decision tree classifier and show its application to medical image retrieval. Hash structure is used to index images. Eucledian distance measure is used for similarity measurement. Both the methods are compared using precision and recall measures. Semantic indexing is shown to outperform CBIR for MR-T2 axial brain images.

Borehole and Ice Feature Annotation Tool (BIFAT): A program for the automatic and manual annotation of glacier borehole images

We present a stand-alone software platform, Borehole and Ice Feature Annotation Tool (BIFAT), for the examination, annotation and analysis of glacier borehole images. This tool aids in the lengthy and often subjective process of annotating layers and other features in optical and acoustic glacier borehole televiewer logs. Since these view 360° around the full circumference of an exploratory borehole, intersecting planes are reconstructed on the televiewer image as sinusoids, the amplitude and phase of which can be used to calculate, respectively, the dip and direction of dip of each of these planes. The program aids in the annotation and examination of such planes, as well as a number of other features, including clusters and inclusions. BIFAT also provides an automatic layer detection option to aid and speed up the often lengthy process of identifying planar features in glacier borehole images. The software also shows promising results in rock borehole images and in the detection of planar layers in line-scan ice core images. We describe the capability and operation of BIFAT, and illustrate its application with reference to the automatic identification and annotation of sections from an optical televiewer (OPTV) borehole log from Roi Baudouin, Dronning Maud Land, Antarctica. The software is freely available for download from http://users.aber.ac.uk/byh/iceoptv.

AUTOMATIC IMAGE ANNOTATION BASED ON SEMI-SUPERVISED CLUSTERING AND MEMBERSHIP-BASED CROSS MEDIA RELEVANCE MODEL

In this paper, we propose a system for automatic image annotation that has two main components. The first component consists of a novel semi-supervised possibilistic clustering and feature weighting algorithm based on robust modeling of the generalized Dirichlet (GD) finite mixture. This algorithm is used to group image regions into prototypical region clusters that summarize the training data and can be used as the basis of annotating new test images. The constraints consist of pairs of image regions that should not be included in the same cluster. These constraints are deduced from the irrelevance of all concepts annotating the training images to help in guiding the clustering process. The second component of our system consists of a probabilistic model that relies on the possibilistic membership degrees, generated by the clustering algorithm, to annotate unlabeled images. The proposed system was implemented and tested on a data set that include thousands of images using four-fold cross validation.

A Novel JSVM Approach for Automatic Image Annotation and Retrieval

This paper presents a novel image annotation framework for domains with large numbers of images. Automatic image annotation is such a domain, by which a computer system automatically assigns metadata in the form of captioning or keywords to a digital image. This application of computer vision technique is used in image retrieval system to organize and locate images of interest from a database. Many techniques have been proposed for image annotation in the last decade that has given reasonable performance on standard datasets In this work, we propose a new model for image annotation known as JSVM which treats annotation as a retrieval problem. In this work, we introduce an JSVM model for image annotation that treats annotation as a retrieval problem. The proposed technique utilizes low level image features and a simple combination of basic distances using JEC to find the nearest neighbors of a given image; the keywords are then assigned using SVM approach which aims to explore the combination of three different methods. First, the initial annotation of the data using flat wise and axis wise methods, and that takes the hierarchy into consideration by classifying consecutively its instances through position wise method. Finally, we make use of pair wise majority voting between methods by simply summing strings in order to produce a final annotation. The result of the proposed technique shows that this technique outperforms the current state of art methods on the standard datasets.

Web Image Annotation Via Subspace-Sparsity Collaborated Feature Selection

The number of web images has been explosively growing due to the development of network and storage technology. These images make up a large amount of current multimedia data and are closely related to our daily life. To efficiently browse, retrieve and organize the web images, numerous approaches have been proposed. Since the semantic concepts of the images can be indicated by label information, automatic image annotation becomes one effective technique for image management tasks. Most existing annotation methods use image features that are often noisy and redundant. Hence, feature selection can be exploited for a more precise and compact representation of the images, thus improving the annotation performance. In this paper, we propose a novel feature selection method and apply it to automatic image annotation. There are two appealing properties of our method. First, it can jointly select the most relevant features from all the data points by using a sparsity-based model. Second, it can uncover the shared subspace of original features, which is beneficial for multi-label learning. To solve the objective function of our method, we propose an efficient iterative algorithm. Extensive experiments are performed on large image databases that are collected from the web. The experimental results together with the theoretical analysis have validated the effectiveness of our method for feature selection, thus demonstrating its feasibility of being applied to web image annotation.

Assessing size, abundance and habitat preferences of the Ocean Perch Helicolenus percoides using a AUV-borne stereo camera system

Traditional fishery resource assessment methods using trawl gear are unable to sample rocky substratum and are prone to underestimating the biomass of species having partial or strong association with rocky reefs. This study successfully used an Autonomous Underwater Vehicle (AUV) and image-yielding methods to estimate size, abundance and habitat preference of an abundant and commercially important ‘rockfish’ – the Ocean Perch (Helicolenus percoides) – in rocky habitats on the continental shelf off Tasmania, SE Australia. More than half (53%) of the Ocean Perch observed were photogrammetrically measured with known accuracy using a stereo camera system yielding length-frequency distributions. Observations of juvenile and adult H. percoides across a depth gradient showed that adults preferred rocky substrates over soft substrates, whereas juveniles preferred soft substrate over hard substrate. We found a positive relationship between rockfish abundance and increasing depth in most habitat types. These results demonstrate the utility of image-based methods for determining size composition and habitat preferences of some reef-associated species. However, there is scope to improve image-based methods using length estimation procedures that enable higher proportions of individuals to be measured (compared to the proportion achieved in this study), and by incorporating automated image annotation to decrease image analysis times, particularly when examining species/habitat relationships. The importance of analytical procedures that account for autocorrelation in non-independent image data on habitats and associated species is discussed. We conclude that rapidly maturing image-based observational methods have potential utility in complementing fishery stock assessments of some reef-associated species. Image-based methods are also well-suited to simultaneously provide additional quantitative measures of benthic habitats, invertebrate fauna and fishery environments.