Image Annotation Refinement Research Articles

The Image Annotation Refinement in Embedding Feature Space based on Mutual Information

The ever-increasing size of images has made automatic image annotation one of the most important tasks in the fields of machine learning and computer vision. Despite continuous efforts in inventing new annotation algorithms and new models, results of the state-of-the-art image annotation methods are often unsatisfactory. In this paper, to further improve annotation refinement performance, a novel approach based on weighted mutual information to automatically refine the original annotations of images is proposed. Unlike the traditional refinement model using only visual feature, the proposed model use semantic embedding to properly map labels and visual features to a meaningful semantic space. To accurately measure the relevance between the particular image and its original annotations, the proposed model utilize all available information including image-to-image, label-to-label and image-to-label. Experimental results conducted on three typical datasets show not only the validity of the refinement, but also the superiority of the proposed algorithm over existing ones. The improvement largely benefits from our proposed mutual information method and utilizing all available information.

Semantic Concept Network and Deep Walk-based Visual Question Answering

Visual Question Answering (VQA) is a hot-spot in the intersection of computer vision and natural language processing research and its progress has enabled many in high-level applications. This work aims to describe a novel VQA model based on semantic concept network construction and deep walk. Extracting visual image semantic representation is a significant and effective method for spanning the semantic gap. Moreover, current research has shown that co-occurrence patterns of concepts can enhance semantic representation. This work is motivated by the challenge that semantic concepts have complex interrelations and the relationships are similar to a network. Therefore, we construct a semantic concept network adopted by leveraging Word Activation Forces (WAFs), and mine the co-occurrence patterns of semantic concepts using deep walk. Then the model performs polynomial logistic regression on the basis of the extracted deep walk vector along with the visual image feature and question feature. The proposed model effectively integrates visual and semantic features of the image and natural language question. The experimental results show that our algorithm outperforms competitive baselines on three benchmark image QA datasets. Furthermore, through experiments in image annotation refinement and semantic analysis on pre-labeled LabelMe dataset, we test and verify the effectiveness of our constructed concept network for mining concept co-occurrence patterns, sensible concept clusters, and hierarchies.

Image annotation refinement via 2P-KNN based group sparse reconstruction

Image annotation aims at predicting labels that can accurately describe the semantic information of images. In the past few years, many methods have been proposed to solve the image annotation problem. However, the predicted labels of the images by these methods are usually incomplete, insufficient and noisy, which is unsatisfactory. In this paper, we propose a new method denoted as 2PKNN-GSR (Group Sparse Reconstruction) for image annotation and label refinement. First, we get the predicted labels of the testing images using the traditional method, i.e., a two-step variant of the classical K-nearest neighbor algorithm, called 2PKNN. Then, according to the obtained labels, we divide the K nearest neighbors of an image in the training images into several groups. Finally, we utilize the group sparse reconstruction algorithm to refine the annotated label results which are obtained in the first step. Experimental results on three standard datasets, i.e., Corel 5K, IAPR TC12 and ESP Game, show the superior performance of the proposed method compared with the state-of-the-art methods.

Semi-Supervised Learning Model Based Efficient Image Annotation

Automatic image annotation is a promising way to achieve more effective image management and retrieval. However, system performances of the existing state-of-the-art keyword annotation schemes are often not so satisfactory. Therefore, image annotation refinement is crucial to improve the imprecise annotation results. In this paper, a novel approach is developed to automatically annotate image content by a semi-supervised learning model. With perceptual visual characteristics, the candidate annotations of unlabelled images are first obtained based on a progressive model. Then, a transducitive model, random walk with restart algorithm is used to refine these candidate annotations and the top ones are reserved as the final annotations. Experiments conducted on the typical Corel dataset show the effectiveness of the proposed approach.

Knowledge Based Image Annotation Refinement

Recently, images on the Web and personal computers are prevalent around the human's life. To retrieve effectively these images, there are many (Automatic Image Annotation) AIA algorithms. However, it still suffers from low-level accuracy since it couldn't overcome the semantic-gap between low-level features (`color', `texture' and `shape') and high-level semantic meanings (e.g., `sky', `beach'). Namely, AIA techniques annotates images with many noisy keywords. In this paper, we propose a novel approach that augments the classical model with generic knowledge-based, WordNet. Our novel approach strives to prune irrelevant keywords by the usage of WordNet. To identify irrelevant keywords, we investigate various semantic similarity measures between keywords and finally fuse outcomes of all these measures together to make a final decision using Dempster-Shafer evidence combination. Furthermore, We can re-formulate the removal of erroneous keywords from image annotation problem into graph-partitioning problem, which is weighted MAX-CUT problem. It is possible that we have too many candidate keywords for web-images. Hence, we need to have deterministic polynomial time algorithm for MAX-CUT problem. We show that finding optimal solution for removing noisy keywords in the graph is NP-Complete problem and propose a new methodology for Knowledge Based Image Annotation Refinement (KBIAR) using a deterministic polynomial time algorithm, namely, randomized approximation graph algorithm. Finally, we demonstrate the superiority of this algorithm over traditional one including the most recent work for a benchmark dataset.