Annotation-retrieval reinforcement by visual cognition modeling on manifold

Abstract

While content-based image annotation and retrieval have been active research topics over the past decade, their correlation is not well exploited until recently. We argue that offline annotation and online retrieval models should be regarded as a unified long-term learner to reinforce each other. Based on this viewpoint, this paper presents an annotation-retrieval reinforcement framework, in which a dual-structured learning model is presented: the keyword-image association is learned by semantic manifold modeling while the semantic correlation between keywords is learned by Image-Word Net modeling. In keyword-image association modeling level, to effectively model keyword-image manifold, we present a manifold co-training algorithm to address the sample insufficiency problem. In manifold-based image annotation, we view annotation process as semantic feature reduction in keyword space, based on which a Biased Fisher Discriminant Analysis (BFDA) algorithm is presented for Eigen feature (keyword) selection. In semantic correlation modeling level, a novel Image-Word Net is learned from annotation training set and users' retrieval log for (1) irrelevant annotated keyword pruning; (2) semantic-level retrieval enhancement. In retrieval, our framework can effectively reveal user target by improving traditional content-based relevance feedback to linguistic-level interaction using annotation information, based on which BFDA is adopted for keyword selection. Finally, user interaction logs are adopted for manifold model updating and annotation refinement. As presented in our experiments, proposed method outperforms state-of-the-art annotation and retrieval algorithms in COREL Image databases with over 10,000 general-purpose images.