Effective semantic classification of consumer events for automatic content management

Abstract

We study semantic event classification in the consumer domain by incorporating cross-domain and within-domain learning. An event is defined as a set of photos and/or videos that are taken within a common period of time, and have similar visual appearance. Events are generated from unconstrained consumer photo and video collections, by an automatic content management system, e.g., an automatic albuming system. Such consumer events have the following characteristics: an event can contain both photos and videos; there usually exist noisy/erroneous images resulting from imperfect albuming; and event data taken by different users, although from the same semantic category, can have highly diverse visual content. To accommodate these characteristics, we develop a general two-step Event-Level Feature (ELF) learning framework that enables the use of external data sources by cross-domain learning and the use of region-level representations, to enhance classification. Specifically, in the first step an elementary-level feature is used to represent images and videos. Then in the second step an ELF is constructed on top of the elementary feature to model each event as a feature vector. Semantic event classifiers can be directly built based on the ELF. Various ELFs are generated from different types of elementary-level features by using both cross-domain and within-domain learning: cross-domain approaches use two sets of concept scores at both image and region level that are learned from two external data sources; within-domain approaches use low-level visual features at both image and region level. Different types of ELFs complement each other for improved classification. Experiments over a large real consumer data set confirm significant improvements, e.g., over 90% MAP gain compared to the previous semantic event classification method.