Geo-Temporal Mining and Searching of Events from Web-based Image Collections

Abstract

The proliferation of Web- and social media-based photo-sharing applications have not only opened many possibilities but also resulted in new needs and challenges. They have resulted in a large amount of personal photos being available for public access. One of the most interesting characteristics of these data is that they are surrounded by 1) textual annotations, also called tags, which are intended to describe and categorize, by collective user efforts, the uploaded resources 2) temporal information referring to when a pictures has been taken and often by 3) a locational information describing where the picture has been taken. Despite the recent developments and technological advances in Web-based mediasharing applications, the continuously increasing amount of available information has made the access to the photos a demanding task. In general, we can address this challenge by allowing photo collections to be organized and browsed through the concept of events. We also believe most users are familiar with searching photo collections using events as starting points. Aiming at supporting the detection and search of event-related photos, this thesis proposes a novel framework for extracting pictures related to real-life events from a collection of Web-images by leveraging on their temporal geographical and textual annotations and comparing the proposed approach with existing related state-of-the-art approaches. Second, a set of geographical features is proposed describing the characteristics of the geographical profile of query terms deriving concepts from exploratory analysis. Third, the thesis provides two different tag-based search framework to improve the effectiveness of searching images related to events. The first framework is based on temporal and geographical proximity of query terms to the temporal neighbours of a given timestamped query, while the second framework is based on a novel machine-learning based query expansion method combining the heterogeneous textual, temporal and geographical similarity between query terms and candidate expansion terms for the selection of the expansion terms given a free text textual query. All the proposed methods have been evaluated by performing extensive experiments on real data gathered from media-sharing applications on the Web. Where possible, comparison with related techniques has been performed to reinforce the validity of the presented approaches. The proposed methods have shown promising results in both the extraction and clustering of event-related images and searching event-related pictures by using metrics from the state of the art. This doctoral work has been performed at the Department of Computer and Information Science at the Norwegian University of Science and Technology (NTNU) under advise of Associate Professor Heri Ramampiaro and with co-supervisors Associate Professor Roger Midtstraum and Associate Professor Randi Karlsen. Professor Ramesh Jain (University of California Irvine), Assistant Professor Claudia Hauff (TU Delft) and Associate Professor Trond Aalberg (NTNU) served as dissertation committee members. Available online at: http://www.idi.ntnu.no/research/doctor_theses/ruocco.pdf.