Temporally sorting images from real-world events

Abstract

• A data-driven method for chronologically sorting images of real-world events by analyzing the appearance of the scene. • The duration of the event is split into successive intervals and their relationships are modeled hierarchically. • An evaluation of several data augmentation techniques in scenarios with limited data availability. • A visualization of the concepts learned by the model to sort images offers additional insights about the problem. As smartphones become ubiquitous in modern life, every major event — from musical concerts to terrorist attempts — is massively captured by multiple devices and instantly uploaded to the Internet. Once shared through social media, the chronological order between available media pieces cannot be reliably recovered, hindering the understanding and reconstruction of that event. In this work, we propose data-driven methods for temporally sorting images originated from heterogeneous sources and captured from distinct angles, viewpoints, and moments. We model the chronological sorting task as an ensemble of binary classifiers whose answers are combined hierarchically to estimate an image’s temporal position within the duration of the event. We evaluate our method on images from the Notre-Dame Catedral fire and the Grenfell Tower fire events and discuss research challenges for analyzing data from real-world forensic events. Finally, we employ visualization techniques to understand what our models have learned, offering additional insights to the problem.