Chapter 12 - Self-supervised temporal event segmentation inspired by cognitive theories

Abstract

We consider the event segmentation problem, i.e., how to mark the event's temporal boundaries and spatially locate them in the image. We draw heavily from cognitive science research to define the problem of event segmentation and to design highly effective computer vision algorithms for spatio-temporal segmentation of events in videos. These approaches do not require any annotated data. They can process streaming video data while learning robust representations for segmenting events. First, we introduce the Event Segmentation Theory (EST) model based on the perceptual prediction model to compute the event boundaries proposed by Zacks et al. (Radvansky and Zacks, 2014). Then we present our computer vision solutions based on the perceptual prediction model from EST in three progressive versions: temporal segmentation using perceptual prediction framework, temporal segmentation along with event working models based on attention maps, and finally spatial and temporal localization of events.