Recognition and prediction of manipulation actions using Enriched Semantic Event Chains

Abstract

Human activity understanding has attracted much attention in recent years, because it plays a key role in a wide range of applications such as human–computer interfaces, visual surveillance, video indexing, intelligent humanoids robots, ambient intelligence and more. Activity understanding strongly benefits from fast, predictive action recognition. Here we present a new prediction algorithm for manipulation action classes in natural scenes. Manipulations are first represented by their temporal sequence of changing static and dynamic spatial relations between the objects that take part in the manipulation. This creates a transition matrix, called “Enriched Semantic Event Chain (ESEC)”. We use these ESECs to classify and predict a large set of manipulations. We find that manipulations can be correctly predicted after only (on average) 45% of their total execution time and that we are almost twice as fast as a standard HMM-based method used for comparison.