Spatial-Temporal Semantic Grouping of Instructional Video Content

Abstract

This paper presents a new approach for content analysis and summarization of instructional videos of blackboard presentations. We first use low-level image processing techniques to segment frames into content regions, regions occluded by instructors, and irrelevant areas, then measure the number of chalk pixels in the content areas of each frame. Using the number of chalk pixels as heuristic measurement of video content, we derive a content figure which describes the actual rather than apparent fluctuation of video content. By searching for local maxima in the content figure, and by detecting camera motions and tracking movements of instructors, we can then define and retrieve key frames. Since some video content may not be contained in any one of the key frames due to occlusion by instructors or camera motion, we use an image registration method to make board content that are free of occlusions and not bound by frame boundaries. Extracted key frames and content images are combined together to summarize and index the video. We further introduce the concept of semantic teaching unit, which is defined as a more natural temporalspatial unit of teaching content. We propose a model to detect teaching units, based on the recognition of actions of instructors, and on the measurement of temporal duration and spatial location of content. We demonstrate experiments on instructional videos which are taken in non-instrumented classrooms, and show examples of the construction of content images and the detection of teaching units within them.