Identifying an efficient and robust sub-shot segmentation method for home movie summarisation

Abstract

With the increasing volumes of home video footage and the need for effectively managing such archives, home movie summarisation has become an important and key research topic in the recent past. Despite growing interest from the research community, automatic summarisation remains a challenging research problem due to unrestricted capture and lack of storyline present in the home video content. In this paper, we address a sub-problem of the broad summarisation problem, namely sub-shot segmentation, as an important content analysis step in home movie summarisation. More specifically, we conduct an experimental study to identify an efficient and robust sub-shot segmentation method through a performance comparison of a number of optical flow field computation methods. Formulating the task of sub-shot segmentation as a process of decomposing a raw video into different video segments in line with the change in dominant camera motion, we are interested in detecting the most useful camera motions we have observed in real collections, namely pan, tilt and zoom. To this end, a number of feature-based optical flow field computation methods, including two well-known local feature descriptors called SIFT and SURF, and the Pyramidal Lucas-Kanade (PLK) local feature tracker, are investigated as potentially useful approaches for facilitating efficient and robust sub-shot segmentation in this paper. We then compare the performance of those algorithms with 5 different block matching motion estimation algorithms. Simulation results show that the PLK algorithm outperforms the SIFT and SURF algorithms in terms of both accuracy and efficiency. Furthermore, despite its slightly lower accuracy compared to that of the most optimal block matching algorithms (BMA), we conclude that a local feature tracker like the PLK algorithm is the preferred choice for sub-shot detection in real-life home movie archives due to the trade-off it provides between accuracy and efficiency.