Video browsing and retrieval based on multimodal integration

Abstract

The rapid growth of multimedia data requires more effective content-based video browsing and retrieval. We present a system developed for video browsing and retrieval based on multimedia integration. First, a basic structure of the system is defined. Second, a robust scene segmentation method is presented, which analyzes audio and visual information and accounts for their interrelations and coincidence to semantically identify video scenes. We then extract text from key frames with video OCR technique and extract text transcriptions by speech recognition to classify video scenes and form the full-text indices. Finally, natural language understanding technique is used to automatically classify video scenes on the basis of the texts obtained from close caption, video OCR process and speech recognition. In this way, we have developed the content-based video database system which integrates multimodality to browse and retrieve video data. The experimental results show that multimodal integration is effective for video scene segmentation. Our system built on the idea of multimodal integration makes content-based browsing and retrieval of video data, key-frame-based video abstract and search by keywords practical.