A semantic based video indexing and retrieval system for maritime surveillance

Abstract

Content-based video retrieval from archived image/video is a very attractive capability of modern intelligent video surveillance systems. This paper presents an innovative Semantic-Based Video Indexing and Retrieval (SBVIR) software toolkit to help users of intelligent video surveillance to easily and rapidly search the content of large video archives to conduct video-based forensic and image intelligence. Tailored for maritime environment, SBVIR is suited for surveillance applications in harbor, sea shores, or around ships. The system comprises two major modules: a video analytic module that performs automatic target detection, tracking, classification, activities recognition, and a retrieval module that performs data indexing, and information retrieval. SBVIR is capable of detecting and tracking objects from multiple cameras robustly in condition of dynamic water background and illumination changes. The system provides hierarchical target classification among a large ontology of watercraft classes, and is capable of recognizing a variety of boat activities. Video retrieval is achieved with both query-by-keyword and query-by-example. Users can query video content using semantic concepts selected from a large dictionary of objects and activities, display the history linked to a given target/activity, and search for anomalies. The user can interact with the system and provide feedbacks to tune the system for improved accuracy and relevance of retrieved data. SBVIR has been tested for real maritime surveillance scenarios and shown to be able to generate highly-semantic metadata tags that can be used during the retrieval to provide user with relevant and accurate data in real-time.