A method and real-time implementation for annotating broadcast video images based on alignment with a set of panoramic images

Abstract

In this paper we propose a panorama-based annotation scheme for superimposing annotation information on broadcast video images by using a set of panoramic images, annotation information for each image and the adjacency relations between these images as information sources. The proposed method aligns an input video sequence with a set of panoramic reference images and displays annotation information on those parts of the video image that correspond to locations annotated in these panoramic images. The method first aligns locations on each of the panoramic images with those in the input video sequence and then selects that panoramic image that was taken from the position that is closest to that of the input image and uses this as a reference for annotation. The method uses mean squared error (MSE) of pixel intensities between the two images as a criterion for alignment. Annotation information is then displayed based on the alignment results with the selected panoramic image. In order to display annotation information appropriately when the camera viewpoint moves, we perform alignment of the input image with the panoramic images that are adjacent to the reference image in parallel, and switch over to another panoramic image taking it as the new reference image if this gives better alignment results than the current reference image. By monitoring the progress of such changeovers between images we can estimate the trajectory of the camera and its position. We have implemented the proposed method on a computer system and obtained favorable results regarding its effectiveness and the suitability of the method to real-time processing. © 2006 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 89(11): 51–60, 2006; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjc.20282