Design and Implementation for Interactive Augmented Broadcasting System

Abstract

In a typical augmented reality (AR), users can rotate, resize, or move AR objects, usually in application programs on handheld devices. In addition, users can instruct them to do an animation. Moreover, various types of application programs are provided and various ways of interactions are tried out for more natural user experience. On the other hand, in regular TV, an AR-applied TV program may be a composite video of original video and AR objects, which was already processed by the transmit side of broadcasting system and thus the TV viewer cannot influence the representation of TV program. Thus, until now, it can be said that an authentic AR service on TV broadcasting is not practically provided. In this paper, a new alternative paradigm of AR service in TV broadcasting is introduced. To provide an interactive AR experience to the viewer, first of all, we propose that the place where the original video and the augmented objects are composited needs to be moved from the transmit side to the TV terminal. Therefore, all augmented objects should be graphically rendered on the TV terminal. Considering the prerequisite, we propose an augmented broadcasting service system based on a hybrid framework that combines the conventional TV broadcasting and the broadband Internet. In the proposed system, the augmented objects are downloaded and rendered in compliance with instructions described in a metadata. The metadata includes the time information to synchronize the representation of the original video images and the rendered augmented-object images. And the metadata is delivered in compliance with MPEG-2 system. Inevitably, the architecture of the MPEG-2 transmission stream multiplexer should be slightly modified to insert the time information for the synchronously compositing. The metadata should be packetized in compliance with the MPEG-2 system. So, we propose a packetized elementary stream format for the metadata. Additionally, we propose an interaction mechanism using a second-screen device that supports touch-screen capability. With second-screen devices, TV viewers can manipulate the augmented objects displayed on TV screen. Moreover, a lot of additive contents synchronized or correlated to TV programs can be automatically popped up on the screen of second-screen device.