Interactive modelling and tracking for mixed and augmented reality

Abstract

Some tasks vital to many mixed and augmented reality systems are either too time consuming or complex to be carried out whilst the system is active. 3D scene modelling and labelling are two such tasks commonly performed by skilled operators in an off-line initialisation phase. Because this phase sometimes needs specialist software and/or expertise it can be a considerable limiting factor for new mixed reality system developers. If a mixed reality system is to operate in real-time, where artificial graphics are woven into real world live images, the way in which these off-line processes are tackled is critical. In this paper we propose a flexible new approach that reduces the time spent during the off-line initialisation phase by adopting an on-line interactive primitive modelling technique. Our solution combines two existing and freely available packages, the Augmented Reality Toolkit Plus (ARToolKitPlus) and the Mixed Reality Toolkit (MRT), to enable rapid interactive modelling over live video using a freely moving camera. As a demonstration we show how these can be used to rapidly seed an object appearance-based tracking algorithm.