An automatic occlusion handling method in augmented reality

Abstract

PurposeTo make an augmented image realistic, the virtual objects should be correctly occluded by foreground objects. The purpose of this paper is to propose a new approach that resolves occlusion problems in augmented reality (AR). The main interest is that it can automatically obtain the proper spatial relationship between virtual and real objects in real time.Design/methodology/approachThe approach is divided into two steps: off‐line disparity map constructing and on‐line occlusion handling. In the off‐line stage, the disparity map of the real scene is constructed using the global stereo matching method prior and then the disparities are refined by means of the fast mean shift method. Since the depth values of objects in different positions are different, the real object that occludes the virtual object can be specified according to the depth value. In the on‐line stage, the contour of the specified object is tracked using the real time object tracking method with the combination of feature tracking method and minimum s‐t cut method. The augmented image with correct occlusions is produced by redrawing all the tracked object pixels on the augmented image.FindingsCompared with the existing methods, the proposed approach can automatically resolve occlusion problem in real time. The effectiveness of the method is demonstrated with several experimental results.Originality/valueThis paper makes three contributions. First, a novel framework is proposed to handle occlusion problem in AR. This framework is different to the previously proposed methods. The main procedure includes: obtain occluding real object, track the object, and redraw the pixels of the object on the composed image. It is much easier to implement and can achieve satisfactory results. Second, the disparity map is used to automatically obtain the contour of the occluding real object. To get the contour of the occluding real object precisely, the mean shift method is used to refine the disparity map. By comparing the depth value, the occluding real object can be extracted automatically. Third, the tracking method combining feature tracking method and minimum s‐t cut method ensures the real‐time requirement. The occlusion problem can be handled in real‐time.