Modeling and rendering the invisibles and the impossibles from single images : a human-computer interaction approach

Abstract

A human being has a remarkable ability to make 3D connection on seeing 2D images. For example, although transparent objects are invisible to a large extent, we can mentally infer the 3D solid almost instantly and effortlessly. This ability, however, sometimes leads to interesting problems. M. C. Escher was a renowned artist who specialized in embedding impossible figures into architectural drawings. Impossible figure is a special kind of drawing consisting of multiple geometrically possible units connected by linear structures. When the figure is viewed as a whole, structural inconsistencies arise and confuse our visual perception. The theme of this thesis consists of an appearance-based computational framework for modeling the invisibles and the impossibles, that is transparent objects and impossible figures. We applied our appearance-based model in computer graphics, allowing for the first time the rendering of transparent objects in a new background scene, as well as the rendering of impossible figures at high frame rates, both without any tedious 3D modeling. The key in our approach lies in bringing the user into the modeling loop, i.e., human-computer interaction (HCI). Human-computer interaction, which can be understood as the supplying of a small amount of hints to help automatic computer algorithm to solve difficult problems, has recently gained a lot of attention in research in computer vision and interactive techniques. The main issue lies in improving computer's performance by user interaction using a simple interface. This thesis presents a HCI approach to model and render transparent objects (the invisibles) and impossible figures (the impossibles) which are traditionally very difficult in computer vision and graphics. In this thesis, we exploit our remarkable human visual system to provide prior knowledge, and propose an HCI approach to transfer such prior information in the form of a few simple hints via an easy and intuitive user interface. The computer algorithm then automatically performs the rest of the processing. We first transform the problem of transparent layer extraction into a soft-segmentation problem, where simple user's hints are available in the form of rough strokes drawn on the image. Then, with the aid of a human user who can easily recognize transparent objects given a single photo, we derive a practical and interactive approach to solve the problem of matting and compositing of transparent and refractive objects. Traditionally, these tasks can only be achieved using multiple images or 3D models. Finally, we present two approaches to model and render impossible figures, which is a long-standing problem in computer vision and computer graphics.