Image-based Scene Representations Research Articles

Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax.

We propose a novel 360° scene representation for converting real scenes into stereoscopic 3D virtual reality content with head-motion parallax. Our image-based scene representation enables efficient synthesis of novel views with six degrees-of-freedom (6-DoF) by fusing motion fields at two scales: (1) disparity motion fields carry implicit depth information and are robustly estimated from multiple laterally displaced auxiliary viewpoints, and (2) pairwise motion fields enable real-time flow-based blending, which improves the visual fidelity of results by minimizing ghosting and view transition artifacts. Based on our scene representation, we present an end-to-end system that captures real scenes with a robotic camera arm, processes the recorded data, and finally renders the scene in a head-mounted display in real time (more than 40 Hz). Our approach is the first to support head-motion parallax when viewing real 360° scenes. We demonstrate compelling results that illustrate the enhanced visual experience - and hence sense of immersion-achieved with our approach compared to widely-used stereoscopic panoramas.

Depth-of-field rendering with multiview synthesis

We present a GPU-based real-time rendering method that simulates high-quality depth-of-field effects, similar in quality to multiview accumulation methods. Most real-time approaches have difficulties to obtain good approximations of visibility and view-dependent shading due to the use of a single view image. Our method also avoids the multiple rendering of a scene, but can approximate different views by relying on a layered image-based scene representation. We present several performance and quality improvements, such as early culling, approximate cone tracing, and jittered sampling. Our method achieves artifact-free results for complex scenes and reasonable depth-of-field blur in real time.

RDTC Optimized Compression of Image-Based Scene Representations (Part II): Practical Coding

Interactive streaming of compressed image-based scene representations requires random access to the reference image data. The degree of interframe dependencies exploited during encoding has an impact on the transmission and decoding time and, at the same time, delimits the (storage) rate-distortion (RD) tradeoff that can be achieved. The transmission data rate and the decoding complexity at the client have received attention in the literature, but their incorporation into the optimization procedure for compression and streaming is missing. If scenario-specific measures are considered, the traditional RD optimization can be extended to a tradeoff between the (storage) rate (R), distortion (D), transmission data rate (T), and decoding complexity (C). In the first part of this sequel of papers, we have theoretically analyzed the RDTC space for the compression of densely sampled image-based scene representations. In this second part, we consider practical RDTC optimization. We propose a modeling and encoding parameter selection procedure that allows us to adapt the compression to scenario-specific properties. The impact of client side caching is considered and evaluated using an experimental testbed. Our results show a significant reduction of the user perceived delay, memory consumption or required minimum channel and storage bitrate for RDTC optimized streams compared to classical RD optimized or independently encoded scene representations.

RDTC Optimized Compression of Image-Based Scene Representations (Part I): Modeling and Theoretical Analysis

Rendering of virtual views in interactive streaming of compressed image-based scene representations requires random access to arbitrary parts of the reference image data. The degree of interframe dependencies exploited during encoding has an impact on the transmission and decoding time and, at the same time, delimits the (storage) rate-distortion (RD) tradeoff that can be achieved. In this work, we extend the classical RD optimization approach using hybrid video coding concepts to a tradeoff between the storage rate (R), distortion (D), transmission data rate (T), and decoding complexity (C). We present a theoretical model for this RDTC space with a focus on the decoding complexity and, in addition, the impact of client side caching on the RDTC measures is considered and evaluated. Experimental results qualitatively match those predicted by our theoretical models and show that an adaptation of the encoding process to scenario specific parameters like computational power of the receiver and channel throughput can significantly reduce the user-perceived delay or required storage for RDTC optimized streams compared to RD optimized or independently encoded scene representations.

RAYSET: A TAXONOMY FOR IMAGE-BASED RENDERING

A new concept, referred to as rayset, is discussed in this paper. It is a parametric function consisting of two mapping relations. The first one maps from a parameter space to the ray space, while the second one maps from the parameter space to the attribute space. A taxonomy is proposed based on the rayset concept whereby scene representations and scene reconstruction techniques used in image-based rendering are individually classified. Existing image-based rendering techniques are surveyed under the proposed classification. The review shows that different image-based scene representations, such as multiple-center-of-projection image and concentric mosaics, can be cast as different kinds of raysets. Different scene reconstruction approaches can be regarded as attempts to render different raysets. The concepts of rayset warping and rayset editing are also formulated. Under the rayset taxonomy, both techniques try to alter one of the mapping relations defined by a rayset without changing the other one.

Selection of reference imaged for image-based scene representations

In computer vision, an attention has been devoted to image-based scene representations. They allow to construct an arbitrary view of a rigid 3-D scene using transfer from two real 2-D reference images, rather than by rendering an explicit 3-D model. We focus on selecting the optimal set of reference images from a large set of images representing the scene. The selected set minimizes a weighted sum of the number of reference views and the total fit error. We propose two different algorithms for solving this optimization problem. The experimental results on synthetic and real data indicate the feasibility of the approach for one-parameter camera motion. The possibility is shown to extend one of the algorithms for more general cases.

Plenoptic Image Editing

This paper presents a new class of interactive image editing operations designed to maintain consistency between multiple images of a physical 3D scene. The distinguishing feature of these operations is that edits to any one image propagate automatically to all other images as if the (unknown) 3D scene had itself been modified. The modified scene can then be viewed interactively from any other camera viewpoint and under different scene illuminations. The approach is useful first as a power-assist that enables a user to quickly modify many images by editing just a few, and second as a means for constructing and editing image-based scene representations by manipulating a set of photographs. The approach works by extending operations like image painting, scissoring, and morphing so that they alter a scene's plenoptic function in a physically-consistent way, thereby affecting scene appearance from all viewpoints simultaneously. A key element in realizing these operations is a new volumetric decomposition technique for reconstructing an scene's plenoptic function from an incomplete set of camera viewpoints.

3-D scene reconstruction with viewpoint adaptation on stereo displays

We propose a generic algorithm for the geometrically correct reconstruction of 3-D scenes on stereo displays with viewpoint adaptation. This forms the basis of multiviewpoint systems, which are currently the most promising candidates for real-time implementations of 3-D visual communication systems. The reconstruction algorithm needs 3-D tracking of the viewers' eyes with respect to the display. We analyze the effect of eye-tracking errors. A simple bound is derived, below which reconstruction errors cannot be observed. We design a multiviewpoint system using a recently introduced image-based scene representation. The design formed the basis of the real-time multiviewpoint system that was recently built in the European PANORAMA project. Experiments with both natural and synthetic scenes show that the proposed reconstruction algorithm performs well. The experiments are performed by computer simulations and the real-time PANORAMA system.

Trifocal motion modeling for object-based video compression and manipulation

Following an overview of two-dimensional (2-D) parametric motion models commonly used in video manipulation and compression, we introduce trifocal transfer, which is an image-based scene representation used in computer vision, as a motion compensation method that uses three frames at a time to implicitly capture camera/scene motion and scene depth. Trifocal transfer requires a trifocal tensor that is computed by matching image features across three views and a dense correspondence between two of the three views. We propose approximating the dense correspondence between two of the three views by a parametric model in order to apply the trifocal transfer for object-based video compression and background mosaic generation. Backward, forward, and bidirectional motion compensation methods based on trifocal transfer are presented. The performance of the proposed motion compensation approaches using the trifocal model has been compared with various other compensation methods, such as dense motion, block motion, and global affine transform on several video sequences. Finally, video compression and mosaic synthesis based on the trifocal motion model are implemented within the MPEG-4 Video Verification Model (VM), and the results are compared with those of the standard MPEG-4 video VM. Experimental results show that the trifocal motion model is superior to block and affine models when there is depth variation and camera translation.