Virtual Viewpoint Research Articles

View generation with 3D warping using depth information for FTV

In this paper, we propose a new method of depth-image-based rendering (DIBR) for free-viewpoint TV (FTV). In the conventional method, we estimated the depth of an object on the virtual image plane, which is called view-dependent depth estimation, and the virtual view images are rendered using the view-dependent depth map. In this method, virtual viewpoint images are rendered with 3D warping instead of estimating the view-dependent depth, since depth estimation is usually costly and it is desirable to eliminate it from the rendering process. However, 3D warping causes some problems that do not occur in the method with view-dependent depth estimation; for example, the appearance of holes on the rendered image, and the occurrence of depth discontinuity on the surface of the object at virtual image plane. Depth discontinuity causes artifacts on the rendered image. In this paper, these problems are solved by projecting depth map to the virtual image plane and performing post-filtering on the projected depth map. In the experiments, high-quality arbitrary viewpoint images were obtained by rendering images from relatively small number of cameras.

Virtual Viewpoint Replay for a Soccer Match by View Interpolation From Multiple Cameras

This paper presents a novel method for virtual view synthesis that allows viewers to virtually fly through real soccer scenes, which are captured by multiple cameras in a stadium. The proposed method generates images of arbitrary viewpoints by view interpolation of real camera images near the chosen viewpoints. In this method, cameras do not need to be strongly calibrated since projective geometry between cameras is employed for the interpolation. For avoiding the complex and unreliable process of 3-D recovery, object scenes are segmented into several regions according to the geometric property of the scene. Dense correspondence between real views, which is necessary for intermediate view generation, is automatically obtained by applying projective geometry to each region. By superimposing intermediate images for all regions, virtual views for the entire soccer scene are generated. The efforts for camera calibration are reduced and correspondence matching requires no manual operation; hence, the proposed method can be easily applied to dynamic events in a large space. An application for fly-through observations of soccer match replays is introduced along with the algorithm of view synthesis and experimental results. This is a new approach for providing arbitrary views of an entire dynamic event.

Stepwise approach for view synthesis

This paper presents some techniques for synthesizing novel view for a virtual viewpoint from two given views captured at different viewpoints to achieve both high quality and high efficiency. The whole process consists of three passes. The first pass recovers depth map. We formulate it as pixel labelling and propose a bisection approach to solve it. It is accomplished in log2 n (n is the number of depth levels) steps, each of which involves a single graph cut computation. The second pass detects occluded pixels and reasons about their depth. It fits a foreground depth curve and a background depth curve using depth of nearby foreground and background pixels, and then distinguishes foreground and background pixels by minimizing a global energy, which involves only one graph cut computation. The third pass finds for each pixel in the novel view the corresponding pixels in the input views and computes its color. The whole process involves only a small number of graph cut computations, therefore it is efficient. And, visual artifacts in the synthesized view can be removed successfully by correcting depth of the occluded pixels. Experimental results demonstrate that both high quality and high efficiency are achieved by the proposed techniques.

Active Rearranged Capturing of Image-Based Rendering Scenes—Theory and Practice

In this paper, we propose to capture image-based rendering scenes using a novel approach called active rearranged capturing (ARC). Given the total number of available cameras, ARC moves them strategically on the camera plane in order to minimize the sum of squared rendering errors for a given set of light rays to be rendered. Assuming the scene changes slowly so that the optimized camera locations are valid in the next time instance, we formulate the problem as a recursive weighted vector quantization problem, which can be solved efficiently. The ARC approach is verified on both synthetic and real-world scenes. In particular, a large self-reconfigurable camera array is built to demonstrate ARC's performance on real-world scenes. The system renders virtual views at 5-10 frames/s depending on the scene complexity on a moderately equipped computer. Given the virtual view point, the cameras move on a set of rails to perform ARC and improve the rendering quality on the fly

Realizing any central projection with a mirror pair

We show that, for any rotationally symmetric projection with a single virtual viewpoint, it is possible to design a two-mirror rotationally symmetric system that realizes the projection exactly. These mirror pairs are derived from two coupled differential equations. We give examples in which the projections from the sphere at infinity are stereographic, perspective, and equiresolution.

Image interpolation for virtual sports scenarios

View interpolation has been explored in the scientific community as a means to avoid the complexity of full 3D in the construction of photo-realistic interactive scenarios. EVENTS project attempts to apply state of the art view interpolation to the field of professional sports. The aim is to populate a wide scenario such as a stadium with a number of cameras and, via computer vision, to produce photo-realistic moving or static images from virtual viewpoints, i.e where there is no physical camera. EVENTS proposes an innovative view interpolation scheme based on the Joint View Triangulation algorithm developed by the project participants. Joint View Triangulation is combined within the EVENTS framework with new initiatives in the field of multiple view layered representation, automatic seed matching, image-based rendering, tracking occluding layers and constrained scene analysis. The computer vision software has been implemented on top of a novel high performance computing platform with the aim to achieve real-time interpolation.

View generation for a virtual camera using multiple depth maps

AbstractThis paper proposes a method of generation of images taken from a virtual viewpoint, based on depth information on the object when viewed from multiple positions. First, the depth map from a viewpoint is converted to the depth map of the object as viewed from a virtual point. Then, based on the depth map from another viewpoint, the occlusion area of the virtual viewpoint depth map is complemented and the depth value is corrected. By appropriate use of real images through interpolation of depth values and smoothing, a virtual viewpoint image is generated. Applying the proposed method makes it possible to generate a virtual viewpoint image with less occlusion, even for a viewpoint deviating greatly from the actual camera position. © 2004 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 87(10): 81–88, 2004; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjc.20110

多視点テニス画像からのプレーヤ視点画像の生成

We propose a new method for synthesizing player viewpoint images from multiple view tennis videos.Our method uses two key techniques : virtual-view synthesis and player's viewpoint estimation. In the former, we divide the object tennis scene into sub-regions, which are the background, tennis court ground, players, and ball. For each sub-region, a virtualviewpoint image is synthesized by considering the geometrical conditions of each region. Then the virtual viewpoint images of the sub-regions are merged into a virtual viewpoint image of the whole scene. In virtual viewpoint image generation, view interpolation, which is a way of synthesizing images at an intermediate viewpoint to two views of the real object, restricts the viewpoint to a position between the two views. To avoid this restriction, we propose our first key technique, whereby we are able to flexibly position the viewpoint. In our second technique, the viewpoint is computed using epipolar geometry from the center of gravity of a player. By applying the computed player's viewpoint to the former technique, we can synthesize player viewpoint images. Experimental results demonstrate that the proposed method can successfully provide a video of the tennis player's viewpoint from multiple-view video images.

Assessing the Visual Impact of Agroforestry Management with Landscape Design Software

In a case study in the Cévennes (France), various agroforestry management options have been simulated with the AMAP landscape design and visualization software. The terrain was described through a digital elevation model. Information concerning the farm structure was integrated in a geographic information system. The AMAP database provides accurate 3-dimensional plant architectural models, built according to the botanical concepts developed by the Montpellier Institute of Botany. Individual tree computer mock-ups were computed for each species present, at the desired ages, in order to simulate landscape evolution with time. Images of the scene were computed and observed on the screen from several virtual viewpoints. Several management options have been tested and visualized, providing a basis for discussion between partners concerning spatial organization, such as the layout of forest and agricultural plots.

Real time vision for intelligent vehicles

The authors first discuss the Intelligent StopandGo system, a sophisticated cruise control that autonomously follows a lead vehicle, pays attention to the relevant elements of the traffic infrastructure, and accounts for other traffic participants. The Intelligent StopandGo represents a multiyear effort by DaimlerChrysler to build a sophisticated cruise control that can function on highways, on secondary roads, and in urban environments. It combines the following capabilities: extracting lane boundaries, even when they are not clearly marked and do not contain the typical structure of highways; detecting a vehicle that can be followed, estimating ifs distance, speed, and acceleration; detecting stationary obstacles, such as parked cars, which limit the available free space; recognising the traffic signs and traffic lights that are relevant; and detecting and classifying additional traffic participants, such as pedestrians, who might cut in between the lead vehicle and the host vehicle. The DaimlerChrysIer demonstrator Urban Traffic Assistant (UTA) has devoted special attention to information, warning, and assistant functions in an inner-city environment. UTA is an E-class Mercedes-Bent containing sensors for longitudinal speed, longitudinal and lateral acceleration, yaw and pitch rate, and the sfeering wheel angle. It is equipped with a stereo black-and-white camera system as well as a color camera. UTA has access to throttle, brake, and steering. Furthermore, it displays the results of the perceptual modules in a graphical environment from either the driver's perspective or a virtual viewpoint. The computer hardware in UTA comprises three 400-MHz Linux/Pentium II (SMP) PCs for the perception of the environment and one Lynx/604e PowerPC for the control of sensors and actuators.

Multiview video sequence analysis, compression, and virtual viewpoint synthesis

This paper considers the problem of structure and motion estimation in multiview teleconferencing-type sequences and its application for video-sequence compression and intermediate-view generation. First, we introduce a new approach for structure estimation from a stereo pair acquired by two parallel cameras. It is based on a 2-D mesh representation of both views of the imaged scene and a parametrization of the structure information by the disparity between corresponding nodes in the image pair. Next, we describe a novel image alignment approach which can convert images captured using nonparallel cameras to coplanar-like images. This approach greatly eases the computational burden incurred by the nonparallel camera geometry, where one must consider both horizontal and vertical disparities. Finally, we present a coder for multiview sequences, which exploits the proposed alignment and structure estimation algorithm. By extracting the foreground objects and estimating the disparity field between a selected view and a reference view the coder can compress the image pair very efficiently. In the meantime, by using the coded structure information, the decoder can generate virtual viewpoints between decoded views, which can be very helpful for telepresence applications.

透視変換下における仮想視点からの画像合成の一手法

透視変換下における仮想視点からの画像合成の一手法

A generalized method for 3D object location from single 2D images

A method is presented to solve for the three-dimensional (3D) object position and orientation from single two-dimensional (2D) images based on a systematic hierarchy of features: primitive features, generalized features, and compound structures. A virtual viewpoint analysis shows that a feature-centered coordinate system allows us to drastically reduce the mathematical complexity of the inverse projection problem for higher order curves. At the heart of the matchings are the matching functions for primitive features, which are: point-to-point correspondence, line-to-line correspondence, and ellipse-to-ellipse correspondence. Parametric matching by generalized features is used to solve for viewpoints at any stage of matching. The generalized features can be viewed as the generalization of features used by previous researchers. The matching method goes “top-down” and then “bottom- up”, if necessary. Models of the objects are analyzed first. In the top-down process, search starts from the most complex features to the least complex features. In the bottom-up process, the least complex generalized features are used to solve for initial viewpoints. The viewpoints are further refined by matching the compound structures which are obtained by merging the consistent generalized features. The contributions include: (1) solving the problem of ellipse-to-ellipse correspondence, thus extending the class of recognizable objects; (2) the generalized features can assume various configurations, thus enabling the system to detect a multitude of features; (3) parametric matching instead of point-point matching makes the algorithm robust; (4) the top-down-bottom-up strategy combines efficiency with robustness.