Free Viewpoint Video Research Articles

3DTV view generation using uncalibrated pure rotating and zooming cameras

This paper proposes a novel method for synthesizing free viewpoint video captured by uncalibrated pure rotating and zooming cameras. Neither intrinsic nor extrinsic parameters of our cameras are known. Projective grid space (PGS), which is the 3D space defined by the epipolar geometry of two basis cameras, is employed for weak camera calibration. Trifocal tensors are used to relate non-basis cameras to PGS. Given trifocal tensors in the initial frame, our method automatically computes trifocal tensors in the other frames. Scale invariant feature transform (SIFT) is used for finding corresponding points in a natural scene between the initial frame and the other frames. Finally, free viewpoint video is synthesized based on the reconstructed visual hull. In the experimental results, free viewpoint video captured by uncalibrated hand-held cameras is successfully synthesized using the proposed method.

Objective quality assessment in free-viewpoint video production

This paper addresses the problem of objectively quantifying accuracy in free-viewpoint video production. Free-viewpoint video makes use of geometric scene reconstruction and renders novel views using the appearance sampled in multiple camera images. Previous work typically adopts an objective evaluation of geometric accuracy against ground-truth data or a subjective evaluation of visual quality in view synthesis. We consider two production scenarios, human performance capture in a highly constrained studio environment and sports production in a large-scale external environment. The accuracy of scene reconstruction is typically limited and absolute geometric accuracy does not necessarily reflect the quality of free-viewpoint rendering. A framework is introduced to quantify error at the point of view synthesis. The approach can be applied as a full-reference metric to measure fidelity to a ground-truth image or as a no-reference metric to measure the error in rendering. The framework is applied to a data set with known geometric accuracy and a comparison is presented for studio based and sports production scenarios.

View synthesis prediction for multiview video coding

We propose a rate-distortion-optimized framework that incorporates view synthesis for improved prediction in multiview video coding. In the proposed scheme, auxiliary information, including depth data, is encoded and used at the decoder to generate the view synthesis prediction data. The proposed method employs optimal mode decision including view synthesis prediction, and sub-pixel reference matching to improve prediction accuracy of the view synthesis prediction. Novel variants of the skip and direct modes are also presented, which infer the depth and correction vector information from neighboring blocks in a synthesized reference picture to reduce the bits needed for the view synthesis prediction mode. We demonstrate two multiview video coding scenarios in which view synthesis prediction is employed. In the first scenario, the goal is to improve the coding efficiency of multiview video where block-based depths and correction vectors are encoded by CABAC in a lossless manner on a macroblock basis. A variable block-size depth/motion search algorithm is described. Experimental results demonstrate that view synthesis prediction does provide some coding gains when combined with disparity-compensated prediction. In the second scenario, the goal is to use view synthesis prediction for reducing rate overhead incurred by transmitting depth maps for improved support of 3DTV and free-viewpoint video applications. It is assumed that the complete depth map for each view is encoded separately from the multiview video and used at the receiver to generate intermediate views. We utilize this information for view synthesis prediction to improve overall coding efficiency. Experimental results show that the rate overhead incurred by coding depth maps of varying quality could be offset by utilizing the proposed view synthesis prediction techniques to reduce the bitrate required for coding multiview video.

Color correction for multi-view video using energy minimization of view networks

Systems using numerous cameras are emerging in many fields due to their ease of production and reduced cost, and one of the fields where they are expected to be used more actively in the near future is in image-based rendering (IBR). Color correction between views is necessary to use multi-view systems in IBR to make audiences feel comfortable when views are switched or when a free viewpoint video is displayed. Color correction usually involves two steps: the first is to adjust camera parameters such as gain, brightness, and aperture before capture, and the second is to modify captured videos through image processing. This paper deals with the latter, which does not need a color pattern board. The proposed method uses scale invariant feature transform (SIFT) to detect correspondences, treats RGB channels independently, calculates lookup tables with an energy-minimization approach, and corrects captured video with these tables. The experimental results reveal that this approach works well.

On-line free-viewpoint video: From single to multiple view rendering

In recent years, many image-based rendering techniques have advanced from static to dynamic scenes and thus become video-based rendering (VBR) methods. But actually, only few of them can render new views on-line. We present a new VBR system that creates new views of a dynamic scene in live. This system provides high quality images and does not require any background subtraction. Our method follows a plane-sweep approach and reaches real-time rendering using consumer graphic hardware, graphics processing unit (GPU). Only one computer is used for both acquisition and rendering. The video stream acquisition is performed by at least 3 webcams. We propose an additional video stream management that extends the number of webcams to 10 or more. These considerations make our system low-cost and hence accessible for everyone. We also present an adaptation of our plane-sweep method to create simultaneously multiple views of the scene in real-time. Our system is especially designed for stereovision using autostereoscopic displays. The new views are computed from 4 webcams connected to a computer and are compressed in order to be transfered to the mobile phone. Due to GPU programming, our method provides up to 16 images of the scene in real-time. The use of both GPU and CPU makes this method work on only one consumer grade computer.

New color correction method of multi-view images for view rendering in free-viewpoint television

Color inconsistency between views is an important problem to be solved in multi-view video imaging application, such as free viewpoint television. Up to now, some color correction methods have been...

System architecture for free-viewpoint video and 3D-TV

This paper presents a system architecture of an acquisition, compression and rendering system for 3D- TV and free-viewpoint video applications. We show that the proposed system yields two distinct advantages. First, it achieves an efficient compression of 3D/multi-view video by extending a standard H.264 encoder such that near backward compatibility is retained. Second, the proposed system can efficiently compress both 3D-TV and free- viewpoint multi-view video datasets using the single proposed system architecture.

Multi-epipolar lines matching-based ray-space interpolation for Free Viewpoint Video system

Ray-space based arbitrary viewpoint rendering without complex object segmentation or model construction is the main technology to realize Free Viewpoint Video (FVV) system for complex scenes. Ray-space interpolation and compression are two key techniques for the solution. In this paper, correlation among multiple epipolar lines in ray-space data is analyzed, and a new method of ray-space interpolation with multi-epipolar lines matching is proposed. Comparing with the pixel-based matching interpolation method and the block-based matching interpolation method, the proposed method can achieve higher Peak Signal to Noise Ratio (PSNR) in interpolating ray-space data and rendering arbitrary viewpoint images.

GPU-Based Optimization of a Free-Viewpoint Video System

We present a method for optimizing the reconstruction and rendering of 3D objects from multiple images by utilizing the latest features of consumer-level graphics hardware based on shader model 4.0. We accelerate visual hull reconstruction by rewriting a shape-from-silhouette algorithm to execute on the GPU's parallel architecture. Rendering is optimized through the application of geometry shaders to generate billboarding micr facets textured with captured images. We also present a method for handling occlusion in the camera selection process that is optimized for execution on the GPU. Execution time is further improved by rendering intermediate results directly to texture to minimize the number of data transfers between graphics and main memory. We show our GPU based system to be significantly more efficient than a purely CPUbased approach, due to the parallel nature of the GPU, while maintaining graphical quality.

FTV (Free viewpoint TV) and Creation of Ray-Based Image Engineering

Free viewpoint TV (FTV) enables us to view a distant 3D world by freely changing our viewpoints as if we were there. FTV will bring an epochal change in the history of television since this function has not yet been achieved by conventional TV technology. We propose the concept of FTV and veri¯ed its feasibility using the world's ¯rst real-time system including the complete chain of operation from image capture to display. The international standardization of FTV technologies is now underway. FTV is a ray based system rather than a pixel-based system. We are creating ray-based image engineering through the development of FTV.

基于视差矢量信息的多视点视频颜色校正方法

Color inconsistency is an urgent problem to be solved in free viewpoint television. In this letter, a new color correction method is proposed by using disparity vector information. At first, we separate foreground and background from the scene with a method of mean-removed disparity estimation. Then the correction parameters are estimated by adopting linear fitting for foreground and background regions, respectively. Next, with expectation-maximization algorithm, we integrate correction parameters of foreground and background to get the final corrected image. Finally, video tracking technique is performed to correct multi-view video. Experimental results show that the proposed method is quite effective.

Occlusion-Aware View Interpolation

View interpolation is an essential step in content preparation for multiview 3D displays, free-viewpoint video, and multiview image/video compression. It is performed by establishing a correspondence among views, followed by interpolation using the corresponding intensities. However, occlusions pose a significant challenge, especially if few input images are available. In this paper, we identify challenges related to disparity estimation and view interpolation in presence of occlusions. We then propose an occlusion-aware intermediate view interpolation algorithm that uses four input images to handle the disappearing areas. The algorithm consists of three steps. First, all pixels in view to be computed are classified in terms of their visibility in the input images. Then, disparity for each pixel is estimated from different image pairs depending on the computed visibility map. Finally, luminance/color of each pixel is adaptively interpolated from an image pair selected by its visibility label. Extensive experimental results show striking improvements in interpolated image quality over occlusion-unaware interpolation from two images and very significant gains over occlusion-aware spline-based reconstruction from four images, both on synthetic and real images. Although improvements are obvious only in the vicinity of object boundaries, this should be useful in high-quality 3D applications, such as digital 3D cinema and ultra-high resolution multiview autostereoscopic displays, where distortions at depth discontinuities are highly objectionable, especially if they vary with viewpoint change.

High-Quality Reconstruction from Multiview Video Streams

Three-dimensional (3-D) video processing is currently an active area of research that attracts scientists from many disciplines, including computer graphics, computer vision, electrical engineering, and video processing. They join their expertise to attack the very hard problem of reconstructing dynamic representations of real- world scenes from a sparse set of synchronized video streams. To put this idea into practice, a variety of challenging engineering and algorithmic problems have to be efficiently solved, ranging from acquisition, over reconstruction in itself, to realistic rendering. This article is a tutorial style review of methods from the literature aiming at reconstruction of 3-D humans as well as of a variety of model-based approaches that we developed to reconstruct, render, and encode free-viewpoint videos of human actors. We will show that the commitment to an a priori shape representation of a person in the real world allows us to solve many of the previously described reconstruction problems in an efficient way.

Applicability of Camera Works to Free Viewpoint Videos with Annotation and Planning

This paper shows the effectiveness of a cinematographic camera for controlling 3D video by measuring its effects on viewers with several typical camera works. 3D free-viewpoint video allows us to set its virtual camera on arbitrary positions and postures in 3D space. However, there have been neither investigations on adaptability nor on dependencies between the camera parameters of the virtual camera (i.e., positions, postures, and transitions) nor the impressions of viewers. Although camera works on 3D video based on expertise seems important for making intuitively understandable video, it has not yet been considered. When applying camera works to 3D video using the planning techniques proposed in previous research, generating ideal output video is difficult because it may include defects due to image resolution limitation, calculation errors, or occlusions as well as others caused by positioning errors of the virtual camera in the planning process. Therefore, we conducted an experiment with 29 subjects with camera-worked 3D videos created using simple annotation and planning techniques to determine the virtual camera parameters. The first point of the experiment examines the effects of defects on viewer impressions. To measure such impressions, we conducted a semantic differential (SD) test. Comparisons between ground truth and 3D videos with planned camera works show that the present defects of camera work do not significantly affect viewers. The experiment's second point examines whether the cameras controlled by planning and annotations affected the subjects with intentional direction. For this purpose, we conducted a factor analysis for the SD test answers whose results indicate that the proposed virtual camera control, which exploits annotation and planning techniques, allows us to realize camera working direction on 3D video.

2-2 局所領域分割を用いた自由視点画像生成における縦方向視点移動方法(第2部門 自由視点・3次元画像)

2-2 局所領域分割を用いた自由視点画像生成における縦方向視点移動方法(第2部門 自由視点・3次元画像)

A Free-Viewpoint Video System for Visualization of Sport Scenes

This paper introduces a new approach to using multicamera images of an event to provide a free-viewpoint video visualization of sport scenes. This allows a virtual camera to be moved freely around, and to view the action from any angle. The system is based on 3-D reconstruction techniques previously developed for studio use. A number of different 3-D representations, including billboards, visual hulls, and view-dependent geometry are evaluated for the purpose. First results show the potential of the new approach for action replay and strategy analysis of sport scenes. Current limitations of the 3-D representations are discussed in the context of a practical use of the system.

A Framework for Representation and Processing of Multi-view Video Using the Concept of Layered Depth Image

The multi-view video is a collection of multiple videos, capturing the same scene at different viewpoints. Since it contains more affluent information than a single video, it can be applied to various applications, such as 3DTV, free viewpoint TV, surveillance, sports matches, and so on. However, the data size of the multi-view video linearly increases as the number of cameras, therefore it is necessary to develop an effective framework to represent, process, and transmit those huge amounts of data. In recent, multi-view video coding is getting lots of attention as efficient video coding technologies are being developed. Although most of multi-view video coding algorithms are based on the state-of-the-art H.264/AVC video coding technology, they do not utilize rich 3-D information. In this paper, we propose a new framework using the concept of layered depth image (LDI), one of the efficient image-based rendering techniques, to efficiently represent and process multi-view video data. We describe how to represent natural multi-view video based on the LDI approach and the overall framework to process those converted data.

A perceptive remote-control 3D TV interface

3D TV is a turning point in the history of television. Here, technologies from computer graphics, computer vision, multimedia, telecommunications, broadcasting, and other related fields merge to expand the sensations provided by watching conventional 2D TV.1 As a result, there has been much research on multi-view image processing and 3D display.2, 3 However, most of the previous studies have focused on acquiring and displaying images while optimizing their ‘naturalness’, simplicity of computation, and real-time operation. Free-viewpoint TV (FTV)4 takes this idea forward. Free-viewpoint systems allow a viewer to look at a scene from the perspective that they want rather than what a director chooses to provide.1 In order to generate a scene according to the user’s requirements, a position in 3D space has to be defined as the view point. To achieve this, we suggest an intelligent approach that automatically generates reference viewpoints based on a theory of human visual attention.Many studies of visual attention and eye movements have shown that humans generally only attend to a few areas in an image rather than scanning the whole, and visual attention models provide a general approach to controlling the activities of active vision systems.5 These models of selective visual attention have been suggested on the basis of evidence in psychology, psychophysics, physiology, etc., and can also be exploited for our purposes. Figure 1 shows an examples of how our artificial human visual attention system can be used to obtain a basis position to generate a scene that corresponds to a user’s viewpoint. The system also simultaneously computes the intensity of salience of an object in a given region. In fact, several objects or regions can be detected in terms of their their individual saliencies, which can in turn be used to describe them. The system computes early visual features from a set of pre-attentive feature maps in a massively parallel way. Activity from all feature maps is combined Figure 1. Schematic diagram showing the proposed system based on a bottom-up approach to attention.

Free viewpoint video synthesis and presentation from multiple sporting videos

AbstractThis paper presents a new framework for observation of sporting events using HMD by applying the method of free viewpoint video synthesis for dynamic events to mixed reality. According to the viewpoint position of an observer, a virtual view image is generated by view interpolation among multiple sporting videos captured in a stadium, and then overlaid onto the real world via HMD. This makes the observer feel as if the match is played in front of his/her eyes. The proposed method performs virtual view synthesis and geometric registration between the real world and the virtual view images using projective geometry between cameras, which can be estimated from correspondence of natural feature points. It does not require calibration of multiple cameras imaging the sporting match and HMD camera capturing the real world. In this paper, we have constructed a mixed reality presentation system for a soccer match in order to demonstrate the utility of the proposed method. This system can overlay soccer scenes captured with multiple cameras in a soccer stadium onto a desktop field model with HMD. © 2006 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 90(2): 40–49, 2007; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjc.20311

Rate-distortion-optimized predictive compression of dynamic 3D mesh sequences

Compression of computer graphics data such as static and dynamic 3D meshes has received significant attention in recent years, since new applications require transmission over channels and storage on media with limited capacity. This includes pure graphics applications (virtual reality, games) as well as 3DTV and free viewpoint video. Efficient compression algorithms have been developed first for static 3D meshes, and later for dynamic 3D meshes and animations. Standard formats are available for instance in MPEG-4 3D mesh compression for static meshes, and Interpolator Compression for the animation part. For some important types of 3D objects, e.g. human head or body models, facial and body animation parameters have been introduced. Recent results for compression of general dynamic meshes have shown that the statistical dependencies within a mesh sequence can be exploited well by predictive coding approaches. Coders introduced so far use experimentally determined or heuristic thresholds for tuning the algorithms. In video coding, rate-distortion (RD) optimization is often used to avoid fixed thresholds and to select the optimum prediction mode. We applied these ideas and present here an RD-optimized dynamic 3D mesh coder. It includes different prediction modes as well as an RD cost computation that controls the mode selection across all possible spatial partitions of a mesh to find the clustering structure together with the associated prediction modes. The general coding structure is derived from statistical analysis of mesh sequences and exploits temporal as well as spatial mesh dependencies. To evaluate the coding efficiency of the developed coder, comparative coding results for mesh sequences at different resolutions were carried out.