Nearby Viewpoints Research Articles

Quantifiable Fine-Grain Occlusion Removal Assistance for Efficient VR Exploration.

This article presents an occlusion management approach that handles fine-grain occlusions, and that quantifies and localizes occlusions as a user explores a virtual environment (VE). Fine-grain occlusions are handled by finding the VE region where they occur, and by constructing a multiperspective visualization that lets the user explore the region from the current location, with intuitive head motions, without first having to walk to the region. VE geometry close to the user is rendered conventionally, from the user's viewpoint, to anchor the user, avoiding disorientation and simulator sickness. Given a viewpoint, residual occlusions are quantified and localized as VE voxels that cannot be seen from the given viewpoint but that can be seen from nearby viewpoints. This residual occlusion quantification and localization helps the user ascertain that a VE region has been explored exhaustively. The occlusion management approach was tested in three controlled studies, which confirmed the exploration efficiency benefit of the approach, and in perceptual experiments, which confirmed that exploration efficiency does not come at the cost of reducing spatial awareness and sense of presence, or of increasing simulator sickness.

Automatic structuring of organic shapes from a single drawing

Complex vector drawings serve as convenient and expressive visual representations, but they remain difficult to edit or manipulate. For clean-line vector drawings of smooth organic shapes, we describe a method to automatically extract a layered structure for the drawn object from the current or nearby viewpoints. The layers correspond to salient regions of the drawing, which are often naturally associated to ‘parts’ of the underlying shape. We present a method that automatically extracts salient structure, organized as parts with relative depth orderings, from clean-line vector drawings of smooth organic shapes. Our method handles drawings that contain complex internal contours with T-junctions indicative of occlusions, as well as internal curves that may either be expressive strokes or substructures. To extract the structure, we introduce a new part-aware metric for complex 2D drawings, the radial variation metric, which is used to identify salient parts. These parts are then considered in a priority-ordered fashion, which enables us to identify and recursively process new shape parts while keeping track of their relative depth ordering. The output is represented in terms of scalable vector graphics layers, thereby enabling meaningful sketch editing and manipulation. We evaluate the method on multiple input drawings and show that the structure we compute is convenient for subsequent posing and animation from nearby viewpoints. Lastly, to further demonstrate the usefulness of our system, we develop two direct applications, namely the creation of cardboard articulated puppets and part-based 3D modeling from a single sketch.

Non-redundant rendering for efficient multi-view scene discretization

A powerful approach for managing scene complexity is to sample the scene with a set of images. However, conventional images from nearby viewpoints have a high level of redundancy, which reduces scene sampling efficiency. We present non-redundant rendering, which detects and avoids redundant samples as the image is computed. We show that non-redundant rendering leads to improved scene sampling quality according to several view-independent and view-dependent metrics, compared to conventional scene discretization using redundant images and compared to depth peeling. Non-redundant images have a higher degree of fragmentation and, therefore, conventional approaches for scene reconstruction from samples are ineffective. We present a novel reconstruction approach that is well suited to scene discretization by non-redundant rendering. Finally, we apply non-redundant rendering and scene reconstruction techniques to soft shadow rendering where we show that our approach has an accuracy advantage over conventional images and over depth peeling.

Encoder-Driven Inpainting Strategy in Multiview Video Compression.

In free viewpoint video systems, a user has the freedom to select a virtual view from which an image of the 3D scene is rendered, and the scene is commonly represented by color and depth images of multiple nearby viewpoints. In such representation, there exists data redundancy across multiple dimensions: 1) a 3D voxel may be represented by pixels in multiple viewpoint images (inter-view redundancy); 2) a pixel patch may recur in a distant spatial region of the same image due to self-similarity (inter-patch redundancy); and 3) pixels in a local spatial region tend to be similar (inter-pixel redundancy). It is important to exploit these redundancies during inter-view prediction toward effective multiview video compression. In this paper, we propose an encoder-driven inpainting strategy for inter-view predictive coding, where explicit instructions are transmitted minimally, and the decoder is left to independently recover remaining missing data via inpainting, resulting in lower coding overhead. In particular, after pixels in a reference view are projected to a target view via depth-image-based rendering at the decoder, the remaining holes in the target view are filled via an inpainting process in a block-by-block manner. First, blocks are ordered in terms of difficulty-to-inpaint by the decoder. Then, explicit instructions are only sent for the reconstruction of the most difficult blocks. In particular, the missing pixels are explicitly coded via a graph Fourier transform or a sparsification procedure using discrete cosine transform, leading to low coding cost. For blocks that are easy to inpaint, the decoder independently completes missing pixels via template-based inpainting. We apply our proposed scheme to frames in a prediction structure defined by JCT-3V where inter-view prediction is dominant, and experimentally we show that our scheme achieves up to 3-dB gain in peak-signal-to-noise-ratio in reconstructed image quality over a comparable 3D-High Efficiency Video Coding implementation using fixed 16 $\times $ 16 block size.

ConstructAide

We describe a set of tools for analyzing, visualizing, and assessing architectural/construction progress with unordered photo collections and 3D building models. With our interface, a user guides the registration of the model in one of the images, and our system automatically computes the alignment for the rest of the photos using a novel Structure-from-Motion (SfM) technique; images with nearby viewpoints are also brought into alignment with each other. After aligning the photo(s) and model(s), our system allows a user, such as a project manager or facility owner, to explore the construction site seamlessly in time, monitor the progress of construction, assess errors and deviations, and create photorealistic architectural visualizations. These interactions are facilitated by automatic reasoning performed by our system: static and dynamic occlusions are removed automatically, rendering information is collected, and semantic selection tools help guide user input. We also demonstrate that our user-assisted SfM method outperforms existing techniques on both real-world construction data and established multi-view datasets.

Stereo vision–based depth of field rendering on a mobile device

The depth of field (DoF) effect is a useful tool in photography and cinematography because of its aesthetic value. However, capturing and displaying dynamic DoF effect were until recently a quality unique to expensive and bulky movie cameras. A computational approach to generate realistic DoF effects for mobile devices such as tablets is proposed. We first calibrate the rear-facing stereo cameras and rectify the stereo image pairs through FCam API, then generate a low-res disparity map using graph cuts stereo matching and subsequently upsample it via joint bilateral upsampling. Next, we generate a synthetic light field by warping the raw color image to nearby viewpoints, according to the corresponding values in the upsampled high-resolution disparity map. Finally, we render dynamic DoF effect on the tablet screen with light field rendering. The user can easily capture and generate desired DoF effects with arbitrary aperture sizes or focal depths using the tablet only, with no additional hardware or software required. The system has been examined in a variety of environments with satisfactory results, according to the subjective evaluation tests.

Seamless View Synthesis Through Texture Optimization

In this paper, we present a novel view synthesis method named Visto, which uses a reference input view to generate synthesized views in nearby viewpoints. We formulate the problem as a joint optimization of inter-view texture and depth map similarity, a framework that is significantly different from other traditional approaches. As such, Visto tends to implicitly inherit the image characteristics from the reference view without the explicit use of image priors or texture modeling. Visto assumes that each patch is available in both the synthesized and reference views and thus can be applied to the common area between the two views but not the out-of-region area at the border of the synthesized view. Visto uses a Gauss–Seidel-like iterative approach to minimize the energy function. Simulation results suggest that Visto can generate seamless virtual views and outperform other state-of-the-art methods.

Racking focus and tracking focus on live video streams: a stereo solution

The ability to produce dynamic Depth of Field effects in live video streams was until recently a quality unique to movie cameras. In this paper, we present a computational camera solution coupled with real-time GPU processing to produce runtime dynamic Depth of Field effects. We first construct a hybrid-resolution stereo camera with a high-res/low-res camera pair. We recover a low-res disparity map of the scene using GPU-based Belief Propagation, and subsequently upsample it via fast Cross/Joint Bilateral Upsampling. With the recovered high-resolution disparity map, we warp the high-resolution video stream to nearby viewpoints to synthesize a light field toward the scene. We exploit parallel processing and atomic operations on the GPU to resolve visibility when multiple pixels warp to the same image location. Finally, we generate racking focus and tracking focus effects from the synthesized light field rendering. All processing stages are mapped onto NVIDIA's CUDA architecture. Our system can produce racking and tracking focus effects for the resolution of 640×480 at 15 fps.

Content-preserving warps for 3D video stabilization

We describe a technique that transforms a video from a hand-held video camera so that it appears as if it were taken with a directed camera motion. Our method adjusts the video to appear as if it were taken from nearby viewpoints, allowing 3D camera movements to be simulated. By aiming only for perceptual plausibility, rather than accurate reconstruction, we are able to develop algorithms that can effectively recreate dynamic scenes from a single source video. Our technique first recovers the original 3D camera motion and a sparse set of 3D, static scene points using an off-the-shelf structure-from-motion system. Then, a desired camera path is computed either automatically (e.g., by fitting a linear or quadratic path) or interactively. Finally, our technique performs a least-squares optimization that computes a spatially-varying warp from each input video frame into an output frame. The warp is computed to both follow the sparse displacements suggested by the recovered 3D structure, and avoid deforming the content in the video frame. Our experiments on stabilizing challenging videos of dynamic scenes demonstrate the effectiveness of our technique.

GENERALIZATION OF THE DESARGUES THEOREM FOR SPARSE 3D RECONSTRUCTION

Visual perception for walking machines needs to handle more degrees of freedom than for wheeled robots. For humanoid, four- or six-legged robots, camera motion is 6D instead of 3D or planar motion. Classical 3D reconstruction methods cannot be applied directly, because explicit sensor motion is needed. In this paper, we propose an algorithm for 3D reconstruction of an unstructured environment using motion-free uncalibrated single camera. Computer vision techniques are employed to obtain an incremental geometrical reconstruction of the environment, therefore using vision as a sensor for robot control tasks like navigation, obstacle avoidance, manipulation, tracking, etc. and 3D model acquisition. The main contribution is that the offline 3D reconstruction problem is considered as a point trajectory search through the video stream. The algorithm takes into account the temporal aspect of the sequence of images in order to have an analytical expression of the geometrical locus of the point trajectories through the sequence of images. The approach is a generalization of the Desargues theorem applied to multiple views taken from nearby viewpoints. Experiments on both synthetic and real image sequences show the simplicity and efficiency of the proposed method. This method provides an alternative technical solution easy to use, flexible in the context of robotic applications and can significantly improve the 3D estimation accuracy.

Real‐Time Consensus‐Based Scene Reconstruction Using Commodity Graphics Hardware†

Abstract We present a novel use of commodity graphics hardware that effectively combines a plane‐sweeping algorithm with view synthesis for real‐time, online 3D scene acquisition and view synthesis. Using real‐time imagery from a few calibrated cameras, our method can generate new images from nearby viewpoints, estimate a dense depth map from the current viewpoint, or create a textured triangular mesh. We can do each of these without any prior geometric information or requiring any user interaction, in real time and online. The heart of our method is to use programmable Pixel Shader technology to square intensity differences between reference image pixels, and then to choose final colors (or depths) that correspond to the minimum difference, i.e. the most consistent color. In this paper we describe the method, place it in the context of related work in computer graphics and computer vision, and present some results. ACM CSS: I.3.3 Computer Graphics—Bitmap and framebuffer operations, I.4.8 Image Processing and Computer Vision—Depth cues, Stereo