Intermediate View Synthesis Research Articles

Resolution-enhanced holographic stereogram based on integral imaging using an intermediate-view synthesis technique

We proposed a resolution-enhanced integral imaging (II) based holographic stereogram (HS) using an intermediate-view synthesis (IVS) technique. The HS can be generated fast by Fourier transforming the elemental images of II into hogels. However, the low sampling rate of II is inherited in HS. In this paper, a simple IVS technique is proposed to synthesize intermediate-view elemental images and then adopted to improve the sampling rate of II-based HS. The proposed method avoids multiple captures which are usually realized through the moving array lenslet technique (MALT). Multiple intermediate-view elemental image arrays (IEIAs) are digitally synthesized from an original elemental image array (EIA) with IVS. The original EIA and synthesized IEIAs are transformed to the corresponding HS sequences. All the HS sequences are shifted and added together in one hologram pattern to synthesize the HS with a high sampling rate. Superior to the MALT in II, this method does not need the time-multiplexing technique in both capture stage and display stage. The 3D image quality improvement was verified in numerical and optical reconstruction.

Accurate and efficient ground-to-aerial model alignment

To produce a complete 3D reconstruction of a large-scale architectural scene, both ground and aerial images are usually captured. A common approach is to first reconstruct the models from different image sources separately, and align them afterwards. Using this pipeline, this work proposes an accurate and efficient approach for ground-to-aerial model alignment in a coarse-to-fine manner. First, both the ground model and aerial model are transformed into the geo-referenced coordinate system using GPS meta-information for coarse alignment. Then, the coarsely aligned models are refined by a similarity transformation that is estimated based on 3D point correspondences between them, and the 3D point correspondences are determined in a 2D-image-matching manner by considering the rich textural and contextual information in the 2D images. Due to the dramatic differences in viewpoint and scale between ground and aerial images, which make matching them directly nearly impossible, we perform an intermediate view-synthesis step to mitigate the matching difficulty. To this end, the following three key issues are addressed: (a) selecting a suitable subset of aerial images to cover the ground model properly; (b) synthesizing images from the ground model under the viewpoints of the selected aerial images; and finally, (c) obtaining the 2D point matches between the synthesized images and the selected aerial images. The experimental results show that the proposed model alignment approach is quite effective and outperforms several state-of-the-art techniques in terms of both accuracy and efficiency.

Wide-Baseline Foreground Object Interpolation Using Silhouette Shape Prior.

We consider the synthesis of intermediate views of an object captured by two widely spaced and calibrated cameras. This problem is challenging because foreshortening effects and occlusions induce significant differences between the reference images when the cameras are far apart. That makes the association or disappearance/appearance of their pixels difficult to estimate. Our main contribution lies in disambiguating this ill-posed problem by making the interpolated views consistent with a plausible transformation of the object silhouette between the reference views. This plausible transformation is derived from an object-specific prior that consists of a nonlinear shape manifold learned from multiple previous observations of this object by the two reference cameras. The prior is used to estimate the evolution of the epipolar silhouette segments between the reference views. This information directly supports the definition of epipolar silhouette segments in the intermediate views, as well as the synthesis of textures in those segments. It permits to reconstruct the epipolar plane images (EPIs) and the continuum of views associated with the EPI volume, obtained by aggregating the EPIs. Experiments on synthetic and natural images show that our method preserves the object topology in intermediate views and deals effectively with the self-occluded regions and the severe foreshortening effect associated with wide-baseline camera configurations.

Disparity Estimation of 3-D Mesh for Stereo Video Coding

This paper proposed a disparity estimation method based on the three-dimensional (3-D) mesh model. The disparity estimation is a key step in stereo video coding. By employing intermediate view synthesis, the proposed technique simulate the nodals in the 3-D sureface of stereo image efficiently. Simulated results show that the proposed technique yields a visually more accurate prediciton than the block-matching algorithm.

New Network Bandwidth-limited Multi-view Video plus Depth Coding Method for 3D Video

Prospective 3D video transmission systems that would use compression of both multiview video and depth maps was dealt with in this paper. To get the optimal synthesized view quality with the transmission bit rate being constant, it firstly analyzed the overall behavior of combined coding and view synthesis, color and depth data with different coding qualities were combined for intermediate view synthesis. Then, individual coding results for color and depth data were evaluated in order to investigate, how well the synthesized view quality was at certain compression. Finally, a new joint coding method for multi-view video plus depth (MVD) was proposed by utilizing the optimal combinations of the quantization parameters of color videos and the corresponding depth maps. Simulation results demonstrated that the proposed method decreased the bit-rate by 12.72%~24.02% in comparison with JMVC and can be used to guide sequence-level texture/depth coding for 3D video compression. The proposed method has better performance in optimizing the tradeoff between bit-rate and visual quality.

Consistent Stereo Matching Under Varying Radiometric Conditions

A consistent stereo matching (CSM) algorithm under varying radiometric conditions, such as lighting and exposure variations, for intermediate view synthesis is proposed in this work. First, we transform the colors of stereo images adaptively so that they are similar at corresponding pixels. Since the correspondences are generally unknown before stereo matching, we estimate pseudo-disparity vectors by sorting pixels based on the cumulative color histograms and use those pseudo vectors in the color transform. Then, to improve the accuracy of stereo matching, we jointly estimate the disparity maps for virtual intermediate views as well as those for real views, based on the consistency criterion that an object point should have the same disparity through all the views. Specifically, we compute matching costs using the reliability term and aggregate the costs to obtain initial disparity maps. We then refine the initial disparity maps by minimizing an energy function, which includes the consistency term. Experimental results show that the proposed CSM algorithm significantly reduces the error rate of disparity estimation under different radiometric conditions and synthesizes high quality intermediate views.

Intermediate View Synthesis Based on Adaptive BP Algorithm and View Interpolation

This paper presents a novel intermediate view synthesis method based on adaptive belief propagation (BP) algorithm and view interpolation. First, we raise the accuracy in disparity estimation by introducing an adaptive BP algorithm. Then, image pairs are divided into three kinds of regions. Finally, intermediate view is obtained by applying a new interpolation method to each region. Experimental results indicate that the proposed method can synthesize virtual images with high quality and performs better than a latest method.

Joint coding of multiview video and depth data using virtual view synthesis

AbstractTo compress multiview video and depth information, we synthesize a virtual image for the current view using color and depth data of neighboring views. In this article, we then use a view interpolation prediction scheme at the virtual image to improve the inter‐view prediction. We also propose a solution for overlapping regions and empty holes that are generated during the intermediate view synthesis process due to occlusion and disocclusion situations. Experimental results show that the proposed methods achieve approximately 0.65 dB of Peak Signal‐to‐Noise Ratio (PSNR) gain on average for multiview depth data and 0.17 dB of PSNR gain for multiview video coding, compared with the reference software, Joint Multiview Video Model 1.0. We also show that our method is even more powerful for smaller search ranges. Furthermore, we examine the effects of inter‐view prediction on hierarchical B‐pictures. © 2010 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 20, 370–377, 2010

Intermediate view synthesis algorithm using mesh clustering for rectangular multiview camera system

A multiview video-based three-dimensional (3-D) video system offers a realistic impression and a free view navigation to the user. The efficient compression and intermediate view synthesis are key technologies since 3-D video systems deal multiple views. We propose an intermediate view synthesis using a rectangular multiview camera system that is suitable to realize 3-D video systems. The rectangular multiview camera system not only can offer free view navigation both horizontally and vertically but also can employ three reference views such as left, right, and bottom for intermediate view synthesis. The proposed view synthesis method first represents the each reference view to meshes and then finds the best disparity for each mesh element by using the stereo matching between reference views. Before stereo matching, we separate the virtual image to be synthesized into several regions to enhance the accuracy of disparities. The mesh is classified into foreground and background groups by disparity values and then affine transformed. By experiments, we confirm that the proposed method synthesizes a high-quality image and is suitable for 3-D video systems.

A polygon soup representation for multiview coding

This paper presents a polygon soup representation for multiview data. Starting from a sequence of multiview video plus depth (MVD) data, the proposed quad-based representation takes into account, in a unified manner, different issues such as compactness, compression, and intermediate view synthesis. The representation is extracted from MVD data in two steps. First, a set of 3D quads is extracted thanks to quadtree decomposition performed on depth maps. Second, a selective elimination of the quads is performed in order to reduce inter-view redundancies and thus provide a compact representation. Moreover, the proposed methodology for extracting the representation allows to reduce ghosting artifacts. Finally, an adapted compression technique is proposed that limits coding artifacts. The results presented on two real sequences show that the proposed representation provides a good trade-off between rendering quality and data compactness.

View Synthesis for Advanced 3D Video Systems

Interest in 3D video applications and systems is growing rapidly and technology is maturating. It is expected that multiview autostereoscopic displays will play an important role in home user environments, since they support multiuser 3D sensation and motion parallax impression. The tremendous data rate cannot be handled efficiently by representation and coding formats such as MVC or MPEG-C Part 3. Multiview video plus depth (MVD) is a new format that efficiently supports such advanced 3DV systems, but this requires high-quality intermediate view synthesis. For this, a new approach is presented that separates unreliable image regions along depth discontinuities from reliable image regions, which are treated separately and fused to the final interpolated view. In contrast to previous layered approaches, our algorithm uses two boundary layers and one reliable layer, performs image-based 3D warping only, and was generically implemented, that is, does not necessarily rely on 3D graphics support. Furthermore, different hole-filling and filtering methods are added to provide high-quality intermediate views. As a result, high-quality intermediate views for an existing 9-view auto-stereoscopic display as well as other stereo- and multiscopic displays are presented, which prove the suitability of our approach for advanced 3DV systems.

Quadtree-based disparity estimation for intermediate view synthesis of stereoscopic image sequences

In stereoscopic or multiview 3-D display systems, the synthesis of intermediate sequences is essential to assure lookaround capability and continuous motion parallax so that it can enhance comfortable 3-D perception. The quadtree-based disparity estimation is one of the most popular methods for the synthesis of intermediate sequences, due to the simplicity of its algorithm and hardware implementation. We propose two solutions in order to reduce annoying flicker at object boundaries of synthesized intermediate sequences using quadtree-based disparity estimation. The first requires a new splitting scheme, providing more consistent quadtree splitting strategies during the disparity estimation. The second involves adaptive temporal smoothing, using the dissimilarity between the present frame and the previous one to reduce the error of disparity estimation. These two proposals are tested by using several stereoscopic sequences, and the results show that flicker is remarkably reduced by them.

Stereo-based intermediate view synthesis with realistic ‘look around’ capability

An algorithm is presented for synthesising intermediate views from a stereo image pair that attempts to produce a realistic 3D ‘look around’ effect. The task is accomplished by suitably incorporating a term proportional to the corresponding disparities to the co-ordinates of the ‘virtual’ intermediate view points to be reconstructed.

An object-based system for stereoscopic viewpoint synthesis

This paper describes algorithms that were developed for a real-time stereoscopic videoconferencing systems with viewpoint adaptation. The goal is a real telepresence illusion, which is achieved by synthesis of intermediate views from a stereoscopic camera shot with a rather large baseline. The actual viewpoint will be adapted according to the head position of the viewer, such that the impression of motion parallax is produced. The object-based system first identifies foreground and background regions and applies disparity estimation to the foreground object. A hierarchical block matching algorithm is employed for this purpose which takes into account the position of high-activity feature points and the object/background border positions. Using the disparity estimator's output, it is possible to generate arbitrary intermediate views by projections from the left- and right-view images. For this purpose, we have also developed an object-based interpolation algorithm, taking into account a very simple convex-surface model of a person's face and body. Though the algorithms had to be held rather simple under the constraint of hardware feasibility, we obtain a good quality of the intermediate-view images. Finally, we describe the hardware concept for the disparity estimator, which is the most complicated part of the algorithm.

Intermediate view synthesis considering occluded and ambiguously referenced image regions

In this paper, we present an algorithm for synthesizing intermediate views from a single stereo-pair. The key contributions of this algorithm are the incorporation of scene assumptions and a disparity estimation confidence measure that lead to the accurate synthesis of occluded and ambiguously referenced regions. The synthesized views have been displayed on a multi-view binocular imaging system, with subjectively effective motion parallax and diminished eye strain.