View Synthesis Method Research Articles

Local Foreground Removal Disocclusion Filling Method for View Synthesis

View synthesis is an effective method to generate the contents of multiple views based on a limited number of reference views, which can be used in 2D to 3D conversion, free viewpoint video and multiview video rendering. Depth-image-based rendering (DIBR) is a practical technique to generate virtual view by using a 2D reference view and its depth image. However, a critical problem in DIBR process is that disocclusions might be produced in the synthesized image because the background occluded by the foreground objects in the reference view may be exposed in the virtual view. In this paper, a local foreground removal method is proposed for disocclusion filling. Morphology-based depth image preprocessing is performed before DIBR, aiming to correct the depth value of the ghosts and remove ghost artifacts. In the synthesized virtual image, pixels on the disocclusion edge are identified and classified. Then they are positioned in the reference image by inverse 3D warping. Local foreground regions that occlude the corresponding background are removed from both the reference image and its depth image based on the disocclusion edge pixels. Removed region is filled with surrounding background contents, and depth information is used in this process to prevent foreground penetration. The predicted background contents are warped to the disocclusion region, thereby achieving the hole filling. Experimental results show that the proposed method performs better than the other methods in disocclusion filling, and improves the subjective and objective quality of the synthesized view. In the evaluation results of PSNR, SSIM, FSIMc and VSI, our method improves by 0.32-2.43dB, 0.0036-0.0155, 0.0041-0.0198 and 0.0012-0.0057 respectively compared with competitive methods.

Light Field Compression Using Global Multiplane Representation and Two-Step Prediction

Due to its spatio-angular structure, light field image allows for a wealth of post-processing techniques like digital refocusing and depth estimation. In order to compress the data of the two domains, the current proposal intends to embed the disparity-based view synthesis method into the decoder. However, predicting each view separately or in local groups means bringing more computational burden to the decoder and destroying the light field structure. Since disparity contains the relationship between all light rays in the light field, the proposed solution is to predict a disparity-based global representation as the first step. In the second step, all the views can be predicted easily based on this representation. In this letter, we use the recently proposed multiplane as the form of this global representation. The experimental results show the effectiveness of the proposed solution, and the better RD performance compared to other schemes especially under low bitrates.

High-fidelity View Synthesis for Light Field Imaging With Extended Pseudo 4DCNN

Multi-view properties of light field (LF) imaging enable exciting applications such as auto-refocusing, depth estimation and 3D reconstruction. However, limited angular resolution has become the main bottleneck of microlens-based plenoptic cameras towards more practical vision applications. Existing view synthesis methods mainly break the task into two steps, i.e. depth estimating and view warping, which are usually inefficient and produce artifacts over depth ambiguities. We have proposed an end-to-end deep learning framework named Pseudo 4DCNN to solve these problems in a conference paper. Rethinking on the overall paradigm, we further extend pseudo 4DCNN and propose a novel loss function which is applicable for all tasks of light field reconstruction i.e. EPI Structure Preserving (ESP) loss function. This loss function is proposed to attenuate the blurry edges and artifacts caused by averaging effect of ${L_2}$ norm based loss function. Furthermore, the extended Pseudo 4DCNN is compared with recent state-of-the-art (SOTA) approaches on more publicly available light field databases, as well as self-captured light field biometrics and microscopy datasets. Experimental results demonstrate that the proposed framework can achieve better performances than vanilla Pseudo 4DCNN and other SOTA methods, especially in the terms of visual quality under occlusions. The source codes and self-collected datasets for reproducibility will be available online soon.

Light Field Reconstruction With a Hybrid Sparse Regularization-Pseudo 4DCNN Framework

The densely-sampled light field (LF) is highly desirable in many applications, such as 3D reconstruction, post-capture refocusing and virtual/augmented reality. However, it is costly and challenging to capture them because of the high dimensional nature. Existing view synthesis methods employ depth information for the densely-sampled LF reconstruction from the undersampled LF with a large disparity range, but fail to achieve non-Lambertian performance. In this article, a novel coarse-to-fine LF reconstruction method is proposed. We will develop a hybrid reconstruction framework that fuses model-based sparse regularization with deep learning. Specifically, sparse regularization with directional filter bank is utilized to solve the large baseline problem and gives a coarse densely-sampled LF. In addition, for those that cannot be recovered by classical model-based methods due to limited angular resolution, is estimated by learning a pseudo 4D convolutional neural network, and thereby offering a refinement on the intermediate LF. Extensive experiments demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods.

Latent transformations neural network for object view synthesis

We propose a fully convolutional conditional generative neural network, the latent transformation neural network, capable of rigid and non-rigid object view synthesis using a lightweight architecture suited for real-time applications and embedded systems. In contrast to existing object view synthesis methods which incorporate conditioning information via concatenation, we introduce a dedicated network component, the conditional transformation unit. This unit is designed to learn the latent space transformations corresponding to specified target views. In addition, a consistency loss term is defined to guide the network toward learning the desired latent space mappings, a task-divided decoder is constructed to refine the quality of generated views of objects, and an adaptive discriminator is introduced to improve the adversarial training process. The generalizability of the proposed methodology is demonstrated on a collection of three diverse tasks: multi-view synthesis on real hand depth images, view synthesis of real and synthetic faces, and the rotation of rigid objects. The proposed model is shown to be comparable with the state-of-the-art methods in structural similarity index measure and $$L_{1}$$ metrics while simultaneously achieving a 24% reduction in the compute time for inference of novel images.

A Semi-Supervised Monocular Stereo Matching Method

Supervised monocular depth estimation methods based on learning have shown promising results compared with the traditional methods. However, these methods require a large number of high-quality corresponding ground truth depth data as supervision labels. Due to the limitation of acquisition equipment, it is expensive and impractical to record ground truth depth for different scenes. Compared to supervised methods, the self-supervised monocular depth estimation method without using ground truth depth is a promising research direction, but self-supervised depth estimation from a single image is geometrically ambiguous and suboptimal. In this paper, we propose a novel semi-supervised monocular stereo matching method based on existing approaches to improve the accuracy of depth estimation. This idea is inspired by the experimental results of the paper that the depth estimation accuracy of a stereo pair as input is better than that of a monocular view as input in the same self-supervised network model. Therefore, we decompose the monocular depth estimation problem into two sub-problems, a right view synthesized process followed by a semi-supervised stereo matching process. In order to improve the accuracy of the synthetic right view, we innovate beyond the existing view synthesis method Deep3D by adding a left-right consistency constraint and a smoothness constraint. To reduce the error caused by the reconstructed right view, we propose a semi-supervised stereo matching model that makes use of disparity maps generated by a self-supervised stereo matching model as the supervision cues and joint self-supervised cues to optimize the stereo matching network. In the test, the two networks are able to predict the depth map directly from a single image by pipeline connecting. Both procedures not only obey geometric principles, but also improve estimation accuracy. Test results on the KITTI dataset show that this method is superior to the current mainstream monocular self-supervised depth estimation methods under the same condition.

VUNet: Dynamic Scene View Synthesis for Traversability Estimation Using an RGB Camera

We present VUNet, a novel view(VU) synthesis method for mobile robots in dynamic environments, and its application to the estimation of future traversability. Our method predicts future images for given virtual robot velocity commands using only RGB images at previous and current time steps. The future images result from applying two types of image changes to the previous and current images: first, changes caused by different camera pose. Second, changes due to the motion of the dynamic obstacles. We learn to predict these two types of changes disjointly using two novel network architectures, SNet and DNet. We combine SNet and DNet to synthesize future images that we pass to our previously presented method GONet [N. Hirose, A. Sadeghian, M. Vazquez, P. Goebel, and S. Savarese, “Gonet: A semi-supervised deep learning approach for traversability estimation,” in Proc. IEEE International Conference on Intelligent Robots and Systems, 2018, pp. 3044–3051] to estimate the traversable areas around the robot. Our quantitative and qualitative evaluation indicate that our approach for view synthesis predicts accurate future images in both static and dynamic environments. We also show that these virtual images can be used to estimate future traversability correctly. We apply our view synthesis-based traversability estimation method to two applications for assisted teleoperation.

Generating Stereoscopic Images With Convergence Control Ability From a Light Field Image Pair

With the advances in commercial light field cameras, light field image processing has attracted considerable attention from researchers. In this paper, we propose a novel method for generating stereoscopic images from a light field image pair with flexible control over the convergence of the virtual stereo cameras. We have developed a light field image-capturing prototype that consists of two horizontally arranged light field cameras (i.e., with their optical axes being parallel to each other). When using our proposed device for image/video capture, stereo photographers can concentrate on how to capture the desired visual experience without being frequently disturbed by having to manipulate the stereo camera parameters, i.e., the convergence angle of a stereo camera. During postprocessing, our method estimates accurate disparity maps for the light field image pair and then generates the target stereoscopic images that satisfy the desired stereo camera convergence requirements by adopting a novel view synthesis method for light field images. We have conducted extensive experiments to demonstrate the effectiveness of our proposed method.

Fast and high-quality virtual view synthesis from multi-view plus depth videos

Depth image based rendering (DIBR) is an effective method for virtual view synthesis from Multi-view Plus Depth(MVD) video. Synthetic images, however, often contain ghost effect and some holes of varying sizes. This paper uses color correction of reference views, and combines depth-based image fusion with direct color image fusion to decrease the ghost effect. Meanwhile, the cracks are filled using depth filtering and inverse warping. What’s more, the image depth-aided inpainting with GPU acceleration is used to fill the remaining big disocclusions. Experimental results show that our proposed method improved the quality of virtual view synthetic images and reduced the processing time sharply.

Computational Light Field Generation Using Deep Nonparametric Bayesian Learning

In this paper, we present a deep nonparametric Bayesian method to synthesize a light field from a single image. Conventionally, light-field capture requires special optical architecture, and the gain in angular resolution often comes at the expense of a reduction in spatial resolution. Techniques for computationally generating the light field from a single image can be expanded further to a variety of applications, ranging from microscopy and materials analysis to vision-based robotic control and autonomous vehicles. We treat the light field as multiple sub-aperture views, and to compute the novel viewpoints, our model contains three major components. First, a convolutional neural network is used for predicting the depth probability map from the image. Second, a multi-scale feature dictionary is constructed within a multi-layer dictionary learning network. Third, the novel views are synthesized taking into account both the probabilistic depth map and the multi-scale feature dictionary. The experiments show that our method outperforms several state-of-the-art novel view synthesis methods in delivering good image resolution.

An Artifact-Type Aware DIBR Method for View Synthesis

Depth-image-based rendering is a popular way to produce content for three-dimensional television and free viewpoint video, allowing the synthesis of numerous viewpoints using a single reference view and its depth map. Due to the synthesis process and nature of the input, artifacts and holes appear, and solving these problems becomes a challenge. In this letter, we propose solutions to remove those artifacts and apply different filling strategies depending on the nature of each hole. Cracks are identified and filled using very local neighborhood information. Regions classified as ghosts are projected to their correct place. The remaining holes are classified as disocclusions or out-of-field areas, and filled with an appropriate adaptation of a popular inpainting method. In both adaptations, patch matching explores the spatial locality concept, using dynamically adaptive patch sizes from the reference image. For disocclusions we propose a filling order using depth and background terms, and a searching process that considers only background patches. We show that our method outperforms several view synthesis methods in the quantitative evaluation, besides presenting consistent visual results for both large baselines and severely occluded scenes.

View synthesis using foreground object extraction for disparity control and image inpainting

Among the rapidly growing three-dimensional technologies, multiview displays have drawn great research interests in three-dimensional television due to their adaption to the motion parallax and wider viewing angles. However, multiview displays still suffer from dazzling discomfort on the border of viewing zones. Leveraging on the separability of scene via foreground segmentation, we propose a novel virtual view synthesis method for depth-image-based rendering to alleviate the discomfort. Foreground objects of interest are extracted to segment the whole image into multiple layers, which are further warped to the virtual viewpoint in order. To alleviate the visual discomfort, global disparity adjustments and local depth control are performed for specific objects in each layer. For the post-processing, we improve an exemplar-based inpainting algorithm to tackle the disoccluded areas. Experimental results demonstrate that our method achieves effective disparity control and generates high-quality virtual view images.

View synthesis method for 3D video coding based on temporal and inter view correlation

Recently, in three-dimensional (3D) television, the temporal correlation between consecutive frames of the intermediate view is used together with the inter-view correlation to improve the quality of the synthesised view. However, most temporal methods are based on the motion vector fields (MVFs) calculated by the optical flow or block-based motion estimation which has very high computational complexity. To alleviate this issue, the authors propose a temporal-disparity-based view synthesis (TDVS) method, which uses the MVFs extracted from the bitstreams of side views and motion warping technique to create the temporal correlation between views in the intermediate position. Then a motion compensation technique is used to create a temporal-based view. Finally, the temporal-based view is fused with a disparity-based view which is generated by a traditional depth image-based rendering technique to create the final synthesised view. The fusion of these views is performed based on the side information which is determined and encoded at the sender-side of the 3D video system using a dynamic programming algorithm and rate-distortion optimisation scheme. Experimental results show that the proposed method can achieve the synthesised view with appreciable improvements in comparison with the view synthesis reference software 1D fast (VSRS-1D Fast) for several test sequences.

Stereo magnification

The view synthesis problem---generating novel views of a scene from known imagery---has garnered recent attention due in part to compelling applications in virtual and augmented reality. In this paper, we explore an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones. We call this problem stereo magnification , and propose a learning framework that leverages a new layered representation that we call multiplane images (MPIs). Our method also uses a massive new data source for learning view extrapolation: online videos on YouTube. Using data mined from such videos, we train a deep network that predicts an MPI from an input stereo image pair. This inferred MPI can then be used to synthesize a range of novel views of the scene, including views that extrapolate significantly beyond the input baseline. We show that our method compares favorably with several recent view synthesis methods, and demonstrate applications in magnifying narrow-baseline stereo images.

No-reference quality assessment of DIBR-synthesized videos by measuring temporal flickering

Depth-image-based-rendering (DIBR) is the most popular view synthesis method. The rendering process in DIBR introduces artifacts to synthesized views. Temporal inconsistency of these artifacts causes flickering, which is one of the dominant distortions in DIBR-synthesized videos. We propose a no-reference quality index for DIBR-synthesized videos. A two-stage flickering region detection method is first proposed by calculating the gradient distance between the matching blocks in adjacent frames followed by a refinement operation. Subsequently, the distortion intensity of flickering regions is measured in singular value decomposition domain. The average flickering intensity is computed as the quality score. Experimental results show that our method outperforms the state-of-the-arts on the IRCCyN/IVC database and ranks the second on the SIAT database, which is only slightly worse than the best-performing one. Furthermore, the presented method is applied to improve the performances of existing metrics and benchmark DIBR algorithms, which achieves very promising results for both tasks.

Reliability-Based View Synthesis for Free Viewpoint Video

View synthesis is a crucial technique for free viewpoint video and multi-view video coding because of its capability to render an unlimited number of virtual viewpoints from adjacent captured texture images and corresponding depth maps. The accuracy of depth maps is very important to the rendering quality, since depth image–based rendering (DIBR) is the most widely used technology among synthesis algorithms. There are some issues due to the fact that stereo depth estimation is error-prone. In addition, filling occlusions is another challenge in producing desirable synthesized images. In this paper, we propose a reliability-based view synthesis framework. A depth refinement method is used to check the reliability of depth values and refine some of the unreliable pixels, and an adaptive background modeling algorithm is utilized to construct a background image aiming to fill the remaining empty regions after a proposed weighted blending process. Finally, the proposed approach is implemented and tested on test video sequences, and experimental results indicate objective and subjective improvements compared to previous view synthesis methods.

Micro-Lens-Based Matching for Scene Recovery in Lenslet Cameras.

Since a light-field camera is able to capture more information than a traditional camera, a lot of methods, such as depth estimation, image super-resolution, and view synthesis, are explored for recovering scene information. In this paper, we propose a novel framework for scene recovery based on lenslet-based light-field camera images. Instead of using traditional matching terms, we design a new micro-lens-based matching term to calculate structure information and recover several kinds of scene information simultaneously. On the one hand, inherent information in micro-lens images is selected to complement details in sub-aperture images. On the other hand, sub-aperture images are used to expand micro-lens images and synthesize new view images. A new micro-lens-based consistency metric is introduced for the matching term to handle occlusions in depth estimation and image reconstruction. The newly appeared and newly occluded areas in synthesized views are analyzed and recovered based on information from surrounding points. Experimental results show that the proposed depth estimation method outperforms state-of-the-art methods on both synthetic and lenslet-based light-field images, especially in low-texture and occlusion regions. Furthermore, the super-resolution and view synthesis methods are able to acquire view images with more details and less aliasing artifacts.

Video synthesis from stereo videos with iterative depth refinement

We propose a novel depth maps refinement algorithm and generate multi-view video sequences from two-view video sequences for modern autostereoscopic display. In order to generate realistic contents for virtual views, high-quality depth maps are very critical to the view synthesis results. Therefore, refining the depth maps is the main challenging problem in the task. We propose an iterative depth refinement algorithm, including error detection and error correction, to correct errors in depth map. Error detection aims at two types of error: across-view color-depth-inconsistency error and local color-depth-inconsistency error. Then, error pixels are corrected based on sampling local candidates. A trilateral filter that considers intensity, spatial and temporal terms into the filter weighting is applied to enhance the spatial and temporal consistency across frames. So the virtual views can be better synthesized according to the refined depth maps. To combine both warped images, disparity-based view interpolation is introduced to alleviate the translucent artifacts. Finally, a directional filter is applied to reduce the aliasing around the object boundaries to generate multiple high-quality virtual views between the two views. We demonstrate the superior image quality of the synthesized virtual views by using the proposed algorithm over the state-of-the-art view synthesis methods through experiments on benchmarking image and video datasets.

Depth-Perception Preserved Content Remapping Based on Cornsweet Illusion for 3D Display

Usually, there is a limited depth of field (DOF) for automultiscopic three-dimensional (3D) displays, which requires modifying light fields of 3D scenes to fit into the DOF constraint of a target display. A novel depth-perception preserved 3D content remapping method is presented based on Cornsweet illusion, which uses Cornsweet profile to enhance the depth-perception by augmenting the contrast at the edges. The original depth map is decomposed into multifrequency bands and remapped by reducing the low-frequency bands while preserving the high-frequency parts, which leads to Cornsweet profiles with the preserved perceptual depth. The manipulation of the multifrequency bands is processed in logarithmic domain with a well-designed monotonic mapping function. The tradeoff between displayed blur and depth preservation is achieved by adopting the available objective metric based on human vision system. A warping-based view synthesis method is adopted to retarget the light field according to the modified depth map. Experimental results show that the displayed modified 3D light field appears sharp, and the original perception of depth is maximally preserved.

Real-time virtual view synthesis using light field

Virtual view synthesis technique renders a virtual view image from several pre-collected viewpoint images. The hotspot on virtual view synthesis area is depth image-based rendering (DIBR), which has low one-time imaging quality. To achieve high imaging quality artifacts, the holes must be inpainted after image warping which means high computational complexity. This paper proposed a real-time virtual view synthesis method from light field. Then the light field is transformed into frequency domain. The light field is parameterized and reconstructed from image array. The virtual view is rendered by resampling the light field in frequency domain. After resampling the image by performing Fourier slice, the virtual view image is obtained by inverse Fourier transform. Experiments show that our method can get high one-time imaging quality in real time.