View Synthesis Reference Software Research Articles

View synthesis method for 3D video coding based on temporal and inter view correlation

Recently, in three-dimensional (3D) television, the temporal correlation between consecutive frames of the intermediate view is used together with the inter-view correlation to improve the quality of the synthesised view. However, most temporal methods are based on the motion vector fields (MVFs) calculated by the optical flow or block-based motion estimation which has very high computational complexity. To alleviate this issue, the authors propose a temporal-disparity-based view synthesis (TDVS) method, which uses the MVFs extracted from the bitstreams of side views and motion warping technique to create the temporal correlation between views in the intermediate position. Then a motion compensation technique is used to create a temporal-based view. Finally, the temporal-based view is fused with a disparity-based view which is generated by a traditional depth image-based rendering technique to create the final synthesised view. The fusion of these views is performed based on the side information which is determined and encoded at the sender-side of the 3D video system using a dynamic programming algorithm and rate-distortion optimisation scheme. Experimental results show that the proposed method can achieve the synthesised view with appreciable improvements in comparison with the view synthesis reference software 1D fast (VSRS-1D Fast) for several test sequences.

Quality Assessment Based on Edge and Saliency for Synthesized Virtual View

In multi-view video technology based on DIBR, virtual view quality assessment is more difficult question due to the virtual view's special distortion. Virtual view synthesized by depth estimation reference software (DERS) and view synthesis reference software (VSRS) has relationship with adjacent views. The main distortion in virtual view occurs on the edge, so we extract edges synthesized by adjacent views edge information and the corresponding depth map. In this paper we compared the edge information of virtual view and combined with stereoscopic visual saliency prediction to assess virtual view quality. Experiments proved that the results of this metric are advantageous and consistent with human visual perception.

Fast Physically Correct Refocusing for Sparse Light Fields Using Block-Based Multi-Rate View Interpolation.

Digital refocusing has a tradeoff between complexity and quality when using sparsely sampled light fields for low-storage applications. In this paper, we propose a fast physically correct refocusing algorithm to address this issue in a twofold way. First, view interpolation is adopted to provide photorealistic quality at infocus-defocus hybrid boundaries. Regarding its conventional high complexity, we devised a fast line-scan method specifically for refocusing, and its 1D kernel can be 30× faster than the benchmark View Synthesis Reference Software (VSRS)-1D-Fast. Second, we propose a block-based multi-rate processing flow for accelerating purely infocused or defocused regions, and a further 3- 34× speedup can be achieved for high-resolution images. All candidate blocks of variable sizes can interpolate different numbers of rendered views and perform refocusing in different subsampled layers. To avoid visible aliasing and block artifacts, we determine these parameters and the simulated aperture filter through a localized filter response analysis using defocus blur statistics. The final quadtree block partitions are then optimized in terms of computation time. Extensive experimental results are provided to show superior refocusing quality and fast computation speed. In particular, the run time is comparable with the conventional single-image blurring, which causes serious boundary artifacts.

Multiview Plus Depth Video Coding With Temporal Prediction View Synthesis

Multiview video (MVV) plus depths formats use view synthesis to build intermediate views from existing adjacent views at the receiver side. Traditional view synthesis exploits the disparity information to interpolate an intermediate view by considered inter-view correlations. However, temporal correlation between different frames of the intermediate view can be used to improve the synthesis. We propose a new coding scheme for 3-D High Efficiency Video Coding (HEVC) that allows us to take full advantage of temporal correlations in the intermediate view and improve the existing synthesis from adjacent views. We use optical flow techniques to derive dense motion vector fields (MVF) from the adjacent views and then warp them at the level of the intermediate view. This allows us to construct multiple temporal predictions of the synthesized frame. A second contribution is an adaptive fusion method that judiciously selects between temporal and inter-view prediction to eliminate artifacts associated with each prediction type. The proposed system is compared against the state-of-the-art view synthesis reference software 1-D Fast technique used in 3-D HEVC standardization. Three intermediary views are synthesized. Gains of up to 1.21-dB Bjontegaard Delta peak SNR are shown when evaluated on several standard MVV test sequences.

View Synthesis Distortion Estimation With a Graphical Model and Recursive Calculation of Probability Distribution

Depth-image-based rendering (DIBR) is frequently used in multiview video applications such as free-viewpoint television. In this paper, we consider the two DIBR algorithms used in the Moving Picture Experts Group view synthesis reference software, and develop a scheme for the encoder to estimate the distortion of the synthesized virtual view at the decoder when the reference texture and depth sequences experience transmission errors such as packet loss. We first develop a graphical model to analyze how random errors in the reference depth image affect the synthesized virtual view. The warping competition rule adopted in the DIBR algorithms is explicitly represented by the graphical model. We then consider the case where packet loss occurs to both the encoded texture and depth images during transmission and develop a recursive optimal distribution estimation (RODE) method to calculate the per-pixel texture and depth probability distributions in each frame of the reference views. The RODE is then integrated with the graphical model method to estimate the distortion in the synthesized view caused by packet loss. Experimental results verify the accuracy of the graphical model method, the RODE, and the combined estimation scheme.

View-spatial–temporal post-refinement for view synthesis in 3D video systems

Depth image based rendering is one of key techniques to realize view synthesis for three-dimensional television and free-viewpoint television, which provide high quality and immersive experiences to end viewers. However, artifacts of rendered images, including holes caused by occlusion/disclosure and boundary artifacts, may degrade the subjective and objective image quality. To handle these problems and improve the quality of rendered images, we present a novel view-spatial–temporal post-refinement method for view synthesis, in which new hole-filling and boundary artifact removal techniques are proposed. In addition, we propose an optimal reference frame selection algorithm for a better trade-off between the computational complexity and rendered image quality. Experimental results show that the proposed method can achieve a peak signal-to-noise ratio gain of 0.94dB on average for multiview video test sequences when compared with the benchmark view synthesis reference software. In addition, the subjective quality of the rendered image is also improved.

Non-reference virtual view quality evaluation of MVD

The quality of virtual view based on multi-view video (MVD) plus depth format is often evaluated by PSNR or subjectively judged. However, due to synthesizing arbitrary view images, the virtual view images mostly have no reference images and are only assessed using non-reference. Virtual view images synthesized by depth estimation reference software (DERS) and view synthesis reference software (VSRS) often accompanied with blockiness and other distortions on the edge. In addition, matching level for the depth map and the corresponding texture maps of left and right views also affects the quality of the virtual view. This paper compares the edge similarity of the depth and the corresponding texture maps which generate the intermediate virtual view and combined with the virtual view’s blockiness which causing blur to evaluate the quality of the virtual view. Experiment results show that the proposed method can reflect the quality of the virtual view better.