Quality Of Virtual Views Research Articles

Learning-Based Stereoscopic View Synthesis with Cascaded Deep Neural Networks

Depth image-based rendering (DIBR) is an important technique in the 2D to 3D conversion process, which renders virtual views with a texture image and the associated depth map. However, certain problems, such as disocclusion, still exist in current DIBR systems. In this study, a new learning-based framework that models conventional DIBR synthesis pipelines is proposed to solve these problems. The proposed model adopts a coarse-to-fine approach to realize virtual view prediction and disocclusion region refinement sequentially in a unified deep learning framework that includes two cascaded joint filter block-based convolutional neural networks (CNNs) and one residual learning-based generative adversarial network (GAN). An edge-guided global looping optimization strategy is adopted to progressively reconstruct the scene structures on the novel view, and a novel directional discounted reconstruction loss is proposed for better training. In this way, our framework performs well in terms of virtual view quality and is more suitable for 2D to 3D conversion applications. The experimental results demonstrate that the proposed method can generate visually satisfactory results.

Depth estimation using modified cost function for occlusion handling

This paper presents a novel approach to the occlusion handling problem in depth estimation using three views. A solution based on modification of similarity cost function is proposed. During the depth estimation via optimization algorithms, like Graph Cuts, the similarity metric is constantly updated so that only non-occluded fragments in the side views are considered. At each iteration of the algorithm, non-occluded fragments are detected based on side view virtual depth maps synthesized from currently the best estimated depth map of the center view. Then, the similarity metric is updated for correspondence search only in non-occluded regions of the side views. The experimental results, conducted on well-known 3D video test sequences, show that the depth maps estimated with the proposed approach provide about 1.25 dB virtual view quality improvement in comparison with the virtual view synthesized based on depth maps generated with the use of the state-of-the-art MPEG Depth Estimation Reference Software.

Artifact Handling Based on Depth Image for View Synthesis

The depth image based rendering (DIBR) is a popular technology for 3D video and free viewpoint video (FVV) synthesis, by which numerous virtual views can be generated from a single reference view and its depth image. However, some artifacts are produced in the DIBR process and reduce the visual quality of virtual view. Due to the diversity of artifacts, effectively handling them becomes a challenging task. In this paper, an artifact handling method based on depth image is proposed. The reference image and its depth image are extended to fill the holes that belong to the out-of-field regions. A depth image preprocessing method is applied to project the ghosts to their correct place. The 3D warping process is optimized by an adaptive one-to-four method to deal with the cracks and pixel overlapping. For disocclusions, we calculate depth and background terms of the filling priority based on depth information. The search for the best matching patch is performed simultaneously in the reference image and the virtual image. Moreover, adaptive patch size is used in all hole-filling processes. Experimental results demonstrate the effectiveness of the proposed method, which has better performance compared with previous methods in subjective and objective evaluation.

Joint Power and Channel Resource Optimization in Soft Multi-View Video Delivery

Existing wireless multi-view video (MVV) transmission schemes use digital compression to achieve a better coding efficiency. However, the digital schemes suffer from the cliff effect, which refers to the phenomenon that the video quality is a step function of wireless channel quality. In this paper, we first consider a soft MVV transmission scheme where the correlations between the inter-view data and texture-depth data are exploited by a 5-dimensional discrete cosine transform (5D-DCT). The linearly transformed 5D-DCT signals are modulated in an analog manner so that the video quality gracefully improves when the channel quality becomes better. The cumbersome bit and rate controls in digital solutions are replaced by simple power controls. Second, as with the increase of the number of cameras and data depths, the data size of MVV increases linearly. To reduce the heavy data traffic in soft MVV transmission, we proposed efficient resource (bandwidth and power) allocation algorithms. Simulations results demonstrate that the proposed distortion-resource (DR) optimization algorithm can ensure a best viewing quality under a resource constraint and the proposed resource-distortion (RD) optimization algorithm can minimize the resource usage for a target video quality requirement. Third, the impact of power control across texture and depth frame and the impact of view positions on synthesized virtual view quality are investigated. The efficacy of the proposed algorithm on both the reference viewpoint as well as the virtual viewpoint is verified via simulations.

A novel distortion criterion of rate-distortion optimization for depth map coding

In 3D video coding systems, depth maps are not displayed to the viewers, but provide the geometric information to generate virtual views. To ensure the quality of virtual views, the rate-distortion optimization (RDO) in depth map coding adopts the virtual view distortion as the distortion item. The virtual view distortion comes from the reconstructed color video distortion and depth distortion. It is usually recognized that the virtual view distortion caused by reconstructed color video distortion is independent of that in depth map coding. Preliminary experiments reveal that the virtual view distortion in depth map coding is also influenced by the reconstructed color video distortion. Therefore, we proposed a novel distortion criterion of depth map coding in which the reconstructed color video distortion is modeled and joins into the virtual view distortion calculation. Correspondingly, the associated Lagrange multiplier is also proposed. Experimental results demonstrate that the method by integrating the proposed distortion criterion into RDO for depth map coding can achieve an average 12.72% bitrate saving compared with SSD based RDO method and can also lead a bitrate reduction (0.64%) compared with the existing distortion estimation method in the current 3D-HEVC reference software. With the associated Lagrange multiplier, the proposed distortion criterion can achieve 12.98% bitrate saving compared with SSD based RDO method on average.

Quality Assessment Based on Edge and Saliency for Synthesized Virtual View

In multi-view video technology based on DIBR, virtual view quality assessment is more difficult question due to the virtual view's special distortion. Virtual view synthesized by depth estimation reference software (DERS) and view synthesis reference software (VSRS) has relationship with adjacent views. The main distortion in virtual view occurs on the edge, so we extract edges synthesized by adjacent views edge information and the corresponding depth map. In this paper we compared the edge information of virtual view and combined with stereoscopic visual saliency prediction to assess virtual view quality. Experiments proved that the results of this metric are advantageous and consistent with human visual perception.

Fine Virtual View Distortion Estimation Method for Depth Map Coding

In three-dimensional (3-D) video coding systems, depth maps represent the geometric information of a 3-D scene. Since depth maps are not displayed to viewers but to generate virtual views, the quality of depth maps needs to be measured by its effect on the virtual view quality, which is indicated by the virtual view distortion (VVD) in depth map coding. In this letter, a fine VVD estimation method is proposed based on the analysis of the VVD. Specifically, the depth distortion of the current pixel, the texture gradient of the colocated color video and the depth distortions of adjacent pixels are all taken into consideration to estimate the VVD accurately. Experimental results demonstrate that the proposed method can improve 13.1% bitrate saving compared with the sum of squared differences based depth distortion calculation method and can improve 1.2% bitrate saving compared with the VVD estimation method in three dimensional high efficiency video coding (3D-HEVC) reference software.

A Noncoverage Field Model for Improving the Rendering Quality of Virtual Views

Rendering quality optimization theory is one of the most basic and fascinating components of image-based rendering (IBR). The rendering quality of virtual views is related to the information in the source images. Capturing the image information depends on the geometric configuration of the camera (GCC) and, in particular, the positions and shooting directions of the cameras. Therefore, the rendering quality of virtual views can be improved by optimizing the GCC. This paper investigates the relationship between the GCC and the geometric configurations of the virtual view (GCVV). The influence of the GCC on the rendering quality is also analyzed. Based on the relationships among the GCC, GCVV, and rendering quality, a mathematical model of the noncoverage field (NCF) is proposed to quantify the rendering quality. The performance of the NCF is also analyzed using a set of GCVVs and GCCs. Furthermore, an optimization algorithm based on the NCF that simultaneously optimizes the position and direction of the GCC is presented. The proposed technique can be applied to obtain the optimal rendering quality of IBR for the linear case of camera positions and virtual views. Finally, experimental results are presented and compared with the theoretical results.

Object Shape Approximation and Contour Adaptive Depth Image Coding for Virtual View Synthesis

A depth image provides partial geometric information of a 3D scene, namely the shapes of physical objects as observed from a particular viewpoint. This information is important when synthesizing images of different virtual camera viewpoints via depth-image-based rendering (DIBR). It has been shown that depth images can be efficiently coded using contour-adaptive codecs that preserve edge sharpness, resulting in visually pleasing DIBR-synthesized images. However, contours are typically losslessly coded as side information, which is expensive if the object shapes are complex. In this paper, we pursue a new paradigm in depth image coding for color-plus-depth representation of a 3D scene: in a pre-processing step, we pro-actively simplify object shapes in a depth and color image pair to reduce depth coding cost, at a penalty of a slight increase in synthesized view distortion. Specifically, we first mathematically derive a distortion upper-bound proxy for 3DSwIM—a quality metric tailored for DIBR-synthesized images. This proxy reduces inter-dependency among pixel rows in a block to ease optimization. We then approximate object contours via a dynamic programming algorithm to optimally tradeoff coding the cost of contours using arithmetic edge coding with our proposed view synthesis distortion proxy. We modify the depth and color images according to the approximated object contours in an inter-view consistent manner. These are then coded, respectively, using a contour-adaptive image codec based on graph Fourier transform for edge preservation and High Efficiency Video Coding (HEVC) intra. Experimental results show that by maintaining sharp but simplified object contours during contour-adaptive coding, for the same visual quality of DIBR-synthesized virtual views, our proposal can reduce depth image coding rate by up to 22% in 3DSwIM and 42% in peak signal-to-noise ratio compared with alternative coding strategies, such as HEVC intra.

Virtual-view PSNR prediction based on a depth distortion tolerance model and support vector machine.

Quality prediction of virtual-views is important for free viewpoint video systems, and can be used as feedback to improve the performance of depth video coding and virtual-view rendering. In this paper, an efficient virtual-view peak signal to noise ratio (PSNR) prediction method is proposed. First, the effect of depth distortion on virtual-view quality is analyzed in detail, and a depth distortion tolerance (DDT) model that determines the DDT range is presented. Next, the DDT model is used to predict the virtual-view quality. Finally, a support vector machine (SVM) is utilized to train and obtain the virtual-view quality prediction model. Experimental results show that the Spearman's rank correlation coefficient and root mean square error between the actual PSNR and the predicted PSNR by DDT model are 0.8750 and 0.6137 on average, and by the SVM prediction model are 0.9109 and 0.5831. The computational complexity of the SVM method is lower than the DDT model and the state-of-the-art methods.

Depth map upsampling using joint edge-guided convolutional neural network for virtual view synthesizing

In texture-plus-depth format of three-dimensional visual data, both texture and depth maps are required to synthesize a desired view via depth-image-based rendering. However, the depth maps captured or estimated always exist with low resolution compared to their corresponding texture images. We introduce a joint edge-guided convolutional neural network that upsamples the resolution of a depth map on the premise of synthesized view quality. The network takes the low-resolution depth map as an input using a joint edge feature extracted from the depth map and the registered texture image as a reference, and then produces a high-resolution depth map. We further use local constraints that preserve smooth regions and sharp edges so as to improve the quality of the depth map and synthesized view. Finally, a global looping optimization is performed with virtual view quality as guidance in the recovery process. We train our model using pairs of depth maps and texture images and then make tests on other depth maps and video sequences. The experimental results demonstrate that our scheme outperforms existing methods both in the quality of the depth maps and synthesized views.

Virtual view quality assessment based on shift compensation and visual masking effect

Depth image based rendering technology is essential for free viewpoint video systems. Because of the compression of depth images and limitations of rendering algorithms, various types of distortion might occur in the virtual viewpoints and cannot effectively be evaluated by traditional two-dimensional assessment methods. Hence, this paper proposes a method for virtual viewpoint quality assessment using the visual masking effect. First, shift is compensated for the distorted virtual viewpoint and then the compensated virtual viewpoint is objectively assessed. Next, according to human visual characteristics such as texture, magnitude, and distribution masking, the corresponding visual sensitivity map and visual masks are extracted. Finally, the visual masking and all factors are pooled to create the final quality score. As verified by the experimental results, the method proposed in this paper corresponds with the characteristics of human vision and can serve as a more effective method for assessing the quality of virtual viewpoints.

Novel Bitrate Saving and Fast Coding for Depth Videos in 3D-HEVC

With the advance of coding and network technology, the multiview video (MVV) system has received growing attention in the 3-D TV market since it can provide consumers with more realistic scenes from different viewpoints. In the MVV system, each video sequence consists of one color-map video and one depth video to be encoded and the depth-image-based rendering technique is often used to create/synthesize virtual views for different viewpoints. To attain the same quality of virtual views, encoding depth videos in a fast and bitrate-saving manner is crucial. Based on the depth no-synthesis-error model developed by Zhao et al. , this paper proposes a new intra-/inter-prediction scheme first and then a fast quadtree structure determination scheme for encoding depth videos in 3-D High Efficiency Video Coding (3D-HEVC). The first proposed scheme has bitrate-saving merit because for each coding unit, each depth value is modified to approach the predicted intra-/inter-depth value as closely as possible, but without causing any synthesis errors in the resultant virtual views. The second proposed scheme has the merit of low computational cost because it can terminate the quadtree structure determination for coding tree units as early as possible. Based on four typical test video sequences, the empirical results demonstrate that on average, our proposed encoding method for depth videos has 16.3% bitrate and 13.8% encoding-time improvement ratios when compared with the traditional method in 3D-HEVC test model, while preserving the quality of the rendered virtual views. In addition, our proposed method outperforms that of Lee et al. in terms of the required bitrate and encoding time.

Adaptive Bit Allocation for 3D Video Coding

In the multi-view video plus depth 3D video coding, texture image and depth map are coded jointly. The texture image is utilized for displaying and synthesizing the virtual view as reference image. The depth map provides the scene geometry information and is utilized to synthesize the virtual view at the terminal through Depth-Image Based Rendering technique. The distortion of the compressed texture image and depth map will be propagated to the synthesized virtual view. Besides the coding efficiency of texture image and depth map, bit allocation between texture image and depth map also has a great effect on the synthesized virtual view quality. Several methods are proposed for bit allocation between texture image and depth map, but most of them attempt to allocate a fixed target bitrate based on virtual view distortion model to achieve optimal synthesized virtual view quality, and the modeling process brings extra complexity. In practical application, the video sequence has different contents and fixed bit ratio cannot achieve optimal performance. In this paper, we propose an adaptive bit allocation algorithm for 3D video coding. First, we present a model to estimate the synthesized virtual view distortion, and then adjust the bit ratio between adjacent views and between texture image and depth map at Group of Picture level based on the virtual view quality fluctuation. We adjust the bit ratio to achieve the optimal virtual view quality for different video contents. Experimental results demonstrate that the proposed algorithm can optimally allocate bits to achieve optimal virtual view quality under different target bitrates and for different video contents, and the computational complexity of the proposed algorithm is extremely low.

Efficient Lagrange multiplier selection algorithm for depth maps coding

The rate–distortion optimisation criterion of depth maps coding is evaluated by virtual view distortion, which motivates research modelling the distortion and corresponding Lagrange multiplier for depth maps coding. Current studies follow the assumption that the virtual view distortion introduced by texture videos distortion and depth maps distortion can be explored separately. Then the texture-videos-distortion-induced virtual view distortion is considered as a constant value during depth maps coding. A relationship between the optimal Lagrange multiplier of depth maps and texture videos distortion is investigated. Based on this relationship, an efficient Lagrange multiplier selection algorithm is proposed for depth maps coding based on quantisation parameters of texture videos and depth maps. With the proposed Lagrange multiplier, 1.36% bitrate saving with the same virtual view quality can be achieved compared with 3D high efficiency video coding reference software.

A depth video processing algorithm based on cluster dependent and corner-ware filtering

In free viewpoint video system, color video and the associated depth video are utilized to synthesize arbitrary virtual viewpoint. Hence, high quality of depth video is a necessity for virtual view rendering. However, depth video estimated by depth estimate software is inconsistent and inaccurate which decreases the quality of virtual views. To solve the problem, a depth video processing algorithm is proposed in this paper. Firstly, depth video is divided into five clusters adaptively by Fuzzy C-means clustering method. Meanwhile, the edges of depth video are detected and expanded into 8×8 block-wise. Secondly, for pixels in non-edge regions of depth video, a cluster dependent filtering method is adopted according to the feature of each cluster. Finally, corners in the corresponding texture video are detected. For pixels in edge regions of depth video, a corner-aware filtering method is used. Experimental results show that the proposed algorithm enhances the depth video and significantly improves the quality of the virtual views. The peak signal to noise ratio of virtual views rendered by using the proposed algorithm is 0.43dB higher than that of the virtual views rendered by using the original depth video. The proposed algorithm also outperforms Martin's algorithm in terms of virtual view rendering performance.

Fast depth map coding based on virtual view quality

Multi-view video plus depth (MVD) format is a three-dimensional (3D) video representation. The depth map in MVD provides the scene distance information and is used to synthesize the virtual view with texture image through Depth Image Based Rendering (DIBR) technique. In this paper, we first present a virtual view distortion estimation (VVDE) scheme to estimate the synthesized virtual view distortion induced by depth map compression. Then based on the synthesized virtual view distortion, we propose region based Quantization Parameter (QP) adjustment scheme to improve the perceived virtual view quality, fast VVDE scheme in depth map coding Rate Distortion Optimization (RDO) to improve the coding efficiency and fast Coding Unit (CU) partition scheme to reduce the depth map coding complexity. Experimental results demonstrate that the proposed algorithm can achieve 0.50dB gain with 47.37% coding time reduction.

Depth video spatial and temporal correlation enhancement algorithm based on just noticeable rendering distortion model

Spatial and temporal inconsistency of depth video deteriorates encoding efficiency in three dimensional video systems. A depth video processing algorithm based on human perception is presented. Firstly, a just noticeable rendering distortion (JNRD) model is formulated by combining the analyses of the influence of depth distortion on virtual view rendering with human visual perception characteristics. Then, depth video is processed based on the JNRD model from two aspects, spatial and temporal correlation enhancement. During the process of spatial correlation enhancement, depth video is segmented into edge, foreground, and background regions, and smoothened by Gaussian and mean filters. The operations of the temporal correlation enhancement include temporal–spatial transpose (TST), temporal smoothing filter and inverse TST. Finally, encoding and virtual view rendering experiments are conducted to evaluate the proposed algorithm. Experimental results show that the proposed algorithm can greatly reduce the bit rate while it maintains the quality of virtual view.

Disparity Refinement near the Object Boundaries for Virtual-View Quality Enhancement

Stereo matching algorithm is usually used to obtain a disparity map from a pair of images. However, the disparity map obtained by using stereo matching contains lots of noise and error regions. In this paper, we propose a virtual-view synthesis algorithm using disparity refinement in order to improve the quality of the synthesized image. First, the error region is detected by examining the consistency of the disparity maps. Then, motion information is acquired by applying optical flow to texture component of the image in order to improve the performance. Then, the occlusion region is found using optical flow on the texture component of the image in order to improve the performance of the optical flow. The refined disparity map is finally used for the synthesis of the virtual view image. The experimental results show that the proposed algorithm improves the quality of the generated virtual-view.

A DASH-based HEVC multi-view video streaming system

Recent advancement in cameras and image processing technology has generated a paradigm shift from traditional 2D and 3D video to multi-view video (MVV) technology, while at the same time improving video quality and compression through standards such as high efficiency video coding (HEVC). In multi-view, cameras are placed in predetermined positions to capture the video from various views. Delivering such views with high quality over the Internet is a challenging prospect, as MVV traffic is several times larger than traditional video, since it consists of multiple video sequences, each captured from a different angle, requiring more bandwidth than single-view video to transmit MVV. In addition, the Internet is known to be prone to packet loss, delay, and bandwidth variation, which adversely affect MVV transmission. Another challenge is that end users' devices have different capabilities in terms of computing power, display, and access link capacity, requiring MVV to be adapted to each user's context. In this paper, we propose an HEVC multi-view system using Dynamic Adaptive Streaming over HTTP to overcome the above-mentioned challenges. Our system uses an adaptive mechanism to adjust the video bit rate to the variations of bandwidth in best effort networks. We also propose a novel scalable way for the multi-view video and depth content for 3D video in terms of the number of transmitted views. Our objective measurements show that our method of transmitting MVV content can maximize the perceptual quality of virtual views after the rendering and hence increase the user's quality of experience.