Quality Of DIBR-synthesized Images Research Articles

Multi-layer and Multi-scale feature aggregation for DIBR-Synthesized image quality assessment

Depth-Image-Based Rendering (DIBR) is one of the main fundamental techniques for generating new viewpoints in 3D video applications such as multi-viewpoint video (MVV), free viewpoint video (FVV) and virtual reality (VR). Due to the imperfections of color images, depth maps or texture restoration techniques, several types of distortions occur in synthesized views. However, most of related works evaluated the quality of DIBR-synthesized views by only detecting a specific type of distortion, such as stretching, black holes, blurring, etc., which were unable to accurately evaluate the quality of DIBR-synthesized views. In this paper, a new no-reference image quality assessment method is proposed to evaluate the quality of DIBR-synthesized images by combining multi-layer and multi-scale features of images. To be specific, the distortions introduced by different stages of virtual viewpoint synthesis are first analyzed, and then multi-layer and multi-scale features are extracted to estimate the degree of texture and structure distortions. As a result, individual quality scores associated with two types of distortions (e.g., structural distortion and texture distortion) are aggregated to an overall image quality. Experimental results on two publicly available DIBR datasets show that the method has better performance than the state-of-the-art models.Index Terms: image quality assessment, DIBR-synthesized image, distortion correction, BIQA.

Quality Prediction of View Synthesis Based on Curriculum-Style Structure Generation

Existing quality metrics of view synthesis usually perform on synthesized images, which are produced based on a computationally expensive depth-image-based rendering (DIBR) process. Moreover, current metrics quantify quality by extracting hand-crafted features, which may fail to fully capture the complex distortion characteristics. With the success of deep learning on numerous computer vision tasks, it has become possible to utilize convolutional neural networks to predict the quality of DIBR-synthesized images. In this letter, we propose a deep model to predict the quality of view synthesis based on Curriculum-style Structure Generation without conducting the DIBR process. Specifically, considering that the distortion of view synthesis is mainly manifested in the destruction of image structure, a structure generation network is first built to learn the structure of the new view from the original one by curriculum-style training. Then, we transfer the prior knowledge learned from the last phase into the quality prediction network for measuring the structure distortion, based on which a regressor is introduced to produce the quality score. Experimental results prove the advantages of the proposed model.

Blind Quality Prediction for View Synthesis Based on Heterogeneous Distortion Perception.

The quality of synthesized images directly affects the practical application of virtual view synthesis technology, which typically uses a depth-image-based rendering (DIBR) algorithm to generate a new viewpoint based on texture and depth images. Current view synthesis quality metrics commonly evaluate the quality of DIBR-synthesized images, where the DIBR process is computationally expensive and time-consuming. In addition, the existing view synthesis quality metrics cannot achieve robustness due to the shallow hand-crafted features. To avoid the complicated DIBR process and learn more efficient features, this paper presents a blind quality prediction model for view synthesis based on HEterogeneous DIstortion Perception, dubbed HEDIP, which predicts the image quality of view synthesis from texture and depth images. Specifically, the texture and depth images are first fused based on discrete cosine transform to simulate the distortion of view synthesis images, and then the spatial and gradient domain features are extracted in a Two-Channel Convolutional Neural Network (TCCNN). Finally, a fully connected layer maps the extracted features to a quality score. Notably, the ground-truth score of the source image cannot effectively represent the labels of each image patch during training due to the presence of local distortions in view synthesis image. So, we design a Heterogeneous Distortion Perception (HDP) module to provide effective training labels for each image patch. Experiments show that with the help of the HDP module, the proposed model can effectively predict the quality of view synthesis. Experimental results demonstrate the effectiveness of the proposed model.

DIBR-Synthesized Image Quality Assessment With Texture and Depth Information.

Accurately predicting the quality of depth-image-based-rendering (DIBR) synthesized images is of great significance in promoting DIBR techniques. Recently, many DIBR-synthesized image quality assessment (IQA) algorithms have been proposed to quantify the distortion that existed in texture images. However, these methods ignore the damage of DIBR algorithms on the depth structure of DIBR-synthesized images and thus fail to accurately evaluate the visual quality of DIBR-synthesized images. To this end, this paper presents a DIBR-synthesized image quality assessment metric with Texture and Depth Information, dubbed as TDI. TDI predicts the quality of DIBR-synthesized images by jointly measuring the synthesized image's colorfulness, texture structure, and depth structure. The design principle of our TDI includes two points: (1) DIBR technologies bring color deviation to DIBR-synthesized images, and so measuring colorfulness can effectively predict the quality of DIBR-synthesized images. (2) In the hole-filling process, DIBR technologies introduce the local geometric distortion, which destroys the texture structure of DIBR-synthesized images and affects the relationship between the foreground and background of DIBR-synthesized images. Thus, we can accurately evaluate DIBR-synthesized image quality through a joint representation of texture and depth structures. Experiments show that our TDI outperforms the competing state-of-the-art algorithms in predicting the visual quality of DIBR-synthesized images.

Quality Assessment of 3D Synthesized Images Based on Textural and Structural Distortion Estimation

Emerging 3D-related technologies such as augmented reality, virtual reality, mixed reality, and stereoscopy have gained remarkable growth due to their numerous applications in the entertainment, gaming, and electromedical industries. In particular, the 3D television (3DTV) and free-viewpoint television (FTV) enhance viewers’ television experience by providing immersion. They need an infinite number of views to provide a full parallax to the viewer, which is not practical due to various financial and technological constraints. Therefore, novel 3D views are generated from a set of available views and their depth maps using depth-image-based rendering (DIBR) techniques. The quality of a DIBR-synthesized image may be compromised for several reasons, e.g., inaccurate depth estimation. Since depth is important in this application, inaccuracies in depth maps lead to different textural and structural distortions that degrade the quality of the generated image and result in a poor quality of experience (QoE). Therefore, quality assessment DIBR-generated images are essential to guarantee an appreciative QoE. This paper aims at estimating the quality of DIBR-synthesized images and proposes a novel 3D objective image quality metric. The proposed algorithm aims to measure both textural and structural distortions in the DIBR image by exploiting the contrast sensitivity and the Hausdorff distance, respectively. The two measures are combined to estimate an overall quality score. The experimental evaluations performed on the benchmark MCL-3D dataset show that the proposed metric is reliable and accurate, and performs better than existing 2D and 3D quality assessment metrics.

Quality Index for View Synthesis by Measuring Instance Degradation and Global Appearance

Virtual view synthesis plays a vital role in the application of multi-view and free-viewpoint videos. Depth-image-based rendering (DIBR) is the most commonly used approach in view synthesis, and many DIBR algorithms have been proposed. However, how to evaluate the quality of DIBR-synthesized images and benchmark the DIBR algorithms are still very challenging, which may hinder the further development of the view synthesis technique. Hence, an effective quality metric for evaluating the distortions in view synthesis is urgently needed. With this motivation, this paper presents a quality index for view synthesis by simultaneously measuring local Instance DEgradation and global Appearance (IDEA). Due to the imperfection of rendering algorithms, local geometric distortions are easily introduced around instance contours, causing instance degradation, which is the dominant distortion in synthesized views. In this work, image instances are first detected and local instance degradation is measured based on discrete orthogonal moments. Meantime, we propose to measure the global appearance of synthesized images based on the superpixel representation. By integrating both local and global aspects of the distortions, a more accurate quality model is built for view synthesis. Extensive experiments and comparisons have demonstrated the superiority of the proposed method in evaluating the quality of DIBR-synthesized images and benchmarking the performance of view synthesis algorithms.

Quality Assessment for DIBR-Synthesized Images With Local and Global Distortions

Depth-Image-Based-Rendering (DIBR), as one important technique in 3D video system, can be used to generate virtual views. Unfortunately, the DIBR algorithms will introduce various distortions and induce an annoying viewing experience. And it has been proved that traditional 2D assessment quality metrics are not suitable for the DIBR-synthesized views. In this paper, we propose a novel approach to assess the quality of DIBR-synthesized images. The proposed method mainly considers three kinds of DIBR-related distortions: holes distortion, strip-sharped distortion and global sharpness. Holes and strip distortions as two local features are used to characterize the local quality of DIBR-synthesized image, respectively. For the global sharpness we consider the Just Notice Difference (JND) model of human eyes and use it to extract the JND-based global difference for analyzing the global quality. Finally, we combine the holes distortion evaluation, strip distortion evaluation and global quality to infer the overall perceptual quality. Extensive experiments indicate that our method achieves higher accuracy of quality prediction than most competing metrics.

Multiscale Natural Scene Statistical Analysis for No-Reference Quality Evaluation of DIBR-Synthesized Views

This paper proposes to blindly evaluate the quality of images synthesized via a depth image-based rendering (DIBR) procedure. As a significant branch of virtual reality (VR), superior DIBR techniques provide free viewpoints in many real applications, including remote surveillance and education; however, limited efforts have been made to measure the performance of DIBR techniques, or equivalently the quality of DIBR-synthesized views, especially in the condition when references are unavailable. To achieve this aim, we develop a novel blind image quality assessment (IQA) method via multiscale natural scene statistical analysis (MNSS). The design principle of our proposed MNSS metric is based on two new natural scene statistics (NSS) models specific to the DBIR-synthesized IQA. First, the DIBR-introduced geometric distortions damage the local self-similarity characteristic of natural images, and the damage degrees of self-similarity present particular variations at different scales. Systematically combining the measurements of the variations mentioned above can gauge the naturalness of the input image and thus indirectly reflect the quality changes of images generated using different DIBR methods. Second, it was found that the degradations in main structures of natural images at different scales remain almost the same, whereas the statistical regularity is destroyed in the DIBR-synthesized views. Estimating the deviation of degradations in main structures at different scales between one DIBR-synthesized image and the statistical model, which is constructed based on a large number of natural images, can quantify how a DIBR method damages the main structures and thus infer the image quality. Via trials, the two NSS-based features extracted above can well predict the quality of DIBR-synthesized images. Further, the two features come from distinct points of view, and we hence integrate them via a straightforward multiplication to derive the proposed blind MNSS metric, which achieves better performance than each component and state-of-the-art quality methods.

Blind Quality Metric of DIBR-Synthesized Images in the Discrete Wavelet Transform Domain

Free viewpoint video (FVV) has received considerable attention owing to its widespread applications in several areas such as immersive entertainment, remote surveillance and distanced education. Since FVV images are synthesized via a depth image-based rendering (DIBR) procedure in the "blind" environment (without reference images), a real-time and reliable blind quality assessment metric is urgently required. However, the existing image quality assessment metrics are insensitive to the geometric distortions engendered by DIBR. In this research, a novel blind method of DIBR-synthesized images is proposed based on measuring geometric distortion, global sharpness and image complexity. First, a DIBR-synthesized image is decomposed into wavelet subbands by using discrete wavelet transform. Then, the Canny operator is employed to detect the edges of the binarized low-frequency subband and high-frequency subbands. The edge similarities between the binarized low-frequency subband and high-frequency subbands are further computed to quantify geometric distortions in DIBR-synthesized images. Second, the log-energies of wavelet subbands are calculated to evaluate global sharpness in DIBR-synthesized images. Third, a hybrid filter combining the autoregressive and bilateral filters is adopted to compute image complexity. Finally, the overall quality score is derived to normalize geometric distortion and global sharpness by the image complexity. Experiments show that our proposed quality method is superior to the competing reference-free state-of-the-art DIBR-synthesized image quality models.

Combining Local and Global Measures for DIBR-Synthesized Image Quality Evaluation.

Depth-Image-Based-Rendering (DIBR) techniques are significant for three-dimensional (3D) video applications, e.g., 3D television and free viewpoint video (FVV). Unfortunately, the DIBR-synthesized image suffers from various distortions, which induce an annoying viewing experience for the entire FVV. Proposing a quality evaluator for DIBR-synthesized images is fundamental for the design of perceptual friendly FVV systems. Since the associated reference image is usually not accessible, full-reference (FR) methods cannot be directly applied for quality evaluation of the synthesized image. In addition, most traditional no-reference (NR) methods fail to effectively measure the specifically DIBR-related distortions. In this paper, we propose a novel NR quality evaluation method accounting for two categories of DIBR-related distortions, i.e., geometric distortions and sharpness. First, the disoccluded regions, as one of the most obvious geometric distortions, are captured by analyzing local similarity. Then, another typical geometric distortion (i.e., stretching) is detected and measured by calculating the similarity between it and its equal-size adjacent region. Second, considering the property of scale invariance, the global sharpness is measured as the distance between the distorted image and its downsampled version. Finally, the perceptual quality is estimated by linearly pooling the scores of two geometric distortions and sharpness together. Experimental results verify the superiority of the proposed method over the prevailing FR and NR metrics. More specifically, it is superior to all competing methods except APT in terms of effectiveness, but greatly outmatches APT in terms of implementation time.

DIBR-Synthesized Image Quality Assessment via Statistics of Edge Intensity and Orientation

DIBR-Synthesized Image Quality Assessment via Statistics of Edge Intensity and Orientation

DIBR-synthesized image quality assessment based on morphological multi-scale approach

The depth-image-based rendering (DIBR) algorithms used for 3D video applications introduce new types of artifacts mostly located around the disoccluded regions. As the DIBR algorithms involve geometric transformations, most of them introduce non-uniform geometric distortions affecting the edge coherency in the synthesized images. Such distortions are not handled efficiently by the common image quality assessment metrics which are primarily designed for other types of distortions. In order to better deal with specific geometric distortions in the DIBR-synthesized images, we propose a full-reference metric based on multi-scale image decomposition applying morphological filters. Using non-linear morphological filters in multi-scale image decomposition, important geometric information such as edges is maintained across different resolution levels. Edge distortion between the multi-scale representation subbands of the reference image and the DIBR-synthesized image is measured precisely using mean squared error. In this way, areas around edges that are prone to synthesis artifacts are emphasized in the metric score. Two versions of morphological multiscale metric have been explored: (a) Morphological Pyramid Peak Signal-to-Noise Ratio metric (MP-PSNR) based on morphological pyramid decomposition, and (b) Morphological Wavelet Peak Signal-to-Noise Ratio metric (MW-PSNR) based on morphological wavelet decomposition. The performances of the proposed metrics have been tested using two databases which contain DIBR-synthesized images: the IRCCyN/IVC DIBR image database and MCL-3D stereoscopic image database. Proposed metrics achieve significantly higher correlation with human judgment compared to the state-of-the-art image quality metrics and compared to the tested metric dedicated to synthesis-related artifacts. The proposed metrics are computationally efficient given that the morphological operators involve only integer numbers and simple computations like min, max, and sum as well as simple calculation of MSE. MP-PSNR has slightly better performances than MW-PSNR. It has very good agreement with human judgment, Pearson’s 0.894, Spearman 0.77 when it is tested on the MCL-3D stereoscopic image database. We have demonstrated that PSNR has particularly good agreement with human judgment when it is calculated between images at higher scales of morphological multi-scale representations. Consequently, simplified and in essence reduced versions of multi-scale metrics are proposed, taking into account only detailed images at higher decomposition scales. The reduced version of MP-PSNR has very good agreement with human judgment, Pearson’s 0.904, Spearman 0.863 using IRCCyN/IVC DIBR image database.