Full-reference Quality Assessment Research Articles

NeRF-NQA: No-Reference Quality Assessment for Scenes Generated by NeRF and Neural View Synthesis Methods.

Neural View Synthesis (NVS) has demonstrated efficacy in generating high-fidelity dense viewpoint videos using a image set with sparse views. However, existing quality assessment methods like PSNR, SSIM, and LPIPS are not tailored for the scenes with dense viewpoints synthesized by NVS and NeRF variants, thus, they often fall short in capturing the perceptual quality, including spatial and angular aspects of NVS-synthesized scenes. Furthermore, the lack of dense ground truth views makes the full reference quality assessment on NVS-synthesized scenes challenging. For instance, datasets such as LLFF provide only sparse images, insufficient for complete full-reference assessments. To address the issues above, we propose NeRF-NQA, the first no-reference quality assessment method for densely-observed scenes synthesized from the NVS and NeRF variants. NeRF-NQA employs a joint quality assessment strategy, integrating both viewwise and pointwise approaches, to evaluate the quality of NVS-generated scenes. The viewwise approach assesses the spatial quality of each individual synthesized view and the overall inter-views consistency, while the pointwise approach focuses on the angular qualities of scene surface points and their compound inter-point quality. Extensive evaluations are conducted to compare NeRF-NQA with 23 mainstream visual quality assessment methods (from fields of image, video, and light-field assessment). The results demonstrate NeRF-NQA outperforms the existing assessment methods significantly and it shows substantial superiority on assessing NVS-synthesized scenes without references. An implementation of this paper are available at https://github.com/VincentQQu/NeRF-NQA.

A full reference quality assessment method with fused monocular and binocular features for stereo images.

Aiming to automatically monitor and improve stereoscopic image and video processing systems, stereoscopic image quality assessment approaches are becoming more and more important as 3D technology gains popularity. We propose a full-reference stereoscopic image quality assessment method that incorporate monocular and binocular features based on binocular competition and binocular integration. To start, we create a three-channel RGB fused view by fusing Gabor filter bank responses and disparity maps. Then, using the monocular view and the RGB fusion view, respectively, we extract monocular and binocular features. To alter the local features in the binocular features, we simultaneously estimate the saliency of the RGB fusion image. Finally, the monocular and binocular quality scores are calculated based on the monocular and binocular features, and the quality scores of the stereo image prediction are obtained by fusion. Performance testing in the LIVE 3D IQA database Phase I and Phase II. The results of the proposed method are compared with newer methods. The experimental results show good consistency and robustness.

Effects of Different Full-Reference Quality Assessment Metrics in End-to-End Deep Video Coding

Visual quality assessment is often used as a key performance indicator (KPI) to evaluate the performance of electronic devices. There exists a significant association between visual quality assessment and electronic devices. In this paper, we bring attention to alternative choices of perceptual loss function for end-to-end deep video coding (E2E-DVC), which can be used to reduce the amount of data generated by electronic sensors and other sources. Thus, we analyze the effects of different full-reference quality assessment (FR-QA) metrics on E2E-DVC. First, we select five optimization-suitable FR-QA metrics as perceptual objectives, which are differentiable and thus support back propagation, and use them to optimize an E2E-DVC model. Second, we analyze the rate–distortion (R-D) behaviors of an E2E-DVC model under different loss function optimizations. Third, we carry out subjective human perceptual tests on the reconstructed videos to show the performance of different FR-QA optimizations on subjective visual quality. This study reveals the effects of the competing FR-QA metrics on E2E-DVC and provides a guide for further future study on E2E-DVC in terms of perceptual loss function design.

Reduced-Reference Quality Assessment of Point Clouds via Content-Oriented Saliency Projection

Many dense 3D point clouds have been exploited to represent visual objects instead of traditional images or videos. To evaluate the perceptual quality of various point clouds, in this letter, we propose a novel and efficient Reduced-Reference quality metric for point clouds, which is based on Content-oriented sAliency Projection (RR-CAP). Specifically, we make the first attempt to simplify reference and distorted point clouds into projected saliency maps with a downsampling operation. Through this process, we tackle the issue of transmitting large-volume original point clouds to user-ends for quality assessment. Then, motivated by the characteristics of the human visual system (HVS), the objective quality scores of distorted point clouds are produced by combining content-oriented similarity and statistical correlation measurements. Finally, extensive experiments are conducted on SJTU-PCQA and WPC databases. The experimental results demonstrate that our proposed algorithm outperforms existing reduced-reference and no-reference quality metrics, and significantly reduces the performance gap between state-of-the-art full-reference quality assessment methods. In addition, we show the performance variation of each proposed technical component by ablation tests.

Multi-Angle Projection Based Blind Omnidirectional Image Quality Assessment

Most of the existing blind omnidirectional image quality assessment (BOIQA) methods are based on data-driven approach where the end-to-end neural network or deep learning tools are mainly used for feature extraction. However, it usually lacks interpretability and is difficult to discover the perceptual mechanism behind. In this paper, from the perspective of perception modeling, we propose a novel multi-angle projection based BOIQA (MP-BOIQA) method. Considering the omnibearing and near eye display characteristics with head mounted display, multiple color cubemap projection images with respect to different viewpoints are grouped as the color omnidirectional distortion (COD) units so as to simulate the user’s viewing behavior in subjective quality assessment. In the designed multi-angle projection based feature extractor, tensor decomposition is implemented on each COD unit for dimensionality reduction, and piecewise exponential fitting is used to get the distribution of mean subtracted contrast normalized coefficients of the unit’s feature matrices in tensor domain. Finally, the extracted features are pooled with random forest. The experimental results on three omnidirectional image quality datasets show that the MP-BOIQA method can deliver highly competitive performance compared with some representative full-reference quality assessment methods, as well as some state-of-the-art BOIQA methods.

Comparative Analysis of Image Enhancement Algorithms

In the complex instruments utilised in essential fields such as satellite cameras, CT scanners, and High-Resolution Cameras (Underwater), image capture is critical without human-rated aberrations, sounds, or atmospheric disturbances. Even full reference QA (quality assessment) approaches have a limited ability to predict quality accurately. As a result, the difficulty of evaluating and enhancing photographs is further subdivided into domain-specific issues by focusing on a small set of artefacts. The most popular is entropy, which is usually relevant in picture coding: it is a lower limit for the average coding length in bits per pixel that may be attained without any loss of information by an optimal coding scheme. The word 'specific' is significant because it establishes right away that the strategies covered in this paper are primarily problemsolving techniques. For example, a procedure that works well for improving X-ray images may not be the ideal option. Thus, a method that works well for boosting X-ray photos may not be the greatest option for enhancing photographs obtained by a satellite thousands of miles away from the Earth. Image enhancement algorithms proposed in this paper are Intensity-Hue-Saturation transformation, Histogram Equalization algorithms, Edge Detection techniques and Retinex theory algorithms. These algorithms are implemented under satellite imagery, medical scans, underwater images, and their parameter analysis.

Image Quality Assessment of No Reference JPEG Compressed Images Using Various Spatial Domain Features

In today's world of technological advancement higher data rate and data transmission with minimal memoryrequirement is gaining importance. At the same time it also important to preserve quality of data to be transferredfrom various types of distortions and to assess the quality at the receiving end. Hence perceptual image qualityassessment is becoming more popular. This paper introduces a new approach in domain of no reference imagequality assessment of jpeg compressed images using spatial features .Considering the fact that pixel distortion,blurring and edge information are important aspects as far as distortions are concern. A novel approach is presentedin this paper to address the distortion categories. The results are obtained in conjunction with the subjectiveimage quality assessment of train and test image categories. Also results are compared with full reference imagequality assessment technique and parameters of full reference quality assessment. To achieve better accuracyalgorithm is tested on LIVE Texas' image database and our own developed database. Results are found to be wellcorrelating to the various parameters considered for the comparison purpose.

Perceptual Quality Assessment of Low-light Image Enhancement

Low-light image enhancement algorithms (LIEA) can light up images captured in dark or back-lighting conditions. However, LIEA may introduce various distortions such as structure damage, color shift, and noise into the enhanced images. Despite various LIEAs proposed in the literature, few efforts have been made to study the quality evaluation of low-light enhancement. In this article, we make one of the first attempts to investigate the quality assessment problem of low-light image enhancement. To facilitate the study of objective image quality assessment (IQA), we first build a large-scale low-light image enhancement quality (LIEQ) database. The LIEQ database includes 1,000 light-enhanced images, which are generated from 100 low-light images using 10 LIEAs. Rather than evaluating the quality of light-enhanced images directly, which is more difficult, we propose to use the multi-exposure fused (MEF) image and stack-based high dynamic range (HDR) image as a reference and evaluate the quality of low-light enhancement following a full-reference (FR) quality assessment routine. We observe that distortions introduced in low-light enhancement are significantly different from distortions considered in traditional image IQA databases that are well-studied, and the current state-of-the-art FR IQA models are also not suitable for evaluating their quality. Therefore, we propose a new FR low-light image enhancement quality assessment (LIEQA) index by evaluating the image quality from four aspects: luminance enhancement, color rendition, noise evaluation, and structure preserving, which have captured the most key aspects of low-light enhancement. Experimental results on the LIEQ database show that the proposed LIEQA index outperforms the state-of-the-art FR IQA models. LIEQA can act as an evaluator for various low-light enhancement algorithms and systems. To the best of our knowledge, this article is the first of its kind comprehensive low-light image enhancement quality assessment study.

Full-Reference Quality Assessment for Screen Content Images Based on the Concept of Global-Guidance and Local-Adjustment

Benefiting from the development of multimedia communication terminals, the visual content presented to people on mobile devices is no longer a single form, but contains text, natural images, and other computer-generated graphics, which is called screen content images (SCIs). Inspired by the different visual stimuli that text and images bring to human eyes and the concept of global-guidance and local-adjustment, we design a novel full-reference image quality assessment (IQA) model using the structural features of the text, the perceptual features of pictures, and a score integration model (SPSIM) to evaluate SCIs quality. Firstly, we split the SCIs into textual and pictorial regions through a fully convolutional network (FCN) to conduct separate analyses. For textual regions, we take advantage of narrow edge extensions and high edge steps as structural features to compute the textual score. For pictorial regions, we extract the just noticeable difference (JND) features, which measure the human eye's ability to detect distortion as perceptual features to calculate the pictorial score. Finally, an innovative score integration method based on the global-guidance and local-adjustment is designed to better analyze the relationship between the above regional scores and the whole global SCIs score. Abundant experiments in SCIs databases have shown that the SPSIM model can achieve better consistency with the human eyes system (HVS) in predicting the visual quality of SCIs.

HDR-BVQM: High dynamic range blind video quality model

Considerable progress has been made toward developing standard dynamic range (SDR) blind video quality assessment (BVQA) models that do not require any baseline reference for quality prediction. However, there is no such method for the high dynamic range (HDR) content. Unlike SDR video, HDR video represents a high-fidelity representation of the real-world scene by preserving the wide luminance range and color gamut. Therefore, SDR BVQA models are not suitable for HDR BVQA. Towards ameliorating this, a first-of-its-kind BVQA model for HDR content is presented in this work. The proposed HDR blind video quality model (HDR-BVQM) is inspired by the spatio-temporal natural scene statistics model, previously employed in SDR blind quality assessment metrics. To build our proposed model, we first develop a comprehensive subjective HDR video quality dataset, including 228 distorted videos generated through three different (H.264, HEVC, packet drop) distortion processes from 19 pristine HDR videos. The developed dataset is then used to extract HDR relevant features, which vary in different distortion types, to train and test the proposed HDR-BVQM. The features are based on the pointwise, pairwise log-derivative, and motion coherence based statistics. Finally, detailed validation and performance comparison is performed with full-reference HDR and no-reference SDR quality assessment methods. The results reveal that the quality prediction by HDR-BVQM correlates with the human judgment of quality.

Quality Assessment of Super-Resolved Omnidirectional Image Quality Using Tangential Views

Omnidirectional images (ODIs), also known as 360-degree images, enable viewers to explore all directions of a given 360-degree scene from a fixed point. Designing an immersive imaging system with ODI is challenging as such systems require very large resolution coverage of the entire 360 viewing space to provide an enhanced quality of experience (QoE). Despite remarkable progress on single image super-resolution (SISR) methods with deep-learning techniques, no study for quality assessments of super-resolved ODIs exists to analyze the quality of such SISR techniques. This paper proposes an objective, full-reference quality assessment framework which studies quality measurement for ODIs generated by GAN-based and CNN-based SISR methods. The quality assessment framework offers to utilize tangential views to cope with the spherical nature of a given ODIs. The generated tangential views are distortion-free and can be efficiently scaled to high-resolution spherical data for SISR quality measurement. We extensively evaluate two state-of-the-art SISR methods using widely used full-reference SISR quality metrics adapted to our designed framework. In addition, our study reveals that most objective metric show high performance over CNN based SISR, while subjective tests favors GAN-based architectures.

Objective quality assessment of synthesized images by local variation measurement

Due to the rapid development of free-viewpoint television (FVT), Depth-Image-Based Rendering (DIBR) technology has been widely used to synthesize images of virtual view-points. However, the types of distortions in the synthesized images are different from those of natural images, such as discontinuity, flickering, stretching, etc. To measure the distortion occurred in the synthesized images, we propose a full-reference (FR) quality assessment method by local variation measurement consisting of three-modules. Firstly, since the distortion in the synthesized image mainly occurs in the region with high-frequency structure information, the Neutrosophic domain is employed to evaluate the degradation of local image structure. Secondly, by considering that the texture of the synthesized image might be damaged due to the warping of 2D image or the loss of information in the occlusion region, we evaluate the visual quality of local texture by using the features obtained from frequency domain. Thirdly, to measure the stretching distortion which is unique in the synthesized image, the visual quality of extracted stretching area is measured by entropy. Finally, a pooling operation is used to combine the quality scores of the three modules to obtain the final predicted quality score. Experimental results show that the performance of the proposed algorithm is competitive with state-of-the-art FR and no-reference image quality assessment metrics.

Parameters optimization of the Structural Similarity Index

We exploit evolutionary computation to optimize the handcrafted Structural Similarity method (SSIM) through a datadriven approach. We estimate the best combination of luminance, contrast and structure components, as well as the sliding window size used for processing, with the objective of optimizing the similarity correlation with human-expressed mean opinion score on a standard dataset. We experimentally observe that better results can be obtained by penalizing the overall similarity only for very low levels of luminance similarity. Finally, we report a comparison of SSIM with the optimized parameters against other metrics for full reference quality assessment, showing superior performance on a different dataset.

Subjective annotation for a frame interpolation benchmark using artefact amplification

Current benchmarks for optical flow algorithms evaluate the estimation either directly by comparing the predicted flow fields with the ground truth or indirectly by using the predicted flow fields for frame interpolation and then comparing the interpolated frames with the actual frames. In the latter case, objective quality measures such as the mean squared error are typically employed. However, it is well known that for image quality assessment, the actual quality experienced by the user cannot be fully deduced from such simple measures. Hence, we conducted a subjective quality assessment crowdscouring study for the interpolated frames provided by one of the optical flow benchmarks, the Middlebury benchmark. It contains interpolated frames from 155 methods applied to each of 8 contents. For this purpose, we collected forced-choice paired comparisons between interpolated images and corresponding ground truth. To increase the sensitivity of observers when judging minute difference in paired comparisons we introduced a new method to the field of full-reference quality assessment, called artefact amplification. From the crowdsourcing data (3720 comparisons of 20 votes each) we reconstructed absolute quality scale values according to Thurstone’s model. As a result, we obtained a re-ranking of the 155 participating algorithms w.r.t. the visual quality of the interpolated frames. This re-ranking not only shows the necessity of visual quality assessment as another evaluation metric for optical flow and frame interpolation benchmarks, the results also provide the ground truth for designing novel image quality assessment (IQA) methods dedicated to perceptual quality of interpolated images. As a first step, we proposed such a new full-reference method, called WAE-IQA, which weights the local differences between an interpolated image and its ground truth.

A Genetically Based Combination of Visual Saliency and Roughness for FR 3D Mesh Quality Assessment: A Statistical Study

Abstract In this paper, we present a full-reference quality assessment metric based on the information of visual saliency. The saliency information is provided under the form of degrees associated to each vertex of the surface mesh. From these degrees, statistical attributes reflecting the structures of the reference and distorted meshes are computed. These are used by four comparisons functions genetically optimized that quantify the structure differences between a reference and a distorted mesh. We also present a statistical comparison study of six full-reference quality assessment metrics for 3D meshes. We compare the objective metrics results with humans subjective scores of quality considering the 3D meshes in one hand and the distorsion types in the other hand. Also, we show which metrics are statistically superior to their counterparts. For these comparisons we use the Spearman Rank Ordered Correlation Coefficient and the hypothetic test of Student (ttest). To attest the pertinence of the proposed approach, a comparison with a ground truth saliency and an application associated to the assessment of the visual rendering of smoothing algorithms are presented. Experimental results show that the proposed metric is very competitive with the state-of-the-art.

Visual attention-aware quality estimation framework for omnidirectional video using spherical Voronoi diagram

Omnidirectional video (ODV) enables viewers to look at every direction from a fixed point and provides a much more immersive experience than traditional 2D video. Assessing the video quality is important for delivering ODV to the end-user with the best possible quality. For this goal, two aspects of ODV should be considered. The first is the spherical nature of ODV and the related projection distortions when the ODV is stored in a planar format. The second is the interactive look-around consumption nature of ODV. Related to this aspect, visual attention, that identifies the regions that attract the viewer’s attention, is important for ODV quality assessment. Considering these aspects, in this paper, we study in particular objective full-reference quality assessment for ODV. To this end, we propose a quality assessment framework based on the spherical Voronoi diagram and visual attention. In this framework, a given ODV is subdivided into multiple planar patches with low projection distortions using the spherical Voronoi diagram. Afterwards, each planar patch is analyzed separately by a quality metric for traditional 2D video, obtaining a quality score for each patch. Then, the patch scores are combined based on visual attention into a final quality score. To validate the proposed framework, we create a dataset of ODVs with scaling and compression distortions, and conduct subjective experiments in order to gather the subjective quality scores and the visual attention data for our ODV dataset. The evaluation of the proposed framework based on our dataset shows that both the use of the spherical Voronoi diagram and visual attention are crucial for achieving state-of-the-art performance.

A 3D Image Quality Assessment Method Based on Vector Information and SVD of Quaternion Matrix under Cloud Computing Environment

With the increasing demands of end-users to the visual perception in three-dimension (3D) image, quality assessment for 3D imageis dominantly required as the feedback information for multimedia transmission systems. In this paper, a novel full-reference quality assessment method by considering the depth and integral color information of 3D image under cloud computing environment is proposed. Based on the property of the depth information in 3D image, the depth map is firstly separated into different planes according to the perception of human visual system (HVS). Then, after express the image pixels of every separated plane through quaternions, the structural and energy information are separated by quaternion singular value decomposition (QSVD). The distortion of structural and energy in every plane are calculated in various formulas respectively. The final result is calculated in terms of the global score, which synthesizes the structural and energy distortion scores in every individual depth plane. It should be pointed out that the chrominance information is employed in our mechanism to evaluate the color image quality because of its useful characteristic for 3D color image quality assessment, and its spatial correlation is used for calculating structural distortion through vector cross-product. Our experimental results confirm that the proposed method has achieves better performance under cloud computing environments compared with other existing 3D image quality assessment methods.

Perceptual stereoscopic images quality assessment method based on visual adaptable characteristics

The quality assessment of stereoscopic images plays an important role in various three-dimensional (3-D) areas and encounters more problems than two-dimensional (2-D) image quality assessment. We propose a perceptual full-reference quality assessment model by considering human adaptable double visual channel. Human binocular combination characteristics and depth perception are both considered in this model, and an adaptable gain-control model is adopted to assign appropriate weights to the information of human visual channels. Experimental results indicate that the proposed algorithm can serve as an efficient predictive image quality feature, which delivers not only highly competitive prediction accuracy but also moderate computational complexity.

Blind quality assessment for screen content images via convolutional neural network

With the wide propagation of cloud and mobile computing, screen content images (SCIs) have become more indispensable in our daily lives. Compared to natural scene images (NSIs), SCIs possess many particular characteristics, like mixed contents, extremely sharp edges, and text graphics. Consequently, more challenges occur in the feature extraction, which is used to reflect the distortion, during the quality assessment of SCIs. Recently, some convolutional neural network (CNN) models have been designed by automatically learning feature to evaluate the quality. In this paper, we develop a novel blind quality assessment method for SCIs via the CNN. First, compared with existing CNN-based methods, the proposed method avoids the disadvantage of training with image patches, and it is the pioneering attempt that takes the entire image as inputs. Second, instead of the image gray value, the original image is decomposed into two portions, i.e., the predicted and unpredicted portions, according to the internal generative mechanism (IGM) theory as the input of CNN. Through the CNN, all features of the image are learned automatically from beginning to end, and the network finally outputs the predicted score. Since existing SCI database is too small, to fully train the network, we collected 30000 SCIs and employed a high-accuracy full-reference quality assessment metric of SCI to compute scores as the training labels. Experimental results on SIQAD database demonstrate that the proposed method is comparable to reference-based SCI quality assessment metrics and is superior to the state-of-the-art NSI quality assessment metrics.

Full-Reference 3-D Video Quality Assessment Using Scene Component Statistical Dependencies

In this letter, we present a full-reference (FR) quality assessment algorithm to assess the perceptual quality of natural stereoscopic three-dimensional (S3-D) videos. Toward the end, we rely on an empirical model for the joint statistics of motion and depth subband coefficients of an S3-D video frame. Specifically, we use the recently proposed bivariate generalized Gaussian distribution (BGGD) model for the joint statistics. In this letter, we show that the coherence of the covariance matrix of the BGGD varies in proportion with the perceptual video quality. We compute the coherence scores from the eigenvalues of the covariance matrix to estimate the amount of directional dependency between the motion and depth components. To estimate the overall spatial quality score, we apply off-the-shelf 2-D FR image quality assessment metrics on a frame-by-frame basis on both the views and average the frame wise scores. Finally, we pool the coherence and spatial quality scores to derive the overall quality for the S3-D video. An evaluation of the proposed algorithm on the IRCCYN, WaterlooIVC Phase I, and LFOVIA S3D video databases demonstrates its robust performance. The proposed algorithm is called depth- and motion-based 3-D video quality evaluator.