Perceptual Quality Assessment Research Articles

TCDM: Transformational Complexity Based Distortion Metric for Perceptual Point Cloud Quality Assessment.

The goal of objective point cloud quality assessment (PCQA) research is to develop quantitative metrics that measure point cloud quality in a perceptually consistent manner. Merging the research of cognitive science and intuition of the human visual system (HVS), in this article, we evaluate the point cloud quality by measuring the complexity of transforming the distorted point cloud back to its reference, which in practice can be approximated by the code length of one point cloud when the other is given. For this purpose, we first make space segmentation for the reference and distorted point clouds based on a 3D Voronoi diagram to obtain a series of local patch pairs. Next, inspired by the predictive coding theory, we utilize a space-aware vector autoregressive (SA-VAR) model to encode the geometry and color channels of each reference patch with and without the distorted patch, respectively. Assuming that the residual errors follow the multi-variate Gaussian distributions, the self-complexity of the reference and transformational complexity between the reference and distorted samples are computed using covariance matrices. Additionally, the prediction terms generated by SA-VAR are introduced as one auxiliary feature to promote the final quality prediction. The effectiveness of the proposed transformational complexity based distortion metric (TCDM) is evaluated through extensive experiments conducted on five public point cloud quality assessment databases. The results demonstrate that TCDM achieves state-of-the-art (SOTA) performance, and further analysis confirms its robustness in various scenarios.

EPQ-GAN: Evolutionary Perceptual Quality Assessment Generative Adversarial Network for Image Dehazing

EPQ-GAN: Evolutionary Perceptual Quality Assessment Generative Adversarial Network for Image Dehazing

Low-dose computed tomography perceptual image quality assessment

In computed tomography (CT) imaging, optimizing the balance between radiation dose and image quality is crucial due to the potentially harmful effects of radiation on patients. Although subjective assessments by radiologists are considered the gold standard in medical imaging, these evaluations can be time-consuming and costly. Thus, objective methods, such as the peak signal-to-noise ratio and structural similarity index measure, are often employed as alternatives. However, these metrics, initially developed for natural images, may not fully encapsulate the radiologists’ assessment process. Consequently, interest in developing deep learning-based image quality assessment (IQA) methods that more closely align with radiologists’ perceptions is growing. A significant barrier to this development has been the absence of open-source datasets and benchmark models specific to CT IQA. Addressing these challenges, we organized the Low-dose Computed Tomography Perceptual Image Quality Assessment Challenge in conjunction with the Medical Image Computing and Computer Assisted Intervention 2023. This event introduced the first open-source CT IQA dataset, consisting of 1,000 CT images of various quality, annotated with radiologists’ assessment scores. As a benchmark, this challenge offers a comprehensive analysis of six submitted methods, providing valuable insight into their performance. This paper presents a summary of these methods and insights. This challenge underscores the potential for developing no-reference IQA methods that could exceed the capabilities of full-reference IQA methods, making a significant contribution to the research community with this novel dataset. The dataset is accessible at https://zenodo.org/records/7833096.

ConIQA: A deep learning method for perceptual image quality assessment with limited data

Effectively assessing the realism and naturalness of images in virtual (VR) and augmented (AR) reality applications requires Full Reference Image Quality Assessment (FR-IQA) metrics that closely align with human perception. Deep learning-based IQAs that are trained on human-labeled data have recently shown promise in generic computer vision tasks. However, their performance decreases in applications where perfect matches between the reference and the distorted images should not be expected, or whenever distortion patterns are restricted to specific domains. Tackling this issue necessitates training a task-specific neural network, yet generating human-labeled FR-IQAs is costly, and deep learning typically demands substantial labeled data. To address these challenges, we developed ConIQA, a deep learning-based IQA that leverages consistency training and a novel data augmentation method to learn from both labeled and unlabeled data. This makes ConIQA well-suited for contexts with scarce labeled data. To validate ConIQA, we considered the example application of Computer-Generated Holography (CGH) where specific artifacts such as ringing, speckle, and quantization errors routinely occur, yet are not explicitly accounted for by existing IQAs. We developed a new dataset, HQA1k, that comprises 1000 natural images each paired with an image rendered using various popular CGH algorithms, and quality-rated by thirteen human participants. Our results show that ConIQA achieves superior Pearson (0.98), Spearman (0.965), and Kendall’s tau (0.86) correlations over fifteen FR-IQA metrics by up to 5%, showcasing significant improvements in aligning with human perception on the HQA1k dataset.

No-reference Bitstream-based Perceptual Quality Assessment of Octree-Lifting Encoded 3D Point Clouds.

No-reference point cloud quality assessment (PCQA) based on bitstreams uses information extracted from the bitstream for quality monitoring at network nodes. We develop a no-reference PCQA model based on bitstreams for the perceived quality assessment of Octree-Lifting coded point clouds. At first, our research explores the essential correlation between subjective visual quality degradation and the texture quantization parameter (TQP) when using lossless geometric coding. Then, we enhance the proposed model by incorporating texture complexity (TC) while taking into account the dependence of perceptual coding distortion on the texture characteristics of a point cloud. We estimate TC by utilizing TQP and calculating the average standard deviation of the Y-component of the attribute value ( Y_ std), both of which are extracted from the bitstream. Then, a texture distortion assessment model is constructed based on TQP and Y_ std. The integration of the texture distortion model with the position quantization scale (PQS) results in the derivation of an overall no-reference bitstream-based PCQA model, named streamPCQ-OL. The findings from the conducted experiments highlight a significant superiority of the proposed model over existing approaches in terms of performance. The dataset and source code will be publicly released and made available for access at https://github.com/qdushl/Waterloo-Point-Cloud-Database-4.0.

Perceptual point cloud qality assessment for immersive metaverse experience

Perceptual quality assessment for point cloud is critical for immersive metaverse experience and is a challenging task. Firstly, because point cloud is formed by unstructured 3D points that makes the topology more complex. Second, the quality impairment generally involves both geometric attributes and color properties, where the measurement of the geometric distortion becomes more complex. We proposed a perceptual point cloud quality assessment model that follows the perceptual features of Human Visual System (HVS) and the intrinsic characteristics of the point cloud. The point cloud is first pre-processed to extract the geometric skeleton keypoints with graph filtering-based re-sampling, and local neighboring regions around the geometric skeleton keypoints are constructed by K-Nearest Neighbors (KNN) clustering. For geometric distortion, the Point Feature Histogram (PFH) is extracted as the feature descriptor, and the Earth Mover's Distance (EMD) between the PFHs of the corresponding local neighboring regions in the reference and the distorted point clouds is calculated as the geometric quality measurement. For color distortion, the statistical moments between the corresponding local neighboring regions are computed as the color quality measurement. Finally, the global perceptual quality assessment model is obtained as the linear weighting aggregation of the geometric and color quality measurement. The experimental results on extensive datasets show that the proposed method achieves the leading performance as compared to the state-of-the-art methods with less computing time. Meanwhile, the experimental results also demonstrate the robustness of the proposed method across various distortion types. The source codes are available at https://github.com/llsurreal919/PointCloudQualityAssessment.

Enhanced Solar Coronal Imaging: A GAN Approach with Fused Attention and Perceptual Quality Enhancement

The activity of the solar corona has a significant impact on all aspects of human life. People typically use images obtained from astronomical telescopes to observe coronal activities, among which the Atmospheric Imaging Assembly (AIA) of the Solar Dynamics Observatory (SDO) is particularly widely used. However, due to resolution limitations, we have begun to study the application of generative adversarial network super-resolution techniques to enhance the image data quality for a clearer observation of the fine structures and dynamic processes in the solar atmosphere, which improves the prediction accuracy of solar activities. We aligned SDO/AIA images with images from the High-Resolution Coronal Imager (Hi-C) to create a dataset. This research proposes a new super-resolution method named SAFCSRGAN, which includes a spatial attention module that incorporates channel information, allowing the network model to better capture the corona’s features. A Charbonnier loss function was introduced to enhance the perceptual quality of the super-resolution images. Compared to the original method using ESRGAN, our method achieved an 11.9% increase in Peak Signal-to-Noise Ratio (PSNR) and a 4.8% increase in Structural Similarity (SSIM). Additionally, we introduced two perceptual image quality assessment metrics, the Natural Image Quality Evaluator (NIQE) and Learned Perceptual Image Patch Similarity (LPIPS), which improved perceptual quality by 10.8% and 1.3%, respectively. Finally, our experiments demonstrated that our improved model surpasses other models in restoring the details of coronal images.

Perceptual Quality Assessment of NeRF and Neural View Synthesis Methods for Front‐Facing Views

AbstractNeural view synthesis (NVS) is one of the most successful techniques for synthesizing free viewpoint videos, capable of achieving high fidelity from only a sparse set of captured images. This success has led to many variants of the techniques, each evaluated on a set of test views typically using image quality metrics such as PSNR, SSIM, or LPIPS. There has been a lack of research on how NVS methods perform with respect to perceived video quality. We present the first study on perceptual evaluation of NVS and NeRF variants. For this study, we collected two datasets of scenes captured in a controlled lab environment as well as in‐the‐wild. In contrast to existing datasets, these scenes come with reference video sequences, allowing us to test for temporal artifacts and subtle distortions that are easily overlooked when viewing only static images. We measured the quality of videos synthesized by several NVS methods in a well‐controlled perceptual quality assessment experiment as well as with many existing state‐of‐the‐art image/video quality metrics. We present a detailed analysis of the results and recommendations for dataset and metric selection for NVS evaluation.

Knowledge Guided Semi-supervised Learning for Quality Assessment of User Generated Videos

Perceptual quality assessment of user generated content (UGC) videos is challenging due to the requirement of large scale human annotated videos for training. In this work, we address this challenge by first designing a self-supervised Spatio-Temporal Visual Quality Representation Learning (ST-VQRL) framework to generate robust quality aware features for videos. Then, we propose a dual-model based Semi Supervised Learning (SSL) method specifically designed for the Video Quality Assessment (SSL-VQA) task, through a novel knowledge transfer of quality predictions between the two models. Our SSL-VQA method uses the ST-VQRL backbone to produce robust performances across various VQA datasets including cross-database settings, despite being learned with limited human annotated videos. Our model improves the state-of-the-art performance when trained only with limited data by around 10%, and by around 15% when unlabelled data is also used in SSL. Source codes and checkpoints are available at https://github.com/Shankhanil006/SSL-VQA.

Perceptual Quality Assessment of Omnidirectional Images: A Benchmark and Computational Model

Compared with traditional 2D images, omnidirectional images (also referred to as 360 ∘ images) have more complicated perceptual characteristics due to the particularities of imaging and display. How humans perceive omnidirectional images in an immersive environment and form the immersive quality of experience are important problems. Thus, it is crucial to measure the quality of omnidirectional images under different viewing conditions, which suffer from realistic distortions. In this article, we build a large-scale subjective assessment database for omnidirectional images and carry out a comprehensive psychophysical experiment to study the relationships between different factors (viewing conditions and viewing behaviors) and the perceptual quality of omnidirectional images. In addition, we collect both subjective ratings and head movement data. A thorough analysis of the collected subjective data is also provided, where we make several interesting findings. Moreover, with the proposed database, we propose a novel transformer-based omnidirectional image quality assessment model. To be consistent with the human viewing process, viewing conditions and behaviors are naturally incorporated into the proposed model. Specifically, the proposed model mainly consists of three parts: viewport sequence generation, multi-scale feature extraction, and perceptual quality prediction. Extensive experimental results conducted on the proposed database demonstrate the effectiveness of the proposed method over existing image quality assessment methods.

Blind Perceptual Quality Assessment of LFI Based on Angular-Spatial Effect Modeling

Blind Perceptual Quality Assessment of LFI Based on Angular-Spatial Effect Modeling

CN-BSRIQA: Cascaded network - blind super-resolution image quality assessment

High resolution (HR) images consist of higher quality and more detail information in comparison to low-resolution images. But obtaining HR images entails higher costs and requires a larger workforce. The solution is using super-resolution (SR) images. Single image SR is the process of creating HR images from a single low-resolution image. Obtaining SR images from low-resolution images is a well-known problem in the domains of computer vision and image processing. Advancement in technology has given rise to many SR algorithms. The perceptual quality assessment of super-resolved images can be used to benchmark the techniques employed for image SR. In this work, a cascaded network-blind super-resolution image quality assessment (CN-BSRIQA) methodology is proposed. The proposed approach works under the cascaded architecture where a convolutional neural network (CNN) is cascaded with a deep belief network (DBN). CNN is employed as a shallow network in the proposed methodology to extract low-level information after partitioning the input image into patches. To assess the visual quality of the SR images, the features retrieved from CNN are incorporated into a DBN. Three databases are used to evaluate the performance of proposed CN-BSRIQA i.e., SR Quality Database (SRQD), SR Image Quality Database (SRID), and visual quality evaluation for super-resolved images (QADS). When compared to other state-of-the-art methodologies for assessing the visual quality of SR images, CN-BSRIQA outperforms them. For the perceptual quality assessment of SR images, the experimental results reveal that CNN-based techniques outperform techniques based on hand-crafted features. Furthermore, the shallow CNN proposed in the CN-BSRIQA can extract features that are content-independent i.e., they show better performance over cross-database evaluation in comparison to existing state-of-the-art techniques.

FF-PPQA: Face frontalization without glasses based on perceptual quality and pixel-level quality assessment

FF-PPQA: Face frontalization without glasses based on perceptual quality and pixel-level quality assessment

Perceptual Information Completion-Based Siamese Omnidirectional Image Quality Assessment Network

Perceptual Information Completion-Based Siamese Omnidirectional Image Quality Assessment Network

Perceptual Quality Assessment of Retouched Face Images

Perceptual Quality Assessment of Retouched Face Images

Perceptual Quality Assessment of Face Video Compression: A Benchmark and An Effective Method

Perceptual Quality Assessment of Face Video Compression: A Benchmark and An Effective Method

Social Media Health-Related Information Credibility and Reliability: An Integrated User Perceived Quality Assessment

Social Media Health-Related Information Credibility and Reliability: An Integrated User Perceived Quality Assessment

USCFormer: Unified Transformer With Semantically Contrastive Learning for Image Dehazing

Haze severely degrades the visibility of scene objects and deteriorates the performance of autonomous driving, traffic monitoring, and other vision-based intelligent transportation systems. As a potential remedy, we propose a novel unified Transformer with semantically contrastive learning for image dehazing, dubbed USCFormer. USCFormer has three key contributions. First, USCFormer absorbs the respective strengths of CNN and Transformer by incorporating them into a unified Transformer format. Thus, it allows the simultaneous capture of global-local dependency features for better image dehazing. Second, by casting clean/hazy images as the positive/negative samples, the contrastive constraint encourages the restored image to be closer to the ground-truth images (positives) and away from the hazy ones (negatives). Third, we regard the semantic information as important prior knowledge to help USCFormer mitigate the effects of haze on the scene and preserve image details and colors by leveraging intra-object semantic correlation. Experiments on synthetic datasets and real-world hazy photos fully validate the superiority of USCFormer in both perceptual quality assessment and subjective evaluation. Code is available at https://github.com/yz-wang/USCFormer.

Perceptual Quality Assessment of Enhanced Colonoscopy Images: A Benchmark Dataset and an Objective Method

In colonoscopy, the captured images are usually with low-quality appearance, such as non-uniform illumination, low contrast, etc., due to the specialized imaging environment, which may provide poor visual feedback and bring challenges to subsequent disease analysis. Many low-light image enhancement (LIE) algorithms have recently proposed to improve the perceptual quality. However, how to fairly evaluate the quality of enhanced colonoscopy images (ECIs) generated by different LIE algorithms remains a rarely-mentioned and challenging problem. In this study, we carry out a pioneering investigation on perceptual quality assessment of ECIs. Firstly, considering the lack of specific datasets, we collect 300 low-light images with diverse contents during the real-world colonoscopy and conduct rigorous subjective studies to compare the performance of 8 popular LIE methods, resulting in a benchmark dataset (named ECIQAD) for ECIs. Secondly, in view of the distinctive distortion characteristics of ECIs, we propose an effective no-reference Enhanced Colonoscopy Image Quality (ECIQ) method to automatically evaluate the perceptual quality of ECIs via analysis of brightness, contrast, colorfulness, naturalness, and noise. Extensive experiments on ECIQAD demonstrate the superiority of our proposed ECIQ method over 14 mainstream no-reference image quality assessment methods.

Improving Structural MRI Preprocessing with Hybrid Transformer GANs.

Magnetic resonance imaging (MRI) is a technique that is widely used in practice to evaluate any pathologies in the human body. One of the areas of interest is the human brain. Naturally, MR images are low-resolution and contain noise due to signal interference, the patient's body's radio-frequency emissions and smaller Tesla coil counts in the machinery. There is a need to solve this problem, as MR tomographs that have the capability of capturing high-resolution images are extremely expensive and the length of the procedure to capture such images increases by the order of magnitude. Vision transformers have lately shown state-of-the-art results in super-resolution tasks; therefore, we decided to evaluate whether we can employ them for structural MRI super-resolution tasks. A literature review showed that similar methods do not focus on perceptual image quality because upscaled images are often blurry and are subjectively of poor quality. Knowing this, we propose a methodology called HR-MRI-GAN, which is a hybrid transformer generative adversarial network capable of increasing resolution and removing noise from 2D T1w MRI slice images. Experiments show that our method quantitatively outperforms other SOTA methods in terms of perceptual image quality and is capable of subjectively generalizing to unseen data. During the experiments, we additionally identified that the visual saliency-induced index metric is not applicable to MRI perceptual quality assessment and that general-purpose denoising networks are effective when removing noise from MR images.