No-reference Research Articles

Point Cloud Quality Assessment Using a One-Dimensional Model Based on the Convolutional Neural Network.

Recent advancements in 3D modeling have revolutionized various fields, including virtual reality, computer-aided diagnosis, and architectural design, emphasizing the importance of accurate quality assessment for 3D point clouds. As these models undergo operations such as simplification and compression, introducing distortions can significantly impact their visual quality. There is a growing need for reliable and efficient objective quality evaluation methods to address this challenge. In this context, this paper introduces a novel methodology to assess the quality of 3D point clouds using a deep learning-based no-reference (NR) method. First, it extracts geometric and perceptual attributes from distorted point clouds and represent them as a set of 1D vectors. Then, transfer learning is applied to obtain high-level features using a 1D convolutional neural network (1D CNN) adapted from 2D CNN models through weight conversion from ImageNet. Finally, quality scores are predicted through regression utilizing fully connected layers. The effectiveness of the proposed approach is evaluated across diverse datasets, including the Colored Point Cloud Quality Assessment Database (SJTU_PCQA), the Waterloo Point Cloud Assessment Database (WPC), and the Colored Point Cloud Quality Assessment Database featured at ICIP2020. The outcomes reveal superior performance compared to several competing methodologies, as evidenced by enhanced correlation with average opinion scores.

GMS-3DQA: Projection-Based Grid Mini-patch Sampling for 3D Model Quality Assessment

Nowadays, most three-dimensional model quality assessment (3DQA) methods have been aimed at improving accuracy. However, little attention has been paid to the computational cost and inference time required for practical applications. Model-based 3DQA methods extract features directly from the 3D models, which are characterized by their high degree of complexity. As a result, many researchers are inclined towards utilizing projection-based 3DQA methods. Nevertheless, previous projection-based 3DQA methods directly extract features from multi-projections to ensure quality prediction accuracy, which calls for more resource consumption and inevitably leads to inefficiency. Thus, in this article, we address this challenge by proposing a no-reference (NR) projection-based G rid M ini-patch S ampling 3D Model Q uality A ssessment (GMS-3DQA) method. The projection images are rendered from six perpendicular viewpoints of the 3D model to cover sufficient quality information. To reduce redundancy and inference resources, we propose a multi-projection grid mini-patch sampling strategy (MP-GMS), which samples grid mini-patches from the multi-projections and forms the sampled grid mini-patches into one quality mini-patch map (QMM). The Swin-Transformer tiny backbone is then used to extract quality-aware features from the QMMs. The experimental results show that the proposed GMS-3DQA outperforms existing state-of-the-art NR-3DQA methods on the point cloud quality assessment databases for both accuracy and efficiency. The efficiency analysis reveals that the proposed GMS-3DQA requires far less computational resources and inference time than other 3DQA competitors. The code is available at https://github.com/zzc-1998/GMS-3DQA .

Auxiliary Information Guided Self-attention for Image Quality Assessment

Image quality assessment (IQA) is an important problem in computer vision with many applications. We propose a transformer-based multi-task learning framework for the IQA task. Two subtasks: constructing an auxiliary information error map and completing image quality prediction, are jointly optimized using a shared feature extractor. We use visual transformers (ViT) as a feature extractor for feature extraction and guide ViT to focus on image quality-related features by building auxiliary information error map subtask. In particular, we propose a fusion network that includes a channel focus module. Unlike the fusion methods commonly used in previous IQA methods, we use the fusion network, including the channel attention module, to fuse the auxiliary information error map features with the image features, which facilitates the model to mine the image quality features for more accurate image quality assessment. And by jointly optimizing the two subtasks, ViT focuses more on extracting image quality features and building a more precise mapping from feature representation to quality score. With slight adjustments to the model, our approach can be used in both no-reference (NR) and full-reference (FR) IQA environments. We evaluate the proposed method in multiple IQA databases, showing better performance than state-of-the-art FR and NR IQA methods.

FAVER: Blind quality prediction of variable frame rate videos

Video quality assessment (VQA) remains an important and challenging problem that affects many applications at the widest scales. Recent advances in mobile devices and cloud computing techniques have made it possible to capture, process, and share high resolution, high frame rate (HFR) videos across the Internet nearly instantaneously. Being able to monitor and control the quality of these streamed videos can enable the delivery of more enjoyable content and perceptually optimized rate control. Accordingly, there is a pressing need to develop VQA models that can be deployed at enormous scales. While some recent effects have been applied to full-reference (FR) analysis of variable frame rate and HFR video quality, the development of no-reference (NR) VQA algorithms targeting frame rate variations has been little studied. Here, we propose a first-of-a-kind blind VQA model for evaluating HFR videos, which we dub the Framerate-Aware Video Evaluator w/o Reference (FAVER). FAVER uses extended models of spatial natural scene statistics that encompass space–time wavelet-decomposed video signals, and leverages the advantages of the deep neural network to provide motion perception, to conduct efficient frame rate sensitive quality prediction. Our extensive experiments on several HFR video quality datasets show that FAVER outperforms other blind VQA algorithms at a reasonable computational cost. To facilitate reproducible research and public evaluation, an implementation of FAVER is being made freely available online: https://github.com/uniqzheng/HFR-BVQA.

TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessment.

Image Quality Assessment (IQA) is a fundamental task in computer vision that has witnessed remarkable progress with deep neural networks. Inspired by the characteristics of the human visual system, existing methods typically use a combination of global and local representations (i.e., multi-scale features) to achieve superior performance. However, most of them adopt simple linear fusion of multi-scale features, and neglect their possibly complex relationship and interaction. In contrast, humans typically first form a global impression to locate important regions and then focus on local details in those regions. We therefore propose a top-down approach that uses high-level semantics to guide the IQA network to focus on semantically important local distortion regions, named as TOPIQ. Our approach to IQA involves the design of a heuristic coarse-to-fine network (CFANet) that leverages multi-scale features and progressively propagates multi-level semantic information to low-level representations in a top-down manner. A key component of our approach is the proposed cross-scale attention mechanism, which calculates attention maps for lower level features guided by higher level features. This mechanism emphasizes active semantic regions for low-level distortions, thereby improving performance. TOPIQ can be used for both Full-Reference (FR) and No-Reference (NR) IQA. We use ResNet50 as its backbone and demonstrate that TOPIQ achieves better or competitive performance on most public FR and NR benchmarks compared with state-of-the-art methods based on vision transformers, while being much more efficient (with only ~13% FLOPS of the current best FR method). Codes are released at https://github.com/chaofengc/IQA-PyTorch.

Hallucinated-PQA: No reference point cloud quality assessment via injecting pseudo-reference features

Point clouds (PCs) have been increasingly applied in business and life, but a variety of PC transmission and generation systems inevitably produce various types of distortions. Therefore, it is of great significance to design an objective point cloud quality assessment (PCQA), particularly in a no-reference manner to evaluate the PC systems in the actual situation. However, due to the lack of original PCs, the existing no-reference (NR) PCQA metrics cannot perceive the feature changes caused by distortions, resulting in inaccurate quality prediction. In addition, dependent on the viewing habits of the subjects in the subjective evaluation experiment, a strong contextual correlation among intra- and inter-view local regions at different scales naturally exists, but the existing projection-based no-reference PCQA only using concatenation or average pooling operations cannot reflect this relationship of multi-view features fusion. Considering the above challenges, we propose a novel hallucination-guided NR PCQA framework, namely Hallucinated-PQA. Specifically, we introduce a distortion restoration network to correct multiple projected images in preprocessing to provide pseudo-reference information for NR PCQA. In the feature extraction of distorted PCs, we designed a hallucination injection block (HIB) by utilizing feature differences to assist the feature description of distorted PCs, and a multi-view and multi-scale context fusion (MMCF) module to construct the contextual correlation among intra- and inter-view local regions at different scales. Experimental results show that our Hallucinated-PQA can achieve comparable or better performance than state-of-the-art (SOTA) metrics on four open PCQA databases. The source code will be released at https://github.com/QSBAOYANGMU/Hallucinated-PQA.

Subjective and Objective Quality Assessment for in-the-Wild Computer Graphics Images

Computer graphics images (CGIs) are artificially generated by means of computer programs and are widely perceived under various scenarios, such as games, streaming media, etc. In practice, the quality of CGIs consistently suffers from poor rendering during production, inevitable compression artifacts during the transmission of multimedia applications, and low aesthetic quality resulting from poor composition and design. However, few works have been dedicated to dealing with the challenge of computer graphics image quality assessment (CGIQA). Most image quality assessment (IQA) metrics are developed for natural scene images (NSIs) and validated on databases consisting of NSIs with synthetic distortions, which are not suitable for in-the-wild CGIs. To bridge the gap between evaluating the quality of NSIs and CGIs, we construct a large-scale in-the-wild CGIQA database consisting of 6,000 CGIs (CGIQA-6k) and carry out the subjective experiment in a well-controlled laboratory environment to obtain the accurate perceptual ratings of the CGIs. Then, we propose an effective deep learning–based no-reference (NR) IQA model by utilizing both distortion and aesthetic quality representation. Experimental results show that the proposed method outperforms all other state-of-the-art NR IQA methods on the constructed CGIQA-6k database and other CGIQA-related databases. The database is released athttps://github.com/zzc-1998/CGIQA6K.

Full-Reference Image Quality Assessment via Low-Level and High-Level Feature Fusion

We propose a full-reference image quality assessment (FR-IQA) method by incorporating low-level and high-level image features. First, in contrast to the preexisting deep IQA methods, which only use the features extracted by the deep network, we not only use the image gradient to replace the low-level features in the first two stages of the deep network, but also combine them with the middle-stage features of the deep network to construct the new low-level features. The deep features of shallow layers contain some unwanted noise and further result in a decline in IQA performance. Second, we combine the global features extracted by the self-attention-based model with the semantic features extracted by the convolutional neural network to form the high-level features. Instead of directly using the self-attention-based model trained on the classification task, we first train a no-reference (NR) IQA regression model on a larger dataset and then use the global features of this NR-IQA model. The self-attention-based model can capture the internal connections of the image and is more effective in extracting global information due to its larger perceptual field. In the final pooling stage, we combine the average pooling and the standard deviation pooling to obtain the dispersion and concentration of the similarity maps for a more comprehensive description of quality. Experiments show that our FR-IQA method is able to obtain competitive results on three standard IQA datasets.

Perception-Oriented U-Shaped Transformer Network for 360-Degree No-Reference Image Quality Assessment

Generally, 360-degree images have absolute senses of reality and three-dimensionality, providing a wide range of immersive interactions. Due to the novel rendering and display technology of 360-degree images, they have more complex perceptual characteristics than other images. It is challenging to perform comprehensive image quality assessment (IQA) learning by simply stacking multichannel neural network architectures for pre/postprocessing, compression, and rendering tasks. To thoroughly learn the global and local features in 360-degree images, reduce the complexity of multichannel neural network models and simplify the training process, this paper proposes a joint architecture with user perception and an efficient transformer dedicated to 360-degree no-reference (NR) IQA. The input of the proposed method is a 360-degree cube map projection (CMP) image. Furthermore, the proposed 360-degree NRIQA method includes a saliency map-based non-overlapping self-attention selection module and a U-shaped transformer (U-former)-based feature extraction module to account for perceptual region importance and projection distortion. The transformer-based architecture and the weighted average technique are jointly utilized for predicting local perceptual quality. Experimental results obtained on widely used databases show that the proposed model outperforms other state-of-the-art methods in NR 360-degree image quality evaluation cases. Furthermore, a cross-database evaluation and an ablation study also demonstrate the inherent robustness and generalization ability of the proposed model.

An End-to-End Blind Image Quality Assessment Method Using a Recurrent Network and Self-Attention

In this paper, we propose a blind image quality assessment (BIQA) method using self-attention and a recurrent neural network (RNN); this approach can effectively capture both local and global information from an input image. The implementation of our constructed deep no-reference (NR) assessment framework does not rely on any convolutional operations. First, the capture step for obtaining locally significant information is performed by a self-attention operation inside a divided window. Second, we design a serialized feature input memory subnetwork to fuse the global features of the image. Finally, all the integrated features are uniformly mapped to the target score. The experimental results obtained on publicly available benchmark IQA databases show that our approach outperforms other state-of-the-art algorithms.

Toward a No-Reference Quality Metric for Camera-Captured Images.

Existing no-reference (NR) image quality assessment (IQA) metrics are still not convincing for evaluating the quality of the camera-captured images. Toward tackling this issue, we, in this article, establish a novel NR quality metric for quantifying the quality of the camera-captured images reliably. Since the image quality is hierarchically perceived from the low-level preliminary visual perception to the high-level semantic comprehension in the human brain, in our proposed metric, we characterize the image quality by exploiting both the low-level image properties and the high-level semantics of the image. Specifically, we extract a series of low-level features to characterize the fundamental image properties, including the brightness, saturation, contrast, noiseness, sharpness, and naturalness, which are highly indicative of the camera-captured image quality. Correspondingly, the high-level features are designed to characterize the semantics of the image. The low-level and high-level perceptual features play complementary roles in measuring the image quality. To infer the image quality, we employ the support vector regression (SVR) to map all the informative features to a single quality score. Thorough tests conducted on two standard camera-captured image databases demonstrate the effectiveness of the proposed quality metric in assessing the image quality and its superiority over the state-of-the-art NR quality metrics. The source code of the proposed metric for camera-captured images is released at https://github.com/YT2015?tab=repositories.

No-Reference Image Quality Assessment of Magnetic Resonance images with multi-level and multi-model representations based on fusion of deep architectures

Accurate quality assessment of Magnetic Resonance (MR) images is essential for effective medical diagnostics, as it impacts the time spent on image acquisition and image interpretation by radiologists. Therefore, this study aims to provide a novel deep learning-based No-Reference (NR) MR Image Quality Assessment (IQA) method for the quality prediction of MR images. Therefore, in this work, an internal fusion of two complementary deep learning architectures is introduced that offers MR-specific quality-aware features. Apart from obtaining multi-level image representations of joint networks, to further enhance the quality prediction performance, features from re-trained single deep learning architectures are also used. Finally, as one of the main findings of this study, the quality assessment is performed by a high-level quality model trained on scores of quality models obtained for layers of the networks. The superiority of the method against the state-of-the-art techniques is verified by using two publicly available MRIQA benchmarks containing MR images and subjective scores provided by a large number of radiologists. As reported, the method offers superior IQA of MR images as the obtained scores are highly correlated with subjective opinions of medical specialists. It is characterized by the weighted average values of the Spearman Rank-order Correlation Coefficient, Kendall Rank-order Correlation Coefficient, and Pearson Linear Correlation Coefficient of 0.8754, 0.7185, and 0.9062, respectively.

No-reference quality assessment for low-light image enhancement: Subjective and objective methods

No-Reference (NR) quality assessment is a crucial approach for evaluating the quality of low-light enhanced images, as it is often difficult to acquire high-quality reference images in applications such as night-time automatic driving. However, current NR evaluation methods for low-light enhanced images often lack consideration of important characteristics such as color, structure, and naturalness. This paper proposes a novel NR quality assessment method for NR low-light enhanced images from both subjective and objective aspects. On the subjective side, we construct the Low-light Enhanced Images Subjective Dataset (LEISD) containing 2040 images with 255 different image contents. Each image was evaluated based on the Single Stimulus (SS) method by 20 subjects. On the objective side, we propose Multi-Features Reconciliation-based Quality Assessment (MFRQA) methods for low-light enhanced images by observing the low-light enhanced images. The MFRQA summarized four key feature perspectives: brightness, color, structure and naturalness, and employed the traditional machine learning model to reconcile the multi-features. Experimental results on the LEISD dataset demonstrate competitive performance and low complexity of our method compared to the representative quality metrics.

Why No Reference Metrics for Image and Video Quality Lack Accuracy and Reproducibility

This article provides a comprehensive overview of no reference (NR) metrics for image quality analysis (IQA) and video quality analysis (VQA). We examine 26 independent evaluations of NR metrics (previously published) and analyze 32 NR metrics on six IQA datasets and six VQA datasets (new results). Where NR metric developers claim Pearson correlation values between 0.66 and 0.99, our measurements range from 0.0 to 0.63. None of the NR metrics we analyzed are accurate enough to be deployed by industry. Performance evaluations that indicate otherwise are based on insufficient data and highly inaccurate. We will examine development strategies, tools, datasets, root cause analysis, and our baseline metric for collaboration, Sawatch.

Self-Supervised Representation Learning for Video Quality Assessment

No-reference (NR) video quality assessment (VQA) is a challenging problem due to the difficulty in model training caused by insufficient annotation samples. Previous work commonly utilizes transfer learning to directly migrate pre-trained models on the image database, which suffers from domain inadaptation. Recently, self-supervised representation learning has become a hot spot for the independence of large-scale labeled data. However, existing self-supervised representation learning method only considers the distortion types and contents of the video, there needs to investigate the intrinsic properties of videos for the VQA task. To amend this, here we propose a novel multi-task self-supervised representation learning framework to pre-train a video quality assessment model. Specifically, we consider the effects of distortion degrees, distortion types, and frame rates on the perceived quality of videos, and utilize them as guidance to generate self-supervised samples and labels. Then, we optimize the ability of the VQA model in capturing spatio-temporal differences between the original video and the distorted version using three pretext tasks. The resulting framework not only eases the requirements for the quality of the original video but also benefits from the self-supervised labels as well as the Siamese network. In addition, we propose a Transformer-based VQA model, where short-term spatio-temporal dependencies of videos are modeled by 3D-CNN and 2D-CNN, and then the long-term spatio-temporal dependencies are modeled by Transformer because of its excellent long-term modeling capability. We evaluated the proposed method on four public video quality assessment databases and found that it is competitive with all compared VQA algorithms.

Spatiotemporal Feature Hierarchy-Based Blind Prediction of Natural Video Quality via Transfer Learning

In this paper, we propose a pyramidal spatiotemporal feature hierarchy (PSFH)-based no-reference (NR) video quality assessment (VQA) method using transfer learning. First, we generate simulated videos by a generative adversarial network (GAN)-based image restoration model. The residual maps between the distorted frames and simulated frames, which can capture rich information, are utilized as one input of the quality regression network. Second, we use 3D convolution operations to construct a PSFH network with five stages. The spatiotemporal features incorporating the shared features transferred from the pretrained image restoration model are fused stage by stage. Third, with the guidance of the transferred knowledge, each stage generates multiple feature mapping layers that encode different semantics and degradation information using 3D convolution layers and gated recurrent units (GRUs). Finally, five approximate perceptual quality scores and a precise prediction score are obtained by fully connected (FC) networks. The whole model is trained under a finely designed loss function that combines pseudo-Huber loss and Pearson linear correlation coefficient (PLCC) loss to improve the robustness and prediction accuracy. According to the extensive experiments, outstanding results can be obtained compared with other state-of-the-art methods. Both the source code and models are available online. <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</xref>

UID2021: An Underwater Image Dataset for Evaluation of No-Reference Quality Assessment Metrics

Achieving subjective and objective quality assessment of underwater images is of high significance in underwater visual perception and image/video processing. However, the development of underwater image quality assessment (UIQA) is limited for the lack of publicly available underwater image datasets with human subjective scores and reliable objective UIQA metrics. To address this issue, we establish a large-scale underwater image dataset, dubbed UID2021, for evaluating no-reference (NR) UIQA metrics. The constructed dataset contains 60 multiply degraded underwater images collected from various sources, covering six common underwater scenes (i.e., bluish scene, blue-green scene, greenish scene, hazy scene, low-light scene, and turbid scene), and their corresponding 900 quality improved versions are generated by employing 15 state-of-the-art underwater image enhancement and restoration algorithms. Mean opinion scores with 52 observers for each image of UID2021 are also obtained by using the pairwise comparison sorting method. Both in-air and underwater-specific NR IQA algorithms are tested on our constructed dataset to fairly compare their performance and analyze their strengths and weaknesses. Our proposed UID2021 dataset enables ones to evaluate NR UIQA algorithms comprehensively and paves the way for further research on UIQA. The dataset is available at https://github.com/Hou-Guojia/UID2021 .

Point Cloud Quality Assessment: Dataset Construction and Learning-based No-reference Metric

Full-reference (FR) point cloud quality assessment (PCQA) has achieved impressive progress in recent years. However, in many cases, obtaining the reference point clouds is difficult, so no-reference (NR) metrics have become a research hotspot. Few researches about NR-PCQA are carried out due to the lack of a large-scale PCQA dataset. In this article, we first build a large-scale PCQA dataset named LS-PCQA, which includes 104 reference point clouds and more than 22,000 distorted samples. In the dataset, each reference point cloud is augmented with 31 types of impairments (e.g., Gaussian noise, contrast distortion, local missing, and compression loss) at 7 distortion levels. Besides, each distorted point cloud is assigned with a pseudo-quality score as its substitute of Mean Opinion Score. Inspired by the hierarchical perception system and considering the intrinsic attributes of point clouds, we propose a NR metric ResSCNN based on sparse convolutional neural network (CNN) to accurately estimate the subjective quality of point clouds. We conduct several experiments to evaluate the performance of the proposed NR metric. The results demonstrate that ResSCNN exhibits the state-of-the-art performance among all the existing NR-PCQA metrics and even outperforms some FR metrics. The dataset presented in this work will be made publicly accessible at https://smt.sjtu.edu.cn . The source code for the proposed ResSCNN can be found at https://github.com/lyp22/ResSCNN .

A large-scale image database for benchmarking mobile camera quality and NR-IQA algorithms

Camera-oriented quality assessment (CQA) differs from traditional image quality assessment (IQA) in that “distorted” images are straight out of real devices instead of various types and levels of artificial operations. However, despite its value for both customers and manufacturers, academic and industrial fields, there are few CQA databases created years ago. To reflect recent mobile imaging advancements, we present a new massive Phone Camera Benchmarking database (PCB2021) in this paper. In PCB2021, 40 modern phone units featuring photography are simultaneously compared in 182 scenes for a total of 7280 images, which can be classified into six categories (sub-datasets) based on different focal lengths and user cases: main camera (107), ultra-wide (20), 2×, 3×, 5× zoom lenses (26, 8, 10) and night-mode (11). The shooting process begins from 7:00 am to 11:00 pm and lasts more than a month. In the subjective study, to overcome the high-resolution-induced overall quality evaluation difficulty, five image attributes: exposure/contrast, color, sharpness, graininess, artifacts are assessed separately on each dataset. To reduce ranking complexity for large-scale cameras, a dynamic anchor ruler method is proposed to obtain quality orders efficiently. With the constructed PCB2021, we further evaluate 15 mainstream no-reference (NR) IQA algorithms. The finding is that for zoomed images, sharpness metrics can achieve Spearman correlation coefficients above 0.8, while for the subtle main camera and night-mode images, performances of all fifteen algorithms drop down quickly, i.e. 0.1-0.2 for the former and ∼0.5 for the latter. The entire database, expert rankings and algorithm performance reports will be freely available on request.

Quadratic fitting model in no-reference image quality assessment

The perceptual quality of image is affected by distortions during compression, delivery and storage. Distortions also impact automatic image quality assessment (IQA) that needs to be highly correlated with subjective scores. In the absence of reference, which is a typical scenario in practice, no-reference (NR) metrics are necessary for quality measurements. Recently such methods are proposed, and they employ natural scene statistics (NSS). The experimental analysis performed in this paper takes into consideration two fitting or regression models of several NR-IQA metrics relying on different distortion types. The results show quadratic model as promising for making relations in terms of difference mean opinion score and Shannon entropy.