Full-reference Metrics Research Articles

GPA-Net:No-Reference Point Cloud Quality Assessment With Multi-Task Graph Convolutional Network.

With the rapid development of 3D vision, point cloud has become an increasingly popular 3D visual media content. Due to the irregular structure, point cloud has posed novel challenges to the related research, such as compression, transmission, rendering and quality assessment. In these latest researches, point cloud quality assessment (PCQA) has attracted wide attention due to its significant role in guiding practical applications, especially in many cases where the reference point cloud is unavailable. However, current no-reference metrics which based on prevalent deep neural network have apparent disadvantages. For example, to adapt to the irregular structure of point cloud, they require preprocessing such as voxelization and projection that introduce extra distortions, and the applied grid-kernel networks, such as Convolutional Neural Networks, fail to extract effective distortion-related features. Besides, they rarely consider the various distortion patterns and the philosophy that PCQA should exhibit shift, scaling, and rotation invariance. In this paper, we propose a novel no-reference PCQA metric named the Graph convolutional PCQA network (GPA-Net). To extract effective features for PCQA, we propose a new graph convolution kernel, i.e., GPAConv, which attentively captures the perturbation of structure and texture. Then, we propose the multi-task framework consisting of one main task (quality regression) and two auxiliary tasks (distortion type and degree predictions). Finally, we propose a coordinate normalization module to stabilize the results of GPAConv under shift, scale and rotation transformations. Experimental results on two independent databases show that GPA-Net achieves the best performance compared to the state-of-the-art no-reference PCQA metrics, even better than some full-reference metrics in some cases.

A No-Reference Quality Assessment Method for Hyperspectral Sharpened Images via Benford’s Law

In recent years, hyperspectral (HS) sharpening technology has received high attention and HS sharpened images have been widely applied. However, the quality assessment of HS sharpened images has not been well addressed and is still limited to the use of full-reference quality evaluation. In this paper, a novel no-reference quality assessment method based on Benford’s law for HS sharpened images is proposed. Without a reference image, the proposed method detects fusion distortion by performing first digit distribution on three quality perception features in HS sharpened images, using the standard Benford’s law as a benchmark. The experiment evaluates 10 HS fusion methods on three HS datasets and selects four full-reference metrics and four no-reference metrics to compare with the proposed method. The experimental results demonstrate the superior performance of the proposed method.

An image quality assessment index based on image features and keypoints for X-ray CT images.

Optimization tasks in diagnostic radiological imaging require objective quantitative metrics that correlate with the subjective perception of observers. However, although one such metric, the structural similarity index (SSIM), is popular, it has limitations across various aspects in its application to medical images. In this study, we introduce a novel image quality evaluation approach based on keypoints and their associated unique image feature values, focusing on developing a framework to address the need for robustness and interpretability that are lacking in conventional methodologies. The proposed index quantifies and visualizes the distance between feature vectors associated with keypoints, which varies depending on changes in the image quality. This metric was validated on images with varying noise levels and resolution characteristics, and its applicability and effectiveness were examined by evaluating images subjected to various affine transformations. In the verification of X-ray computed tomography imaging using a head phantom, the distances between feature descriptors for each keypoint increased as the image quality degraded, exhibiting a strong correlation with the changes in the SSIM. Notably, the proposed index outperformed conventional full-reference metrics in terms of robustness to various transformations which are without changes in the image quality. Overall, the results suggested that image analysis performed using the proposed framework could effectively visualize the corresponding feature points, potentially harnessing lost feature information owing to changes in the image quality. These findings demonstrate the feasibility of applying the novel index to analyze changes in the image quality. This method may overcome limitations inherent in conventional evaluation methodologies and contribute to medical image analysis in the broader domain.

Point Cloud Quality Assessment: Dataset Construction and Learning-based No-reference Metric

Full-reference (FR) point cloud quality assessment (PCQA) has achieved impressive progress in recent years. However, in many cases, obtaining the reference point clouds is difficult, so no-reference (NR) metrics have become a research hotspot. Few researches about NR-PCQA are carried out due to the lack of a large-scale PCQA dataset. In this article, we first build a large-scale PCQA dataset named LS-PCQA, which includes 104 reference point clouds and more than 22,000 distorted samples. In the dataset, each reference point cloud is augmented with 31 types of impairments (e.g., Gaussian noise, contrast distortion, local missing, and compression loss) at 7 distortion levels. Besides, each distorted point cloud is assigned with a pseudo-quality score as its substitute of Mean Opinion Score. Inspired by the hierarchical perception system and considering the intrinsic attributes of point clouds, we propose a NR metric ResSCNN based on sparse convolutional neural network (CNN) to accurately estimate the subjective quality of point clouds. We conduct several experiments to evaluate the performance of the proposed NR metric. The results demonstrate that ResSCNN exhibits the state-of-the-art performance among all the existing NR-PCQA metrics and even outperforms some FR metrics. The dataset presented in this work will be made publicly accessible at https://smt.sjtu.edu.cn . The source code for the proposed ResSCNN can be found at https://github.com/lyp22/ResSCNN .

Perceptual quality assessment for fine-grained compressed images

Recent years have witnessed the rapid development of image storage and transmission systems, in which image compression plays an important role. Generally speaking, image compression algorithms are developed to ensure good visual quality at limited bit rates. However, due to the different compression optimization methods, the compressed images may have different levels of quality, which needs to be evaluated quantificationally. Nowadays, the mainstream full-reference (FR) metrics are effective to predict the quality of compressed images at coarse-grained levels (the bit rates differences of compressed images are obvious), however, they may perform poorly for fine-grained compressed images whose bit rates differences are quite subtle. Therefore, to better improve the Quality of Experience (QoE) and provide useful guidance for compression algorithms, we propose a full-reference image quality assessment (FR-IQA) method for compressed images of fine-grained levels. Specifically, the reference images and compressed images are first converted to YCbCr color space. The gradient features are extracted from regions that are sensitive to compression artifacts. Then we employ the Log-Gabor transformation to further analyze the texture difference. Finally, the obtained features are fused into a quality score. The proposed method is validated on the fine-grained compression image quality assessment (FGIQA) database, which is especially constructed for assessing the quality of compressed images with close bit rates. The experimental results show that our metric outperforms mainstream FR-IQA metrics on the FGIQA database. We also test our method on other commonly used compression IQA databases and the results show that our method obtains competitive performance on the coarse-grained compression IQA databases as well.

No-Reference Quality Assessment for 3D Colored Point Cloud and Mesh Models

To improve the viewer's Quality of Experience (QoE) and optimize computer graphics applications, 3D model quality assessment (3D-QA) has become an important task in the multimedia area. Point cloud and mesh are the two most widely used digital representation formats of 3D models, the visual quality of which is quite sensitive to lossy operations like simplification and compression. Therefore, many related studies such as point cloud quality assessment (PCQA) and mesh quality assessment (MQA) have been carried out to measure the visual quality degradations of 3D models. However, a large part of previous studies utilize full-reference (FR) metrics, which indicates they can not predict the quality level with the absence of the reference 3D model. Furthermore, few 3D-QA metrics consider color information, which significantly restricts their effectiveness and scope of application. In this paper, we propose a no-reference (NR) quality assessment metric for colored 3D models represented by both point cloud and mesh. First, we project the 3D models from 3D space into quality-related geometry and color feature domains. Then, the 3D natural scene statistics (3D-NSS) and entropy are utilized to extract quality-aware features. Finally, machine learning is employed to regress the quality-aware features into visual quality scores. Our method is validated on the colored point cloud quality assessment database (SJTU-PCQA), the Waterloo point cloud assessment database (WPC), and the colored mesh quality assessment database (CMDM). The experimental results show that the proposed method outperforms most compared NR 3D-QA metrics with competitive computational resources and greatly reduces the performance gap with the state-of-the-art FR 3D-QA metrics. The code of the proposed model is publicly available now to facilitate further research.

Full-Reference Objective Quality Metric for Three-Dimensional Deformed Models

Three-dimensional data are generally represented by triangular meshes. The 3D data are used in several fields including remote 3D games, 3D medical application, 3D virtual worlds and 3D augmented reality application. These applications require displaying, printing or exchanging the 3D models through the network to optimize the rendering of the 3D models and 3D applications, which include different treatments, for example, smoothing, compression, re-meshing, simplification, watermarking, etc. However, these processes generate distortions that affect the quality of the rendered 3D data. Thus, subjective or objective metrics are required for assessing the visual quality of the deformed models to evaluate the efficiency of the applied algorithms. In this context, we introduce a new perceptual full-reference metric that compare two 3D meshes based on their 3D content information. The proposed metric integrates the relativity and selectivity properties of the Human visual system (HVS) independent of the mesh type and connectivity (e.g. Triangular, Quadrilateral, Tetrahedron, Hexahedron), which represent a limit in the existing method, in order to capture the perceptual quantity of the distortion by the observer. The results of the proposed approach outperform the existing metrics and have a high correlation with the subjective measures. We use the two correlation coefficients Spearman Rank ([Formula: see text]) and Pearson Rank (Rp) in order to assess the performance of the proposed metric.

From synthetic to natural — single natural image dehazing deep networks using synthetic dataset domain randomization

Image dehazing methods aim to solve the problem of poor visibility in images due to haze. Techniques proposed for image dehazing in literature focus on image priors, haze lines or data driven statistical models. Variations of the classical methods relying on prior model or haze line model use no-reference image quality metrics to prove their dehazing performance. Recently developed deep learning models rely on huge amounts of hazy, haze-free pairs for training, and uses PSNR and SSIM like image reconstruction metrics to show their performance. These methods perform poorly on no-reference image quality assessments and also dehazes poorly at the depths of the image. These methods though can be optimized for memory usage and are faster. This work presents a deep learning model (Feature Fusion Attention Network) trained on a domain randomized synthetic dataset generated in simulation. The proposed model achieves the highest scores on blind image assessments through the gradient rationing technique for a deep learning-based approach by a significant margin. The images were evaluated on full-reference metrics as well and obtained favorable results. This approach also yields one of the highest edge sharpness obtained after dehazing. The training procedure adopted to obtain significant gains on real-world dehazing, without using any real-world data is also detailed in this paper.

Automated assessment of transthoracic echocardiogram image quality using deep neural networks

BackgroundStandard views in two-dimensional echocardiography are well established but the qualities of acquired images are highly dependent on operator skills and are assessed subjectively. This study was aimed at providing an objective assessment pipeline for echocardiogram image quality by defining a new set of domain-specific quality indicators. Consequently, image quality assessment can thus be automated to enhance clinical measurements, interpretation, and real-time optimization. MethodsWe developed deep neural networks for the automated assessment of echocardiographic frames that were randomly sampled from 11,262 adult patients. The private echocardiography dataset consists of 33,784 frames, previously acquired between 2010 and 2020. Unlike non-medical images where full-reference metrics can be applied for image quality, echocardiogram's data are highly heterogeneous and requires blind-reference (IQA) metrics. Therefore, deep learning approaches were used to extract the spatiotemporal features and the image's quality indicators were evaluated against the mean absolute error. Our quality indicators encapsulate both anatomical and pathological elements to provide multivariate assessment scores for anatomical visibility, clarity, depth-gain and foreshortedness. ResultsThe model performance accuracy yielded 94.4%, 96.8%, 96.2%, 97.4% for anatomical visibility, clarity, depth-gain and foreshortedness, respectively. The mean model error of 0.375±0.0052 with computational speed of 2.52 ms per frame (real-time performance) was achieved. ConclusionThe novel approach offers new insight to the objective assessment of transthoracic echocardiogram image quality and clinical quantification in A4C and PLAX views. It also lays stronger foundations for the operator's guidance system which can leverage the learning curve for the acquisition of optimum quality images during the transthoracic examination.

Automated image quality appraisal through partial least squares discriminant analysis.

Automatic retinal fundus image quality analysis is one of the most essential preliminary stages in automatic computer-aided retinal disease diagnosis system, which allows good-quality fundus images for accurate disease prediction through localization and segmentation of retinal regions. This paper presents new feature extraction methods using full-reference and no-reference image quality metrics for image quality classification. Basic image features, reference and no-reference features are extracted from the fundus image and applied through different classification techniques to determine the image quality for further diagnosis. In this paper, human-made categorization including good and non-good-quality fundus image classification is constructed by considering major features of retinal fundus images are illumination, clarity, image intensity, contrast and region visibility. The proposed system presented fundus image quality classification by automatic extraction of features from fundus images through image processing techniques and automatic classification of image quality through different classification algorithm. This system was thoroughly investigated on 2674 retinal fundus images from publically available datasets, namely MESSIDOR, Drishti-GS1, DRIVE, HRF, DRIONS-DB, DIARETDB0, DIARETDB1, IDRiD, INSPIRE-AVR, CHASE-DB1, ONHSD, DRIMDB and e-ophtha-EX with better performance results in terms of sensitivity, accuracy, precision and F1 score of 99.36%, 96.79%, 96.29% and 97.79%, respectively. The proposed system results were compared to the existing state-of-the-art approaches and outperform existing methods for image quality assessment representing the efficiency and robustness of our system is most suitable for automatic image analysis during retinal disease diagnosis.

Combined No-Reference Image Quality Metrics for Visual Quality Assessment Optimized for Remote Sensing Images

No-reference image quality assessment is one of the most demanding areas of image analysis for many applications where the results of the analysis should be strongly correlated with the quality of an input image and the corresponding reference image is unavailable. One of the examples might be remote sensing since the transmission of such obtained images often requires the use of lossy compression and they are often distorted, e.g., by the presence of noise and blur. Since the practical usefulness of acquired and/or preprocessed images is directly related to their quality, there is a need for the development of reliable and adequate no-reference metrics that do not need any reference images. As the performance and universality of many existing metrics are quite limited, one of the possible solutions is the design and application of combined metrics. Several possible approaches to their composition have been previously proposed and successfully used for full-reference metrics. In the paper, three possible approaches to the development and optimization of no-reference combined metrics are investigated and verified for the dataset of images containing distortions typical for remote sensing. The proposed approach leads to good results, significantly improving the correlation of the obtained results with subjective quality scores.

Point Cloud Quality Assessment via 3D Edge Similarity Measurement

In this letter, a new full-reference metric is presented to assess the perceptual quality of the point clouds (PCs). The human visual system (HVS) always shows a high sensitivity to the three-dimensional (3D) edge features inherent in the PCs. With this motivation, the three-dimensional edge similarity-based model (TDESM) is proposed, which makes the first attempt to apply 3D Difference of Gaussian (3D-DOG) on point cloud quality assessment (PCQA). Specifically, the 3D edge features are captured by convolving the dual-scale 3D-DOG filters with both reference and distorted PCs. The quality scores of distorted PCs are generated by combining the 3D edge similarity measured from different scales. The experiments are conducted on four publicly available PCQA datasets, i.e., Torlig2018, M-PCCD, ICIP2020, and SJTU-PCQA. Compared with multiple state-of-the-art PCQA metrics, our proposed approach is able to be higher consistent with the subjective perception on the PCs.

Noise Reduction Methods for Charge Stability Diagrams of Double Quantum Dots

Operating semiconductor quantum dots as quantum bits requires isolating single electrons by adjusting gate voltages. The transitions of electrons to and from the dots appear as a honeycomb-like pattern in recorded charge stability diagrams (CSDs). Thus, detecting the pattern is essential to tune a double dot, but manual tuning is seriously time-consuming. However, automation of this process is difficult because the transitions’ contrast is often low, and noise and background disorder potential shifts disturb the CSDs. Therefore, the signal-to-noise ratio needs to be increased to improve the detection of the line pattern. For this purpose, we evaluate a representative set of edge-preserving smoothing filters and compare them both quantitatively and qualitatively by suitable metrics and visual assessment. We generate artificial data to use full-reference metrics for the evaluation procedure and to optimize the filter parameters. Based on the results of this article, the methods attain a moderate to good amount of noise reduction and structure improvement dependent on the different CSD qualities. In conclusion, we suggest introducing the block-matching and three-dimensional transform-domain filter into the automated tuning processing pipeline. If the data are corrupted by significant amounts of random telegraph noise, the bilateral filter and the rolling guidance filter are also good choices.

A Point-Matching Method of Moment with Sparse Bayesian Learning Applied and Evaluated in Dynamic Lung Electrical Impedance Tomography.

Dynamic lung imaging is a major application of Electrical Impedance Tomography (EIT) due to EIT’s exceptional temporal resolution, low cost and absence of radiation. EIT however lacks in spatial resolution and the image reconstruction is very sensitive to mismatches between the actual object’s and the reconstruction domain’s geometries, as well as to the signal noise. The non-linear nature of the reconstruction problem may also be a concern, since the lungs’ significant conductivity changes due to inhalation and exhalation. In this paper, a recently introduced method of moment is combined with a sparse Bayesian learning approach to address the non-linearity issue, provide robustness to the reconstruction problem and reduce image artefacts. To evaluate the proposed methodology, we construct three CT-based time-variant 3D thoracic structures including the basic thoracic tissues and considering 5 different breath states from end-expiration to end-inspiration. The Graz consensus reconstruction algorithm for EIT (GREIT), the correlation coefficient (), the root mean square error () and the full-reference () metrics are applied for the image quality assessment. Qualitative and quantitative comparison with traditional and more advanced reconstruction techniques reveals that the proposed method shows improved performance in the majority of cases and metrics. Finally, the approach is applied to single-breath online in-vivo data to qualitatively verify its applicability.

Combined Full-Reference Image Quality Metrics for Objective Assessment of Multiply Distorted Images

In the recent years, many objective image quality assessment methods have been proposed by different researchers, leading to a significant increase in their correlation with subjective quality evaluations. Although many recently proposed image quality assessment methods, particularly full-reference metrics, are in some cases highly correlated with the perception of individual distortions, there is still a need for their verification and adjustment for the case when images are affected by multiple distortions. Since one of the possible approaches is the application of combined metrics, their analysis and optimization are discussed in this paper. Two approaches to metrics’ combination have been analyzed that are based on the weighted product and the proposed weighted sum with additional exponential weights. The validation of the proposed approach, carried out using four currently available image datasets, containing multiply distorted images together with the gathered subjective quality scores, indicates a meaningful increase of correlations of the optimized combined metrics with subjective opinions for all datasets.

An Imaged Based Method for Universal Performance Evaluation of Electrical Impedance Tomography Systems.

This paper describes a simple and reproducible method for universal evaluation of the performance of electrical impedance tomography (EIT) systems using reconstructed images. To address the issues where common electrical parameters are not directly related to the quality of EIT images, based on objective full reference (FR) image quality assessment, the method provides a visually distinguishable hot colormap and two new FR metrics, the global and the more specific 'region of interest'. A passive 16 electrode EIT system using an application specific integrated circuit front-end was used to evaluate the proposed method. The measured results show, both visually and in terms of the proposed FR metrics, the impact on recorded EIT images with different design parameters and non-idealities. The paper also compares the image results of a passive electrode system with a matched 'single variable' active electrode system and demonstrates the merit of an active electrode system for noise interference. A figure of merit based on the FR metrics is proposed.

QoS monitoring in real-time streaming overlays based on lock-free data structures

Peer-to-peer streaming is a well-known technology for the large-scale distribution of real-time audio/video contents. Delay requirements are very strict in interactive real-time scenarios (such as synchronous distance learning), where playback lag should be of the order of seconds. Playback continuity is another key aspect in these cases: in presence of peer churning and network congestion, a peer-to-peer overlay should quickly rearrange connections among receiving nodes to avoid freezing phenomena that may compromise audio/video understanding. For this reason, we designed a QoS monitoring algorithm that quickly detects broken or congested links: each receiving node is able to independently decide whether it should switch to a secondary sending node, called “fallback node”. The architecture takes advantage of a multithreaded design based on lock-free data structures, which improve the performance by avoiding synchronization among threads. We will show the good responsiveness of the proposed approach on machines with different computational capabilities: measured times prove both departures of nodes and QoS degradations are promptly detected and clients can quickly restore a stream reception. According to PSNR and SSIM, two well-known full-reference video quality metrics, QoE remains acceptable on receiving nodes of our resilient overlay also in presence of swap procedures.

Single Image Dehazing by Predicting Atmospheric Scattering Parameters

In this work we propose a method for single image dehazing that exploits a physical model to recover the haze-free image by estimating the atmospheric scattering parameters. Cycle consistency is used to further improve the reconstruction quality of local structures and objects in the scene as well. Experimental results on four real and synthetic hazy image datasets show the effectiveness of the proposed method in terms of two commonly used full-reference image quality metrics.

Quality Assessment of 3D Printed Surfaces Using Combined Metrics Based on Mutual Structural Similarity Approach Correlated with Subjective Aesthetic Evaluation

Quality assessment of the 3D printed surfaces is one of the crucial issues related to fast prototyping and manufacturing of individual parts and objects using the fused deposition modeling, especially in small series production. As some corrections of minor defects may be conducted during the printing process or just after the manufacturing, an automatic quality assessment of object’s surfaces is highly demanded, preferably well correlated with subjective quality perception, considering aesthetic aspects. On the other hand, the presence of some greater and more dense distortions may indicate a reduced mechanical strength. In such cases, the manufacturing process should be interrupted to save time, energy, and the filament. This paper focuses on the possibility of using some general-purpose full-reference image quality assessment methods for the quality assessment of the 3D printed surfaces. As the direct application of an individual (elementary) metric does not provide high correlation with the subjective perception of surface quality, some modifications of similarity-based methods have been proposed utilizing the calculation of the average mutual similarity, making it possible to use full-reference metrics without the perfect quality reference images, as well as the combination of individual metrics, leading to a significant increase of correlation with subjective scores calculated for a specially prepared dataset.

Cloud-based elastic architecture for distributed video encoding: Evaluating H.265, VP9, and AV1

Areas with social and business impact such as entertainment, healthcare, surveillance, and e-learning would benefit from improvements in video coding and transcoding services. New codecs, such as AV1, are being developed to deal with new demands for high video resolutions with bandwidth constraints and quality requirements. However, these new codecs have high computational requirements and new strategies are needed to accelerate their processing.Cloud computing offers interesting features such as on-demand resource allocation, multitenancy, elasticity, and resiliency among others. Deploying video coding and transcoding services on these infrastructures is suitable because it allows the adaptation of the resources to the workload, offers high availability, and provides ubiquitous access.This work proposes a cloud-based distributed architecture, tuned for video coding, that relies on an elastic pool of workers and media servers to provide fault-tolerance. A distributed application is deployed on the top of the architecture to split the video encoding process of a video in several jobs that can be dynamically assigned among the elastic pool of workers. The proposed solution is analyzed in terms of scalability, resource usage, and job distribution varying the number of workers for three well-known video codecs: H.265, VP9, and AV1. Moreover, the quality of the encoded videos has been evaluated for different bit rates and number of frames per job using full reference metrics like: PSNR, MS-SSIM, and VIF. Results show that our solution achieve similar quality and bitrate compared with full video coding while improving the total encoding time, which can decrease more than 90% depending on the encoder and the number of workers.