Predict Quality Scores Research Articles

Perceptually-calibrated synergy network for night-time image quality assessment with enhancement booster and knowledge cross-sharing

Image quality assessment (IQA) and image enhancement (IE) of night-time images are highly correlated tasks. On the one hand, IQA task could obtain more complementary information from the enhanced image. On the other hand, IE task would benefit from the prior knowledge of quality-aware attributes. Thus, we propose a Perceptually-calibrated Synergy Network (PCSNet) to simultaneously predict and enhance image quality of night-time images. More specifically, a shared shallow network is applied to extract the shared features for both tasks by leveraging complementary in-formation. The shared features are then fed to task-specific sub-networks to predict quality scores and generate enhanced images in parallel. In order to better exploit the interaction of complementary information, intermediate Cross-Sharing Modules are used to form efficient feature representations for the image quality assessment (IQA) and image enhancement (IE) subnetworks. Experimental results of the night-time image datasets show that the proposed approach achieves state-of-the-art performance on both quality prediction and image enhancement tasks.

Automatic ploidy prediction and quality assessment of human blastocysts using time-lapse imaging

Assessing fertilized human embryos is crucial for in vitro fertilization, a task being revolutionized by artificial intelligence. Existing models used for embryo quality assessment and ploidy detection could be significantly improved by effectively utilizing time-lapse imaging to identify critical developmental time points for maximizing prediction accuracy. Addressing this, we develop and compare various embryo ploidy status prediction models across distinct embryo development stages. We present BELA, a state-of-the-art ploidy prediction model that surpasses previous image- and video-based models without necessitating input from embryologists. BELA uses multitask learning to predict quality scores that are thereafter used to predict ploidy status. By achieving an area under the receiver operating characteristic curve of 0.76 for discriminating between euploidy and aneuploidy embryos on the Weill Cornell dataset, BELA matches the performance of models trained on embryologists’ manual scores. While not a replacement for preimplantation genetic testing for aneuploidy, BELA exemplifies how such models can streamline the embryo evaluation process.

No‐reference image quality assessment via a dual‐branch residual network

AbstractThe No‐Reference Image Quality Assessment (NR‐IQA) method can predict quality scores of distorted images without the reference image. However, due to the variability in both color and structure of images, existing NR‐IQA methods struggle to accurately predict quality scores of distorted images. Therefore, an NR‐IQA method based on a Dual‐Branch Residual Network (DBRIQA) for evaluating the quality scores of color‐distorted images is proposed. First, guided filtering is applied to the hue images in the HSV space to extract distortion information from the image's color. Then, due to significant differences in distorted images, enhancements are made to the traditional residual blocks, forming a feature extraction module that captures multi‐scale features from the image. In order to capture the global relationships in the distorted image, a Global‐Level Attention Block (GLAB) is introduced, facilitating the interaction of information among the extracted features. Experiments were conducted across four publicly available IQA datasets, including LIVE, CSIQ, TID2008, and TID2013. The experimental results demonstrate that the proposed method exhibits strong performance and generalization capabilities in predicting image quality compared to peer methods.

Blind Quality Assessment of Images Containing Objects of Interest.

To monitor objects of interest, such as wildlife and people, image-capturing devices are used to collect a large number of images with and without objects of interest. As we are recording valuable information about the behavior and activity of objects, the quality of images containing objects of interest should be better than that of images without objects of interest, even if the former exhibits more severe distortion than the latter. However, according to current methods, quality assessments produce the opposite results. In this study, we propose an end-to-end model, named DETR-IQA (detection transformer image quality assessment), which extends the capability to perform object detection and blind image quality assessment (IQA) simultaneously by adding IQA heads comprising simple multi-layer perceptrons at the top of the DETRs (detection transformers) decoder. Using IQA heads, DETR-IQA carried out blind IQAs based on the weighted fusion of the distortion degree of the region of objects of interest and the other regions of the image; the predicted quality score of images containing objects of interest was generally greater than that of images without objects of interest. Currently, the subjective quality score of all public datasets is in accordance with the distortion of images and does not consider objects of interest. We manually extracted the images in which the five predefined classes of objects were the main contents of the largest authentic distortion dataset, KonIQ-10k, which was used as the experimental dataset. The experimental results show that with slight degradation in object detection performance and simple IQA heads, the values of PLCC and SRCC were 0.785 and 0.727, respectively, and exceeded those of some deep learning-based IQA models that are specially designed for only performing IQA. With the negligible increase in the computation and complexity of object detection and without a decrease in inference speeds, DETR-IQA can perform object detection and IQA via multi-tasking and substantially reduce the workload.

Combining pairwise structural similarity and deep learning interface contact prediction to estimate protein complex model accuracy in CASP15.

Estimating the accuracy of quaternary structural models of protein complexes and assemblies (EMA) is important for predicting quaternary structures and applying them to studying protein function and interaction. The pairwise similarity between structural models is proven useful for estimating the quality of protein tertiary structural models, but it has been rarely applied to predicting the quality of quaternary structural models. Moreover, the pairwise similarity approach often fails when many structural models are of low quality and similar to each other. To address the gap, we developed a hybrid method (MULTICOM_qa) combining a pairwise similarity score (PSS) and an interface contact probability score (ICPS) based on the deep learning inter-chain contact prediction for estimating protein complex model accuracy. It blindly participated in the 15th Critical Assessment of Techniques for Protein Structure Prediction (CASP15) in 2022 and performed very well in estimating the global structure accuracy of assembly models. The average per-target correlation coefficient between the model quality scores predicted by MULTICOM_qa and the true quality scores of the models of CASP15 assembly targets is 0.66. The average per-target ranking loss in using the predicted quality scores to rank the models is 0.14. It was able to select good models for most targets. Moreover, several key factors (i.e., target difficulty, model sampling difficulty, skewness of model quality, and similarity between good/bad models) for EMA are identified and analyzed. The results demonstrate that combining the multi-model method (PSS) with the complementary single-model method (ICPS) is a promising approach to EMA.

Blind Video Quality Assessment for Ultra-High-Definition Video Based on Super-Resolution and Deep Reinforcement Learning.

Ultra-high-definition (UHD) video has brought new challenges to objective video quality assessment (VQA) due to its high resolution and high frame rate. Most existing VQA methods are designed for non-UHD videos-when they are employed to deal with UHD videos, the processing speed will be slow and the global spatial features cannot be fully extracted. In addition, these VQA methods usually segment the video into multiple segments, predict the quality score of each segment, and then average the quality score of each segment to obtain the quality score of the whole video. This breaks the temporal correlation of the video sequences and is inconsistent with the characteristics of human visual perception. In this paper, we present a no-reference VQA method, aiming to effectively and efficiently predict quality scores for UHD videos. First, we construct a spatial distortion feature network based on a super-resolution model (SR-SDFNet), which can quickly extract the global spatial distortion features of UHD videos. Then, to aggregate the spatial distortion features of each UHD frame, we propose a time fusion network based on a reinforcement learning model (RL-TFNet), in which the actor network continuously combines multiple frame features extracted by SR-SDFNet and outputs an action to adjust the current quality score to approximate the subjective score, and the critic network outputs action values to optimize the quality perception of the actor network. Finally, we conduct large-scale experiments on UHD VQA databases and the results reveal that, compared to other state-of-the-art VQA methods, our method achieves competitive quality prediction performance with a shorter runtime and fewer model parameters.

Visual Interaction Perceptual Network for Blind Image Quality Assessment

In observing images, the perception of the human visual system (HVS) is affected by both image contents and distortions. Obviously, the visual quality of the same image varies under different distortion types and intensities. Furthermore, the visual masking effects reveal that image content and distortion have a visual interaction, where the HVS presents different visibility of the identical distortion for different image contents. Based upon this, we propose a visual interaction perceptual network that can perceive both content and distortion of an image. The proposed model consists of three sub-modules: content perception module (CPM), distortion perception module (DPM), and visual interaction module (VIM). However, the subjective quality score cannot guide the model to explicitly learn the feature representations of image content and distortion. Thus, we perform a two-stage training procedure. In the first stage, we obtain CPM and DPM, where semantic features are extracted to recognize the image content in CPM, and distortion features are extracted to capture the image distortion type and intensity in DPM. In the second stage, the VIM is applied to model the interaction between semantic and distortion features, and the final predicted quality score is given by a fully connected layer. Experimental results demonstrate that the proposed method can achieve state-of-the-art performance on multiple benchmark databases, e.g., CSIQ, TID2013, KADID-10K, and KonIQ-10K.

Dynamically attentive viewport sequence for no-reference quality assessment of omnidirectional images.

Omnidirectional images (ODIs) have drawn great attention in virtual reality (VR) due to the capability of providing an immersive experience to users. However, ODIs are usually subject to various quality degradations during different processing stages. Thus, the quality assessment of ODIs is of critical importance to the community of VR. The quality assessment of ODIs is quite different from that of traditional 2D images. Existing IQA methods focus on extracting features from spherical scenes while ignoring the characteristics of actual viewing behavior of humans in continuously browsing an ODI through HMD and failing to characterize the temporal dynamics of the browsing process in terms of the temporal order of viewports. In this article, we resort to the law of gravity to detect the dynamically attentive regions of humans when viewing ODIs. In this article, we propose a novel no-reference (NR) ODI quality evaluation method by making efforts on two aspects including the construction of Dynamically Attentive Viewport Sequence (DAVS) from ODIs and the extraction of Quality-Aware Features (QAFs) from DAVS. The construction of DAVS aims to build a sequence of viewports that are likely to be explored by viewers based on the prediction of visual scanpath when viewers are freely exploring the ODI within the exploration time via HMD. A DAVS that contains only global motion can then be obtained by sampling a series of viewports from the ODI along the predicted visual scanpath. The subsequent quality evaluation of ODIs is performed merely based on the DAVS. The extraction of QAFs aims to obtain effective feature representations that are highly discriminative in terms of perceived distortion and visual quality. Finally, we can adopt a regression model to map the extracted QAFs to a single predicted quality score. Experimental results on two datasets demonstrate that the proposed method is able to deliver state-of-the-art performance.

A CNN-based no reference image quality metric exploiting content saliency

Assessing the quality of images is a challenging task. To achieve this goal, images must be evaluated by a pool of subjects following a well-defined protocol or an objective quality metric must be defined. In this work, an objective quality metric based on deep neural network is proposed. The metric takes into account the human vision system by computing the saliency map and natural scene statistics features of the image under test. The neural network is composed by two modules: the convolutional layers and the regression units. The first one is trained by using preprocessed distorted images. The feature weights of the first module are smoothed by exploiting the estimated saliency map. The latter module is fit with the ground truth quality scores of the input image and the scaled feature weights obtained from first module by using visual sensitivity factor of image obtained using natural scene statistics features. The performances of the proposed metric have been evaluated by using four datasets: LIVEIQA, TID2013, CSIQ, and KADID10K. The achieved results show the effectiveness of the proposed system in closely matching the predicted quality scores with the ground truth ones.

Motion measurement and quality variation driven video quality assessment

Many video quality assessment (VQA) methods generally use frame difference to represent motion information which may cause foreground apertures. Besides, quality variations of video contents significantly affect visual quality predictions. To remedy these, we develop a novel video quality assessment model that considers the impact of motion estimation and quality variations on distortion perception. The model firstly decomposes a video into a reflectance component and an illumination component via the intrinsic decomposition, and extracts scene features from these two components. To alleviate the foreground apertures, a simple structure difference is calculated as the motion measurement between adjacent frames. Motion difference features are extracted from the structure difference. We additionally consider analyzing quality variations to boost video quality predictions. Finally, a support vector regressor (SVR) is used to map generated features to predicted quality scores. We evaluated our proposed model on the LIVE, CSIQ, CVD2014 and LIVE-VQC video quality databases. The results show that the proposed model achieved competitive performance in comparison with state-of-the-art methods. Source code is freely available at: https://github.com/Aca4peop/QVDVQA.

CPDINet: Blind image quality assessment via a content perception and distortion inference network

Nowadays, it is still challenging for the blind image quality assessment (BIQA) to accurately predict quality scores of distorted images, since distorted images have rich content information and complex distortions. To solve these problems, a content perception and distortion inference network for BIQA is proposed, which divides IQA task into content perception and distortion inference processes. Since humans try to understand image content before perceiving quality scores, a content feature extractor is designed to explore content information in an image to deal with the content variation problem. To handle the distortion diversity problem, a distortion feature extractor is proposed to capture distortion features in images. Because extracted content features and distortion ones have different characteristics, attention-based fusion blocks to fuse multi-scale content features and distortion ones as guidance to selectively enhance important features based on calculated weight scores are proposed. With fused features, a quality prediction module is designed to regress multi-scale features to quality scores. Experiments are performed on six public IQA datasets, including LIVE, CSIQ, TID2013, LIVEC, KonIQ-10k, and SPAQ. Experimental results show that the proposed method can effectively predict quality scores for both synthetically and authentically distorted images than its peers, including the state-of-the-art methods.

No-reference stereoscopic image quality assessment on both complex contourlet and spatial domain via Kernel ELM

Stereoscopic imaging is widely used in many fields. To guarantee the best quality of experience, it is necessary to design a robust and accurate quality assessment model for stereoscopic content. In this paper, we proposed a no-reference stereoscopic image quality assessment (NR-SIQA) model using both complex contourlet and spatial domain features of monocular and binocular images. Monocular features extracted from the CIELAB color space are exploited to characterize quality degradation, including the across-scale and across-orientation correlation in complex contourlet domain and natural scene statistics-based (NSS) features in spatial domain. Then, the binocular features consist of energy along with energy difference and structural correlation in complex contourlet domain, and statistics distribution in spatial domain extracted from the synthesized cyclopean image and the 3D visual discomfort measure based on the statistics of disparity map. Finally, the above features are mapped into the predicted quality scores by the Kernel ELM (KELM) regression model. Experimental results on four public datasets show that the proposed model is highly consistent with human subjective perception in terms of accuracy and generalization.

DeepRPN-BIQA: Deep architectures with region proposal network for natural-scene and screen-content blind image quality assessment

With the emerging use of technology and screen-oriented applications in our daily life, screen content images have gained the same importance as natural scene images. This results in many natural-scene and screen-content blind image quality assessment (BIQA) models to evaluate the perceptual quality without any prior information regarding the reference image. Recently, patch-based techniques for image quality assessment (IQA) have shown promising results. As per our knowledge, no IQA technique in literature is available that can be equally effective for both natural-scene and screen-content images. In this work, we have proposed a deep architecture with a region proposal network (RPN) for blind natural-scene and screen-content image quality assessment, named DeepRPN-BIQA. The proposed architecture computes visual saliency using RPN to extract important regions having a high contribution towards the image quality. Important regions are extracted by utilizing the texture and edges of images by sliding the network over the extracted feature map from deep architectures i.e., VGGNet and ResNet. The regions proposed (RP) that overlap more than 60% are merged into one proposal and are called the region of interest (ROI). The overlap between RPs is computed using anchors having 3 different scales and aspect ratios. A local quality score is computed over each ROI and the total quality score is computed by taking the average of all the local quality scores. Experimental results show that the DeepRPN-BIQA shows a high correlation between mean observer score and predicted quality score and performs better than other models for screen content images, synthetically distorted images, images taken in real-life conditions using mobile phone cameras, and large scale image quality assessment database.

An efficient approach for no-reference image quality assessment based on statistical texture and structural features

This paper presents a rotation-invariant and computationally efficient no-reference image quality assessment (NR-IQA) model. It estimates the image quality based on texture and structural information associated with the images. The human visual system (HVS) uses perceptual features such as texture and structure as primary information to understand the visual scene and the image content. Moreover, the texture and structural information capture the loss of naturalness due to distortions in the image. Therefore, in this work, the important texture features are extracted using the local binary patterns (LBP).The modified LBP, also known as the hyper-smoothing LBP (H-LBP) and Laplacian of H-LBP (LH-LBP), represents the image structure. Further, the image quality prediction model computes the quality of the image based on the statistical feature measures of the texture and structural information. In the proposed approach, the image quality prediction model uses support vector regression (SVR) to measure image quality. Various experimentations are carried out on the LIVE and TID2013 database to test the effectiveness of the proposed NR-IQA model. The performance metrics such as Spearman rank-ordered correlation coefficient, Pearson linear correlation coefficient, and root mean square error is computed to show the efficiency of the presented approach. The experimental results illustrate a high correlation between the predicted quality score and the human visual perceptions. It is also found to be competitive with the best-performing full-reference and no-reference IQA models.

PIQI: perceptual image quality index based on ensemble of Gaussian process regression

Digital images contain a lot of redundancies, therefore, compression techniques are applied to reduce the image size without loss of reasonable image quality. Same become more prominent in the case of videos which contains image sequences and higher compression ratios are achieved in low throughput networks. Assessment of quality of images in such scenarios has become of particular interest. Subjective evaluation in most of the scenarios is infeasible so objective evaluation is preferred. Among the three objective quality measures, full-reference and reduced-reference methods require an original image in some form to calculate the image quality which is unfeasible in scenarios such as broadcasting, acquisition or enhancement. Therefore, a no-reference Perceptual Image Quality Index (PIQI) is proposed in this paper to assess the quality of digital images which calculates luminance and gradient statistics along with mean subtracted contrast normalized products in multiple scales and color spaces. These extracted features are provided to a stacked ensemble of Gaussian Process Regression (GPR) to perform the perceptual quality evaluation. The performance of the PIQI is checked on six benchmark databases and compared with twelve state-of-the-art methods and competitive results are achieved. The comparison is made based on RMSE, Pearson and Spearman’s correlation coefficients between ground truth and predicted quality scores. The scores of 0.0552, 0.9802 and 0.9776 are achieved respectively for these metrics on CSIQ database. Two cross-dataset evaluation experiments are performed to check the generalization of PIQI.

Objective quality assessment of synthesized images by local variation measurement

Due to the rapid development of free-viewpoint television (FVT), Depth-Image-Based Rendering (DIBR) technology has been widely used to synthesize images of virtual view-points. However, the types of distortions in the synthesized images are different from those of natural images, such as discontinuity, flickering, stretching, etc. To measure the distortion occurred in the synthesized images, we propose a full-reference (FR) quality assessment method by local variation measurement consisting of three-modules. Firstly, since the distortion in the synthesized image mainly occurs in the region with high-frequency structure information, the Neutrosophic domain is employed to evaluate the degradation of local image structure. Secondly, by considering that the texture of the synthesized image might be damaged due to the warping of 2D image or the loss of information in the occlusion region, we evaluate the visual quality of local texture by using the features obtained from frequency domain. Thirdly, to measure the stretching distortion which is unique in the synthesized image, the visual quality of extracted stretching area is measured by entropy. Finally, a pooling operation is used to combine the quality scores of the three modules to obtain the final predicted quality score. Experimental results show that the performance of the proposed algorithm is competitive with state-of-the-art FR and no-reference image quality assessment metrics.

Study of assessment of green tea’ grades in GC‐MS determination of aromatic components based on principal component analysis (PCA)

Aroma compounds are one of the most important components in tea, as well as an important indicator for its quality. In this paper, using HS-SPME/GC-MS method, we established the fingerprint for aromatic components in green tea produced in Guizhou Province, China. In this fingerprint, there are nine volatile aromatic components which have significant correlation with the quality scores of green teas’ aroma (p < .05). They contain ethers and aldehydes (dimethyl sulfide and phenylacetaldehyde) of aromatic compounds, and alcohols and esters (1-penten-3-ol, linalool, phenethyl alcohol, geraniol, and Methyl salicylate) of aliphatic compounds, and pyrrole compounds of heterocyclic compounds (3-ethyl-1-pyrrole and 1-ethyl-2-carbaldehyde-1-pyrrole). The Regression Model has been established for predicting evaluation scores of green teas’ aroma. A good result was obtained via verification, indicating that GC-MS is a feasible way to analyze key volatile aromatic components in green tea, which also can be used to predict quality scores aroma and different grades. Practical applications The aromatic components of tea are considered as the important indicator for evaluating the quality of green tea. We can use GC-MS and other technologies to determine the representative aroma substances in green tea that can evaluate the quality of green tea. In this way, we can use the instrumental analysis method of the aromatic components to assist the traditional sensory evaluation, which make the green tea quality evaluation results more scientific.

Redundancy Removing Aggregation Network With Distance Calibration for Video Face Recognition

Attention-based techniques have been successfully used for rating image quality, and have been widely employed for set-based face recognition. Nevertheless, for video face recognition, where the base convolutional neural network (CNN) trained on large-scale data already provides discriminative features, fusing features with only predicted quality scores to generate representation are likely to cause duplicate sample dominant problem, and degrade performance correspondingly. To resolve the problem mentioned above, we propose a redundancy removing aggregation network (RRAN) for video face recognition. Compared with other quality-aware aggregation schemes, RRAN can take advantage of similarity information to tackle the noise introduced by redundant video frames. By leveraging metric learning, RRAN introduces a distance calibration scheme to align distance distributions of negative pairs of different video representations, which improves the accuracy under a uniform threshold. A series of experiments is conducted on multiple realistic data sets to evaluate the performance of RRAN, including YouTube Faces, IJB-A, and IJB-C. In comprehensive experiments, we demonstrate that our method can diminish the overall influence of poor quality components with large proportion in the video and further improve the overall recognition performance with individual difference. Specifically, RRAN achieves a 96.84% accuracy on YouTube Face, outperforming all existing aggregation schemes.

Real distorted images quality assessment based on multi-layer visual perception mechanism and high-level semantics

Most of the existing image quality assessment (IQA) methods are directed to artificially synthesized distorted images, in which the types and characteristics of distortion are different from those in the real world. In view of the fact that the existing non-reference IQA methods can not accurately evaluate the quality of the real distortion image, combined with the theoretical analysis of multi-layer visual perception mechanism, we propose a real image distortion IQA method based on image underlying features and high-level semantics. Considering non-linear hierarchical structure of human visual perception, firstly, k-means clustering algorithm is performed according to the underlying feature indexs of the image so that the used image database can be divided into several groups, which aims to improve the accuracy of predicted quality score. Secondly, the deep convolutional neural network (DCNN) is used to extract the first-grade high-level semantic features in each group. Then, second-grade high-level semantic features that can provide better representation of image features are obtained by performing multiple statistical functions on first-grade high-level semantics. Besides, we establish an effective high-capacity regressor with high-level semantics and subjective mean opinion scores (MOS) values of the human eyes. The experimental results show that the proposed model on the KonIQ-10 k image database can predict the quality score effectively and achieve a high consistency with the corresponding MOS value, which is helpful for the subsequent image enhancement.

Quality Assessment on Authentically Distorted Images by Expanding Proxy Labels

In this paper, we propose a no-reference image quality assessment (NR-IQA) approach towards authentically distorted images, based on expanding proxy labels. In order to distinguish from the human labels, we define the quality score, which is generated by using a traditional NR-IQA algorithm, as “proxy labels”. “Proxy” means that the objective results are obtained by computer after the extraction and assessment of the image features, instead of human judging. To solve the problem of limited image quality assessment (IQA) dataset size, we adopt a cascading transfer-learning method. First, we obtain large numbers of proxy labels which denote the quality score of authentically distorted images by using a traditional no-reference IQA method. Then the deep network is trained by the proxy labels, in order to learn IQA-related knowledge from the amounts of images with their scores. Ultimately, we use fine-tuning to inherit knowledge represented in the trained network. During the procedure, the mapping relationship fits in with human visual perception closer. The experimental results demonstrate that the proposed algorithm shows an outstanding performance as compared with the existing algorithms. On the LIVE In the Wild Image Quality Challenge database and KonIQ-10k database (two standard databases for authentically distorted image quality assessment), the algorithm realized good consistency between human visual perception and the predicted quality score of authentically distorted images.