No-reference Research Articles

No-Reference Image Quality Assessment by Hallucinating Pristine Features.

In this paper, we propose a no-reference (NR) image quality assessment (IQA) method via feature level pseudo-reference (PR) hallucination. The proposed quality assessment framework is rooted in the view that the perceptually meaningful features could be well exploited to characterize the visual quality, and the natural image statistical behaviors are exploited in an effort to deliver the accurate predictions. Herein, the PR features from the distorted images are learned by a mutual learning scheme with the pristine reference as the supervision, and the discriminative characteristics of PR features are further ensured with the triplet constraints. Given a distorted image for quality inference, the feature level disentanglement is performed with an invertible neural layer for final quality prediction, leading to the PR and the corresponding distortion features for comparison. The effectiveness of our proposed method is demonstrated on four popular IQA databases, and superior performance on cross-database evaluation also reveals the high generalization capability of our method. The implementation of our method is publicly available on https://github.com/Baoliang93/FPR.

Three-Stream 3D deep CNN for no-Reference stereoscopic video quality assessment

• An end-to-end learning structure for the Stereoscopic Video Quality Assessment with three pipelines to extract the quality degradation from spatial, temporal, and disparity information. • A non-homogeneous data augmentation system that splits the stereoscopic video into small cube patches and removes the outliers from the created dataset using entropy criteria. • A robust deep depth quality-aware feature extractor used to extract motion and depth maps jointly and feed them synchronously to the deep network to extract quality discriminatory features. • Two popular datasets are used to validate the proposed learning architecture for Stereoscopic Video Quality Assessment and received state-of-the-art results in both datasets. Convolutional Neural Networks (CNNs) have achieved great success in learning computer vision tasks, particularly 3D CNNs, for extracting spatio-temporal features from the given videos. However, 3D CNNs have not been well-examined for the Stereoscopic Video Quality Assessment (SVQA). To our best knowledge, most of the state-of-the-art methods used the traditional hand-crafted feature extraction methods for the SVQA. Very few methods used the power of deep learning for SVQA, and they just considered the spatial information, ignoring the disparity and motion information. In this paper, we propose a No-Reference (NR) deep 3D CNN architecture that jointly focuses on spatial, disparity, and temporal information between consecutive frames. A 3-Stream 3D CNN, shortly 3S-3DCNN , by performing 3D CNNs, extracts features from spatial, motion, and depth channels to estimate the stereo video’s quality. It captures the degradations in the quality of the stereoscopic video in multiple dimensions. Firstly, the scene flow, which is the joint prediction of the optical flow and stereo disparity, is calculated. Then, the spatial information, optical flow, and disparity map of a given video are used as input to the 3S-3DCNN model. The extracted features are concatenated and utilized as inputs to the fully connected layers for doing the regression. We split the input videos into cube patches for data augmentation and remove the cubes that confuse our model from the training and testing sets. Two standard stereoscopic video quality assessment benchmarks of LFOVIAS3DPh2 and NAMA3DS1-COSPAD1 were used to evaluate our method. Experimental results show that our 3S-3DCNN method’s objective score significantly correlates with the subjective SVQ scores in multiple video datasets. The RMSE for NAMA3DS1-COSPAD1 dataset is 0.2757, which outperforms other methods by a large margin. The SROCC value for the blur distortion of the LFOVIAS3DPh2 dataset is more than 98%, indicating that the 3S-3DCNN is consistent with human visual perception.

The Development of Three Image Quality Evaluation Metrics Based on a Comprehensive Dataset

In this study, a large scale experiment was carried out to assess the image quality of 2266 images using categorical judgement method by 20 observers. These images were rendered in color contrast, chroma, colorfulness, lightness, and vividness directions. The results were used to derive three No-Reference (NR) Image Quality Estimation Models (IQEMs). The first model was based on color science, (different scales in CIELAB). The second model was a Neural Network model while the third model was a statistics model based on color appearance attributes. Their performances were evaluated using two databases, those developed at Zhejiang University and those available from the public databases in terms of correlation coefficients between the objective and predicted image quality scores.

CNN-based no-reference video quality assessment method using a spatiotemporal saliency patch selection procedure

We propose a yet lightweight no-reference (NR) video quality assessment (VQA) method, which uses a convolution neural network (CNN) architecture. The proposed method implements a spatiotemporal saliency patch selection procedure that crops the frame into small nonoverlapping blocks of images (patches) and selects the most perceptually relevant ones. The selected patches are then forwarded to the CNN. To determine which patches are the most relevant, spatial and temporal saliency features are computed for each frame. The proposed method does not require subjective scores to train the CNN. It uses objective quality scores as target quality scores for each video frame, which are computed using an NR image quality assessment method. Given the lack of large annotated video quality databases, this is an advantage of the proposed method. Finally, although it has much smaller cost of data-processing, compared with other state-of-the-art methods, the proposed NR-VQA obtains robust and competitive results.

RM-IQA: A new no-reference image quality assessment framework based on range mapping method

Significant progress has been made in recent years in image quality assessment (IQA). In particular, the development of deep learning has provided no-reference (NR)-IQA with more impressive solutions. However, improving the generalization of NR-IQA models is still an urgent necessity. In this study, we propose a new framework that uses the range mapping method to map an existing full-reference (FR)-IQA dataset to an NR-IQA dataset, thereby further enhancing the accuracy and generalization of the NR-IQA model. First, an NR-IQA model is employed to score an FR-IQA dataset to obtain the corresponding mean opinion score (MOS) values. Then, the correlation coefficients between these MOS values and the original differential mean opinion score (DMOS) values marked by the FR-IQA dataset itself is calculated. Subsequently, the matching sequence pair is obtained according to these correlation coefficients. Then, a range mapping function is selected based on this sequence pair, and this function is used to map the entire FR-IQA dataset to the existing NR-IQA dataset, and a new NR-IQA dataset is generated. Finally, the new and the existing NR-IQA datasets are merged into a new dataset, which can train an end-to-end multi-task network to obtain the final model RM-IQA. This model exhibits better performance as it exploits more prior information. Based on the largest available NR-IQA dataset KonIQ-10k and FR-IQA dataset KADID-10K, the experimental results proved the effectiveness of the proposed framework.

Fingerprint Presentation Attack Detection in Open-Set Scenario Using Transient Liveness Factor

Background: In recent history, fingerprint presentation attack detection (FPAD) proposal came out in a variety of ways. A close-set approach uses pattern classification technique that best suits to a specific context and goal. Openset approach works fine in wider context, which is relatively robust with new fabrication material and independent of sensor type. In both case results were promising but not too generalizable because of unseen condition not fitting into method used. It is clear, the two key challenges in FPAD system, sensor interoperability and robustness with new fabrication materials not addressed to date. Objective: To address above challenge a liveness detection model is proposed using live sample using transient liveness factor and one-class CNN. Methods: In our architecture, liveness is predicted by using the fusion rule, score level fusion of two decisions. Here, ‘n’ high quality live samples are initially trained for quality. We have observed that fingerprint liveness information is ‘transitory’ in nature, a variation in the different live sample is natural. Thus, each live sample has a ‘transient liveness’ (TL) information. We use no-reference (NR) image quality measure (IQM) as a transient value corresponding to each live sample. A consensus agreement is collectively reached in transient value to predict adversarial input. Further, live sample at server are trained with augmented inputs on the one-class classifier to predict the outlier. So, by using the fusion rule, score level fusion of consensus agreement and appropriately characterized negative cases (or outliers) predicts liveness. Results: Our approach uses high quality 30-live sample only, out of 90 images available in dataset to reduce learning time. We used Time Series images from LivDet competition 2015. It has 90-live images and 45-spoof images made from Bodydouble, Ecoflex and Playdoh of each person. Fusion rule results in 100% accuracy in recognising live as live. Conclusion: We have presented an architecture for liveness-server for extraction/updating transient liveness factor. Our work explained here a significant step forward towards generalized and reproducible process with a consideration towards the provision for the universal scheme as a need of today. The proposed TLF approach has a solid presumption; it will address dataset heterogeneity as it incorporates wider scope-context. Similar results with other dataset are under validation. Implementation seems difficult now but have several advantages when carried out during the transformative process.

No-reference video quality assessment based on modeling temporal-memory effects

This study presents a hybrid network for no-reference (NR) video quality assessment (VQA). Besides spatial cues, the network concerns temporal motion effect and temporal hysteresis effect on the visual quality estimation, and two modules are embedded. One module is dedicated to incorporate short-term spatio-temporal features based on spatial quality maps and temporal quality maps, and the follow-up module explores graph convolutional network to quantify the relationship between image frames in a sequence. The proposed network and several popular models are evaluated on three video quality databases (CSIQ, LIVE, and KoNViD-1K). Experimental results indicate that the network outperforms other involved NR models, and its competitive performance is close to that of state-of-the-art full-reference VQA models. Conclusively, short-term spatio-temporal feature fusion benefits the modeling of interaction between spatial and temporal cues in VQA tasks, long-term sequence fusion further improves the performance, and a strong correlation with human subjective judgment is achieved.

MoNET: no-reference image quality assessment based on a multi-depth output network

When deep convolutional neural networks perform feature extraction, the features computed at each layer express different abstractions of visual information. The earlier layers extract highly compact low-level features such as bandpass and directional primitives, whereas deeper layers extract structural features of increasing abstraction, similar to contours, shapes, and edges, becoming less effable as the depth increases. We propose a different kind of end-to-end no-reference (NR) image quality assessment (IQA) model, which is defined as a multi-depth output convolutional neural network (MoNET). It accomplishes this by mapping both shallow and deep features to perceived quality. MoNET delivers three outputs that express shallow (lower-level) and deep (high-level) features, and maps them to subjective quality scores. The multiple outputs are combined into a single, final quality score. MoNET does this by combining the responses of three learning machines, so it may be viewed as a form of ensemble learning. The experimental results on three public image quality databases show that our proposed model achieves better performance than other state-of-the-art NR IQA algorithms.

Visual perception of computer-generated stereoscopic pictures: Toward the impact of image resolution

In comparison with the generation of monoscopic images, the time cost of rendering stereoscopic images is doubled. When generating stereoscopic images by computer algorithms, it is desirable to save the computational expense by decreasing the image resolution, without degrading the visual perceptual quality of the images. In this work, to evaluate the perceptual visual quality of computer-generated stereoscopic images (CGSIs), a data set consisting of stereoscopic images created with different horizontal and vertical resolutions was constructed. First, a series of subjective experiments for the analysis of various perceptual situations was conducted. The experimental results show that when the original image resolution was reduced by half, the image difference was not perceptible. In addition, based on full-reference (FR) and no-reference (NR) image quality measurement (IQM), a combined FR-and-NR CGSIQA model was established to predict perceptual quality. We perform weighting calculations for different combinations of FR and NR. The experimental results show that the proposed model significantly outperforms all the classical models.

An optimized CNN-based quality assessment model for screen content image

Most existing convolutional neural network (CNN) based models designed for natural image quality assessment (IQA) employ image patches as training samples for data augmentation, and obtain final quality score by averaging all predicted scores of image patches. This brings two problems when applying these methods for screen content image (SCI) quality assessment. Firstly, SCI contains more complex content compared to natural image. As a result, qualities of SCI patches are different, and the subjective differential mean opinion score (DMOS) is not appropriate as qualities of all image patches. Secondly, the average score of image patches does not represent the quality of entire SCI since the human visual system (HVS) is sensitive to image patches containing texture and edge information. In this paper, we propose a novel quadratic optimized model based on the deep convolutional neural network (QODCNN) for full-reference (FR) and no-reference (NR) SCI quality assessment to overcome these two problems. The contribution of our algorithm can be concluded as follows: 1) Considering the characteristics of SCIs, a valid network architecture is designed for both NR and FR visual quality evaluation of SCIs, which makes the networks learn the feature differences for FR-IQA; 2) with the consideration of correlation between local quality and DMOS, a training data selection method is proposed to fine-tune the pre-trained model with valid SCI patches; 3) an adaptive pooling approach is employed to fuse patch quality to obtain image quality, owns strong noise robust and effects on both FR and NR IQA. Experimental results verify that our model outperforms both current no-reference and full-reference image quality assessment methods on the benchmark screen content image quality assessment database (SIQAD). Cross-database evaluation shows high generalization ability and high effectiveness of our model.

RT-BVE—Real-Time no-reference Blocking Visibility Estimation in video frames

During lossy video compression, a blocking artifact is introduced into a video. It can significantly reduce the quality of a compressed video, and consequently end-user Quality of Experience. Hence, it is very important to measure and monitor the quality of compressed videos delivered to the end-user, which can be performed only by using the no-reference (NR) approach, without the original video signal available. Many NR video quality assessment (VQA) metrics are based on measurements of different artifacts visibility, which are then used as an input for an objective NR VQA metric. Thus, reliable artifacts detection and their visibility estimation are the first step in a reliable VQA process. In this paper, a new algorithm for Real-Time NR Blocking Visibility Estimation (RT-BVE) in video frames is proposed. The performance of the proposed RT-BVE algorithm is compared to the performance of two freely available algorithms, i.e., BBT-BDA and MSU Blocking, using videos from the well-known databases (156 video sequences in total and more than 48,000 frames). While RT-BVE achieves performance comparable to BBT-BDA and MSU Blocking algorithms for MPEG-2 videos, it outperforms both algorithms for H.264 videos. The results show that the proposed RT-BVE algorithm can precisely estimate blocking visibility in video frames for videos compressed according to different compression standards by using a fixed set of algorithm parameter values. Additionally, RT-BVE is suitable for usage in real-time applications since it is capable of processing approximately 100 Full HD video frames per second on the current midrange × 86–64 platform.

No-reference quality assessment of HEVC video streams based on visual memory modelling

Providing adequate Quality of Experience (QoE) to end-users is crucial for streaming service providers. In this paper, in order to realize automatic quality assessment, a No-Reference (NR) bitstream Human-Vision-System-(HVS)-based video quality assessment (VQA) model is proposed. Inspired by discoveries from the neuroscience community, which suggest there is a considerable overlap between active areas of the brain when engaging in video quality assessment and saliency detection tasks, saliency maps are used in the proposed method to improve the quality assessment accuracy. To this end, saliency maps are first generated from features extracted from the HEVC bitstream. Then, saliency map statistics are employed to create a model of visual memory. Finally, a support vector regression pipeline learns an estimate of the video quality from the visual memory, saliency, and frame features. Evaluations on SJTU dataset indicate that the proposed bitstream based no-reference video quality assessment algorithm achieves a competitive performance.

Structured Computational Modeling of Human Visual System for No-reference Image Quality Assessment

Objective image quality assessment (IQA) plays an important role in various visual communication systems, which can automatically and efficiently predict the perceived quality of images. The human eye is the ultimate evaluator for visual experience, thus the modeling of human visual system (HVS) is a core issue for objective IQA and visual experience optimization. The traditional model based on black box fitting has low interpretability and it is difficult to guide the experience optimization effectively, while the model based on physiological simulation is hard to integrate into practical visual communication services due to its high computational complexity. For bridging the gap between signal distortion and visual experience, in this paper, we propose a novel perceptual no-reference (NR) IQA algorithm based on structural computational modeling of HVS. According to the mechanism of the human brain, we divide the visual signal processing into a low-level visual layer, a middle-level visual layer and a high-level visual layer, which conduct pixel information processing, primitive information processing and global image information processing, respectively. The natural scene statistics (NSS) based features, deep features and free-energy based features are extracted from these three layers. The support vector regression (SVR) is employed to aggregate features to the final quality prediction. Extensive experimental comparisons on three widely used benchmark IQA databases (LIVE, CSIQ and TID2013) demonstrate that our proposed metric is highly competitive with or outperforms the state-of-the-art NR IQA measures.

Predicting Spatio-Temporal Entropic Differences for Robust No Reference Video Quality Assessment

We consider the problem of robust no reference (NR) video quality assessment (VQA) where the algorithms need to have good generalization performance when they are trained and tested on different datasets. We specifically address this question in the context of predicting video quality for compression and transmission applications. Motivated by the success of the spatio-temporal entropic differences video quality predictor in this context, we design a framework using convolutional neural networks to predict spatial and temporal entropic differences without the need for a reference or human opinion score. This approach enables our model to capture both spatial and temporal distortions effectively and allows for robust generalization. We evaluate our algorithms on a variety of datasets and show superior cross database performance when compared to state of the art NR VQA algorithms.

Comparative Perceptual Assessment of Visual Signals Using Free Energy Features

In this paper, we put forward the concept of comparative perceptual quality assessment (C-PQA), which refers to the judgment of relative qualities of two visual signals of the same content, but subject to different types and levels of distortions. While it is straightforward for human observers to fulfill the CPQA task in daily lives, it remains a difficult challenge for the current research of perceptual quality assessment (PQA). Among the existing PQA algorithms, the full-reference (FR) and reducedreference (RR) methods both need prior knowledge of the original images while the no-reference (NR) algorithms usually work with a single input image. C-PQA is inherently different from those existing methods in that it takes an image pair as input and predicts their relative quality without using any knowledge about the original image. In this paper, we propose a brain theory inspired approach to C-PQA that emulates the process of comparing the relative quality of two visual stimuli as performed by the human visual system (HVS) within the framework of free energy minimization. The brain's internal generative models initialized on the inputs are then used to explain both images. During the internal generative modeling, a group of features are extracted and then integrated to determine the relative quality of two images. We designed a dedicated image database to test the proposed C-PQA algorithm. Experimental results show that the proposed method achieves up to 98% prediction accuracy in line with the subjective ratings, outperforming many state of the art PQA algorithms.

Learning to predict the quality of distorted-then-compressed images via a deep neural network

Being captured by amateur photographers, reciprocally propagated through multimedia pipelines, and compressed with different levels, real-world images usually suffer from a wide variety of hybrid distortions. Faced with this scenario, full-reference (FR) image quality assessment (IQA) algorithms can not deliver promising predictions due to the inferior references. Meanwhile, existing no-reference (NR) IQA algorithms remain limited in their efficacy to deal with different distortion types. To address this obstacle, we explore a NR-IQA metric by predicting the perceptual quality of distorted-then-compressed images using a deep neural network (DNN). First, we propose a novel two-stream DNN to handle both authentic distortions and synthetic compressions and adopt effective strategies to pre-train the two branches of the network. Specifically, we transfer the knowledge learned from in-the-wild images to account for authentic distortions by utilizing a pre-trained deep convolutional neural network (CNN) to provide meaningful initializations. Meanwhile, we build a CNN for synthetic compressions and pre-train it on a dataset including synthetic compressed images. Subsequently, we bilinearly pool these two sets of features as the image representation. The overall network is fine-tuned on an elaborately-designed auxiliary dataset, which is annotated by a reliable objective quality metric. Furthermore, we integrate the output of the authentic-distortion-aware branch with that of the overall network following a two-step prediction manner to boost the prediction performance, which can be applied in the distorted-then-compressed scenario when the reference image is available. Extensive experimental results on several databases especially on the LIVE Wild Compressed Picture Quality Database show that the proposed method achieves state-of-the-art performance with good generalizability and moderate computational complexity.

Staged-Learning: Assessing the Quality of Screen Content Images from Distortion Information

The small volume of the existing screen content images (SCIs) database with human ratings restricts the training processes of no-reference (NR) image quality assessment models based on traditional machine learning and deep learning. In this letter, we propose an NR model called the multi-task distortion-learning network to jointly analyse the distortion types and distortion degree of SCIs to be the prior knowledge for predicting the SCIs quality. Specifically, we first generate sufficient distorted SCIs labelled with the distortion type and degree, which does not need much effort to conduct subjective scoring experiments. Then, relying on these data, we pre-train a multi-task learning network to obtain strong prior knowledge about assessing the image quality. Finally, we further jointly train a quality assessment network with an attention module that simulates the mechanism of processing visual signals in the human eyes. The experimental results on the public SCIs databases show that the proposed model is competitive against other state-of-art approaches and achieves better consistency with the human vision system.

Learning a Unified Blind Image Quality Metric via On-Line and Off-Line Big Training Instances

In this work, we resolve a big challenge that most current image quality metrics (IQMs) are unavailable across different image contents, especially simultaneously coping with natural scene (NS) images or screen content (SC) images. By comparison with existing works, this paper deploys on-line and off-line data for proposing a unified no-reference (NR) IQM, not only applied to different distortion types and intensities but also to various image contents including classical NS images and prevailing SC images. Our proposed NR IQM is developed with two data-driven learning processes following feature extraction, which is based on scene statistic models, free-energy brain principle, and human visual system (HVS) characteristics. In the first process, the scene statistic models and an image retrieve technique are combined, based on on-line and off-line training instances, to derive a novel loose classifier for retrieving clean images and helping to infer the image content. In the second process, the features extracted by incorporating the inferred image content, free-energy and low-level perceptual characteristics of the HVS are learned by utilizing off-line training samples to analyze the distortion types and intensities and thereby to predict the image quality. The two processes mentioned above depend on a gigantic quantity of training data, much exceeding the number of images applied to performance validation, and thus make our model's performance more reliable. Through extensive experiments, it has been validated that the proposed blind IQM is capable of simultaneously inferring the quality of NS and SC images, and it has attained superior performance as compared with popular and state-of-the-art IQMs on the subjective NS and SC image quality databases. The source code of our model will be released with the publication of the paper at https://kegu.netlify.com.

Re-Visiting Discriminator for Blind Free-Viewpoint Image Quality Assessment

Accurate measurement of perceptual quality is important for various immersive multimedia, which demand real-time quality control or quality-based bench-marking for relevant algorithms. For instance, virtual views rendering in Free-Viewpoint (FV) navigation scenarios is a typical case that introduces challenging distortions, particularly the ones around dis-occluded regions. Existing quality metrics, most of which are targeting for impairments caused by compression or network condition, fail to quantify such non-uniform structure-related distortions. Moreover, the lack of quality databases for such distortions makes it even more challenging to develop robust quality metrics. In this work, a Generative Adversarial Networks based No-Reference (NR) quality Metric, namely GANs-NRM, is proposed. We first present an approach to create masks mimicking dis-occlusions/textureless regions, which is applicable on large-scale 2D image databases publicly available in the computer vision domain. Using these synthetic data, we then train a GANs-based context renderer with the capability of rendering those masked regions. Since the naturalness of the rendered dis-occluded regions strongly relates to the perceptual quality, we assume that the discriminator of the trained GANs has an intrinsic ability for quality assessment. We thus use the features extracted from the discriminator to learn a Bag-of-Distortion-Word (BDW) codebook. We show that a quality predictor can be then well trained using only a small amount of subjective quality data for the FV views rendering. Moreover, in the proposed framework, the discriminator is also adapted as a distortion-detector to locate possible distorted regions. According to the experimental results, the proposed model outperforms significantly the state-of-the-art quality metrics. The corresponding context renderer also shows appealing visualized results over other rendering algorithms.

Learning Local Quality-Aware Structures of Salient Regions for Stereoscopic Images via Deep Neural Networks

The perceptual quality of stereoscopic images plays an essential role in the human perception of visual information. However, most available stereoscopic image quality assessment (SIQA) methods evaluate 3D visual experience using hand-crafted features or shallow architectures, which cannot model the visual properties of stereo images well. In this paper, we use convolutional neural networks (CNNs) to learn deeper local quality-aware structures for stereo images. With different inputs, two CNN models are designed for no-reference SIQA tasks. The one-column CNN model directly accepts a cyclopean view as the input, and the three-column CNN model jointly considers the cyclopean, left and right views as CNN inputs. The two SIQA frameworks share the same implementation approach: First, to overcome the obstacle of limited SIQA datasets, we accept image patches that have been cropped from corresponding stereopairs as inputs for local quality-sensitive feature extraction. Next, a local feature selection algorithm is used to remove related features on non-salient patches, which could cause large prediction errors. Finally, the reserved local visual structures of salient regions are aggregated into a final quality score in an end-to-end manner. Experimental results on three public SIQA databases demonstrate that our method outperforms most state-of-the-art no-reference (NR) SIQA methods. The results of a cross-database experiment also show the robustness and generality of the proposed method.