Screen Content Images Research Articles

Blind Quality Assessment Based on Pseudo-Reference Image

Traditional full-reference image quality assessment (IQA) metrics generally predict the quality of the distorted image by measuring its deviation from a perfect quality image called reference image. When the reference image is not fully available, the reduced-reference and no-reference IQA metrics may still be able to derive some characteristics of the perfect quality images, and then measure the distorted image's deviation from these characteristics. In this paper, contrary to the conventional IQA metrics, we utilize a new “reference” called pseudo-reference image (PRI) and a PRI-based blind IQA (BIQA) framework. Different from a traditional reference image, which is assumed to have a perfect quality, PRI is generated from the distorted image and is assumed to suffer from the severest distortion for a given application. Based on the PRI-based BIQA framework, we develop distortion-specific metrics to estimate blockiness, sharpness, and noisiness. The PRI-based metrics calculate the similarity between the distorted image's and the PRI's structures. An image suffering from severer distortion has a higher degree of similarity with the corresponding PRI. Through a two-stage quality regression after a distortion identification framework, we then integrate the PRI-based distortion-specific metrics into a general-purpose BIQA method named blind PRI-based (BPRI) metric. The BPRI metric is opinion-unaware (OU) and almost training-free except for the distortion identification process. Comparative studies on five large IQA databases show that the proposed BPRI model is comparable to the state-of-the-art opinion-aware- and OU-BIQA models. Furthermore, BPRI not only performs well on natural scene images, but also is applicable to screen content images. The MATLAB source code of BPRI and other PRI-based distortion-specific metrics will be publicly available.

Quality Assessment of Screen Content Images via Convolutional-Neural-Network-Based Synthetic/Natural Segmentation.

The recent popularity of remote desktop software and live streaming of composited video has given rise to a growing number of applications which make use of so-called screen content images that contain a mixture of text, graphics, and photographic imagery. Automatic quality assessment (QA) of screen-content images is necessary to enable tasks such as quality monitoring, parameter adaptation, and other optimizations. Although QA of natural images has been heavily researched over the last several decades, QA of screen content images is a relatively new topic. In this paper, we present a QA algorithm, called convolutional neural network (CNN) based screen content image quality estimator (CNN-SQE), which operates via a fuzzy classification of screen content images into plain-text, computergraphics/ cartoons, and natural-image regions. The first two classes are considered to contain synthetic content (text/graphics), and the latter two classes are considered to contain naturalistic content (graphics/photographs), where the overlap of the classes allows the computer graphics/cartoons segments to be analyzed by both text-based and natural-image-based features. We present a CNN-based approach for the classification, an edge-structurebased quality degradation model, and a region-size-adaptive quality-fusion strategy. As we will demonstrate, the proposed CNN-SQE algorithm can achieve better/competitive performance as compared with other state-of-the-art QA algorithms.

Screen content image quality assessment based on the most preferred structure feature

The quality of screen content images (SCIs) plays a crucial benchmark of displaying the system and directly affects the user experience of multidevice communication. However, accurately predicting the quality of SCIs with an objective algorithm still has many challenges. Hence, we propose a new no-reference method for evaluating the quality of SCIs effectively and efficiently. Specifically, we first propose a simple but effective strategy to partition the SCIs into the rough and smooth regions. Then, a most preferred structure (MPS) metric is derived to represent the meaningful part of each image patch according to the free-energy theory of human brain. Subsequently, the MPSs of the rough and smooth regions are pooled for constructing the rough and smooth features of SCIs, separately. Finally, to evaluate the final visual quality, the rough and smooth features are combined as the quality-aware features of SCIs, and a learning-based framework is employed to map the quality-aware features to objective quality scores. The experimental results show that the performance of the proposed method is highly consistent with the human perception and superior to the state-of-the-art quality assessment methods on the SCI-oriented databases. Furthermore, the proposed method delivers a high computational efficiency and is very suitable to be integrated into the displaying system.

Blind Image Quality Estimation via Distortion Aggravation

Traditional blind image quality assessment (IQA) measures generally predict quality from a sole distorted image directly. In this paper, we first introduce multiple pseudo reference images (MPRIs) by further degrading the distorted image in several ways and to certain degrees, and then compare the similarities between the distorted image and the MPRIs. Via such distortion aggravation, we can have some references to compare with, i.e., the MPRIs, and utilize the full-reference IQA framework to compute the quality. Specifically, we apply four types and five levels of distortion aggravation to deal with the commonly encountered distortions. Local binary pattern features are extracted to describe the similarities between the distorted image and the MPRIs. The similarity scores are then utilized to estimate the overall quality. More similar to a specific pseudo reference image (PRI) indicates closer quality to this PRI. Owning to the availability of the created multiple PRIs, we can reduce the influence of image content, and infer the image quality more accurately and consistently. Validation is conducted on four mainstream natural scene image and screen content image quality assessment databases, and the proposed method is comparable to or outperforms the state-of-the-art blind IQA measures. The MATLAB source code of the proposed measure will be publicly available.

Local and Global Feature Learning for Blind Quality Evaluation of Screen Content and Natural Scene Images.

The blind quality evaluation of screen content images (SCIs) and natural scene images (NSIs) has become an important, yet very challenging issue. In this paper, we present an effective blind quality evaluation technique for SCIs and NSIs based on a dictionary of learned local and global quality features. First, a local dictionary is constructed using local normalized image patches and conventional -means clustering. With this local dictionary, the learned local quality features can be obtained using a locality-constrained linear coding with max pooling. To extract the learned global quality features, the histogram representations of binary patterns are concatenated to form a global dictionary. The collaborative representation algorithm is used to efficiently code the learned global quality features of the distorted images using this dictionary. Finally, kernel-based support vector regression is used to integrate these features into an overall quality score. Extensive experiments involving the proposed evaluation technique demonstrate that in comparison with most relevant metrics, the proposed blind metric yields significantly higher consistency in line with subjective fidelity ratings.

No Reference Quality Assessment for Screen Content Images With Both Local and Global Feature Representation.

In this paper, we propose a novel no reference quality assessment method by incorporating statistical luminance and texture features (NRLT) for screen content images (SCIs) with both local and global feature representation. The proposed method is designed inspired by the perceptual property of the human visual system (HVS) that the HVS is sensitive to luminance change and texture information for image perception. In the proposed method, we first calculate the luminance map through the local normalization, which is further used to extract the statistical luminance features in global scope. Second, inspired by existing studies from neuroscience that high-order derivatives can capture image texture, we adopt four filters with different directions to compute gradient maps from the luminance map. These gradient maps are then used to extract the second-order derivatives by local binary pattern. We further extract the texture feature by the histogram of high-order derivatives in global scope. Finally, support vector regression is applied to train the mapping function from quality-aware features to subjective ratings. Experimental results on the public large-scale SCI database show that the proposed NRLT can achieve better performance in predicting the visual quality of SCIs than relevant existing methods, even including some full reference visual quality assessment methods.

Reduced-Reference Quality Assessment of Screen Content Images

The screen content images (SCIs) quality influences the user experience and the interactive performance of remote computing systems. With numerous approaches proposed to evaluate the quality of natural images, much less work has been dedicated to reduced-reference image quality assessment (RR-IQA) of SCIs. Here, we propose an RR-IQA method from the perspective of SCI visual perception. In particular, the quality of the distorted SCI is evaluated by comparing a set of extracted statistical features that consider both primary visual information and unpredictable uncertainty. A unique property that differentiates the proposed method from previous RR-IQA methods for natural images is the consideration of behaviors when human subjects view the screen content, which motivates us to establish the perceptual model according to the distinct properties of SCIs. Validations based on the screen content IQA database show that the proposed algorithm provides accurate predictions across a wide range of SCI distortions with negligible transmission overhead.

Blind quality assessment for screen content images by orientation selectivity mechanism

Currently, most objective image quality assessment (IQA) methods designed for screen content images (SCIs) require reference information, and existing blind IQA metrics cannot obtain consistent results with subjective scores. In this study, we propose a novel blind IQA method for SCIs based on orientation selectivity mechanism. First, we extract orientation features to compute the visual distortion of degraded SCIs by orientation selectivity mechanism. The statistical orientation features are further obtained by the histogram of orientation features. Second, the statistical structure features are calculated as the complementary information of orientation features for quality prediction of degraded SCIs. Finally, we employ support vector regression (SVR) as the mapping function from these extracted statistical features to quality scores. Experimental results show that the proposed method can obtain better performance of visual quality prediction for SCIs than other existing related methods.

Improved Sparse 3D Transform-Domain Collaborative Filter for Screen Content Image Denoising

Screen content videos or images are widely used in applications such as screen sharing. Compressed screen content videos or images may have distortions or noises because of the quantization process. This paper proposes an improved sparse 3D transform-domain collaborative filter to enhance screen content image quality by block classification and block segmentation. Experimental results show that the proposed algorithm achieves a peak signal-to-noise ratio increase and subjective visual quality improvements for reconstructed screen content images.

Evaluating Quality of Screen Content Images Via Structural Variation Analysis.

With the quick development and popularity of computers, computer-generated signals have drastically invaded into our daily lives. Screen content image is a typical example, since it also includes graphic and textual images as components as compared with natural scene images which have been deeply explored, and thus screen content image has posed novel challenges to current researches, such as compression, transmission, display, quality assessment, and more. In this paper, we focus our attention on evaluating the quality of screen content images based on the analysis of structural variation, which is caused by compression, transmission, and more. We classify structures into global and local structures, which correspond to basic and detailed perceptions of humans, respectively. The characteristics of graphic and textual images, e.g., limited color variations, and the human visual system are taken into consideration. Based on these concerns, we systematically combine the measurements of variations in the above-stated two types of structures to yield the final quality estimation of screen content images. Thorough experiments are conducted on three screen content image quality databases, in which the images are corrupted during capturing, compression, transmission, etc. Results demonstrate the superiority of our proposed quality model as compared with state-of-the-art relevant methods.

Saliency-induced reduced-reference quality index for natural scene and screen content images

Massive content composed of both natural scene and screen content has been generated with the increasing use of wireless computing and cloud computing, which call for general image quality assessment (IQA) measures working for both natural scene images (NSIs) and screen content images (SCIs). In this paper, we develop a saliency-induced reduced-reference (SIRR) IQA measure for both NSIs and SCIs. Image quality and visual saliency are two widely studied and closely related research topics. Traditionally, visual saliency is often used as a weighting map in the final pooling stage of IQA. Instead, we detect visual saliency as a quality feature since different types and levels of degradation can strongly influence saliency detection. Image quality is described by the similarity between two images’ saliency maps. In SIRR, saliency is detected through a binary image descriptor called “image signature”, which significantly reduces the reference data. We perform extensive experiments on five large-scale NSI quality assessment databases including LIVE, TID2008, CSIQ, LIVEMD, CID2013, as well as two recently constructed SCI QA databases, i.e., SIQAD and QACS. Experimental results show that SIRR is comparable to state-of-the-art full-reference and reduced-reference IQA measures in NSIs, and it can outperform most competitors in SCIs. The most important is that SIRR is a cross-content-type measure, which works efficiently for both NSIs and SCIs. The MATLAB source code of SIRR will be publicly available with this paper.

Adaptive Chroma Subsampling-Binding and Luma-Guided Chroma Reconstruction Method for Screen Content Images.

In this paper, we propose a novel adaptive chroma subsampling-binding and luma-guided (ASBLG) chroma reconstruction method for screen content images (SCIs). After receiving the decoded luma and subsampled chroma image from the decoder, a fast winner-first voting strategy is proposed to identify the used chroma subsampling scheme prior to compression. Then, the decoded luma image is subsampled as the identified subsampling scheme was performed on the chroma image such that we are able to conclude an accurate correlation between the subsampled decoded luma image and the decoded subsampled chroma image. Accordingly, an adaptive sliding window-based and luma-guided chroma reconstruction method is proposed. The related computational complexity analysis is also provided. We take two quality metrics, the color peak signal-to-noise ratio (CPSNR) of the reconstructed chroma images and SCIs and the gradient-based structure similarity index (CGSS) of the reconstructed SCIs to evaluate the quality performance. Let the proposed chroma reconstruction method be denoted as `ASBLG'. Based on 26 typical test SCIs and 6 JCT-VC test screen content video sequences (SCVs), several experiments show that on average, the CPSNR gains of all the reconstructed UV images by 4:2:0(A)-ASBLG, SCIs by 4:2:0(MPEG-B)-ASBLG, and SCVs by 4:2:0(A)-ASBLG are 2.1, 1.87, and 1.87 dB, respectively, when compared with that of the other combinations. Specifically, in terms of CPSNR and CGSS, CSBILINEAR-ASBLG for the test SCIs and CSBICUBIC-ASBLG for the test SCVs outperform the existing state-of-the-art comparative combinations, where CSBILINEAR and CSBICUBIC denote the luma-aware based chroma subsampling schemes by Wang et al.

Unified Blind Quality Assessment of Compressed Natural, Graphic, and Screen Content Images.

Digital images in the real world are created by a variety of means and have diverse properties. A photographical natural scene image (NSI) may exhibit substantially different characteristics from a computer graphic image (CGI) or a screen content image (SCI). This casts major challenges to objective image quality assessment, for which existing approaches lack effective mechanisms to capture such content type variations, and thus are difficult to generalize from one type to another. To tackle this problem, we first construct a cross-content-type (CCT) database, which contains 1,320 distorted NSIs, CGIs, and SCIs, compressed using the high efficiency video coding (HEVC) intra coding method and the screen content compression (SCC) extension of HEVC. We then carry out a subjective experiment on the database in a well-controlled laboratory environment. Moreover, we propose a unified content-type adaptive (UCA) blind image quality assessment model that is applicable across content types. A key step in UCA is to incorporate the variations of human perceptual characteristics in viewing different content types through a multi-scale weighting framework. This leads to superior performance on the constructed CCT database. UCA is training-free, implying strong generalizability. To verify this, we test UCA on other databases containing JPEG, MPEG-2, H.264, and HEVC compressed images/videos, and observe that it consistently achieves competitive performance.

No-Reference Quality Assessment of Screen Content Pictures

Recent years have witnessed a growing number of image and video centric applications on mobile, vehicular, and cloud platforms, involving a wide variety of digital screen content images. Unlike natural scene images captured with modern high fidelity cameras, screen content images are typically composed of fewer colors, simpler shapes, and a larger frequency of thin lines. In this paper, we develop a novel blind/no-reference (NR) model for accessing the perceptual quality of screen content pictures with big data learning. The new model extracts four types of features descriptive of the picture complexity, of screen content statistics, of global brightness quality, and of the sharpness of details. Comparative experiments verify the efficacy of the new model as compared with existing relevant blind picture quality assessment algorithms applied on screen content image databases. A regression module is trained on a considerable number of training samples labeled with objective visual quality predictions delivered by a high-performance full-reference method designed for screen content image quality assessment (IQA). This results in an opinion-unaware NR blind screen content IQA algorithm. Our proposed model delivers computational efficiency and promising performance. The source code of the new model will be available at: https://sites.google.com/site/guke198701/publications .

ESIM: Edge Similarity for Screen Content Image Quality Assessment.

In this paper, an accurate full-reference image quality assessment (IQA) model developed for assessing screen content images (SCIs), called the edge similarity (ESIM), is proposed. It is inspired by the fact that the human visual system (HVS) is highly sensitive to edges that are often encountered in SCIs; therefore, essential edge features are extracted and exploited for conducting IQA for the SCIs. The key novelty of the proposed ESIM lies in the extraction and use of three salient edge features-i.e., edge contrast, edge width, and edge direction. The first two attributes are simultaneously generated from the input SCI based on a parametric edge model, while the last one is derived directly from the input SCI. The extraction of these three features will be performed for the reference SCI and the distorted SCI, individually. The degree of similarity measured for each above-mentioned edge attribute is then computed independently, followed by combining them together using our proposed edge-width pooling strategy to generate the final ESIM score. To conduct the performance evaluation of our proposed ESIM model, a new and the largest SCI database (denoted as SCID) is established in our work and made to the public for download. Our database contains 1800 distorted SCIs that are generated from 40 reference SCIs. For each SCI, nine distortion types are investigated, and five degradation levels are produced for each distortion type. Extensive simulation results have clearly shown that the proposed ESIM model is more consistent with the perception of the HVS on the evaluation of distorted SCIs than the multiple state-of-the-art IQA methods.

Towards efficient blind quality evaluation of screen content images based on edge‐preserving filter

The problem of quality assessment of screen content images from two respects is addressed, the edge-preserving filter based free energy and structural degradation model. For screen content images which always contain texts, edge information plays an important role in the process of evaluating the screen images quality. Inspired by this, the edge-preserving filter based free energy entropy and structural degradation model were combined to extract the quality features and accordingly the statistical model of screen content images was established. Experimental results prove that the proposed method can produce highly consistent with human perception, particularly superior to state-of-the-art full- and no-reference quality metrics on the SIQAD database dedicated to the quality assessment of screen images.

Perceptual Quality Measure of Screen Content Images

Perceptual Quality Measure of Screen Content Images

Objective Quality Assessment of Screen Content Images by Uncertainty Weighting.

In this paper, we propose a novel full-reference objective quality assessment metric for screen content images (SCIs) by structure features and uncertainty weighting (SFUW). The input SCI is first divided into textual and pictorial regions. The visual quality of textual regions is estimated based on perceptual structural similarity, where the gradient information is adopted as the structural feature. To predict the visual quality of pictorial regions in SCIs, we extract the structural features and luminance features for similarity computation between the reference and distorted pictorial patches. To obtain the final visual quality of SCI, we design an uncertainty weighting method by perceptual theories to fuse the visual quality of textual and pictorial regions effectively. Experimental results show that the proposed SFUW can obtain better performance of visual quality prediction for SCIs than other existing ones.

Naturalization of Screen Content Images for Enhanced Quality Evaluation

Naturalization of Screen Content Images for Enhanced Quality Evaluation

Subjective and Objective Quality Assessment of Compressed Screen Content Images

Objectively accessing the quality of screen content images (SCIs) is a challenging problem, as SCIs may not always have identical properties as natural scenes. Here we conduct comprehensive studies on the subjective and objective quality assessment of the compressed SCIs. Firstly, we build a database that contains the distorted SCIs generated by the high efficiency video coding standard as well as its extension on screen content compression. Subsequently, subjective experiments are conducted to evaluate the perceived quality of these SCIs with compression artifacts. To automatically predict the subjective quality, a reduced-reference quality assessment model is further learnt by a set of wavelet domain features concerning the generalized spectral behavior, the fluctuations of the energy, and the information content with relatively large scale training samples. Our experimental results show that the learnt model is able to achieve better prediction on the SCI quality with a few extracted meaningful features.