Screen Content Images Research Articles

Full-reference Screen Content Image Quality Assessment by Fusing Multilevel Structure Similarity

Screen content images (SCIs) usually comprise various content types with sharp edges, in which artifacts or distortions can be effectively sensed by a vanilla structure similarity measurement in a full-reference manner. Nonetheless, almost all of the current state-of-the-art (SOTA) structure similarity metrics are “locally” formulated in a single-level manner, while the true human visual system (HVS) follows the multilevel manner; such mismatch could eventually prevent these metrics from achieving reliable quality assessment. To ameliorate this issue, this article advocates a novel solution to measure structure similarity “globally” from the perspective of sparse representation. To perform multilevel quality assessment in accordance with the real HVS, the abovementioned global metric will be integrated with the conventional local ones by resorting to the newly devised selective deep fusion network. To validate its efficacy and effectiveness, we have compared our method with 12 SOTA methods over two widely used large-scale public SCI datasets, and the quantitative results indicate that our method yields significantly higher consistency with subjective quality scores than the current leading works. Both the source code and data are also publicly available to gain widespread acceptance and facilitate new advancement and validation.

Subjective and Objective Quality Assessments of Display Products.

In recent years, people’s daily lives have become inseparable from a variety of electronic devices, especially mobile phones, which have undoubtedly become necessity in people’s daily lives. In this paper, we are looking for a reliable way to acquire visual quality of the display product so that we can improve the user’s experience with the display product. This paper proposes two major contributions: the first one is the establishment of a new subjective assessment database (DPQAD) of display products’ screen images. Specifically, we invited 57 inexperienced observers to rate 150 screen images showing the display product. At the same time, in order to improve the reliability of screen display quality score, we combined the single stimulation method with the stimulation comparison method to evaluate the newly created display products’ screen images database effectively. The second one is the development of a new no-reference image quality assessment (IQA) metric. For a given image of the display product, first our method extracts 27 features by analyzing the contrast, sharpness, brightness, etc., and then uses the regression module to obtain the visual quality score. Comprehensive experiments show that our method can evaluate natural scene images and screen content images at the same time. Moreover, compared with ten state-of-the-art IQA methods, our method shows obvious superiority on DPQAD.

A fast algorithm based on gray level co-occurrence matrix and Gabor feature for HEVC screen content coding

To reduce the computational complexity of screen content video coding (SCC), a fast algorithm based on gray level co-occurrence matrix and Gabor feature model for HEVC-SCC, denoted as GGM, is proposed in this paper. By studying the correlation of non-zero number in gray level co-occurrence matrix with different partitioning depth, the coding unit (CU) size of intra coding can be prejudged, which selectively skips the intra prediction process of CU in other depth. With Gabor filter, the edge information reflecting the features of screen content images to the human visual system (HVS) are extracted. According to Gabor feature, CUs are classified into natural content CUs (NCCUs), smooth screen content CUs (SSCUs) and complex screen content CUs (CSCUs), with which, the calculation and judgment of unnecessary intra prediction modes are skipped. Under all-intra (AI) configuration, experimental results show that the proposed algorithm GGM can achieve encoding time saving by 42.13% compared with SCM-8.3, and with only 1.85% bit-rate increasement.

Multiply distorted image quality assessment based on feature level fusion and optimal feature selection

No reference image quality assessment (NR-IQA) has received considerable importance in the last decade due to a rise in the use of multimedia content in our daily lives. Due to limitations in technology, multiple distortions may be introduced in the images that need to be assessed. Recently feature selection has shown promising results for single distorted NR-IQA and their effectiveness on multiple distorted images still need to be addressed. In this paper, impact of feature level fusion and feature selection on multiple distorted image quality assessment is presented. To this end features are extracted from multiple distorted images using six NR-IQA techniques (BLIINDS-II, BRISQUE, CurveletQA, DIIVINE, GM-LOG, SSEQ) that extract features in different (discrete cosine transform, spatial, curvelet transform, wavelet transform, spatial and gradient, spatial and spectral) domains. The extracted features from different domains are fused to generate a single feature vector. All combinations of feature-level fusion from six different techniques have been evaluated. Three different feature selection algorithms (genetic search, linear forward search, particle swarm optimization) are then applied to select optimum features for NR-IQA. The selected features are then used by the support vector regression model to predict the quality score. The performance of the proposed methodology is evaluated for two multiple distorted IQA databases (LIVE multiple distorted image dataset (LIVEMD), multiply distorted image database (MDID2017)), two singly synthetically distorted IQA databases (Tampere image database (TID2013), Computational and subjective image quality database (CSIQ)), and one screen content IQA database (Screen content image quality database (SIQAD)). Experimental results show that the fusion of features from different domains gives better performance in comparison to existing multiple-distorted NR-IQA techniques with SROCC scores of 0.9555, 0.9587, 0.6892, 0.9452, and 0.7682 on the LIVEMD, MDID, TID2013, CSIQ, and SIQAD databases respectively. Moreover, the performance is further improved when the genetic search feature selection algorithm is applied to fused features to remove the redundant and irrelevant features. The SROCC scores are improved to 0.9691, 0.9723, and 0.6897 for LIVEMD, MDID, and TID2013 databases respectively.

An optimized CNN-based quality assessment model for screen content image

Most existing convolutional neural network (CNN) based models designed for natural image quality assessment (IQA) employ image patches as training samples for data augmentation, and obtain final quality score by averaging all predicted scores of image patches. This brings two problems when applying these methods for screen content image (SCI) quality assessment. Firstly, SCI contains more complex content compared to natural image. As a result, qualities of SCI patches are different, and the subjective differential mean opinion score (DMOS) is not appropriate as qualities of all image patches. Secondly, the average score of image patches does not represent the quality of entire SCI since the human visual system (HVS) is sensitive to image patches containing texture and edge information. In this paper, we propose a novel quadratic optimized model based on the deep convolutional neural network (QODCNN) for full-reference (FR) and no-reference (NR) SCI quality assessment to overcome these two problems. The contribution of our algorithm can be concluded as follows: 1) Considering the characteristics of SCIs, a valid network architecture is designed for both NR and FR visual quality evaluation of SCIs, which makes the networks learn the feature differences for FR-IQA; 2) with the consideration of correlation between local quality and DMOS, a training data selection method is proposed to fine-tune the pre-trained model with valid SCI patches; 3) an adaptive pooling approach is employed to fuse patch quality to obtain image quality, owns strong noise robust and effects on both FR and NR IQA. Experimental results verify that our model outperforms both current no-reference and full-reference image quality assessment methods on the benchmark screen content image quality assessment database (SIQAD). Cross-database evaluation shows high generalization ability and high effectiveness of our model.

Staged-Learning: Assessing the Quality of Screen Content Images from Distortion Information

The small volume of the existing screen content images (SCIs) database with human ratings restricts the training processes of no-reference (NR) image quality assessment models based on traditional machine learning and deep learning. In this letter, we propose an NR model called the multi-task distortion-learning network to jointly analyse the distortion types and distortion degree of SCIs to be the prior knowledge for predicting the SCIs quality. Specifically, we first generate sufficient distorted SCIs labelled with the distortion type and degree, which does not need much effort to conduct subjective scoring experiments. Then, relying on these data, we pre-train a multi-task learning network to obtain strong prior knowledge about assessing the image quality. Finally, we further jointly train a quality assessment network with an attention module that simulates the mechanism of processing visual signals in the human eyes. The experimental results on the public SCIs databases show that the proposed model is competitive against other state-of-art approaches and achieves better consistency with the human vision system.

No-Reference Screen Content Image Quality Assessment With Unsupervised Domain Adaptation.

In this paper, we quest the capability of transferring the quality of natural scene images to the images that are not acquired by optical cameras (e.g., screen content images, SCIs), rooted in the widely accepted view that the human visual system has adapted and evolved through the perception of natural environment. Here, we develop the first unsupervised domain adaptation based no reference quality assessment method for SCIs, leveraging rich subjective ratings of the natural images (NIs). In general, it is a non-trivial task to directly transfer the quality prediction model from NIs to a new type of content (i.e., SCIs) that holds dramatically different statistical characteristics. Inspired by the transferability of pair-wise relationship, the proposed quality measure operates based on the philosophy of improving the transferability and discriminability simultaneously. In particular, we introduce three types of losses which complementarily and explicitly regularize the feature space of ranking in a progressive manner. Regarding feature discriminatory capability enhancement, we propose a center based loss to rectify the classifier and improve its prediction capability not only for source domain (NI) but also the target domain (SCI). For feature discrepancy minimization, the maximum mean discrepancy (MMD) is imposed on the extracted ranking features of NIs and SCIs. Furthermore, to further enhance the feature diversity, we introduce the correlation penalization between different feature dimensions, leading to the features with lower rank and higher diversity. Experiments show that our method can achieve higher performance on different source-target settings based on a light-weight convolution neural network. The proposed method also sheds light on learning quality assessment measures for unseen application-specific content without the cumbersome and costing subjective evaluations.

An Effective Bicubic Convolution Interpolation-Based Iterative Luma Optimization for Enhancing Quality in Chroma Subsampling

Traditionally, prior to compressing an RGB full-color image, for each converted $2\times 2$ CbCr block $B^{CbCr}$ , chroma subsampling only downsamples $B^{CbCr}$ , but without changing the luma block $B^{Y}$ at all. In the current research, a special linear interpolation-based, namely the COPY-based, chroma subsampling-first luma modification (CSFLM) study has attempted to change the luma block for enhancing the quality of the reconstructed RGB full-color image. In this paper, a fast and effective nonlinear interpolation, namely the bicubic convolution interpolation (BCI), based iterative luma modification method for CSFLM is proposed. In our iterative method, a BCI-based distortion function and its convex property proof are first provided. Next, based on the proposed convex distortion function, a pseudoinverse technique is applied to obtain the initial luma modification solution, and then an iterative method is proposed to improve the initial luma modification solution. Based on five testing image datasets, namely the IMAX, Kodak, SCI (screen content images), CI (classical images), and Video datasets, the thorough experimental results have demonstrated that on the newly released Versatile Video Coding (VVC) platform VTM-12.0, our iterative luma modification method achieves substantial quality, execution-time, and quality-bitrate tradeoff improvements when compared with the existing state-of-the-art methods.

No-Reference Quality Assessment for Screen Content Images Using Visual Edge Model and AdaBoosting Neural Network.

In this paper, a competitive no-reference metric is proposed to assess the perceptive quality of screen content images (SCIs), which uses the human visual edge model and AdaBoosting neural network. Inspired by the existing theory that the edge information which reflects the visual quality of SCI is effectively captured by the human visual difference of the Gaussian (DOG) model, we compute two types of multi-scale edge maps via the DOG operator firstly. Specifically, two types of edge maps contain contour and edge information respectively. Then after locally normalizing edge maps, L -moments distribution estimation is utilized to fit their DOG coefficients, and the fitted L -moments parameters can be regarded as edge features. Finally, to obtain the final perceptive quality score, we use an AdaBoosting back-propagation neural network (ABPNN) to map the quality-aware features to the perceptual quality score of SCIs. The reason why the ABPNN is regarded as the appropriate approach for the visual quality assessment of SCIs is that we abandon the regression network with a shallow structure, try a regression network with a deep architecture, and achieve a good generalization ability. The proposed method delivers highly competitive performance and shows high consistency with the human visual system (HVS) on the public SCI-oriented databases.

Quality Measurement of Screen Images via Foreground Perception and Background Suppression

The widespread screen sharing poses a great challenge in the visual quality control of industrial applications, as there is a significant difference between the visual quality perception of natural content (NC) and screen content (SC) images (or videos) during acquisition , compression , distribution , and display . Apart from the foreground distortion, artifacts appeared in the smooth background of SC images also play a key role in its perceived quality. In this article, a robust approach is designed for quality measurement of SC images, which embodies both foreground perception and background suppression (FPBS) in a satisfying way. Specifically, the extraction of foreground perceptual features is investigated by considering both the local contrast variation and the local edge response features, while for the first time, a suppression strategy is devised to eliminate the influences of artifacts appeared in the simple and homogeneous background. Experimental results show that, based on only two simple features, the proposed scheme achieves the state-of-the-art results on two latest benchmark databases. We believe that the feature engineering method can further improve the performance of the FPBS framework on the quality measurement of screen images.

Perceptual Redundancy Estimation of Screen Images via Multi-Domain Sensitivities

Visual redundancy detection is essential for image and video communication. Human visual system (HVS) is difficult to perceive the pixel magnitude change below a certain visibility threshold which is also known as just-noticeable-difference (JND). In this letter, we present an efficient JND estimation approach for screen content images by considering high-frequency sensitivity and orientation sensitivity correction. Specifically, to better quantify the visual redundancy, we investigate the visibility threshold based on the high-frequency distortion sensitivity. To obtain the orientation sensitivity correction, we divide the screen image pixels into three levels based on the oblique effect that considers the sensitive integrity of edges. Compared with several state-of-the-art JNDs, experimental results show that our method tolerates more perceptual redundancy, and delivers better visual quality under the same injected-noise energy. The implementation of the proposed method is publicly available at https://sites.google.com/site/wangmiaohui/.

A generalized quality assessment method for natural and screen content images

A generalized objective quality assessment method is proposed for natural images andscreen content images. Since natural images and screen content images have different sta-tistical properties, the modelling of a generalized quality assessment method that worksfor both types of images is complicated because some properties of natural imag es andscreen content images are conflicting to one another. The proposed method assesses theperceptual quality of an image based on edge magnitude and direction. In this method, animage is first separated into regions with high and low gradients. Gradient is used due tothe small perceptual span of the human visual system for textual content. For high gradientregions, small kernel size of Prewitt operators is used to obtain the gradient magnitude anddirection. Correspondingly, bigger kernel size of Prewitt operators is utilized for low gra-dient regions. Visual quality indices are computed from both regions and pooled to obtainthe final quality index. From the performance comparison, it is shown that the proposedmethod could assess the perceived quality of natural images and screen content imageswith high accuracy.

Learning a Unified Blind Image Quality Metric via On-Line and Off-Line Big Training Instances

In this work, we resolve a big challenge that most current image quality metrics (IQMs) are unavailable across different image contents, especially simultaneously coping with natural scene (NS) images or screen content (SC) images. By comparison with existing works, this paper deploys on-line and off-line data for proposing a unified no-reference (NR) IQM, not only applied to different distortion types and intensities but also to various image contents including classical NS images and prevailing SC images. Our proposed NR IQM is developed with two data-driven learning processes following feature extraction, which is based on scene statistic models, free-energy brain principle, and human visual system (HVS) characteristics. In the first process, the scene statistic models and an image retrieve technique are combined, based on on-line and off-line training instances, to derive a novel loose classifier for retrieving clean images and helping to infer the image content. In the second process, the features extracted by incorporating the inferred image content, free-energy and low-level perceptual characteristics of the HVS are learned by utilizing off-line training samples to analyze the distortion types and intensities and thereby to predict the image quality. The two processes mentioned above depend on a gigantic quantity of training data, much exceeding the number of images applied to performance validation, and thus make our model's performance more reliable. Through extensive experiments, it has been validated that the proposed blind IQM is capable of simultaneously inferring the quality of NS and SC images, and it has attained superior performance as compared with popular and state-of-the-art IQMs on the subjective NS and SC image quality databases. The source code of our model will be released with the publication of the paper at https://kegu.netlify.com.

Blind Quality Assessment of Screen Content Images Via Macro-Micro Modeling of Tensor Domain Dictionary

Screen content images (SCIs) have been rapidly and widely applied in interactive multimedia applications. The problem of quality assessment for SCIs is an interesting research topic. Most of the existing methods use subjective and independent features in gray domain to predict the image quality, which cannot comprehensively characterize the image properties or lack unified mathematical explanation for SCIs. To address these problems, we propose a novel blind quality assessment method based on macro-micro modeling of tensor domain dictionary for SCIs in this article. In the proposed method, the tensor decomposition is explored first to avoid the loss of color information, and then a target dictionary is learned more effectively with the principal components. Furthermore, a macro-micro model is established to characterize the micro and macro features in the target dictionary space, which can provide a systematic mathematical interpretation for feature extraction. For the micro features, a log-normal pooling scheme is designed to enhance the effectiveness of feature aggregation by analyzing the particularity of the statistical distribution of sparse codes. Additionally, the statistical properties are mainly discussed and studied based on the Bernoulli law of large numbers, and then a reliable macro feature is generated to describe the relationship between the statistical distribution and quality degradation of SCIs. Experimental results determined by using three public SCI databases show that the proposed method can perform better than relevant existing methods in the prediction of the visual quality of SCIs, especially in terms of the generalization for distortion type and interpretability for feature generation.

No Reference Quality Assessment for Screen Content Images Using Stacked Autoencoders in Pictorial and Textual Regions.

Recently, the visual quality evaluation of screen content images (SCIs) has become an important and timely emerging research theme. This article presents an effective and novel blind quality evaluation metric for SCIs by using stacked autoencoders (SAE) based on pictorial and textual regions. Since the SCI consists of not only the pictorial area but also the textual area, the human visual system (HVS) is not equally sensitive to their different distortion types. First, the textual and pictorial regions can be obtained by dividing an input SCI via an SCI segmentation metric. Next, we extract quality-aware features from the textual region and pictorial region, respectively. Then, two different SAEs are trained via an unsupervised approach for quality-aware features that are extracted from these two regions. After the training procedure of the SAEs, the quality-aware features can evolve into more discriminative and meaningful features. Subsequently, the evolved features and their corresponding subjective scores are input into two regressors for training. Each regressor can obtain one output predictive score. Finally, the final perceptual quality score of a test SCI is computed by these two predicted scores via a weighted model. Experimental results on two public SCI-oriented databases have revealed that the proposed scheme can compare favorably with the existing blind image quality assessment metrics.

Blind Quality Evaluation for Screen Content Images Based on Regionalized Structural Features

Currently, screen content images (SCIs) are widely used in our modern society. However, since SCIs have distinctly different properties compared to natural images, traditional quality assessment methods of natural images cannot precisely evaluate the quality of SCIs. Thus, we propose a blind quality evaluation method for SCIs based on regionalized structural features that are closely relevant to the intrinsic quality of SCIs. Firstly, the features of textual and pictorial regions of SCIs are extracted separately. For textual regions, since they contain noticeable structural information, we propose improved histograms of oriented gradients extracted from multi-order derivatives as structural features. For pictorial regions, since human vision is sensitive to texture information and luminance variation, we adopt texture as the structural feature; meanwhile, luminance is used as the auxiliary feature. The local derivative pattern and the shearlet local binary pattern are used to extract texture in the spatial and shearlet domains, respectively. Secondly, to derive the quality of textual and pictorial regions, two mapping functions are respectively trained from their features to subjective values. Finally, an activity weighting strategy is proposed to combine the quality of textual and pictorial regions. Experimental results show that the proposed method achieves better performance than the state-of-the-art methods.

Perceptual Hashing With Visual Content Understanding for Reduced-Reference Screen Content Image Quality Assessment

Numerous screen content images (SCIs) have been produced to meet the needs of virtual desktop and remote display, which put forward a very urgent requirement for security and management of SCIs. Perceptual hashing is an effective way to deal with this issue. However, since SCIs are generally composed of pictures, graphics and texts, their intrinsic characteristics are different from those of natural images. Thus the previous hashing methods for natural images are not suitable for SCIs. In this article, we propose a perceptual hashing method for SCIs from the perspective of visual content understanding. Specifically, considering that the visual content understanding of SCIs mainly comes from textual regions, while the contours of text always have thinner width and higher contrast, it is decided to generate hash in the gradient field. An input screen image is first performed by some joint preprocessing operations. Then the maximum gradient magnitude and corresponding orientation information are extracted from three color channels R, G and B. Normalized histogram and local frequency coefficient features are further obtained from the maximum gradient magnitude. Finally, a hash sequence is constructed by statistics that are derived from extracted features. Experiments validated on three SCIs databases were conducted to evaluate classification between robustness and discrimination. Receiver operating characteristics (ROC) results demonstrate that the proposed method is superior to the state-of-the-art algorithms. Besides, SIQAD and SCID databases were leveraged to present the application in reduced-reference screen content image quality assessment, and comparisons show that our hashing could provide accurate predictions than other metrics.

Hybrid Adaptive Lossless Image Compression Based on Discrete Wavelet Transform.

A new hybrid transform for lossless image compression exploiting a discrete wavelet transform (DWT) and prediction is the main new contribution of this paper. Simple prediction is generally considered ineffective in conjunction with DWT but we applied it to subbands of DWT modified using reversible denoising and lifting steps (RDLSs) with step skipping. The new transform was constructed in an image-adaptive way using heuristics and entropy estimation. For a large and diverse test set consisting of 499 photographic and 247 non-photographic (screen content) images, we found that RDLS with step skipping allowed effectively combining DWT with prediction. Using prediction, we nearly doubled the JPEG 2000 compression ratio improvements that could be obtained using RDLS with step skipping. Because for some images it might be better to apply prediction instead of DWT, we proposed compression schemes with various tradeoffs, which are practical contributions of this study. Compared with unmodified JPEG 2000, one scheme improved the compression ratios of photographic and non-photographic images, on average, by 1.2% and 30.9%, respectively, at the cost of increasing the compression time by 2% and introducing only minimal modifications to JPEG 2000. Greater ratio improvements, exceeding 2% and 32%, respectively, are attainable at a greater cost.

A local structural information representation method for image quality assessment

Image is a typical example of visual data, and its quality inevitably affects its application. Hence, measuring image quality accurately is a beneficial task. In practical application, there are different image types e.g. natural image and screen content image (SCIs). And the distortion types contained in images are various. Most image quality assessment (IQA) methods concentrate on a single image type with limited distortion types. In this paper, we present a no-reference IQA method which can accurately measure the quality for both natural image and SCI, and is robust for various distortion types. Human visual system is sensitive to the changes in image structural information which are usually caused by image quality degradation. Therefore, the new method employs local structural information representation for IQA. We first analyze the gray-scale fluctuation of each pixel in four detection directions to obtain four gray-scale fluctuation maps (GFMs) and one gray-scale fluctuation direction map (GFD). And then, the structural features extracted from GFMs and GFD are used for representing local structural information. Finally, the mapping function from the features to image subjective scores is trained by support vector regression (SVR). The experimental results on the public databases demonstrate that SVR is suitable for IQA and the proposed method can accurately predict the quality of both natural images and SCIs with various distortion types.

Unified non-uniform scale adaptive sampling model for quality assessment of natural scene and screen content images

Among many multimedia scenarios, digital images are mainly composed of natural scene images (NSIs) and screen content images (SCIs). NSIs are always captured by cameras from the real world while SCIs are typically the mixture of pictures and text. Traditional full-reference image quality assessment (IQA) metrics are designed for natural scene images, and they cannot work well on SCIs, because they do not take account of characteristics of different visual content/regions between NSIs and SCIs. Inspired by early eye movement study that human visual system samples the world non-completely or non-uniformly, we observe that human brain perceive different image content at different scales. Given these discoveries, we propose a non-uniform scale adaptive sampling model (NUSM) which dedicates to preprocess the input images for IQA methods, enabling traditional NSI IQA models to predict the quality of SCIs effectively. The proposed sampling model applies different sample scales into different regions to generate pre-sampled image pairs, then send to existing IQAs to evaluate the quality of images. The experimental results demonstrate that our model has achieved a promising performance on both NSIs and SCIs.