Screen Content Images Research Articles

No‐reference image quality assessment via structural information fluctuation

Image quality assessment (IQA) is a meaningful research topic to meet the increasing demand of high-quality image. The degradation of image quality will cause changes in image structural information. Meanwhile, human visual system is sensitive to changes in structural information. This finding motivates us to utilise structural information for proposing IQA method which is consistent with human visual perception. Recently, IQA methods are mainly focused on individual image type, e.g. natural image or screen content image (SCI), thus, the authors proposed a novel no-reference IQA method which can be suitable for both natural image and SCI. The proposed method is based on structural information analysis. For each image, they first obtain the grey-scale fluctuation maps (GFMs) in four detection directions. After that, the grey-scale fluctuation direction map (GFD) of certain image can be acquired via its GFMs. Based on the GFMs and GFD, the structural features of each image are extracted, and then collected and transformed to feature vectors. Subsequently, the IQA model is trained by support vector regression. The experimental results on the public databases demonstrate the proposed method can predict image quality accurately for both natural image and SCI, and the performance is competitive with prevalent methods.

Screen content image quality assessment based on convolutional neural networks

Screen content image (SCI) is a composite image including textual and pictorial regions resulting in many difficulties in image quality assessment (IQA). Large SCIs are divided into image patches to increase training samples for CNN training of IQA model, and this brings two problems: (1) local quality of each image patch is not equal to subjective differential mean opinion score (DMOS) of an entire image; (2) importance of different image patches is not same for quality assessment. In this paper, we propose a novel no-reference (NR) IQA model based on the convolutional neural network (CNN) for assessing the perceptual quality of SCIs. Our model conducts two designs solving problems which benefits from two strategies. For the first strategy, to imitate full-reference (FR) CNN-based model behavior, a CNN-based model is designed for both FR and NR IQA, and performance of NR-IQA part improves when the image patch scores predicted by FR-IQA part are adopted as the ground-truth to train NR-IQA part. For the second strategy, image patch qualities of one entire SCI are fused to obtain the SCI quality with an adaptive weighting method taking account the effect of the different image patch contents. Experimental results verify that our model outperforms all test NR IQA methods and most FR IQA methods on the screen content image quality assessment database (SIQAD). On the cross-database evaluation, the proposed method outperforms the existing NR IQA method in terms of at least 2.4 percent in PLCC and 2.8 percent in SRCC, which shows high generalization ability and high effectiveness of our model.

Blind quality assessment for screen content images based on edge and structure

Blind quality assessment for screen content images based on edge and structure

Modeling the Screen Content Image Quality via Multiscale Edge Attention Similarity

Screen content image (SCI) prevails because of the explosive growth of screen oriented applications. This leads to extensive studies on SCI quality assessment and modeling for application optimization. In this paper, we propose a full reference multiscale edge attention (MSEA) similarity index to efficiently measure the perceptual quality of a screen image. This model considers the perceptual impacts of fixation attention, edge structure and edge contrast jointly, to accurately capture the masking phenomena (e.g., frequency selectivity, luminance, contrast, etc.) of our human visual system (HVS) when viewing a screen image. Specifically, we decompose the images using Gaussian and Laplacian pyramids which are then used to derive the edge structure, and edge contrast feature maps. Together with the fixation attention map generated by weighted luminance difference between the reference and distorted SCIs, we could eventually offer a MSEA similarity map for final index score. We have evaluated this model using a publicly accessible screen image database. Simulation results have shown that the MSEA similarity index correlates with the collected subjective mean opinion score (MOS) very well. In fact, it is ranked at the first place for both Pearson linear correlation coefficient (PLCC) and Root mean squared error (RMSE), and ranked at the second place for Spearman rank-order correlation coefficient (SROCC) measurements, among existing quality metrics.

No-reference screen content image quality assessment based on multi-region features

Unlike natural image captured with cameras, screen content image (SCI) is a composite image including textual and pictorial regions. The different characteristics lead to many difficulties in image quality assessment (IQA). Most existing models based on the convolutional neural network (CNN) divide large SCIs into image patches to increase training samples for CNN training. This brings two problems: (1) a single image patch can not represent the quality of the entire image, especially in IQA of SCI; (2) SCI patches of an entire image degraded by the same distortion type and strength may have drastically different quality. In addition, these models adopt the mean square error (MSE) between the predicted quality and the subjective differential mean opinion score (DMOS) to train the CNN, without considering quality ranking between different SCIs. In this paper, we propose a novel no-reference (NR) IQA model based on the convolutional neural network (CNN). The contributions of our algorithm can be concluded as follows: (1) considering a large difference exists in different regions in a SCI, the pseudo global features generated with multi-region local features are utilized for quality evaluation, which better reflect image quality than local features of each image patch; (2) the noise classification task is used as an auxiliary task which aids the quality score prediction task to improve the representation ability; (3) the Siamese networks are used to predict the quality scores of two different SCIs, and a new ranking loss is proposed to rank the predicted scores, aiming to enhance the ability of the model to rank image in terms of quality. Experimental results verify that our model outperforms all test NR IQA methods and full-reference (FR) IQA methods on the screen content image quality assessment database (SIQAD).

Perceptual Quality Assessment for Screen Content Images by Spatial Continuity

In this paper, we propose an effective blind quality assessment method for screen content images (SCIs), called perceptual quality measure by spatial continuity (PQSC). With the center-surround mechanism in the human visual system (HVS), the proposed method extracts the statistical features on chromatic and textural variations in SCIs to measure the visual distortion. First, by considering the chromatic continuity between spatially adjacent pixels, photo-metric invariant chromatic descriptors are extracted as zero-order and first-order features. Second, motivated by the perceptual mechanism that the HVS is sensitive to image texture variation, we employ local ternary pattern operator to effectively depict the spatial continuity of texture. With these extracted chromatic and textural features, we further adopt histogram to compute the statistical chromatic and textural features. Support vector regression (SVR) is used to train the quality prediction model from visual features to human ratings. Experimental results on three public benchmark databases demonstrate that the performance of our method is superior to the current blind image quality assessment methods, even better than some full reference image quality assessment counterparts.

No-Reference Quality Assessment for Screen Content Images Based on Hybrid Region Features Fusion

Research on screen content images (SCIs) attracts more attention as they are highly applied to image- and video-centric applications on mobile and other devices. It is important to develop an efficient image-quality assessment (IQA) method for SCIs because IQA can guide and optimize various image-processing methods for SCIs and improve user experience. In this paper, we propose a no-reference objective assessment model for SCIs including SCIs segmentation and the analysis of local and global perceptual feature representations. Since the human visual system is highly sensitive to sharp edges that are commonly encountered in SCIs, we utilize the variance of local standard deviation, which is a noise robust index to distinguish the sharp edge patches (SEPes) and non-SEPes of SCIs. For SEPes, we perform two kinds of feature extractions. First, the entropy and contrast features are extracted with a gray-level co-occurrence matrix, which are highly perceptive of microstructural change. Second, the local phase coherence is utilized to capture the loss in sharpness. Then, average pooling is adopted to fuse features obtained from all of the SEPes to represent the local features. We further combine local features with global features that are derived using the BRISQUE method as the hybrid region (HR)-based features. Finally, a regression module is learned using support vector regression to train the mapping function that maps HR-based features to subjective quality scores. Experimental results on the screen image-quality assessment database show that the proposed method can achieve better performance in visual-quality prediction for SCIs than the performance achieved by state-of-the-art methods.

FQI: feature‐based reduced‐reference image quality assessment method for screen content images

In this study, a reduced-reference image-quality-assessment (IQA) method for screen content images, named as feature-quality-index (FQI) is proposed. The proposed method is based on the fact that the human visual system is more sensitive towards change in features than intensity or structure. Reduced features from the reference and distorted images are first extracted. In order to find the preserved features in the distorted image, a feature matching process with a reduced number of distance calculations is proposed, namely reduced-distance method. To reflect the importance of the matched features and their distance, the inner product between the normalised scale and distance vector is obtained. Extensive comparisons are performed on two available benchmark databases namely SIQAD and QACS, with eight reduced-reference, and nine full-reference state-of-the-art IQA techniques to demonstrate the consistency, accuracy, and robustness of the proposed FQI. The subjective evaluation of mean opinion score shows that FQI outperforms the current state-of-the-art IQA techniques.

Learning content-specific codebooks for blind quality assessment of screen content images

This paper presents a novel blind quality assessment method for screen content images (SCIs) by learning content-specific codebooks. Instead of manually extracting quality-aware features for quality evaluation, the proposed method automatically generates effective features based on a simple feature encoding technique over content-specific codebooks. Considering the mixed content type in SCIs, content-specific codebooks including textual codebook and pictorial codebook are first learned in an offline manner. Given an input SCI, a textual/pictorial segmentation method is first applied to divide the SCI into textual and pictorial regions. Then, patches in different regions are respectively encoded using the learned textual and pictorial codebooks to produce the corresponding feature codes. Finally, the feature codes of each patch are aggregated, by using a percentage-based local pooling scheme, to yield the global feature codes of different regions. The final quality-predictive features used for quality regression are the combined global feature codes of different regions. Despite its simplicity, our method delivers low computational complexity, making it well suitable for real-time applications. Extensive experiments are conducted on three public SCI databases to validate the performance of our method, the results well confirm its superiority over the existing relevant full reference and no reference SCI quality assessment methods.

Screen Content Image Quality Assessment With Edge Features in Gradient Domain

Objective visual quality assessment specific for screen content images (SCIs) has been increasingly investigated over the years. In this paper, an effective full-reference quality evaluation model for SCIs is proposed, in which edge features in gradient domain (EFGD) are extracted for better visual perceptual representation. Unlike traditional edge feature extraction directly in the image pixel domain, all edge features in the proposed EFGD model are extracted based on the gradient map of input SCIs, including edge sharpness, edge brightness/contrast, and edge chrominance. Specifically, the gradient profile model that can well represent the spatial layout of edges is adopted to measure the edge sharpness degree. A novel computation way is reported to measure the edge brightness and contrast change between the reference and distorted SCIs, while color moments are used to account for the color chrominance variation. In addition, an adaptive weighting strategy is designed to adjust the effects of these three kinds of edge features, according to the statistical distributions of the input SCIs. Moreover, the maximum value of edge sharpness features is extracted from the test SCIs as the pooling weight to get the final image quality assessment (IQA) score. The experimental results on two commonly used SCIs databases have verified the superiorities of the EFGD model and show that the EFGD model is in more conformity with the subjective assessment results than most of the existing IQA models.

Blind quality assessment for screen content images via convolutional neural network

With the wide propagation of cloud and mobile computing, screen content images (SCIs) have become more indispensable in our daily lives. Compared to natural scene images (NSIs), SCIs possess many particular characteristics, like mixed contents, extremely sharp edges, and text graphics. Consequently, more challenges occur in the feature extraction, which is used to reflect the distortion, during the quality assessment of SCIs. Recently, some convolutional neural network (CNN) models have been designed by automatically learning feature to evaluate the quality. In this paper, we develop a novel blind quality assessment method for SCIs via the CNN. First, compared with existing CNN-based methods, the proposed method avoids the disadvantage of training with image patches, and it is the pioneering attempt that takes the entire image as inputs. Second, instead of the image gray value, the original image is decomposed into two portions, i.e., the predicted and unpredicted portions, according to the internal generative mechanism (IGM) theory as the input of CNN. Through the CNN, all features of the image are learned automatically from beginning to end, and the network finally outputs the predicted score. Since existing SCI database is too small, to fully train the network, we collected 30000 SCIs and employed a high-accuracy full-reference quality assessment metric of SCI to compute scores as the training labels. Experimental results on SIQAD database demonstrate that the proposed method is comparable to reference-based SCI quality assessment metrics and is superior to the state-of-the-art NSI quality assessment metrics.

Blind quality assessment for screen content images by combining local and global features

Recently, several no-reference image quality assessment (NR-IQA) metrics have been developed for the quality evaluation of screen content images (SCIs). While, most of them are opinion-aware methods, which are limited by the subjective opinion scores of training data. Hence, in this paper, we propose a novel opinion-unaware method to predict the quality of SCIs without any prior information. Firstly, an union feature is proposed by considering the local and global visual characteristics of human visual system simultaneously. Specifically, a local structural feature is extracted from the rough and smooth regions of SCIs by leveraging a sparse representation model. As a supplement, a global feature is obtained by combining the luminance statistical feature and local binary pattern (LBP) feature of entire SCIs. Secondly, to get rid of the limitation of subjective opinion scores, a new large-scale training dataset contained 80,000 distorted SCIs is constructed, and the quality labels of those distorted SCIs are derived by an advanced full-reference IQA metric. Thirdly, a regression model between image features and image quality labels is learned from the training dataset by employing a learning-based framework. And then, the quality scores of test SCIs can be predicted by the pre-trained regression model. The experimental results on two largest SCI-oriented databases show that the proposed method is superior to the state-of-the-art NR-IQA metrics.

Hybrid JND model-guided watermarking method for screen content images

With the prevalence of digital products like cellphone, tablet and personal computer, the screen content images (SCIs) consisting of text, graphic, and natural scene picture becomes a significant media in various communication scenarios. In this paper, we propose a novel Spread Transform Dither Modulation (STDM) watermarking scheme based on Hybrid just noticeable distortion model for screen content images. Firstly, the original image was transformed from RGB into YCrCb to ensure stability robustness and invisibility. Then, we proposed a novel automatic classification method based on AC coefficients feature. Different from pictorial block in screen content images, structural-based contrast masking effects was incorporated to adjust the just noticeable distortion value for textual blocks. Finally, the reference image from the SIQAD image database was used to evaluate the performance of our proposed scheme. Experiments showed that our method has a good performance in term of robustness with better visual quality.

Blind screen content image quality measurement based on sparse feature learning

Recently, the perceived quality measurement of screen content images (SCIs) has become an active research topic. In this paper, a blind image quality measurement (IQM) metric for SCIs based on the learning of sparse features via dictionary learning is proposed. First, to extract the sparse features, histogram representations from multi-scale local gradient patterns are integrated to form a dictionary. Subsequently, using a pursuit algorithm, the sparse features of the distorted SCIs are efficiently coded by this dictionary. Finally, to obtain the final quality of the distorted SCIs, a machine learning algorithm is utilised to combine the sparse features into a final quality score. The results of extensive simulations conducted show that the proposed blind IQM metric consistently obtains competitive performance and is in line with human beings perceive.

Naturalization Module in Neural Networks for Screen Content Image Quality Assessment

Deep learning approaches have demonstrated success in no-reference image quality assessment tasks. However, due to the specific properties of screen content images (SCIs), deep neural networks for SCI quality assessment are not as optimal as those designed for images depicting natural scenes. In order to tackle this discrepancy, a “naturalization” module composed of an upsampling layer and a convolutional layer is proposed to transform SCIs to have characteristics more similar to that of natural images. In addition, a new deep learning model architecture along with data augmentation techniques tailored to SCIs are implemented. The performance of the proposed approach is evaluated on the Screen Image Quality Assessment Database and Screen Content Image Database, and has shown to have superior performance to state-of-the-art methods in predicting the perceptual quality of SCIs.

Effective Content-Aware Chroma Reconstruction Method for Screen Content Images.

In this paper, we propose an effective novel content-aware chroma reconstruction (CACR) method for screen content images (SCIs). After receiving the decoded downsampled YUV image on the client side, our fast chroma-copy approach reconstructs the missing chroma pixels in the flat regions of SCI. Then, for non-flat regions, a non-flat region-based winner-first voting (NRWV) strategy is proposed to identify the chroma subsampling scheme used on the server side prior to compression. Further, an effective adaptive hybrid approach is proposed to reconstruct each missing chroma pixel in the non-flat region by fusing the two reconstructed results, one from our modified NRWV-based chroma subsampling-binding and luma-guided chroma reconstruction scheme, which favors the sharp edges in SCI, as well as the other from the bicubic interpolation scheme, which favors blurred and continuous-tone textures. Further, based on the identified chroma subsampling scheme, a geometry alignment-based error compensation approach is proposed to enhance the reconstructed chroma image. Based on typical test SCIs and JCT-VC screen content videos, comprehensive experiments are carried out in HEVC-16.17 to demonstrate that in terms of quality, visual effect, and quality-bitrate tradeoff of the reconstructed SCIs, our CACR method significantly outperforms the existing state-of-the-art methods.

A Gabor Feature-Based Quality Assessment Model for the Screen Content Images.

In this paper, an accurate and efficient full-reference image quality assessment (IQA) model using the extracted Gabor features, called Gabor feature-based model (GFM), is proposed for conducting objective evaluation of screen content images (SCIs). It is well-known that the Gabor filters are highly consistent with the response of the human visual system (HVS), and the HVS is highly sensitive to the edge information. Based on these facts, the imaginary part of the Gabor filter that has odd symmetry and yields edge detection is exploited to the luminance of the reference and distorted SCI for extracting their Gabor features, respectively. The local similarities of the extracted Gabor features and two chrominance components, recorded in the LMN color space, are then measured independently. Finally, the Gabor-feature pooling strategy is employed to combine these measurements and generate the final evaluation score. Experimental simulation results obtained from two large SCI databases have shown that the proposed GFM model not only yields a higher consistency with the human perception on the assessment of SCIs but also requires a lower computational complexity, compared with that of classical and state-of-the-art IQA models. The source code for the proposed GFM will be available at http://smartviplab.org/pubilcations/GFM.html.

Toward a Blind Quality Predictor for Screen Content Images

Blind quality assessment of screen content images (SCIs) is much challenging than traditional natural images. In this paper, we propose a blind quality predictor for SCIs to explore the issue from the perspective of sparse representation. Specifically, we conduct local sparse representation for the textual and pictorial regions, respectively, and conduct global sparse representation for the global SCIs. Subsequently, the underlying relationship between the feature and quality vectors is bridged in a universal sparse representation framework. The quality pooling is comparatively simply that only need to estimate the local and global quality scores and combine them to a total one. Our experimental results show that the proposed predictor can achieve better prediction performance to be in line with subjective assessment.

Screen Content Image Quality Assessment Using Multi-Scale Difference of Gaussian

In this paper, a novel image quality assessment (IQA) model for the screen content image s (SCIs) is proposed by using multi-scale difference of Gaussian (MDOG). Motivated by the observation that the human visual system (HVS) is sensitive to the edges while the image details can be better explored in different scales, the proposed model exploits MDOG to effectively characterize the edge information of the reference and distorted SCIs at two different scales, respectively. Then, the degree of edge similarity is measured in terms of the smaller-scale edge map. Finally, the edge strength computed based on the larger-scale edge map is used as the weighting factor to generate the final SCI quality score. Experimental results have shown that the proposed IQA model for the SCIs produces high consistency with human perception of the SCI quality and outperforms the state-of-the-art quality models.

No reference quality evaluation for screen content images considering texture feature based on sparse representation

Abstract In this paper, an accurate blind metric evaluating screen content images (SCIs) considering texture feature via sparse representation is proposed. The existing theory on human vision tells us that human visual system (HVS) is sensitive to texture information of images. In addition, gradient direction has not been adequately explored as a predictive information source of SCIs. This method is first based on three kinds of gradient maps of SCIs—i.e., gradient magnitude map, relative gradient direction map, relative gradient magnitude map. Since the existing literatures about human neuroscience have revealed that image texture can be captured by high-order derivatives, texture information of SCIs can be represented by local histogram of oriented gradient (HOG) features extracted from the aforementioned gradient maps. Finally, it is composed of two important procedures: representing HOG features by means of sparse coding, then weighting subjective experimental quality scores via the sparse coding coefficients to predict objective perceptive quality scores of SCIs. Comparison experiments on the public SCI database demonstrate that the designed metric has the better superiority over existing the-state-of-art algorithms and delivers consistency and accuracy in relation with subjective evaluation.