Screen Content Images Research Articles

Screen content image quality measurement based on multiple features

Screen content image quality measurement based on multiple features

Deep Feature Statistics Mapping for Generalized Screen Content Image Quality Assessment.

The statistical regularities of natural images, referred to as natural scene statistics, play an important role in no-reference image quality assessment. However, it has been widely acknowledged that screen content images (SCIs), which are typically computer generated, do not hold such statistics. Here we make the first attempt to learn the statistics of SCIs, based upon which the quality of SCIs can be effectively determined. The underlying mechanism of the proposed approach is based upon the mild assumption that the SCIs, which are not physically acquired, still obey certain statistics that could be understood in a learning fashion. We empirically show that the statistics deviation could be effectively leveraged in quality assessment, and the proposed method is superior when evaluated in different settings. Extensive experimental results demonstrate the Deep Feature Statistics based SCI Quality Assessment (DFSS-IQA) model delivers promising performance compared with existing NR-IQA models and shows a high generalization capability in the cross-dataset settings. The implementation of our method is publicly available at https://github.com/Baoliang93/DFSS-IQA.

Compressed Screen Content Image Super Resolution

Screen content has become one of the prominent mediums in the increasingly connected world. With the prevalence of remote collaboration and communication such as virtual conferences and online education, recent years have witnessed a dramatic increase in the data volume of the screen content. Screen content compression serves as the fundamental technology in fostering the storage, transmission, and exhibition of screen content. In this article, we target the super-resolution of the compressed screen content images, intending to tackle the real-world challenge problems. A dataset is proposed for the super-resolution of the screen contents contaminated with different compression distortion levels. Subsequently, we introduce the principle of the multi-hypothesis into the super resolution and propose a new paradigm for the restoration of the compressed screen content images. The luminance and sharpness similarity metric is adopted in the network learning to better adapt to the screen content characteristic and ensure perceptual fidelity. Experimental results verify the superiority and effectiveness of the proposed method.

Screen content image quality assessment using curvelet transform

Screen content image quality assessment using curvelet transform

Compression for Bayer CFA Images: Review and Performance Comparison.

Bayer color filter array (CFA) images are captured by a single-chip image sensor covered with a Bayer CFA pattern which has been widely used in modern digital cameras. In the past two decades, many compression methods have been proposed to compress Bayer CFA images. These compression methods can be roughly divided into the compression-first-based (CF-based) scheme and the demosaicing-first-based (DF-based) scheme. However, in the literature, no review article for the two compression schemes and their compression performance is reported. In this article, the related CF-based and DF-based compression works are reviewed first. Then, the testing Bayer CFA images created from the Kodak, IMAX, screen content images, videos, and classical image datasets are compressed on the Joint Photographic Experts Group-2000 (JPEG-2000) and the newly released Versatile Video Coding (VVC) platform VTM-16.2. In terms of the commonly used objective quality, perceptual quality metrics, the perceptual effect, and the quality-bitrate tradeoff metric, the compression performance comparison of the CF-based compression methods, in particular the reversible color transform-based compression methods and the DF-based compression methods, is reported and discussed.

A No-Reference Quality Assessment Method for Screen Content Images Based on Human Visual Perception Characteristics

The widespread application of screen content images (SCIs) has met the needs of remote display and online working. It is a topic that is challenging and worthwhile discussing in research on quality assessment for SCIs. However, existing methods focus on extracting artificial features to predict image quality, which are subjective and incomplete, or lack good interpretability. To overcome these problems, we propose an effective quality assessment method for SCIs based on human visual perceptual characteristics. The proposed method simulates the multi-channel working mechanism of the human visual system (HVS) through pyramid decomposition and the information extraction process of brains with the help of dictionary learning and sparse coding. The input SCIs are first decomposed at multiple scales, and then dictionary learning and sparse coding are applied to the images at each scale. Furthermore, the sparse representation results are analyzed from multiple perspectives. First, a pooling scheme about generalized Gaussian distribution and log-normal distribution is designed to describe the sparse coefficients with and without zero values, respectively. Then the sparse coefficients are used to characterize the energy characteristics. Additionally, the probability of each atom is calculated to describe the statistical property of SCIs. Since the above process only deals with brightness, color-related features are also added to make the model more general and robust. Experimental results on three public SCI databases show that the proposed method can achieve better performance than existing methods.

Dual-Anchor Metric Learning for Blind Image Quality Assessment of Screen Content Images

The natural scene statistic is destroyed by the artificial portion in the screen content images (SCIs) and is also impractical for obtaining an accurate statistical model due to the variable composition of the artificial and natural parts in SCIs. To resolve this problem, this paper presents a dual-anchor metric learning (DAML) method that is inspired by metric learning to obtain discriminative statistical features and further identify complex distortions, as well as predict SCI image quality. First, two Gaussian mixed models with prior data are constructed as the target anchors of the statistical model from natural and artificial image databases, which can effectively enhance the metrical discrimination of the mapping relation between the feature representation and quality degradation by conditional probability analysis. Then, the distances of the high-order statistics are softly aggregated to conduct metric learning between the local features and clusters of each target statistical model. Through empirical analysis and experimental verification, only variance differences are used as quality-aware features to benefit the balance of complexity and effectiveness. Finally, the mapping model between the target distances and subjective quality can be obtained by support vector regression. To validate the performance of DAML, multiple experiments are carried out on three public databases: SIQAD, SCD, and SCID. Meanwhile, PLCC, SRCC, and the RMSE are then employed to compute the correlation between subjective and objective ratings, which can estimate the prediction of accuracy, monotonicity, and consistency, respectively. The PLCC and RMSE of the method achieved 0.9136 and 0.7993. The results confirm the good performance of the proposed method.

Dual-Channel Multi-Task CNN for No-Reference Screen Content Image Quality Assessment

Nowadays the problem of image quality assessment (IQA) for screen content images (SCIs) has become a research hotspot as they are ubiquitous in multimedia applications. Although the quality assessment of natural images (NIs) has been continuously developed in the past few decades, few NI-oriented IQA methods can be directly applied on SCIs due to different visual characteristics between them. In this paper, we present a no-reference quality prediction approach considering the content information of SCIs, which is based on dual-channel multi-task convolutional neural network. First, we segment a SCI into small patches and classify them as the textual patches and the pictorial patches. Then, we devise a novel dual-channel convolutional neural network (CNN) to predict the quality of textual patches and pictorial patches. Finally, we propose an effective adaptive weighting strategy for quality score aggregation. The proposed CNN is built on an end-to-end multi-task learning framework, which assists the SCI quality prediction task through the histogram of oriented gradient (HOG) feature prediction task to learn a better mapping between the input patch and its quality score. The adaptive weighting strategy further improves the representation ability of each SCI patch. Experimental results on two largest SCI-oriented databases demonstrate that the proposed method outperforms most of the state-of-the-art no-reference IQA methods and the full-reference IQA methods.

A comprehensive survey on robust image watermarking

With the rapid development and popularity of the Internet, multimedia security has become a general essential concern. Especially, as manipulation of digital images gets much easier, the challenges it brings to authentication certification are increasing. As part of the solution, digital watermarking has made significant contributions to image content security and has attracted increasing attention. In this paper, we present a comprehensive review on digital image watermarking methods that were published in recent years illustrating the conventional schemes in different domains. We provide an overview of geometric invariant techniques and emerging watermarking methods for novel medias, such as depth image based rendering (DIBR), high dynamic range (HDR), screen content images (SCIs), and point cloud model. Particularly, as deep learning has achieved a great success in the field of image processing, and has also successfully been used in the field of digital watermarking, learning-based watermarking methods using various neural networks are summarized according to the utilization of neural networks in the single stage training (SST) and double stage training (DST). Finally, we provide an analysis and summary on those methods, and suggest some future research directions.

Quality Assessment of Screen Content Images in Wavelet Domain

Objective image quality assessment (IQA) aims to predict human judgment on an image’s visual quality. This computational model is applicable in various scenarios, such as monitoring, image compression and image restoration. However, the research in this area began with natural scenes while image contents has become much more diverse. Remote computing and screen sharing applications led to the prevalence of screen content images (SCIs), which are full of texts, flat areas, and also natural scenes. Due to their different characteristics, traditional methods for IQA are insufficient for SCIs. In this paper, we show that capturing the distortions in the horizontal and vertical structures of an SCI is effective for predicting its quality. We also take into account the human visual system’s trend to analyze a scene in a multi-scale fashion. Based on these observations, we propose wavelet analysis as a means to accomplish the mentioned strategies. Experimental results on three SCI quality datasets show that plausible predictions can be made by the proposed full-reference method, while having an acceptable time complexity.

Deep CNN Model for Non-Screen Content and Screen Content Image Quality Assessment

In the current world, user experience in various platforms matters a lot for different organizations. But providing a better experience can be challenging if the multimedia content on online platforms is having different kinds of distortions which impact the overall experience of the user. There can be various reasons behind distortions such as compression or minimal lighting condition while taking photos. In this work, a deep CNN-based Non-Screen Content and Screen Content NR-IQA framework is proposed which solves this issue in a more effective way. The framework is known as DNSSCIQ. Two different architectures are proposed based upon the input image type whether the input is a screen content or non-screen content image. This work attempts to solve this by evaluating the quality of such images.

Asymmetric two-stream network for screen content image quality assessment based on region features

With the latest popularization of remote desktop software and online work, screen content images (SCIs) are increasingly being contacted by people. Many image quality assessment (IQA) methods based on deep learning have been proposed for SCIs. However, most IQA methods for SCIs pay less attention to regional characteristics. An asymmetric two-stream network based on region features is proposed for SCI objective quality assessment, in which features are derived from a channel separation feature extraction (CSFE) sub-network and a spatial analysis feature extraction (SAFE) sub-network. In the CSFE sub-network, three channel separation (CS) modules are utilized to strengthen the ability for capturing deepthwise features in the pictorial regions. In the SAFE sub-network, two spatial analysis (SA) module are applied to focus on the structural information in the spatial domain for the textual regions. Two asymmetric sub-networks can efficiently extract the corresponding region features by separately considering the pictorial and textual regions of SCIs. In addition, an adaptive global weighting strategy is introduced to estimate the quality for SCIs. The experimental results on the screen image quality assessment database show that the proposed model is superior to the other state-of-the-art SCI quality assessment methods.

Blind Image Quality Assessment of Screen Content Images via Fisher Vector Coding

This paper presents an effective blind image quality assessment (BIQA) method for screen content images (SCIs) based on Fisher vector encoding, with the hypothesis that local statistics will be altered with the change of distortions, and can be characterized by the fusion of statistical models and direction vectors. Firstly, a specific Gaussian mixture model (GMM) is generated from a corpus of pristine SCIs to simulate the local distribution of SCIs in spatial domain. Then, discriminative features are generated to characterize the quality of test image with Fisher vector coding and generated GMM. Finally, support vector regression is adopted to learn the mapping between discriminative features and subjective opinion scores. To validate the performance of our method, extensive experiments are conducted on three public SCI databases and the results well confirm its superiority over the existing relevant BIQA method of SCIs.

A Cubic Convolution Interpolation-Based Chroma Subsampling Method for Bayer and RGBW CFA Raw Images

The Bayer and RGBW color filter array (CFA) raw images, denoted by <inline-formula> <tex-math notation="LaTeX">$I^{Bayer}$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$I^{RGB}$ </tex-math></inline-formula>, respectively, have been widely used in consumer markets. In the demosaicking-first compression scheme, chroma subsampling is necessary prior to compressing <inline-formula> <tex-math notation="LaTeX">$I^{RGB}$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$I^{RGBW}$ </tex-math></inline-formula>. Several linear interpolation-based chroma subsampling methods have been developed for <inline-formula> <tex-math notation="LaTeX">$I^{Bayer}$ </tex-math></inline-formula>, but no nonlinear interpolation-based chroma subsampling methods have targeted the above two CFA image types simultaneously. In this paper, we first propose a nonlinear interpolation-based, namely the cubic convolution interpolation-based (CCI-based), <inline-formula> <tex-math notation="LaTeX">$2\times 2$ </tex-math></inline-formula> block-distortion function for each <inline-formula> <tex-math notation="LaTeX">$2\times 2$ </tex-math></inline-formula> CFA block <inline-formula> <tex-math notation="LaTeX">$B^{CbCr}$ </tex-math></inline-formula>. Next, using the Cauchy-Schwarz inequality, we prove that the proposed block-distortion function is a convex function in the real domain, which further serves as the base of the initial subsampled chroma solution. Then, a CCI-based iterative method is proposed to improve the initial subsampled chroma solution. The results of comprehensive experimental tests using Bayer and RGBW CFA images created from the three RGB full-color datasets, namely IMAX, SCI (screen content images), and CI (classical images), demonstrate that on the Versatile Video Coding (VVC) platform VTM-11.0, the proposed method achieves substantial quality enhancement and quality-bitrate tradeoff merits of the reconstructed Bayer and RGBW CFA images compared with existing chroma subsampling methods.

Dual-Anchor Metric Learning for Blind Image Quality Assessment of Screen Content Images

Dual-Anchor Metric Learning for Blind Image Quality Assessment of Screen Content Images

Screen content image quality assessment using distortion-based directional edge and gradient similarity maps

With the prevalent use of applications like Facebook, Twitter, and remote control applications, assessing the quality of the screen content images (SCIs) has become one of the critical fields of researches in image processing. In this paper, we develop a novel full-reference image quality assessment (IQA) method, called distortion-based directional edge and gradient similarity maps (DDEGSM) method, to evaluate the quality of SCIs by efficiently incorporating the effect of two challenging distortion types: contrast change (CC) and color saturation change (CSC). The proposed DDEGSM method has four main steps. First, we form an edge similarity map by extracting two edge and gradient features using the image’s gradient in 12 directions and the Laplacian of Gaussian filter. Next, we use the difference of Gaussian filter to weigh the strength of each pixel in the edge similarity map. Then, we examine the distorted image using two efficient approaches to find out whether the image contains the CC or CSC distortion. Finally, we calculate the quality score of the input SCI from the distortion-based feature maps using a pooling strategy. Our extensive experiments on three commonly-used SCI datasets indicate the proposed method is superior to the state-of-the-art full-reference IQA methods and is more consistent with the subjective assessments.

DeepRPN-BIQA: Deep architectures with region proposal network for natural-scene and screen-content blind image quality assessment

With the emerging use of technology and screen-oriented applications in our daily life, screen content images have gained the same importance as natural scene images. This results in many natural-scene and screen-content blind image quality assessment (BIQA) models to evaluate the perceptual quality without any prior information regarding the reference image. Recently, patch-based techniques for image quality assessment (IQA) have shown promising results. As per our knowledge, no IQA technique in literature is available that can be equally effective for both natural-scene and screen-content images. In this work, we have proposed a deep architecture with a region proposal network (RPN) for blind natural-scene and screen-content image quality assessment, named DeepRPN-BIQA. The proposed architecture computes visual saliency using RPN to extract important regions having a high contribution towards the image quality. Important regions are extracted by utilizing the texture and edges of images by sliding the network over the extracted feature map from deep architectures i.e., VGGNet and ResNet. The regions proposed (RP) that overlap more than 60% are merged into one proposal and are called the region of interest (ROI). The overlap between RPs is computed using anchors having 3 different scales and aspect ratios. A local quality score is computed over each ROI and the total quality score is computed by taking the average of all the local quality scores. Experimental results show that the DeepRPN-BIQA shows a high correlation between mean observer score and predicted quality score and performs better than other models for screen content images, synthetically distorted images, images taken in real-life conditions using mobile phone cameras, and large scale image quality assessment database.

Full-Reference Quality Assessment for Screen Content Images Based on the Concept of Global-Guidance and Local-Adjustment

Benefiting from the development of multimedia communication terminals, the visual content presented to people on mobile devices is no longer a single form, but contains text, natural images, and other computer-generated graphics, which is called screen content images (SCIs). Inspired by the different visual stimuli that text and images bring to human eyes and the concept of global-guidance and local-adjustment, we design a novel full-reference image quality assessment (IQA) model using the structural features of the text, the perceptual features of pictures, and a score integration model (SPSIM) to evaluate SCIs quality. Firstly, we split the SCIs into textual and pictorial regions through a fully convolutional network (FCN) to conduct separate analyses. For textual regions, we take advantage of narrow edge extensions and high edge steps as structural features to compute the textual score. For pictorial regions, we extract the just noticeable difference (JND) features, which measure the human eye's ability to detect distortion as perceptual features to calculate the pictorial score. Finally, an innovative score integration method based on the global-guidance and local-adjustment is designed to better analyze the relationship between the above regional scores and the whole global SCIs score. Abundant experiments in SCIs databases have shown that the SPSIM model can achieve better consistency with the human eyes system (HVS) in predicting the visual quality of SCIs.

Unified quality assessment of natural and screen content images via adaptive weighting on double scales

In multimedia communication systems, digital images may contain various visual contents, among which the natural scene images (NSIs) and screen content images (SCIs) are two important and common types. The existing full-reference image quality assessment (IQA) metrics are designed for only one type of images, but cannot precisely perceive the visual quality of another type. It is still unclear what the different characteristics are between NSIs and SCIs resulting in this failure. Inspired by some psychological studies, we figure out that it is due to the different structural scale levels between NSIs and SCIs. Given this observation, this paper introduces the gradient degradation of Gaussians (GDoG) to analyze the images’ structural scale level, proposes a fast unified IQA index for both NSIs and SCIs by incorporating an adaptive weighting strategy on double scales. Experimental results conducted on several databases verify the effectiveness and efficiency of the proposed unified IQA index for both types of images, also demonstrate that the adaptive weighting strategy based on GDoG can improve the existing models for cross-content-type images.

Quality assessment of screen content images based on multi-stage dictionary learning

In this paper, we propose an effective method for quality assessment of screen content images (SCIs) based on multi-stage dictionary learning. To simulate the brain’s layered processing of signals, we proposed a hierarchical feature extraction strategy, which is called multi-stage dictionary learning, to simulate the hierarchical information processing of brain. First, the standard deviation of normalized map obtained from training image is used to select the training data in a certain proportion, which can ensure the learning efficiency and reduce the training burden. Next, the reconstructed map is weighted as the input of the next-stage dictionary learning. Then using the trained dictionary, the sparse representation is applied to extract features. Meanwhile, considering that some important features may be ignored in the process of multi-stage dictionary learning, we use Log Gabor filter to extract feature maps, and then calculate the correlation between feature maps as another kind of compensation features. Final, for the two feature sets, we choose SVR and feature codebook to learn two objective scores, and then use the adaptive weighting strategy to get the final objective quality score. Experimental results show that the proposed method is superior to several mainstream SCIs metrics on two publicly available databases.