Just Noticeable Distortion Model Research Articles

Surprise-based JND estimation for perceptual quantization in H.265/HEVC codecs

Just noticeable distortion (JND), reflecting the perceptual redundancy directly, has been widely used in image and video compression. However, the human visual system (HVS) is extremely complex and the visual signal processing has not been fully understood, which result in existing JND models are not accurate enough and the bitrate saving of JND-based perceptual compression schemes is limited. This paper presents a novel pixel-based JND model for videos and a JND-based perceptual quantization scheme for HEVC codecs. In particular, positive and negative perception effects of the inter-frame difference and the motion information are analyzed and measured with an information-theoretic approach. Then, a surprise-based JND model is developed for perceptual video coding (PVC). In our PVC scheme, the frame-level perceptual quantization parameter (QP) is derived on the premise that the coding distortion is infinitely close to the estimated JND threshold. On the basis of the frame-level perceptual QP, we determine the perceptual QP for each coding unit through a perceptual adjustment function to achieve better perceptual quality. Experimental results indicate that the proposed JND model outperforms existing models significantly, the proposed perceptual quantization scheme improves video compression efficiency with better perceptual quality and lower coding complexity.

Perceptual ultra-high definition video coding based on adaptive just noticeable distortion model

Ultra-high definition (UHD) video is gradually applied into many fields. The huge amount of data in UHD video limits the realization of these applications. Perception video coding can exploit the visual redundancy, and improve the compression performance. So far, the perpetual video coding based on just noticeable distortion (JND) model cannot fully exploit the perceptual redundancy of ultra-high definition (UHD) video/image in 10-bit depth. Thus, an adaptive JND model based on perceptual noise Bayesian network (PNBN) is firstly established for UHD video/image. Then, combined perceptual noise channel encoding/decoding with human visual characteristics, hidden side information of image is dynamically estimated by PNBN. Thirdly, the adaptive JND model is considered both luminance masking effect and texture complexity, which can adaptively exploit redundancy in different texture regions of image. Finally, an effective perceptual video coding (PVC) scheme is proposed for versatile video coding. A novel distortion compensation factor is appropriately designed for rate-distortion optimization. Experimental results show that the proposed PVC scheme achieves significant bitrate reduction with better subjective video quality than the state-of-the-art PVC schemes.

Blind Photograph Watermarking with Robust Defocus-Based JND Model

Just noticeable distortion (JND) is widely employed to describe the perception redundancy in the quantization-based watermarking framework. However, the existing JND models are generally constructed to treat every region of the photograph with an equal focus level, whereas the defocus effect has never been considered. In this paper, the defocus feature, which can portray the aesthetic emphasis in the photograph, is provided to improve the perceptual JND model. Firstly, two indicators which consider the block energy in the defocus measurement (DM) are proposed. Then, the defocus feature map (DFM) is obtained by integrating the influence of the circumambient blocks, and it is applied to the proposed JND contrast masking (CM) processing. In this way, a new blind photograph watermarking method, with emphasis on defocus-JND estimation combined with the proposed CM, is presented. Simulations show that the proposed JND is more suitable for watermarking framework than some exiting JND models, and the proposed watermarking scheme with the improved defocus-based JND model has superior robustness compared with some watermarking schemes.

Hierarchical Predictive Coding-Based JND Estimation for Image Compression.

The human visual system (HVS) is a hierarchical system, in which visual signals are processed hierarchically. In this paper, the HVS is modeled as a three-level communication system and visual perception is divided into three stages according to the hierarchical predictive coding theory. Then, a novel just noticeable distortion (JND) estimation scheme is proposed. In visual perception, the input signals are predicted constantly and spontaneously in each hierarchy, and neural response is evoked by the central residue and inhibited by surrounding residues. These two types' residues are regarded as the positive and negative visual incentives which cause positive and negative perception effects, respectively. In neuroscience, the effect of incentive on observer is measured by the surprise of this incentive. Thus, we propose a surprise-based measurement method to measure both perception effects. Specifically, considering the biased competition of visual attention, we define the product of the residue self-information (i.e., surprise) and the competition biases as the perceptual surprise to measure the positive perception effect. As for the negative perception effect, it is measured by the average surprise (i.e., the local Shannon entropy). The JND threshold of each stage is estimated individually by considering both perception effects. The total JND threshold is finally obtained by non-linear superposition of three stage thresholds. Furthermore, the proposed JND estimation scheme is incorporated into the codec of Versatile Video Coding for image compression. Experimental results show that the proposed JND model outperforms the relevant existing ones, and over 16% of bit rate can be reduced without jeopardizing the perceptual quality.

Just Noticeable Distortion Profile Inference: A Patch-Level Structural Visibility Learning Approach.

In this paper, we propose an effective approach to infer the just noticeable distortion (JND) profile based on patch-level structural visibility learning. Instead of pixel-level JND profile estimation, the image patch, which is regarded as the basic processing unit to better correlate with the human perception, can be further decomposed into three conceptually independent components for visibility estimation. In particular, to incorporate the structural degradation into the patch-level JND model, a deep learning-based structural degradation estimation model is trained to approximate the masking of structural visibility. In order to facilitate the learning process, a JND dataset is further established, including 202 pristine images and 7878 distorted images generated by advanced compression algorithms based on the upcoming Versatile Video Coding (VVC) standard. Extensive experimental results further show the superiority of the proposed approach over the state-of-the-art. Our dataset is available at: https://github.com/ShenXuelin-CityU/PWJNDInfer.

An improved DCT-based JND estimation model considering multiple masking effects

The just noticeable distortion (JND) in the contour and orderly regions is easy to be overestimated and that in the disorderly areas is usually underestimated. In order to estimate the JND threshold more accurately, this paper proposes an improved DCT-based JND estimation model considering multiple masking effects properly. The contributions of this paper are characterized by twofold. On the one hand, a mean absolute difference based (MAD-based) block classification method is developed at first to classify image blocks into plain, contour and texture types accurately and quickly. And the JND model for contrast masking effect (CM-JND) is constructed as a modulation factor based on the MAD of each block. On the other hand, we propose a distance-based disorder evaluation metric to measure the disorder intensity in block level. Then, the JND model for the disorderly concealment effect (DC-JND) is proposed based on our psychological experiment. Finally, the total JND estimation threshold is modeled by fusing the spatial contrast sensitivity function, the luminance adaptation effect, the CM and DC effects. Experimental results show that the proposed DCT-based JND estimation model outperforms existing models in performance and complexity. Specifically, the proposed model shows more tolerance for distortions, lower computational complexity with better perceptual quality than other JND models.

A novel attention-guided JND Model for improving robust image watermarking

Just noticeable distortion (JND) and visual attention (VA), which are two widely used mathematical models of human visual system (HVS) that aim to simulate the human brain mechanism, are sufficiently explored and applied to many researches including digital watermarking. The activity of human brain, however, is extremely complex and it can be more limited due to complicated fusion of spatial saliency for image domain. In this paper, we propose a novel VA guided JND model in which we fuse the final attention map from the low-level features by using two laws of Gestalt principle. Firstly, we demonstrate a classic JND model in DCT domain, which consists of spatial contrast sensitivity function (CSF), luminance adaptation (LA) and contrast masking (CM). The foveation effect and orientation feature are considered to obtain the CSF and CM factor. The foveation effect is affected by spatial attention, and the orientation features are modeled for CM effect together with traditional block texture strength through three direction-based AC coefficients in DCT domain. The attention features are integrated with a novel Gestalt principle-based weighting mechanism for the final block-based VA model, which is then used to modulate JND profiles with two non-linear functions. Finally, the proposed VA-guided JND model is incorporated into a logarithmic spread transform dither modulation (L-STDM) watermarking scheme. Experimental results show that the newly proposed algorithm can achieve good performance in term of robustness and get better visual quality.

A Superpixel-Wise Just Noticeable Distortion Model

The just noticeable distortion (JND) model reveals the visibility limitation. Human eyes hold different attention and sensitivity to different regions owing to different contributions to the perceptual quality. In this paper, a superpixel-wise JND model based on region (RJND) is proposed. First, an image is segmented into superpixels by the simple linear iterative clustering (SLIC). Then, region color contrast is calculated for each region and foveation regions are selected for the image. Based on the human visual perception, a region weighting model is established by incorporating region color contrast and foveation regions modulation. Considering the contrast masking (CM) effect is not perfect, we introduce the texture coarseness combined with CM effect for a more accurate visual masking effect. Finally, a new region JND model is established by combining the region weighting model and the coarseness modulation. The experimental results demonstrate the proposed RJND model can decrease PSNR more efficiently compared with some existing JND models when achieving nearly the same subjective perceptual quality. It can remove much more visual redundancy.

Color Video JND Model Using Compound Spatial Masking and Structure-Based Temporal Masking

In order to provide high transmission efficiency and visual quality for the color image/video in internet, effective estimation of its inherent just noticeable distortion (JND) is continuously an important topic. In this paper, a discrete cosine transform (DCT)-based perceptual model for estimating spatio-temporal JNDs of color videos is presented. Firstly, the compound masking adjustment integrated with different spatial masking factors is proposed. Based on the base detection threshold for DCT coefficients in luminance and chrominance components of the color image, the adjustment combined by luminance masking, pattern-based contrast masking, and cross masking is exploited to measure visibility thresholds for luminance component, while the adjustment induced by the variance-based statistical properties is utilized to measure visibility thresholds for chrominance components. Then, the local temporal statistics of luminance component are considered to design the structure-based temporal masking adjustment for further estimating visibility thresholds of video signals. To verify the proposed method, a subjective viewing test is designed and a fair rating process of evaluating the visual quality is carried out under the specified viewing condition. Experimental results show that the proposed method is able to measure more visual distortion tolerance than the existing color-based methods at high visual quality.

Surprise-Based JND Estimation for Images

In visual perception, the neural response is evoked by the central stimulus and inhibited by the surrounding stimuli. These two stimuli can be regarded as the positive and negative incentives for perception. In this letter, the aftereffect of two incentives is incorporated into the just noticeable distortion (JND) estimation to avoid the threshold overestimation issue. In neuroscience, how an incentive affects an observer is measured by the surprise of this incentive which is estimated by the residual self-information according to the effective coding theory. Considering the competition biases, we define the product of the residual self-information and the biases, named as perceptual surprise, to measure the positive effect. As for the negative effect, we measure it with the average surprise (i.e., local Shannon entropy). Besides, visual signals are processed hierarchically in the HVS. To simplify the JND modeling process, the HVS is divided into two levels and the perception is simply divided into two stages. In each stage, the threshold is estimated individually and the total JND threshold is obtained by non-linear superposition of two stage thresholds. Experimental results show that the proposed JND model outperforms the existing pixel-based JND models and effectively avoid the threshold overestimation issue.

Perceptual Video Coding Scheme Using Just Noticeable Distortion Model Based on Entropy Filter

Because perceptual video coding (PVC) can reduce bitrates with negligible visual quality loss in video compression, a PVC scheme based on just noticeable distortion (JND) model is proposed for ultra-high definition video. Firstly, the proposed JND model is designed, considering the spatial JND characteristics such as contrast sensitivity, luminance adaptation and saliency weight factor. Secondly, in order to perform precise JND suppression, the Gauss differential entropy (GDE) filter is designed to divide the image into smooth and complex texture region. Thirdly, through incorporating the proposed JND model into the encoding process, the transform coefficients are suppressed in harmonization with the transform/quantization process of high efficiency video coding (HEVC). In order to achieve the JND suppression effectively, a distortion compensation factor and distortion compensation control factor are incorporated to control the extent of distortion in the rate distortion optimization process. The experimental results show that the proposed PVC scheme can achieve a remarkable bitrate reduction of 32.98% for low delay (LD) configuration and 28.61% for random access (RA) configuration with a negligible subjective quality loss. Meanwhile, the proposed method only causes about average 12.94% and 22.45% encoding time increase under LD and RA configuration compared with an HEVC reference software, respectively.

Color image watermarking based on orientation diversity and color complexity

The Just Noticeable Distortion (JND) can reliably measure the perceptual strength in image watermarking, but, it remains a challenge to computationally model the process of embedding watermark without prior knowledge of the image contents. This paper proposed a novel color image watermarking scheme in discrete cosine transform (DCT) domain based on JND, which takes both orientation diversity and color complexity features into account. Firstly, two indicator was introduced which take into account the differences in the texture types and orientation diversity of the Human Visual System (HVS) in the proposed JND contrast masking (CM) processing. In addition, a novel color complexity weight from Cb-channel is used to guarantee the scheme robustness. Then, a novel JND model combined with the proposed contrast masking and color complexity is applied into quantization watermarking scheme. Compared with the state-of-the-art methods for color image watermarking, experimental results using publicly available images show that our proposed scheme is reliable and effective.

Orientation-Aware Saliency Guided JND Model for Robust Image Watermarking

The visual saliency-based just noticeable distortion (VS-JND) model has been widely used in the quantization-based watermarking framework, but it remains a challenge that how to reflect the characteristics of the human visual system (HVS) to the greatest extent in the processing of detecting salient objects. In this paper, given that the HVS has a different perception sensitivity in the different orientation of the image, a compound orientation feature map, which contains the vertical, horizontal, and diagonal information for directional patch extraction, is computed to improve the visual saliency map within just noticeable distortion (JND) profile. First, the DC and three low-frequency coefficients of each block are used to calculate the luminance and texture feature map, respectively. Then, the feature map based on the complex directional feature includes the horizontal, vertical, and diagonal information contained in the image is calculated by three low-frequency coefficients. Finally, we will be able to make a linear fusion of the same three different visual features to get the final VS map, and the JND model is improved according to the new VS model. Later, the new proposed VS-JND model is employed in the quantization watermarking (Q-watermarking) framework. The simulation results show that the proposed watermarking scheme is effective and produces superior robustness compared with existing monochrome image watermarking schemes.

Pattern complexity-based JND estimation for quantization watermarking

The perceptual just noticeable distortion (JND), which plays an important role in perceptual image watermarking and can refers to the optimal tradeoff between imperceptibility and robustness. Major challenges of the perceptual quantization watermarking approach are two-fold: (1) Most DCT-based JND models cannot accurately estimate the contrast masking effect due to the complicated interaction among visual contents. Research on cognitive science shows that the HVS is adaptive to extract the visual pattern for image understanding, and we can formulate the pattern complexity as another factor to determine the total watermarking strength. (2) Moreover, the calculated JND values will change as watermark embedding can affect the pixels of the image, i.e. the operation in cross-domain reduce the watermarking robustness. Therefore, in this regard, the maximum directional energy is calculated by three AC DCT coefficients, which can measure the different direction energy and keep the pattern complexity measurement insensitive to the changes caused by watermarking procedure. The luminance contrast is also calculated in the DCT domain. So a pattern complexity-based perceptual JND estimation model is designed by takeing DCT-based pattern complexity and luminance contrast into account. Furthermore, a new logarithmic quantization watermarking scheme is presented based on the proposed model to verify the feasibility and effectiveness of our proposed JND model. Experimental results show that the new built JND model can effectively enhance the robustness of the quantization watermarking scheme.

Rate distortion optimization for stereoscopic video coding based on joint of binocular just-noticeable distortion and structural similarity index metric

In video coding, the Lagrange-based rate distortion optimization (RDO) plays important roles in improving coding efficiency. We proposed a method to improve the RDO of stereoscopic video coding. It uses two independent perception-based distortion metrics for the left and right views, as the distortion in the RDO. The distortion metric of the right view is to measure binocular perceptual distortion of the two views. The distortion metric of the left view is to measure two-dimensional (2-D) perceptual distortion of the left view. Both distortion metrics are developed based on the structural similarity index metric (SSIM) that is weighted by visual sensitivity. A binocular just-noticeable distortion (JND) model is proposed to measure binocular visual sensitivity for the right view. The commonly used 2-D JND models are adopted for the left view. In addition, a 1/SSIM-based rate-distortion (R-D) model is improved based the proposed distortion metrics. Based on it, the Lagrange multiplier is then derived according to the R-D theory. The experiments demonstrate that the proposed method has better performance than other perceptual RDO methods. Subjective evaluations also show that it can achieve better perceptual quality with less bit rate.

Learning-Based Just-Noticeable-Quantization- Distortion Modeling for Perceptual Video Coding.

Conventional predictive video coding-based approaches are reaching the limit of their potential coding efficiency improvements, because of severely increasing computation complexity. As an alternative approach, perceptual video coding (PVC) has attempted to achieve high coding efficiency by eliminating perceptual redundancy, using just-noticeable-distortion (JND) directed PVC. The previous JNDs were modeled by adding white Gaussian noise or specific signal patterns into the original images, which were not appropriate in finding JND thresholds due to distortion with energy reduction. In this paper, we present a novel discrete cosine transform-based energy-reduced JND model, called ERJND, that is more suitable for JND-based PVC schemes. Then, the proposed ERJND model is extended to two learning-based just-noticeable-quantization-distortion (JNQD) models as preprocessing that can be applied for perceptual video coding. The two JNQD models can automatically adjust JND levels based on given quantization step sizes. One of the two JNQD models, called LR-JNQD, is based on linear regression and determines the model parameter for JNQD based on extracted handcraft features. The other JNQD model is based on a convolution neural network (CNN), called CNN-JNQD. To our best knowledge, our paper is the first approach to automatically adjust JND levels according to quantization step sizes for preprocessing the input to video encoders. In experiments, both the LR-JNQD and CNN-JNQD models were applied to high efficiency video coding (HEVC) and yielded maximum (average) bitrate reductions of 38.51% (10.38%) and 67.88% (24.91%), respectively, with little subjective video quality degradation, compared with the input without preprocessing applied.

A Perceptual Video Coding Based on JND Model

In view of the fact that the current high efficiency video coding standard does not consider the characteristics of human vision, this paper proposes a perceptual video coding algorithm based on the just noticeable distortion model (JND). The adjusted JND model is combined into the transformation quantization process in high efficiency video coding (HEVC) to remove more visual redundancy and maintain compatibility. First of all, we design the JND model based on pixel domain and transform domain respectively, and the pixel domain model can give the JND threshold more intuitively on the pixel. The transform domain model introduces the contrast sensitive function into the model, making the threshold estimation more precise. Secondly, the proposed JND model is embedded in the HEVC video coding framework. For the transformation skip mode (TSM) in HEVC, we adopt the existing pixel domain called nonlinear additively model (NAMM). For the non-transformation skip mode (non-TSM) in HEVC, we use transform domain JND model to further reduce visual redundancy. The simulation results show that in the case of the same visual subjective quality, the algorithm can save more bitrates.

A Novel STDM Watermarking Using Visual Saliency-Based JND Model

The just noticeable distortion (JND) model plays an important role in measuring the visual visibility for spread transform dither modulation (STDM) watermarking. However, the existing JND model characterizes the suprathreshold distortions with an equal saliency level. Visual saliency (VS) has been widely studied by psychologists and computer scientists during the last decade, where the distortions are more likely to be noticeable to any viewer. With this consideration, we proposed a novel STDM watermarking method for a monochrome image by exploiting a visual saliency-based JND model. In our proposed JND model, a simple VS model is employed as a feature to reflect the importance of a local region and compute the final JND map. Extensive experiments performed on the classic image databases demonstrate that the proposed watermarking scheme works better in terms of the robustness than other related methods.

Saliency-Guided Just Noticeable Distortion Estimation Using the Normalized Laplacian Pyramid

The human visual system (HVS), like any other physical system, has limitations. For instance, it is known that the HVS can only sense the content changes that are larger than the so-called just noticeable distortion (JND) threshold. Also, to reduce the computational load on the brain, the visual attention mechanism is deployed such that regions with higher visual saliency are processed with higher priority than other less-salient regions. It is also known that visual saliency has a modulatory effect on JND thresholds. In this letter, we present a novel pixel-wise JND estimation method that considers the interplay between visual saliency and JND thresholds. In the proposed method, the largest JND thresholds of a given image are found such that the perceptual distance between the image and its JND noise-contaminated version is minimized in a perceptual space defined by the coefficients of the image in a normalized Laplacian pyramid. Experimental results indicate that the proposed method outperforms four of the latest JND models for static images.

Estimating just‐noticeable distortion for images/videos in pixel domain

Existing pixel-based just noticeable distortion (JND) models only take into account luminance adaptation and texture masking (TM). Similarly, existing discrete cosine transform (DCT) based models do not take into account foveal vision effects and do not estimate TM efficiently. As human visual system (HVS) is not sensitive to distortion below the JND threshold, estimation of the perceptual visibility threshold is widely used in digital and video processing applications. The authors propose a comprehensive and efficient pixel-based JND model incorporating all major factors which contribute to the JND estimation. The evaluation of contrast masking (CM) is done by distinguishing the edge and TM with respect to the entropy masking properties of the HVS. Similarly, the foveal vision effects are also taken into account for the comprehensive estimation of contrast sensitivity function (CSF). Hence, the proposed pixel-based JND model incorporates the spatio-temporal CSF, foveal vision effects, influence of eye-movement, luminance adaptation and CM to be more consistent with human perception. The incorporation of these important factors makes the proposed model the most comprehensive and efficient in the current literature. Psychophysical experiments were performed to test the proposed model. The results show the proposed model comprehensively outperforms other existing models proving its efficiency.