Just Noticeable Distortion Estimation Research Articles

Audio–video collaborative JND estimation model for multimedia applications

With the rapid development of the Internet and multimedia technologies, multimedia applications integrating audio and video are becoming increasingly prevalent in both everyday life and professional environments. A critical challenge is to significantly enhance compression efficiency and bandwidth utilization while maintaining high-quality user experiences. To address this challenge, the Just Noticeable Distortion (JND) estimation model, which leverages the perceptual characteristics of the Human Visual System (HVS), is widely used in image and video coding for improved data compression. However, human visual perception is an integrative process that involves both visual and auditory stimuli. Therefore, this paper investigates the influence of audio signals on visual perception and presents a collaborative audio–video JND estimation model tailored for multimedia applications. Specifically, we characterize audio loudness, duration, and energy as temporal perceptual features, while assigning the audio saliency superimposed on the image plane as the spatial perceptual feature. An audio JND adjustment factor is then designed using a segmentation function. Finally, the proposed model combines the video-based JND model with the audio JND adjustment factor to form the audio–video collaborative JND estimation model. Compared with existing JND models, the model presented in this paper achieves the best subjective quality, with an average PSNR value of 26.97 dB. The experimental results confirm that audio significantly impacts human visual perception. The proposed audio–video collaborative JND model effectively enhances the accuracy of JND estimation for multimedia data, thereby improving compression efficiency and maintaining high-quality user experiences.

Surprise-based JND estimation for perceptual quantization in H.265/HEVC codecs

Just noticeable distortion (JND), reflecting the perceptual redundancy directly, has been widely used in image and video compression. However, the human visual system (HVS) is extremely complex and the visual signal processing has not been fully understood, which result in existing JND models are not accurate enough and the bitrate saving of JND-based perceptual compression schemes is limited. This paper presents a novel pixel-based JND model for videos and a JND-based perceptual quantization scheme for HEVC codecs. In particular, positive and negative perception effects of the inter-frame difference and the motion information are analyzed and measured with an information-theoretic approach. Then, a surprise-based JND model is developed for perceptual video coding (PVC). In our PVC scheme, the frame-level perceptual quantization parameter (QP) is derived on the premise that the coding distortion is infinitely close to the estimated JND threshold. On the basis of the frame-level perceptual QP, we determine the perceptual QP for each coding unit through a perceptual adjustment function to achieve better perceptual quality. Experimental results indicate that the proposed JND model outperforms existing models significantly, the proposed perceptual quantization scheme improves video compression efficiency with better perceptual quality and lower coding complexity.

A Fast Perceptual Surveillance Video Coding (PSVC) Based on Background Model-Driven JND Estimation

Perceptual video coding (PVC) optimization has been an important video coding technique, which can be consistent with the perception characteristics of the human visual system (HVS). Currently, PVC schemes incorporating the just noticeable distortion (JND) model can obtain better performance gain in all PVC schemes. To further accelerate the JND computation for real-time video coding applications (e.g. surveillance video coding and conference video coding), this paper proposes a fast perceptual surveillance video coding (PSVC) scheme based on background model-driven JND estimation method. First, to utilize the surveillance scene characteristics, the computation complexity of JND estimation can be significantly decreased by reusing the content complexity of background regions. Then we apply the perceptive video coding scheme into the background modeling-based surveillance video codec. The proposed scheme adopts background modeling frame as background anchor. Experimental results show that the proposed scheme can yield remarkable time saving of 42.33% maximum and on average 34.76% with approximate bitrate reductions and similar subjective quality, compared to HEVC and other state-of-the-art schemes.

Hierarchical Predictive Coding-Based JND Estimation for Image Compression.

The human visual system (HVS) is a hierarchical system, in which visual signals are processed hierarchically. In this paper, the HVS is modeled as a three-level communication system and visual perception is divided into three stages according to the hierarchical predictive coding theory. Then, a novel just noticeable distortion (JND) estimation scheme is proposed. In visual perception, the input signals are predicted constantly and spontaneously in each hierarchy, and neural response is evoked by the central residue and inhibited by surrounding residues. These two types' residues are regarded as the positive and negative visual incentives which cause positive and negative perception effects, respectively. In neuroscience, the effect of incentive on observer is measured by the surprise of this incentive. Thus, we propose a surprise-based measurement method to measure both perception effects. Specifically, considering the biased competition of visual attention, we define the product of the residue self-information (i.e., surprise) and the competition biases as the perceptual surprise to measure the positive perception effect. As for the negative perception effect, it is measured by the average surprise (i.e., the local Shannon entropy). The JND threshold of each stage is estimated individually by considering both perception effects. The total JND threshold is finally obtained by non-linear superposition of three stage thresholds. Furthermore, the proposed JND estimation scheme is incorporated into the codec of Versatile Video Coding for image compression. Experimental results show that the proposed JND model outperforms the relevant existing ones, and over 16% of bit rate can be reduced without jeopardizing the perceptual quality.

An improved DCT-based JND estimation model considering multiple masking effects

The just noticeable distortion (JND) in the contour and orderly regions is easy to be overestimated and that in the disorderly areas is usually underestimated. In order to estimate the JND threshold more accurately, this paper proposes an improved DCT-based JND estimation model considering multiple masking effects properly. The contributions of this paper are characterized by twofold. On the one hand, a mean absolute difference based (MAD-based) block classification method is developed at first to classify image blocks into plain, contour and texture types accurately and quickly. And the JND model for contrast masking effect (CM-JND) is constructed as a modulation factor based on the MAD of each block. On the other hand, we propose a distance-based disorder evaluation metric to measure the disorder intensity in block level. Then, the JND model for the disorderly concealment effect (DC-JND) is proposed based on our psychological experiment. Finally, the total JND estimation threshold is modeled by fusing the spatial contrast sensitivity function, the luminance adaptation effect, the CM and DC effects. Experimental results show that the proposed DCT-based JND estimation model outperforms existing models in performance and complexity. Specifically, the proposed model shows more tolerance for distortions, lower computational complexity with better perceptual quality than other JND models.

Pattern complexity-based JND estimation for quantization watermarking

The perceptual just noticeable distortion (JND), which plays an important role in perceptual image watermarking and can refers to the optimal tradeoff between imperceptibility and robustness. Major challenges of the perceptual quantization watermarking approach are two-fold: (1) Most DCT-based JND models cannot accurately estimate the contrast masking effect due to the complicated interaction among visual contents. Research on cognitive science shows that the HVS is adaptive to extract the visual pattern for image understanding, and we can formulate the pattern complexity as another factor to determine the total watermarking strength. (2) Moreover, the calculated JND values will change as watermark embedding can affect the pixels of the image, i.e. the operation in cross-domain reduce the watermarking robustness. Therefore, in this regard, the maximum directional energy is calculated by three AC DCT coefficients, which can measure the different direction energy and keep the pattern complexity measurement insensitive to the changes caused by watermarking procedure. The luminance contrast is also calculated in the DCT domain. So a pattern complexity-based perceptual JND estimation model is designed by takeing DCT-based pattern complexity and luminance contrast into account. Furthermore, a new logarithmic quantization watermarking scheme is presented based on the proposed model to verify the feasibility and effectiveness of our proposed JND model. Experimental results show that the new built JND model can effectively enhance the robustness of the quantization watermarking scheme.

Visual Saliency Based Just Noticeable Difference Estimation in DWT Domain

It has been known that human visual systems (HVSs) can be applied to describe the underlying masking properties for the image processing. In general, HVS can only perceive small changes in a scene when they are greater than the just noticeable distortion (JND) threshold. Recently, the cognitive resources of huma visual attention mechanisms are limited, which can not concentrate on all stimuli. To be specific, only more important stimuli will react from the mechanisms. When it comes to visual attention mechanisms, we need to introduce the visual saliency to model the human perception more accurately. In this paper, we presents a new wavelet-based JND estimation method that takes into account the interrelationship between visual saliency and JND threshold. In the experimental part, we verify it from both subjective and objective aspects. In addition, the experimental results show that extracting the saliency map of the image in the discrete wavelet transform (DWT) domain and then modulating its JND threshold is better than the non-modulated JND effect.

Saliency-Guided Just Noticeable Distortion Estimation Using the Normalized Laplacian Pyramid

The human visual system (HVS), like any other physical system, has limitations. For instance, it is known that the HVS can only sense the content changes that are larger than the so-called just noticeable distortion (JND) threshold. Also, to reduce the computational load on the brain, the visual attention mechanism is deployed such that regions with higher visual saliency are processed with higher priority than other less-salient regions. It is also known that visual saliency has a modulatory effect on JND thresholds. In this letter, we present a novel pixel-wise JND estimation method that considers the interplay between visual saliency and JND thresholds. In the proposed method, the largest JND thresholds of a given image are found such that the perceptual distance between the image and its JND noise-contaminated version is minimized in a perceptual space defined by the coefficients of the image in a normalized Laplacian pyramid. Experimental results indicate that the proposed method outperforms four of the latest JND models for static images.

Estimating just‐noticeable distortion for images/videos in pixel domain

Existing pixel-based just noticeable distortion (JND) models only take into account luminance adaptation and texture masking (TM). Similarly, existing discrete cosine transform (DCT) based models do not take into account foveal vision effects and do not estimate TM efficiently. As human visual system (HVS) is not sensitive to distortion below the JND threshold, estimation of the perceptual visibility threshold is widely used in digital and video processing applications. The authors propose a comprehensive and efficient pixel-based JND model incorporating all major factors which contribute to the JND estimation. The evaluation of contrast masking (CM) is done by distinguishing the edge and TM with respect to the entropy masking properties of the HVS. Similarly, the foveal vision effects are also taken into account for the comprehensive estimation of contrast sensitivity function (CSF). Hence, the proposed pixel-based JND model incorporates the spatio-temporal CSF, foveal vision effects, influence of eye-movement, luminance adaptation and CM to be more consistent with human perception. The incorporation of these important factors makes the proposed model the most comprehensive and efficient in the current literature. Psychophysical experiments were performed to test the proposed model. The results show the proposed model comprehensively outperforms other existing models proving its efficiency.

A saliency-modulated just-noticeable-distortion model with non-linear saliency modulation functions

It is well-known that the human visual system (HVS) cannot sense small variations of visual signals below the so-called just-noticeable distortion (JND) thresholds due to their underlying spatial/temporal masking properties. It is also known that the visual attention mechanism of the human brain can enhance or reduce visual sensitivity. In other words, the visual attention has modulatory effects on JND thresholds. The current knowledge also states that the visual attention is mainly driven by visual saliency in an automatic and involuntary manner. In this paper we present a saliency-modulated JND (SJND) model for static images in the discrete cosine transform (DCT) domain. In the proposed model, the JND thresholds of each block in a given image are elevated by two non-linear modulation functions using the visual saliency of the block. The parameters of the saliency modulation functions are obtained through an optimization framework, which utilizes a state-of-the-art saliency-based objective image quality assessment method. To evaluate the proposed SJND model, two subjective experiments were conducted. The obtained experimental results demonstrated that the proposed method achieves a high accuracy in JND estimation, and also it provides a high distortion-hiding capacity.

Prediction error preprocessing for perceptual color image compression

In this article, a prediction error preprocessor based on the just noticeable distortion (JND) for the color image compression scheme is presented. The dynamic range of prediction error signals we can reduce, the lower bit rate of the reconstructed image we can obtain at high visual quality. We propose a color JND estimator that is incorporated into the design of the preprocessor in the compression scheme. The color JND estimator is carried out in the wavelet domain to present good estimates to the available amount masking. The estimated JND is used to preprocess the signal and is also used to incorporate into the design of the quantization stage in the compression scheme for higher performance. Simulation results show that the bit rate required by the compression scheme with the preprocessor is lower at high visual quality of the reconstructed color image. The preprocessor is further applied to the input color image of the JPEG and JPEG2000 coders for better performance.

Estimating Just-Noticeable Distortion for Video

Just-noticeable distortion (JND), which refers to the maximum distortion that the human visual system (HVS) cannot perceive, plays an important role in perceptual image and video processing. In comparison with JND estimation for images, estimation of the JND profile for video needs to take into account the temporal HVS properties in addition to the spatial properties. In this paper, we develop a spatio-temporal model estimating JND in the discrete cosine transform domain. The proposed model incorporates the spatio-temporal contrast sensitivity function, the influence of eye movements, luminance adaptation, and contrast masking to be more consistent with human perception. It is capable of yielding JNDs for both still images and video with significant motion. The experiments conducted in this study have demonstrated that the JND values estimated for video sequences with moving objects by the model are in line with the HVS perception. The accurate JND estimation of the video towards the actual visibility bounds can be translated into resource savings (e.g., for bandwidth/storage or computation) and performance improvement in video coding and other visual processing tasks (such as perceptual quality evaluation, visual signal restoration/enhancement, watermarking, authentication, and error protection)

Motion-compensated residue preprocessing in video coding based on just-noticeable-distortion profile

We present a motion-compensated residue signal preprocessing scheme in video coding scheme based on just-noticeable-distortion (JND) profile. Human eyes cannot sense any changes below the JND threshold around a pixel due to their underlying spatial/temporal masking properties. An appropriate (even imperfect) JND model can significantly help to improve the performance of video coding algorithms. From the viewpoint of signal compression, smaller variance of signal results in less objective distortion of the reconstructed signal for a given bit rate. In this paper, a new JND estimator for color video is devised in image-domain with the nonlinear additivity model for masking (NAMM) and is incorporated into a motion-compensated residue signal preprocessor for variance reduction toward coding quality enhancement. As the result, both perceptual quality and objective quality are enhanced in coded video at a given bit rate. A solution of adaptively determining the parameter for the residue preprocessor is also proposed. The devised technique can be applied to any standardized video coding scheme based on motion compensated prediction. It provides an extra design option for quality control, besides quantization, in contrast with most of the existing perceptually adaptive schemes which have so far focused on determination of proper quantization steps. As an example for demonstration, the proposed scheme has been implemented in the MPEG-2 TM5 coder, and achieved an average peak signal-to-noise (PSNR) increment of 0.505 dB over the twenty video sequences which have been tested. The perceptual quality improvement has been confirmed by the subjective viewing tests conducted.

Just noticeable distortion model and its applications in video coding

We explore a new perceptually-adaptive video coding (PVC) scheme for hybrid video compression, in order to achieve better perceptual coding quality and operational efficiency. A new just noticeable distortion (JND) estimator for color video is first devised in the image domain. How to efficiently integrate masking effects together is a key issue of JND modelling. We integrate spatial masking factors with the nonlinear additivity model for masking (NAMM). The JND estimator applies to all color components and accounts for the compound impact of luminance masking, texture masking and temporal masking. Extensive subjective viewing confirms that it is capable of determining a more accurate visibility threshold that is close to the actual JND bound in human eyes. Secondly, the image-domain JND profile is incorporated into hybrid video encoding via the JND-adaptive motion estimation and residue filtering process. The scheme works with any prevalent video coding standards and various motion estimation strategies. To demonstrate the effectiveness of the proposed scheme, it has been implemented in the MPEG-2 TM5 coder and demonstrated to achieve average improvement of over 18% in motion estimation efficiency, 0.6 dB in average peak signal-to perceptual-noise ratio (PSPNR) and most remarkably, 0.17 dB in the objective coding quality measure (PSNR) on average. Theoretical explanation is presented for the improvement on the objective coding quality measure. With the JND-based motion estimation and residue filtering process, hybrid video encoding can be more efficient and the use of bits is optimized for visual quality.

Improved estimation for just-noticeable visual distortion

Perceptual visibility threshold estimation, based upon characteristics of the human visual system (HVS), has wide applications in digital image/video processing. An improved scheme for estimating just-noticeable distortion (JND) is proposed in this paper. It is proved to outperform the DCTune model, with the major contributions of a new formula for luminance adaptation adjustment and the incorporation of block classification for contrast masking. The HVS visibility threshold for digital images exhibits an approximately parabolic curve versus gray levels and this has been formulated to yield a more accurate base threshold. Moreover, edge regions have been differentiated via block classification to effectively avoid over-estimation of JND in the said regions. Experiments with different images and the associated subjective tests show improved performance of the proposed scheme over the DCTune model for luminance adaptation (especially in dark regions) and masking effect in edge regions. Our model has further demonstrated to achieve favorable results in perceptual visual distortion gauge and image compression. The improvement in JND estimation facilitates better visual distortion measurement and visual signal compression.