Just Noticeable Distortion Research Articles

Fast and robust JND-guided video watermarking scheme in spatial domain

Generally speaking, those watermarking studies using the spatial domain tend to be fast but with limited robustness and imperceptibility while those performed in other transform domains are robust but have high computational cost. Watermarking applied to digital video has as one of the main challenges the large amount of computational power required due to the huge amount of information to be processed. In this paper we propose a watermarking algorithm for digital video that addresses this problem. To increase the speed, the watermark is embedded using a technique to modify the DCT coefficients directly in the spatial domain, in addition to carrying out this process considering the video scene as the basic unit and not the video frame. In terms of robustness, the watermark is modulated by a Just Noticeable Distortion (JND) scheme computed directly in the spatial domain guided by visual attention to increase the strength of the watermark to the maximum level but without this operation being perceivable by human eyes. Experimental results confirm that the proposed method achieves remarkable performance in terms of processing time, robustness and imperceptibility compared to previous studies.

Audio–video collaborative JND estimation model for multimedia applications

With the rapid development of the Internet and multimedia technologies, multimedia applications integrating audio and video are becoming increasingly prevalent in both everyday life and professional environments. A critical challenge is to significantly enhance compression efficiency and bandwidth utilization while maintaining high-quality user experiences. To address this challenge, the Just Noticeable Distortion (JND) estimation model, which leverages the perceptual characteristics of the Human Visual System (HVS), is widely used in image and video coding for improved data compression. However, human visual perception is an integrative process that involves both visual and auditory stimuli. Therefore, this paper investigates the influence of audio signals on visual perception and presents a collaborative audio–video JND estimation model tailored for multimedia applications. Specifically, we characterize audio loudness, duration, and energy as temporal perceptual features, while assigning the audio saliency superimposed on the image plane as the spatial perceptual feature. An audio JND adjustment factor is then designed using a segmentation function. Finally, the proposed model combines the video-based JND model with the audio JND adjustment factor to form the audio–video collaborative JND estimation model. Compared with existing JND models, the model presented in this paper achieves the best subjective quality, with an average PSNR value of 26.97 dB. The experimental results confirm that audio significantly impacts human visual perception. The proposed audio–video collaborative JND model effectively enhances the accuracy of JND estimation for multimedia data, thereby improving compression efficiency and maintaining high-quality user experiences.

JND-based multi-module cooperative perceptual optimization for HEVC

Since the just noticeable distortion (JND) reflects the tolerance limit of human visual system (HVS) to coding distortion directly, JND-based perceptual video coding (PVC) plays an increasingly significant role in video compression with the developing explosion of video data. In this paper, we focus on the coupling effect among coding modules and provide a JND-based multi-module cooperative perceptual optimization (JMCPO) scheme for HEVC. The main contribution of the proposed JMCPO scheme includes the following three aspects. (1) Based on quantization distortion estimation, an adaptive perceptual quantization scheme is proposed using the binary search approach on the premise of that the quantization distortion is infinitely close to the estimated JND threshold. (2) The coupling effect among coding modules is first analyzed and a novel perceptual residual filtering scheme is presented based on statistic analysis of coupling strength. (3) The JMCPO scheme is finally developed through collaborative optimization of residual filtering, quantization and rate–distortion optimization. Experimental results show that the proposed JMCPO scheme saves more bitrate with better subjective and objective qualities.

Surprise-based JND estimation for perceptual quantization in H.265/HEVC codecs

Just noticeable distortion (JND), reflecting the perceptual redundancy directly, has been widely used in image and video compression. However, the human visual system (HVS) is extremely complex and the visual signal processing has not been fully understood, which result in existing JND models are not accurate enough and the bitrate saving of JND-based perceptual compression schemes is limited. This paper presents a novel pixel-based JND model for videos and a JND-based perceptual quantization scheme for HEVC codecs. In particular, positive and negative perception effects of the inter-frame difference and the motion information are analyzed and measured with an information-theoretic approach. Then, a surprise-based JND model is developed for perceptual video coding (PVC). In our PVC scheme, the frame-level perceptual quantization parameter (QP) is derived on the premise that the coding distortion is infinitely close to the estimated JND threshold. On the basis of the frame-level perceptual QP, we determine the perceptual QP for each coding unit through a perceptual adjustment function to achieve better perceptual quality. Experimental results indicate that the proposed JND model outperforms existing models significantly, the proposed perceptual quantization scheme improves video compression efficiency with better perceptual quality and lower coding complexity.

BL-JUNIPER: A CNN-Assisted Framework for Perceptual Video Coding Leveraging Block-Level JND

Just Noticeable Distortion (JND) finds the minimum distortion level perceivable by humans. This can be a natural solution for setting the compression for each video region in perceptual video coding. However, existing JND-based solutions estimate JND levels for each video frame and ignore the fact that different video regions have different perceptual importance. To address this issue, we propose a Block-Level Just Noticeable Distortion-based Perceptual (BL-JUNIPER) framework for video coding. The proposed four-stage framework combines different perceptual information to further improve the prediction accuracy. The JND mapping in the first stage derives block-level JNDs from frame-level information without the need to collect a new bock-level JND dataset. In the second stage, an efficient CNN-based model is proposed to predict JND levels for each block according to spatial and temporal characteristics. Unlike existing methods, BL-JUNIPER works on raw video frames and avoids re-encoding each frame several times, making it computationally practical. Third, the visual importance of each block is measured using a visual attention model. Finally, a proposed quantization control algorithm uses both JND levels and visual importance to adjust the Quantization Parameter (QP) for each block. The specific algorithm for each stage of the proposed framework can be changed, as long as the input and output formats of each block are followed, without the need to change other stages, based on any current or future methods, providing a flexible and robust solution. Extensive experimental results demonstrate that BL-JUNIPER achieves a mean bitrate reduction of 27.75% with a Delta Mean Opinion Score (DMOS) close to zero and BD-Rate gains of 25.44% based on MOS, compared to the baseline encoding, and also gains a better performance compared to competing methods.

Perceptual ultra-high definition video coding based on adaptive just noticeable distortion model

Ultra-high definition (UHD) video is gradually applied into many fields. The huge amount of data in UHD video limits the realization of these applications. Perception video coding can exploit the visual redundancy, and improve the compression performance. So far, the perpetual video coding based on just noticeable distortion (JND) model cannot fully exploit the perceptual redundancy of ultra-high definition (UHD) video/image in 10-bit depth. Thus, an adaptive JND model based on perceptual noise Bayesian network (PNBN) is firstly established for UHD video/image. Then, combined perceptual noise channel encoding/decoding with human visual characteristics, hidden side information of image is dynamically estimated by PNBN. Thirdly, the adaptive JND model is considered both luminance masking effect and texture complexity, which can adaptively exploit redundancy in different texture regions of image. Finally, an effective perceptual video coding (PVC) scheme is proposed for versatile video coding. A novel distortion compensation factor is appropriately designed for rate-distortion optimization. Experimental results show that the proposed PVC scheme achieves significant bitrate reduction with better subjective video quality than the state-of-the-art PVC schemes.

Vision‐oriented algorithm for fast decision in 3D video coding

This paper designs a novel method to reduce the coding complexity of 3D-HEVC encoder by utilizing the properties of human visual perception. Two vision-oriented edge detections are proposed: for colour texture detection, the authors adopt the Just-Noticeable Distortion (JND); for depth map, the authors combine the Sample Adaptive Offset (SAO) and the Just Noticeable Depth Difference (JNDD) model. The authors also analyse the properties of colour texture and depth map to classify the coding tree unit (CTU) into various kinds of types, including complex-edge CTU, moderate-edge CTU and homogeneous CTU. Besides, fast mode decisions and early termination criteria are performed individually on each type of CTUs according to their characteristics. Especially for those CTUs with more edge information, the proposed projection-based fast mode decision and residual-based early termination preserve important colour texture while speeding up the coding at the same time. The proposed vision-oriented algorithm reduces 31.981% of the overall average coding time with only 1.580% BD-Bitrate increase. Experimental results show that the proposed algorithm can provide considerable time-saving while still maintain the video quality, which outperforms the previous researches.

Fast CU Size Decision Method Based on Just Noticeable Distortion and Deep Learning

With the development of broadband networks and high-definition displays, people have higher expectations for the quality of video images, which also brings new requirements and challenges to video coding technology. Compared with H.265/High Efficiency Video Coding (HEVC), the latest video coding standard, Versatile Video Coding (VVC), can save 50%-bit rate while maintaining the same subjective quality, but it leads to extremely high encoding complexity. To decrease the complexity, a fast coding unit (CU) size decision method based on Just Noticeable Distortion (JND) and deep learning is proposed in this paper. Specifically, the hybrid JND threshold model is first designed to distinguish smooth, normal, or complex region. Then, if CU belongs to complex area, the Ultra-Spherical SVM (US-SVM) classifiers are trained for forecasting the best splitting mode. Experimental results illustrate that the proposed method can save about 52.35% coding runtime, which can realize a trade-off between the reduction of computational burden and coding efficiency compared with the latest methods.

Perceptual VVC quantization refinement with ensemble learning

Compressing videos while maintaining an acceptable level of Quality of Experience (QoE) is indispensable. To this aim, a feasible method is to further increase the Quantization Parameter (QP) of video stream to eliminate visual redundancy, simultaneously utilizing perceptual characteristics of Human Visual System (HVS) to impose a threshold constraint on the maximum QP. In this paper, we employ Just Noticeable Distortion (JND) to characterize the aforementioned threshold constraint, thereby avoiding perceptual loss during QP refinement process. We propose an effective JND-based algorithm for QP optimization, in which a video saliency detection is introduced to extract regions of interest, a refinement model based on a lightweight network is designed to predict QP value and an ensemble learning method to improve generalization performance. Theoretical analysis and experimental results demonstrate that the proposed algorithm has been successfully applied to Versatile Video Coding (VVC) to achieve significant bitrate reduction without sacrificing perceived quality.

Hybrid Distortion-Based Rate-Distortion Optimization and Rate Control for H.265/HEVC

Mean squared error (MSE) is adopted as the distortion metric for the rate-distortion optimization (RDO) and rate control (RC) in most of the existing video coding standards, e.g., H.265/High-Efficiency Video Coding (HEVC) and H.264/Advanced Video Coding (AVC), because the pixel-wise MSE can represent the image fidelity efficiently. As videos are usually used for visual enjoyment, recently, researchers found that the MSE is not always consistent with perceptual quality, and proposed some perceptual quality-based video coding methods by exchanging the MSE with other perceptual quality metrics completely, like the structure similarity index (SSIM), just noticeable distortion (JND), etc. We believe that a sophisticated video encoder should guarantee both the subject quality and the data fidelity. By investigating the properties of the human visual system (HVS), hybrid distortion-based RDO and RC methods are proposed to improve subjective quality while avoiding fidelity degradation. Experimental results demonstrate that the proposed methods can improve subjective quality efficiently, while preserving the image fidelity of the reconstructed videos as much as possible.

A Fast Perceptual Surveillance Video Coding (PSVC) Based on Background Model-Driven JND Estimation

Perceptual video coding (PVC) optimization has been an important video coding technique, which can be consistent with the perception characteristics of the human visual system (HVS). Currently, PVC schemes incorporating the just noticeable distortion (JND) model can obtain better performance gain in all PVC schemes. To further accelerate the JND computation for real-time video coding applications (e.g. surveillance video coding and conference video coding), this paper proposes a fast perceptual surveillance video coding (PSVC) scheme based on background model-driven JND estimation method. First, to utilize the surveillance scene characteristics, the computation complexity of JND estimation can be significantly decreased by reusing the content complexity of background regions. Then we apply the perceptive video coding scheme into the background modeling-based surveillance video codec. The proposed scheme adopts background modeling frame as background anchor. Experimental results show that the proposed scheme can yield remarkable time saving of 42.33% maximum and on average 34.76% with approximate bitrate reductions and similar subjective quality, compared to HEVC and other state-of-the-art schemes.

Blind Photograph Watermarking with Robust Defocus-Based JND Model

Just noticeable distortion (JND) is widely employed to describe the perception redundancy in the quantization-based watermarking framework. However, the existing JND models are generally constructed to treat every region of the photograph with an equal focus level, whereas the defocus effect has never been considered. In this paper, the defocus feature, which can portray the aesthetic emphasis in the photograph, is provided to improve the perceptual JND model. Firstly, two indicators which consider the block energy in the defocus measurement (DM) are proposed. Then, the defocus feature map (DFM) is obtained by integrating the influence of the circumambient blocks, and it is applied to the proposed JND contrast masking (CM) processing. In this way, a new blind photograph watermarking method, with emphasis on defocus-JND estimation combined with the proposed CM, is presented. Simulations show that the proposed JND is more suitable for watermarking framework than some exiting JND models, and the proposed watermarking scheme with the improved defocus-based JND model has superior robustness compared with some watermarking schemes.

Hierarchical Predictive Coding-Based JND Estimation for Image Compression.

The human visual system (HVS) is a hierarchical system, in which visual signals are processed hierarchically. In this paper, the HVS is modeled as a three-level communication system and visual perception is divided into three stages according to the hierarchical predictive coding theory. Then, a novel just noticeable distortion (JND) estimation scheme is proposed. In visual perception, the input signals are predicted constantly and spontaneously in each hierarchy, and neural response is evoked by the central residue and inhibited by surrounding residues. These two types' residues are regarded as the positive and negative visual incentives which cause positive and negative perception effects, respectively. In neuroscience, the effect of incentive on observer is measured by the surprise of this incentive. Thus, we propose a surprise-based measurement method to measure both perception effects. Specifically, considering the biased competition of visual attention, we define the product of the residue self-information (i.e., surprise) and the competition biases as the perceptual surprise to measure the positive perception effect. As for the negative perception effect, it is measured by the average surprise (i.e., the local Shannon entropy). The JND threshold of each stage is estimated individually by considering both perception effects. The total JND threshold is finally obtained by non-linear superposition of three stage thresholds. Furthermore, the proposed JND estimation scheme is incorporated into the codec of Versatile Video Coding for image compression. Experimental results show that the proposed JND model outperforms the relevant existing ones, and over 16% of bit rate can be reduced without jeopardizing the perceptual quality.

Just Noticeable Distortion Profile Inference: A Patch-Level Structural Visibility Learning Approach.

In this paper, we propose an effective approach to infer the just noticeable distortion (JND) profile based on patch-level structural visibility learning. Instead of pixel-level JND profile estimation, the image patch, which is regarded as the basic processing unit to better correlate with the human perception, can be further decomposed into three conceptually independent components for visibility estimation. In particular, to incorporate the structural degradation into the patch-level JND model, a deep learning-based structural degradation estimation model is trained to approximate the masking of structural visibility. In order to facilitate the learning process, a JND dataset is further established, including 202 pristine images and 7878 distorted images generated by advanced compression algorithms based on the upcoming Versatile Video Coding (VVC) standard. Extensive experimental results further show the superiority of the proposed approach over the state-of-the-art. Our dataset is available at: https://github.com/ShenXuelin-CityU/PWJNDInfer.

An Approach for Diagnostically Lossless Coding of Volumetric Medical Data Based on Wavelet and Just-Noticeable-Distortion Model

This paper explores a technique for visually/diagnostically lossless coding in the wavelet domain to effectively compress the three-dimensional medical image data. The quantisation module based on Just Noticeable Distortion (JND) for wavelets guarantees the visual quality in the reconstructed data. This method has been further extended to present the Volume of Interest (VOI) based technique that enables to preserve the quality of the diagnostically significant VOI region. The proposed method tested on several datasets outperforms the state-of-the-art methods. Apart from the conventional quality metric, Human Visual System (HVS) based quality metrics are also used to evaluate the visual quality of the reconstructed image. A subjective and objective evaluation carried out for VOI based coder shows that the quality-compression needs of the medical community are well addressed.

An improved DCT-based JND estimation model considering multiple masking effects

The just noticeable distortion (JND) in the contour and orderly regions is easy to be overestimated and that in the disorderly areas is usually underestimated. In order to estimate the JND threshold more accurately, this paper proposes an improved DCT-based JND estimation model considering multiple masking effects properly. The contributions of this paper are characterized by twofold. On the one hand, a mean absolute difference based (MAD-based) block classification method is developed at first to classify image blocks into plain, contour and texture types accurately and quickly. And the JND model for contrast masking effect (CM-JND) is constructed as a modulation factor based on the MAD of each block. On the other hand, we propose a distance-based disorder evaluation metric to measure the disorder intensity in block level. Then, the JND model for the disorderly concealment effect (DC-JND) is proposed based on our psychological experiment. Finally, the total JND estimation threshold is modeled by fusing the spatial contrast sensitivity function, the luminance adaptation effect, the CM and DC effects. Experimental results show that the proposed DCT-based JND estimation model outperforms existing models in performance and complexity. Specifically, the proposed model shows more tolerance for distortions, lower computational complexity with better perceptual quality than other JND models.

A novel attention-guided JND Model for improving robust image watermarking

Just noticeable distortion (JND) and visual attention (VA), which are two widely used mathematical models of human visual system (HVS) that aim to simulate the human brain mechanism, are sufficiently explored and applied to many researches including digital watermarking. The activity of human brain, however, is extremely complex and it can be more limited due to complicated fusion of spatial saliency for image domain. In this paper, we propose a novel VA guided JND model in which we fuse the final attention map from the low-level features by using two laws of Gestalt principle. Firstly, we demonstrate a classic JND model in DCT domain, which consists of spatial contrast sensitivity function (CSF), luminance adaptation (LA) and contrast masking (CM). The foveation effect and orientation feature are considered to obtain the CSF and CM factor. The foveation effect is affected by spatial attention, and the orientation features are modeled for CM effect together with traditional block texture strength through three direction-based AC coefficients in DCT domain. The attention features are integrated with a novel Gestalt principle-based weighting mechanism for the final block-based VA model, which is then used to modulate JND profiles with two non-linear functions. Finally, the proposed VA-guided JND model is incorporated into a logarithmic spread transform dither modulation (L-STDM) watermarking scheme. Experimental results show that the newly proposed algorithm can achieve good performance in term of robustness and get better visual quality.

FJND-based fuzzy rate control of scalable video for streaming applications

A fuzzy rate controller with buffer constraint in combination with a perceptual quality controller is proposed for streaming applications of the AVC/H.264 scalable (SVC) video. The bit rate of each video layer is controlled separately by the fuzzy controller that adjusts the quantization parameter (QP) on a group of pictures (GOP) basis. The QPs of pictures are computed from the GOP QP by the well-known QP cascading technique. While the fuzzy controller provides the buffering constraint for each video layer, the quality controller tries to improve the perceptual quality of the compressed video based on the foveated just-noticeable distortion (FJND) model. The quality controller regulates the QP of each macroblock around the picture QP based on the visibility threshold of the FJND model. In these applications, the initial buffering allows slight variations of the bit rate leading to produce a variable bit rate (VBR) video bit stream with consistent quality. Experimental results show that the proposed algorithm effectively adapts to the buffer size, while strictly prevents buffer overflow and underflow. In addition, incorporating the perceptual quality controller into the fuzzy rate controller achieves higher perceptual quality at the same bit rate.

Color Video JND Model Using Compound Spatial Masking and Structure-Based Temporal Masking

In order to provide high transmission efficiency and visual quality for the color image/video in internet, effective estimation of its inherent just noticeable distortion (JND) is continuously an important topic. In this paper, a discrete cosine transform (DCT)-based perceptual model for estimating spatio-temporal JNDs of color videos is presented. Firstly, the compound masking adjustment integrated with different spatial masking factors is proposed. Based on the base detection threshold for DCT coefficients in luminance and chrominance components of the color image, the adjustment combined by luminance masking, pattern-based contrast masking, and cross masking is exploited to measure visibility thresholds for luminance component, while the adjustment induced by the variance-based statistical properties is utilized to measure visibility thresholds for chrominance components. Then, the local temporal statistics of luminance component are considered to design the structure-based temporal masking adjustment for further estimating visibility thresholds of video signals. To verify the proposed method, a subjective viewing test is designed and a fair rating process of evaluating the visual quality is carried out under the specified viewing condition. Experimental results show that the proposed method is able to measure more visual distortion tolerance than the existing color-based methods at high visual quality.

Just Noticeable Distortion-Based Perceptual Rate Control in HEVC

In this paper, we propose a just noticeable distortion (JND)-based perceptual rate control method for high efficiency video coding (HEVC). First, the JND factor of a coding unit has been mathematically shown to be an approximation of the average pixel-level JND weight, which means that it can also be used as a weight for bitrate allocation. Second, rate-distortion (R-D) modelling is conducted based on the JND factor. Finally, the proposed R-D model is integrated into an existing rate control framework to improve the coding efficiency, and the proposed algorithm is implemented in the newest video coding standard. As the experimental results reveal, compared with HEVC reference software, our algorithm achieves significantly improved coding performance, subjective coding quality and bitrate accuracy.