Perceptual Video Coding Scheme Research Articles

Surprise-based JND estimation for perceptual quantization in H.265/HEVC codecs

Just noticeable distortion (JND), reflecting the perceptual redundancy directly, has been widely used in image and video compression. However, the human visual system (HVS) is extremely complex and the visual signal processing has not been fully understood, which result in existing JND models are not accurate enough and the bitrate saving of JND-based perceptual compression schemes is limited. This paper presents a novel pixel-based JND model for videos and a JND-based perceptual quantization scheme for HEVC codecs. In particular, positive and negative perception effects of the inter-frame difference and the motion information are analyzed and measured with an information-theoretic approach. Then, a surprise-based JND model is developed for perceptual video coding (PVC). In our PVC scheme, the frame-level perceptual quantization parameter (QP) is derived on the premise that the coding distortion is infinitely close to the estimated JND threshold. On the basis of the frame-level perceptual QP, we determine the perceptual QP for each coding unit through a perceptual adjustment function to achieve better perceptual quality. Experimental results indicate that the proposed JND model outperforms existing models significantly, the proposed perceptual quantization scheme improves video compression efficiency with better perceptual quality and lower coding complexity.

Perceptual ultra-high definition video coding based on adaptive just noticeable distortion model

Ultra-high definition (UHD) video is gradually applied into many fields. The huge amount of data in UHD video limits the realization of these applications. Perception video coding can exploit the visual redundancy, and improve the compression performance. So far, the perpetual video coding based on just noticeable distortion (JND) model cannot fully exploit the perceptual redundancy of ultra-high definition (UHD) video/image in 10-bit depth. Thus, an adaptive JND model based on perceptual noise Bayesian network (PNBN) is firstly established for UHD video/image. Then, combined perceptual noise channel encoding/decoding with human visual characteristics, hidden side information of image is dynamically estimated by PNBN. Thirdly, the adaptive JND model is considered both luminance masking effect and texture complexity, which can adaptively exploit redundancy in different texture regions of image. Finally, an effective perceptual video coding (PVC) scheme is proposed for versatile video coding. A novel distortion compensation factor is appropriately designed for rate-distortion optimization. Experimental results show that the proposed PVC scheme achieves significant bitrate reduction with better subjective video quality than the state-of-the-art PVC schemes.

A Fast Perceptual Surveillance Video Coding (PSVC) Based on Background Model-Driven JND Estimation

Perceptual video coding (PVC) optimization has been an important video coding technique, which can be consistent with the perception characteristics of the human visual system (HVS). Currently, PVC schemes incorporating the just noticeable distortion (JND) model can obtain better performance gain in all PVC schemes. To further accelerate the JND computation for real-time video coding applications (e.g. surveillance video coding and conference video coding), this paper proposes a fast perceptual surveillance video coding (PSVC) scheme based on background model-driven JND estimation method. First, to utilize the surveillance scene characteristics, the computation complexity of JND estimation can be significantly decreased by reusing the content complexity of background regions. Then we apply the perceptive video coding scheme into the background modeling-based surveillance video codec. The proposed scheme adopts background modeling frame as background anchor. Experimental results show that the proposed scheme can yield remarkable time saving of 42.33% maximum and on average 34.76% with approximate bitrate reductions and similar subjective quality, compared to HEVC and other state-of-the-art schemes.

Perceptual coding scheme for ultra-high definition video based on perceptual noise channel model

Perceptual video coding (PVC) can eliminate visual redundancy and effectively improve the compression performance of ultra-high definition video. The conventional PVC schemes, involving the perceptual characteristics related to the video features, do not account for interference in the human visual system caused by external environment changes. Therefore, this paper presents a PVC scheme based on a perceptual noise channel model. In this approach, visual dither is utilized to imitate the adjustment process of human eyes while environment changes. Considering reliable external visual stimulus decoding in the perceptual noise channel and visual characteristics, dynamic estimation of visual dither quantization is performed based on a message passing algorithm on a factor graph. Furthermore, a weight factor based on the perceptual noise channel capacity is incorporated into the rate distortion optimization process. The experimental results show that the proposed PVC scheme achieves significant bitrate reduction with better subjective video quality than competitive PVC schemes.

Perceptual Video Coding Scheme Using Just Noticeable Distortion Model Based on Entropy Filter

Because perceptual video coding (PVC) can reduce bitrates with negligible visual quality loss in video compression, a PVC scheme based on just noticeable distortion (JND) model is proposed for ultra-high definition video. Firstly, the proposed JND model is designed, considering the spatial JND characteristics such as contrast sensitivity, luminance adaptation and saliency weight factor. Secondly, in order to perform precise JND suppression, the Gauss differential entropy (GDE) filter is designed to divide the image into smooth and complex texture region. Thirdly, through incorporating the proposed JND model into the encoding process, the transform coefficients are suppressed in harmonization with the transform/quantization process of high efficiency video coding (HEVC). In order to achieve the JND suppression effectively, a distortion compensation factor and distortion compensation control factor are incorporated to control the extent of distortion in the rate distortion optimization process. The experimental results show that the proposed PVC scheme can achieve a remarkable bitrate reduction of 32.98% for low delay (LD) configuration and 28.61% for random access (RA) configuration with a negligible subjective quality loss. Meanwhile, the proposed method only causes about average 12.94% and 22.45% encoding time increase under LD and RA configuration compared with an HEVC reference software, respectively.

High Efficiency Video Coding Compliant Perceptual Video Coding Using Entropy Based Visual Saliency Model

In past years, several visual saliency algorithms have been proposed to extract salient regions from multimedia content in view of practical applications. Entropy is one of the important measures to extract salient regions, as these regions have high randomness and attract more visual attention. In the context of perceptual video coding (PVC), computational visual saliency models that utilize the charactertistics of the human visual system to improve the compression ratio are of paramount importance. To date, only a few PVC schemes have been reported that use the visual saliency model. In this paper, we conduct the first attempt to utilize entropy based visual saliency models within the high efficiency video coding (HEVC) framework. The visual saliency map generated for each input video frame is optimally thresholded to generate a binary saliency mask. The proposed HEVC compliant PVC scheme adjusts the quantization parameter according to visual saliency relevance at the coding tree unit (CTU) level. Efficient CTU level rate control is achieved by allocating bits to salient and non-salient CTUs by adjusting the quantization parameter values according to their perceptual weighted map. The attention based on information maximization has shown the best performance on newly created ground truth dataset, which is then incorporated in a HEVC framework. An average bitrate reduction of is achieved by the proposed HEVC compliant PVC scheme with the same perceptual quality and a nominal increase in coding complexity of when compared with HEVC reference software. Moreover, the proposed PVC scheme performs better than other HEVC based PVC schemes when encoded at low data rates.

Learning-Based Just-Noticeable-Quantization- Distortion Modeling for Perceptual Video Coding.

Conventional predictive video coding-based approaches are reaching the limit of their potential coding efficiency improvements, because of severely increasing computation complexity. As an alternative approach, perceptual video coding (PVC) has attempted to achieve high coding efficiency by eliminating perceptual redundancy, using just-noticeable-distortion (JND) directed PVC. The previous JNDs were modeled by adding white Gaussian noise or specific signal patterns into the original images, which were not appropriate in finding JND thresholds due to distortion with energy reduction. In this paper, we present a novel discrete cosine transform-based energy-reduced JND model, called ERJND, that is more suitable for JND-based PVC schemes. Then, the proposed ERJND model is extended to two learning-based just-noticeable-quantization-distortion (JNQD) models as preprocessing that can be applied for perceptual video coding. The two JNQD models can automatically adjust JND levels based on given quantization step sizes. One of the two JNQD models, called LR-JNQD, is based on linear regression and determines the model parameter for JNQD based on extracted handcraft features. The other JNQD model is based on a convolution neural network (CNN), called CNN-JNQD. To our best knowledge, our paper is the first approach to automatically adjust JND levels according to quantization step sizes for preprocessing the input to video encoders. In experiments, both the LR-JNQD and CNN-JNQD models were applied to high efficiency video coding (HEVC) and yielded maximum (average) bitrate reductions of 38.51% (10.38%) and 67.88% (24.91%), respectively, with little subjective video quality degradation, compared with the input without preprocessing applied.

A fast and HEVC-compatible perceptual video coding scheme using a transform-domain Multi-Channel JND model

Coding optimization methods incorporating the just noticeable distortion (JND) model, called perceptual video coding (PVC), have drawn much attention in recent years for better video coding performance. To further remove perceptual redundancy in every channel and improve the coding performance, this paper proposes a fast PVC scheme in the latest High Efficiency Video Coding (HEVC) framework based on our proposed variable block-size transform-domain multi-channel JND model. Firstly, through extensive experiments, we find out for the first time that the contrast masking (CM) effects for chroma channels show a lowpass property in frequency, which differs from the luma channel that has a bypass property. Based on this observation, CM effects in chroma blue (Cb) and chroma red (Cr) channels are modeled as a continuous function for variable-sized blocks, respectively. Secondly, since different characteristics of the human visual system (HVS) exhibit quite distinct effects in luma and chroma channels and effects in chroma channels were not well explored, we develop a new JND model through comprehensive consideration for both luma and chroma channels of five typical HVS effects, with especial focus on parameterized modeling of chroma channels in each effect. Finally, to incorporate the proposed JND model into the latest HEVC coding framework, a multi-channel coefficients suppression method based on JND thresholds and quantization parameter (QP) ranges is proposed in the transform and quantization process, which can decrease the computational complexity. Extensive experimental results show that the proposed PVC scheme implemented in HEVC reference software (HM15.0) can yields significant bit saving of up to 25.91% and on average 9.42% with similar subjective quality, compared to HM15.0, and consistently outperforms two PVC schemes with much reduced bitrate and complexity overhead.

HEVC-Based Perceptually Adaptive Video Coding Using a DCT-Based Local Distortion Detection Probability Model.

Discrete Cosine Transform (DCT)-based just noticeable difference (JND) profiles have widely been applied into human perception-based video coding in order to reduce perceptual redundancy, which is one of the main goals of perceptual video coding (PVC). However, there are two problems for this approach: 1) the JND value of each transform coefficient is estimated for a fixed-sized DCT kernel (e.g., 8 × 8), but flexible coding structures with variable-sized transform units have been utilized in standard video coding frameworks, such high efficiency video coding (HEVC) and 2) the DCT transform coefficients are suppressed by the amounts of JND values for the removal of perceptual redundancy, but the DCT transform coefficients of residues are not sufficiently suppressed due to many small transform coefficient values in mid- and high-frequency regions below the JND values. In order to solve these problems, we propose a more generalized visibility model in the DCT domain, called the DCT-based local distortion detection probability (LDDP) model that can estimate a degree of distortion visibility for any distribution of the transform coefficients of any sized DCT kernel for residues. Furthermore, we propose an HEVC-compliant LDDP-based PVC scheme where transform coefficients are sufficiently suppressed based on the LDDP model. The proposed PVC scheme is implemented in the HEVC Test Model (HM 11.0) reference software to show the effectiveness of the LDDP-based PVC scheme. Objective and subjective tests for encoded test sequences are performed. The experimental results show that the LDDP-based PVC scheme achieves a significant performance improvement of bitrate reduction at the similar visual quality levels compared with the original HM 11.0.

An HEVC-Compliant Perceptual Video Coding Scheme Based on JND Models for Variable Block-Sized Transform Kernels

In this paper, a High Efficiency Video Coding (HEVC)-compliant perceptual video coding (PVC) scheme is introduced based on just-noticeable difference (JND) models in both transform and pixel domains. We adopt an existing pixel-domain JND model for the transform skip mode of HEVC and propose a transform-domain JND model for the transform nonskip modes of HEVC. The proposed transform-domain JND model is designed by considering the spatial JND characteristics such as contrast sensitivity, luminance adaptation, and contrast masking effects as well as by considering the summation effects of variable block-sized transforms in HEVC. A temporal JND model is additionally incorporated into the proposed transform-domain JND model to further reduce perceptual redundancy. To incorporate the transform- and pixel-domain JND models into the encoding process in an HEVC-compliant manner, the transform coefficients and residues are suppressed in harmonization with the transform/quantization process and the quantization-only process of HEVC, respectively. To make the JND-based suppression effective, a distortion compensation factor is also proposed to reflect the perceptual distortion in the rate-distortion optimization-based encoding process. Based on subjective quality assessments of the encoded bit streams of test sequences, the proposed HEVC-compliant PVC scheme yields remarkable bitrate reductions of a maximum 49.10% and an average 16.10% with negligible subjective quality loss, compared with an HEVC reference software HEVC test model (HM 11.0). In addition, the proposed HEVC-compliant PVC scheme increases the encoding complexity of HM 11.0 only by an average of 11.25%.

Analysis of the JND-Suppression Effect in Quantization Perspective for HEVC-based Perceptual Video Coding

Transform-domain JND (Just Noticeable Difference)-based for PVC (Perceptual Video Coding) is often performed in quantization processes to effectively remove perceptual redundancy. This study examined the JND-suppression effects on quantized coefficients of transform in HEVC (High Efficiency Video Coding). To reveal the JND-suppression effect in quantization, the properties of the floor functions were used for modeling the quantized coefficients, and a JNDadjustment process in an HEVC-compliant PVC scheme was used to tune the JND values by analyzing the JND suppression effect. In the experimental results, the bitrate reduction decreases slightly, but the PSNR and perceptual quality are improved significantly when the proposed JND adjustment process is applied.