Hierarchical Prediction Structure Research Articles

Light field image coding using a residual channel attention network–based view synthesis

PurposeLight field images (LFIs) have gained popularity as a technology to increase the field of view (FoV) of plenoptic cameras since they can capture information about light rays with a large FoV. Wide FoV causes light field (LF) data to increase rapidly, which restricts the use of LF imaging in image processing, visual analysis and user interface. Effective LFI coding methods become of paramount importance. This paper aims to eliminate more redundancy by exploring sparsity and correlation in the angular domain of LFIs, as well as mitigate the loss of perceptual quality of LFIs caused by encoding.Design/methodology/approachThis work proposes a new efficient LF coding framework. On the coding side, a new sampling scheme and a hierarchical prediction structure are used to eliminate redundancy in the LFI's angular and spatial domains. At the decoding side, high-quality dense LF is reconstructed using a view synthesis method based on the residual channel attention network (RCAN).FindingsIn three different LF datasets, our proposed coding framework not only reduces the transmitted bit rate but also maintains a higher view quality than the current more advanced methods.Originality/value(1) A new sampling scheme is designed to synthesize high-quality LFIs while better ensuring LF angular domain sparsity. (2) To further eliminate redundancy in the spatial domain, new ranking schemes and hierarchical prediction structures are designed. (3) A synthetic network based on RCAN and a novel loss function is designed to mitigate the perceptual quality loss due to the coding process.

Reinforcement Learning for Rate-Distortion Optimized Hierarchical Prediction Structure

Video coding standards use a prediction structure to arrange video frames and exploit temporal correlations. In this aspect, it is crucial to resolve complicated temporal dependencies among frames to improve coding efficiency because the coding of a preceding frame affects the rate-distortion (R-D) performance of the subsequent frames. Previous algorithms have attempted to address the problem using handcrafted features or analytical models even though natural videos display various temporal characteristics. In this paper, we propose a reinforcement learning (RL)-based decision algorithm to build the optimal hierarchical prediction structure under a random-access configuration (RA-HPS) in Versatile Video Coding (VVC). Our goal is to maximize coding efficiency by selecting a series of optimal group of pictures (GOP) structures for coding. Accordingly, we formulate an adaptive GOP selection algorithm with a binary tree to represent a policy. We generate an optimal binary tree to minimize the sum of the R-D costs among all plausible binary trees. A new RL policy representation is defined, and the optimal policy is obtained by a sequential update. The tree grows with a hierarchical state-action and a reward sequence in each node. For efficient learning, the proposed technique uses a deep Q-network architecture to capture the temporal correlation between frames, which helps learn the policy of the tree-based RL framework effectively. Experimental results demonstrate that the proposed technique achieves a significant Bjontegaard-Delta (BD)-rate reduction compared with state-of-the-art GOP size-selection algorithms.

Gaussian-Wiener Representation and Hierarchical Coding Scheme for Focal Stack Images

Focal stack images (FoSIs) are a set of 2D images that captured one scene with serial focal depths. The redundancies of FoSIs mainly come from gradual focused depths changes rather than motion of objects. Conventional coding schemes cannot fully exploit such redundancies, leading to coding inefficiency. In this paper, we propose a new Gaussian-Wiener representation to model the gradual focused depths changes among FoSIs. In the representation, image degradation-restoration relations are utilized to describe the focus-defocus changing characteristics of FoSIs. Based on this representation, we propose a new hierarchical coding scheme for fully exploiting the inter-frame redundancies of FoSIs. In the scheme, a Gaussian-Wiener representation based inter prediction (GWR-IP) is presented by embedding Gaussian convolution and Wiener deconvolution into normal video encoder. Block-wise focus-defocus changing of FoSIs can be predicted in bi-directional manner by solving optimization problem. For higher coding efficiency, a Gaussian-Wiener representation based hierarchical prediction structure (GWR-HPS) is also designed and applied in the coding scheme. The proposed coding scheme is performed on 10 test sequences, including 5 synthetic scenes and 5 realistic scenes. Experimental results show that proposed coding scheme can obtain 2.640 dB PSNR gains and 51.830% bitrate savings on average of all test sequences in Low Delay P configuration, 2.123 dB PSNR gains and 43.975% bitrate savings in Low Delay B configuration, and 1.044 dB PSNR gains and 26.078% bitrate savings in Random Access configuration. Particularly, it achieves up to 65.544% bit rate savings and 3.901 dB PSNR increments for test sequence I09 in Low Delay P configuration. Furthermore, ablation test demonstrates that Gaussian representation contributes more on coding performance than Wiener representation and GWR-HPS.

Rate Distortion Optimization: A Joint Framework and Algorithms for Random Access Hierarchical Video Coding.

This paper revisits the problem of rate distortion optimization (RDO) with focus on inter-picture dependence. A joint RDO framework which incorporates the Lagrange multiplier as one of parameters to be optimized is proposed. Simplification strategies are demonstrated for practical applications. To make the problem tractable, we consider an approach where prediction residuals of pictures in a video sequence are assumed to be emitted from a finite set of sources. Consequently the RDO problem is formulated as finding optimal coding parameters for a finite number of sources, regardless of the length of the video sequence. Specifically, in cases where a hierarchical prediction structure is used, prediction residuals of pictures at the same prediction layer are assumed to be emitted from a common source. Following this approach, we propose an iterative algorithm to alternatively optimize the selections of quantization parameters (QPs) and the corresponding Lagrange multipliers. Based on the results of the iterative algorithm, we further propose two practical algorithms to compute QPs and the Lagrange multipliers for the RA(random access) hierarchical video coding: the first practical algorithm uses a fixed formula to compute QPs and the Lagrange multipliers, and the second practical algorithm adaptively adjusts both QPs and the Lagrange multipliers. Experimental results show that these three algorithms, integrated into the HM 16.20 reference software of HEVC, can achieve considerable RD improvements over the standard HM 16.20 encoder, in the common RA test configuration.

Adaptive Lagrangian Multiplier for Quantization Parameter Cascading in HEVC Hierarchical Coding

Hierarchical prediction structures applied in High Efficiency Video Coding (H.265/HEVC) leads to significant coding efficiency improvement. Quantization parameter cascading (QPC) techniques designed for specific prediction structure employs different quantization intervals for different temporal layers that further improve rate-distortion (R-D) performance of hierarchical coding. The existing QPC schemes focus on quantization parameter allocation, however, ignore the adaptation of Lagrangian multiplier to prediction structure, inter-layer residual dependency and layer-different quantization parameters (QPs). The misalignment of employed Lagrangian multiplier to the optimal one restricts the performance of QPC methods. To promote QPC schemes to their best performance, we propose a Lagrangian multiplier adaptation method based on accurate hyperbolic R-D model, distortion-QP model and rate-QP model. The derived Lagrangian multiplier is adaptive to residual characteristics under different prediction structures and QPC methods. Experimental results show that our proposed Lagrangian multipliers applied to existing QPC methods achieves Bjontegaard delta bit rate (BDBR) gain of 3.59 ${\%}$ to 4.65 ${\%}$ with small computational complexity increase.

Sub-Frame Layer Rate Distortion Optimization for High Efficiency Video Coding

This paper proposes a sub-frame layer RDO algorithm to further improve coding performance of low-delay hierarchical prediction structure in High Efficiency Video Coding. In the proposed algorithm, each frame is segmented into two sub-frames according to motion intensity, and a sub-frame layer is taken as a basic optimizing unit. An inter-frame dependency model is proposed to quantify inter-frame dependency of a frame or a sub-frame. The proposed inter-frame dependency model is composed of two parts, Laplace distribution based direct inter-frame dependency model and distortion propagation chain. The direct inter-frame dependency model is used to evaluate dependency caused by direct reference, and the distortion propagation chain shows the relationship between direct reference and indirect reference. With quantified inter-frame dependency, Lagrange optimization is then used to solve the optimal $\lambda $ -QP pair for each sub-frame layer in frame layer 1 (L1) to frame layer 4 (L4). For frame layer 0 (L0), additional L0 frames are adaptively inserted according to inter-frame dependency of the previous L0 frame, and the optimal $\lambda $ -QP pair for each sub-frame in L0 is solved using Lagrange optimization. When peak signal to noise ratio is used as quality criterion, the proposed algorithm achieves 7.0% and 7.0% bit-rate reduction with lowdelay_P_main (LDP) and lowdelay_main (LDB) configurations respectively; when structural similarity is used as quality criterion, the proposed algorithm achieves 13.3% and 13.1% bit-rate reduction with LDP and LDB configurations respectively.

Periodic Enhanced Frame Based Long-Short-Term Reference in HEVC for Conference and Surveillance Video Coding

The coding tools of the High Efficiency Video Coding (HEVC) standard were essentially designed for ordinary video contents. They have not fully exploited the long-term temporal dependence of surveillance videos and conference videos, in which the backgrounds are generally stationary. In this paper, a periodic-enhanced frame-based long-short-term reference scheme is presented for coding the videos with stationary backgrounds. First, periodic-enhanced frames are proposed based on global rate-distortion optimization, in which the background error propagation characteristic is introduced to reduce the long-term temporal redundancy. Second, two limitations, such as temporary occlusion and quantization error, are analyzed as applying the enhanced frame on a default HEVC hierarchical prediction structure. In order to reduce the limitations, a long-short-term reference scheme is presented based on the enhanced frames. In this scheme, the periodic-enhanced frames are employed as a long-term reference picture, and the adjacent frames are set as short-term reference pictures. The proposed method is verified by testing on the conference videos and surveillance videos.

Long-term prediction for hierarchical-B-picture-based coding of video with repeated shots

The latest video coding standard High Efficiency Video Coding (HEVC) can achieve much higher coding efficiency than previous video coding standards. Particularly, by exploiting the hierarchical B-picture prediction structure, temporal redundancy among neighbor frames is eliminated remarkably well. In practice, videos available to consumers usually contain many repeated shots, such as TV series, movies, and talk shows. According to our observations, when these videos are encoded by HEVC with the hierarchical B-picture structure, the temporal correlation in each shot is well exploited. However, the long-term correlation between repeated shots has not been used. We propose a long-term prediction (LTP) scheme to use the long-term temporal correlation between correlated shots in a video. The long-term reference (LTR) frames of a source video are chosen by clustering similar shots and extracting the representative frames, and a modified hierarchical B-picture coding structure based on an LTR frame is introduced to support long-term temporal prediction. An adaptive quantization method is further designed for LTR frames to improve the overall video coding efficiency. Experimental results show that up to 22.86% coding gain can be achieved using the new coding scheme.

An efficient all-zero block detection algorithm for high efficiency video coding with RDOQ

Rate distortion optimized HEVC video coding is challenged by the extremely high complexity caused by hierarchical prediction structure and advanced coding tools. All-zero-block (AZB) detection is an efficient tool to decrease the complexity of mode decision in HEVC coders in the sense of rate distortion optimization (RDO). Traditional AZB detection schemes were usually designed by employing hard-decision quantization (HDQ), and the AZB detection thresholds were generally derived according to the individual block’s characteristic parameter such as SAD or SATD. However, soft-decision quantization (SDQ) is preferably employed in HEVC encoders for coding performance improvement, and the AZB detection thresholds are supposed to be determined according to the ensemble’s statistical characteristics in the sense of statistical inference. This paper proposes an adaptive AZB detection algorithm well-suited for SDQ, more specifically RD optimized quantization (RDOQ). Inspired by Bayesian inference, a more accurate coefficient-level zero-quantized threshold model in the RDO sense is proposed by fully simulating RDOQ to aid in the following AZB detection. In addition, a local parameter depicting the individual block’s characteristics regarding the inter-coefficient distribution is jointly combined with SATD to derive an adaptive threshold model for AZB detection. The experimental results demonstrate that the proposed algorithm detects 91.9% 4×4 AZB with smaller than 1.9% FAR and 8.1% FRR false alarm rate on average, at the cost of negligible rate-distortion performance loss. This work is well-suited for fast RD optimized HEVC coding.

Adaptive search range control in H.265/HEVC with error propagation resilience and hierarchical adjustment

In the state-of-the-art video coding standards H.26 4/AVC and H.265/HEVC, adaptive search range (ASR) control has achieved a significant performance in reducing the computational complexity of motion estimation. Nevertheless, the application of ASR in adjacent coding units may bring error propagation, and the ASR values may vary subject to different layers in hierarchical prediction structure. Here we propose two separate algorithms for ASR in H.265/HEVC: an error propagation resilience algorithm that eliminates the rate-distortion (RD) loss due to error propagation, and a hierarchical range adjustment algorithm that adjusts the ASR values between hierarchical layers. Simulation results show that our methods greatly improve the RD performances of traditional ASR works whilst keeping almost intact complexity reductions.

Block Recovery Rate-Based Unequal Error Protection for Three-Screen TV

This paper describes a three-screen television system using a block recovery rate (BRR)-based unequal error protection (UEP). The proposed in-home wireless network uses scalable video coding (SVC) and UEP with forward error correction (FEC) for maximizing the quality of service (QoS) over error-prone wireless networks. For efficient FEC packet assignment, this paper proposes a simple and efficient performance metric, a BRR which is defined as a recovery rate of temporal and quality layer from FEC assignment by analyzing the hierarchical prediction structure including the current packet loss. It also explains the SVC layer switching scheme according to network conditions such as packet loss rate (PLR) and available bandwidth (ABW). In the experiments conducted, gains in video quality with the proposed UEP scheme vary from 1 to 3 dB in Y-peak signal-to-noise ratio (PSNR) with corresponding subjective video quality improvements.

Reduction of temporal distortion in video coding based on detection of just-noticeable temporal pumping artifact

At low bit-rates, video coding under the hierarchical prediction structure induces a temporal distortion known as temporal pumping artifact (TPA) when the quantization parameter cascading strategy is used. TPA manifests itself as a pumping effect visually, due to severe quality fluctuations among adjacent pictures, which seriously affects the perceptual quality of a video. In this work, the causes and perception of the TPA is first analyzed. Then a metric of just-noticeable temporal pumping artifact (JNTPA) is formulated based on characteristics of the human visual system. Based on JNTPA, an efficient algorithm has been devised for TPA reduction (denoted as TRA-JNTPA for short). The experimental results have demonstrated that the proposed JNTPA measure is in line with human perception, and compared with JVT-P014, the proposed TRA-JNTPA achieves significant reduction of the TPA with superior subjective video quality at a given bit-rate.

SSIM-Based Coarse-Grain Scalable Video Coding

We propose an improved coarse-grain scalable video coding (SVC) approach based on the structural similarity (SSIM) index as the visual quality criterion, aiming at maximizing the overall coding performance constrained by user-defined quality weightings for all scalable layers. First, we develop an interlayer rate-SSIM dependency model, by investigating bit rate and SSIM relationships between different layers. Second, a reduced-reference SSIM-Q model and a Laplacian R-Q model are introduced for SVC, by incorporating the characteristics of hierarchical prediction structure in each layer. Third, based on the user-defined weightings and the proposed models, we design a rate-distortion optimization approach to adaptively adjust Lagrange multipliers for all layers to maximize the overall rate-SSIM performance of the scalable encoder. Experiments with multiple layers, different layer weightings, and various videos demonstrate that the proposed framework can achieve better rate-SSIM performance than single layer optimization method, and provide better coding efficiency as compared to the conventional SVC scheme. Subjective tests further demonstrate the benefits of the proposed scheme.

Scalable Video Transmission Over Wireless Networks Based on Loss Distribution and Layer Information

We propose an efficient scalable video transmission method using unequal forward error correction (FEC) scheme based on loss distribution and layer information. The actual packet loss rates over wireless networks are variable due to the clustered packet losses. First, we deduce the expected packet loss recovery rate from the estimation of the effective packet loss rate using loss distribution statistics. Second, layer-based distortion metric is presented including error propagation effects from hierarchical prediction structure in compressed scalable video packets. Third, the performance metric for FEC assignment is proposed from the layer-based distortion metric and the expected packet loss recovery rate. Finally, the FEC assignment algorithm is proposed for maximizing the performance metric. The proposed unequal FEC assignment method demonstrates robustness and significantly improved performance for scalable video transmission in various channel conditions compared to previous FEC assignment schemes.

Optimizing the hierarchical prediction and coding in HEVC for surveillance and conference videos with background modeling.

For the real-time and low-delay video surveillance and teleconferencing applications, the newly video coding standard HEVC can achieve much higher coding efficiency over H.264/AVC. However, we still argue that the hierarchical prediction structure in the HEVC low-delay encoder still does not fully utilize the special characteristics of surveillance and conference videos that are usually captured by stationary cameras. In this case, the background picture (G-picture), which is modeled from the original input frames, can be used to further improve the HEVC low-delay coding efficiency meanwhile reducing the complexity. Therefore, we propose an optimization method for the hierarchical prediction and coding in HEVC for these videos with background modeling. First, several experimental and theoretical analyses are conducted on how to utilize the G-picture to optimize the hierarchical prediction structure and hierarchical quantization. Following these results, we propose to encode the G-picture as the long-term reference frame to improve the background prediction, and then present a G-picture-based bit-allocation algorithm to increase the coding efficiency. Meanwhile, according to the proportions of background and foreground pixels in coding units (CUs), an adaptive speed-up algorithm is developed to classify each CU into different categories and then adopt different speed-up strategies to reduce the encoding complexity. To evaluate the performance, extensive experiments are performed on the HEVC test model. Results show our method can averagely save 39.09% bits and reduce the encoding complexity by 43.63% on surveillance videos, whereas those are 5.27% and 43.68% on conference videos.

An efficient algorithm to eliminate temporal pumping artifact in video coding with hierarchical prediction structure

At low bit-rates, hierarchical prediction structure using quantization parameter cascading strategy will introduce a new temporal distortion, i.e., the temporal pumping artifact (TPA) presenting itself as a stumbling effect which seriously affect the subjective video reconstructive quality. This paper analyzes one important key factor which greatly influence the critical perceptual of the TPA and proposes a metric for just noticeable temporal pumping artifact (JNTPA) based on spatial–temporal masking effects in human visual system (HVS). Finally, an efficient TPA eliminating algorithm in video coding based on the JNTPA has been proposed. The subsequent experiments demonstrate that the estimated JNTPA values using the proposed metric are in line with the HVS perception and comparable with JVT-P104, the proposed TPA eliminating algorithm achieves a more smooth subjective video quality and improves the observe comfort of the human eye. At the same bit-rate, better subjective video quality can be observed using the eliminating algorithm.

Packet-Level Layer-Based Interleaving Unequal Forward Error Correction (LIU-FEC) for Robust Wireless Scalable Video Transmission

To improve the error resilience and video quality over wireless networks, we propose a novel packet-level layer-based interleaving unequal forward error correction (LIU-FEC) method. First, a scalable layer-based interleaving architecture is proposed for improving the efficiency of FEC from successive packet losses in variable channel conditions. The interleaved transmission across different scalable layers can efficiently disperse the consecutive packet losses into different scalable layers. Second, a closed form FEC assignment solution is proposed for minimizing video quality degradation using simple layer-based error propagation metric in hierarchical prediction structure. The simulation results show that the proposed algorithm offers higher PSNR values in various channel status, compared to the conventional FEC algorithm.

Motion estimation in low-delay hierarchical p-frame coding using motion vector composition

Low-delay hierarchical prediction structure is currently adopted in various new video coding standards. The only hurdle of this structure is the need of motion estimation in distant reference frames. To maintain high coding efficiency, a large search range for motion estimation can improve the coding efficiency in distant reference pictures. Computational complexity will thus be increased dramatically. In this paper, a fast motion estimation scheme for a low-delay hierarchical P-frame structure is proposed. The proposed scheme adopts a motion vector composition strategy to expedite the motion estimation process for distant reference frames in the hierarchical P structure. In addition, a motion vector composition algorithm is tailor-made with the proposed hierarchical P coding scheme to further improve the coding efficiency. Simulation results show that the proposed scheme can deliver a remarkable complexity savings and coding efficiency improvement on coding a frame in low temporal layers of the hierarchical P structure.

Efficient Fine-Granular Scalable Coding of 3D Mesh Sequences

An efficient fine-granular scalable coding algorithm of 3-D mesh sequences for low-latency streaming applications is proposed in this work. First, we decompose a mesh sequence into spatial and temporal layers to support scalable decoding. To support the finest-granular spatial scalability, we decimate only a single vertex at each layer to obtain the next layer. Then, we predict the coordinates of decimated vertices spatially and temporally based on a hierarchical prediction structure. Last, we quantize and transmit the spatio-temporal prediction residuals using an arithmetic coder. We propose an efficient context model for the arithmetic coding. Experiment results show that the proposed algorithm provides significantly better compression performance than the conventional algorithms, while supporting finer-granular spatial scalability.

In-Layer Multibuffer Framework for Rate-Controlled Scalable Video Coding

Temporal scalability is supported in scalable video coding (SVC) by means of hierarchical prediction structures, where the higher layers can be ignored for frame rate reduction. Nevertheless, this kind of scalability is not totally exploited by the rate control (RC) algorithms since the hypothetical reference decoder (HRD) requirement is only satisfied for the highest frame rate substream of every dependence (spatial or coarse grain scalability) layer. In this paper, we propose a novel RC approach that aims to deliver several HRD-compliant temporal resolutions within a particular dependence layer. Instead of using the common SVC encoder configuration consisting of a dependence layer per each temporal resolution, a compact configuration that does not require additional dependence layers for providing different HRD-compliant temporal resolutions is proposed. Specifically, the proposed framework for rate-controlled SVC uses a set of virtual buffers within a dependence layer so that their levels can be simultaneously controlled for overflow and underflow prevention while minimizing the reconstructed video distortion of the corresponding substreams. This in-layer multibuffer approach has been built on the top of a baseline H.264/SVC RC algorithm for variable bit rate applications. The experimental results show that our proposal achieves a good performance in terms of mean quality, quality consistency, and buffer control using a reduced number of layers.