Scalable High Efficiency Video Coding Research Articles

Hybrid Strategies for Efficient Intra Prediction in Spatial SHVC

With multi-layer encoding and Inter-layer prediction, Spatial Scalable High Efficiency Video Coding (SSHVC) has extremely high coding complexity. It is very crucial to speed up its coding to promote widespread and cost-effective SSHVC applications. Specifically, we first reveal that the average RD cost of Inter-layer Reference (ILR) mode is different from that of Intra mode, but they both follow the Gaussian distribution. Based on this discovery, we apply the classic Gaussian Mixture Model and Expectation Maximization to determine whether ILR mode is the best mode thus skipping Intra mode. Second, when coding units (CUs) in enhancement layer use Intra mode, it indicates very simple texture is presented. We investigate their Directional Mode (DM) distribution, and divide all DMs into three classes, and then develop different methods with respect to classes to progressively predict the best DMs. Third, by jointly considering rate distortion costs, residual coefficients and neighboring CUs, we propose to employ the Conditional Random Fields model to early terminate depth selection. Experimental results demonstrate that the proposed algorithm can significantly improve coding speed with negligible coding efficiency losses.

A Probability-Based Zero-Block Early Termination Algorithm for QSHVC

To seamlessly adapt to time-varying network bandwidths, the Quality Scalable High-Efficiency Video Coding (QSHVC) is developed. However, its coding process is overly complex, and this seriously limits its wide applications in real-time environments. Therefore, it is of great significance to study fast coding algorithms for QSHVC. In this paper, we propose a novel probability-based zero-block early termination algorithm for QSHVC. First, we observed that the generated residual coefficients follow the Laplace distribution if a CU is accurately predicted. According to this observation, we derive the sum of squared differences based All-Zero Block (AZB) decision condition. Second, we develop the Hadamard Transform (HT)-based zero-valued quantized coefficient decision condition to obtain zero-valued quantized coefficients and the corresponding Partial-Zero Block (PZB). Third, the probability of each coding mode and coding depth being chosen as the best ones are combined with both AZBs and PZBs to derive the probability-based early termination condition. The experimental results show that the proposed algorithm can improve the average coding speed by 79.85% with a 0.34% decrease in BDBR.

Scalable extension of HEVC with adaptive de-blocking filter for bandwidth-limited applications

A primary latest video coding standard H.265/SHVC (Scalable High Efficiency Video Coding), an extension of High Efficiency Video Coding (HEVC) is dealt in this paper. To transfer high quality video over any network that usually occupies large bandwidth, suffers bandwidth limitation. It can be controlled by trading-off between the quality and the coding efficiency. The trade-off between these two parameters is considered in this paper to keep the difference at the minimum level. A novel Neural Network based Adaptive De-blocking Filter (NNADF) technique is proposed to maintain good PSNR with minimum bit rate in SHVC encoder. The adaptive de-blocking filter (ADF) performs block matching for nonlinear predictions to all frames in the video. Moreover, the ADF performs frame de-noising hence good quality of the video is achieved and trains a neural network (NN) model. The NN model restrains their outputs to achieve bit rate reduction, thereby coding efficiency is achieved. The simulation results show that the proposed NNADF technique delivers an average increment of 0.62 dB in BD-PSNR and an average decrement of 24 % in BD-BR for combined quality and spatial ratios respectively compared with the previously proposed algorithm.

A Deep Learning Approach in Scalable High Efficiency Video Coding for Fast Coding Unit Size Decision

The High Efficiency Video Coding (HEVC) with scalable extension known as SHVC allows encoding the same video with different resolutions in a single bitstream. However, this process increases the encoding complexity. The complexity is increased mainly due to the motion estimation process and the Rate-Distortion Optimization (RDO) search process. The Test Zonal (TZ) algorithm with fast search patterns helps to accelerate the motion estimation process. However, the search patterns may get trapped to local minima, leading to inaccurate motion vectors. Moreover, the RDO search process used to determine the Coding Unit (CU) size increases the complexity. We proposed the Horizontal Subsampling Motion Estimation (HSME) method to find the accurate motion vectors with reduced complexity. The experimental results prove that the HSME method saves the encoding time by 53.03% with a 6.45% increase in Bjøntegaard delta bit rate and 0.28 dB loss in Bjøntegaard Delta Peak Signal-to-Noise Ratio (BD-PSNR) compared to standard SHM-12.1. In addition, we designed the Early Terminated Long- and Short-Term Memory (ET-LSTM) network that predicts the CU partition by taking the output features of the Early Terminated Convolutional Neural Network (ET-CNN). The ET-CNN learns the CU partitions from the residual Coding Tree Unit (CTU) using the deep learning approach. Our proposed method (HSME + ET-CNN + ET-LSTM) achieves 53% savings in encoding time, which is significantly higher than state-of-the-art methods.

Object Tracking Based Surgical Incision Region Encoding using Scalable High Efficiency Video Coding for Surgical Telementoring Applications

Object Tracking Based Surgical Incision Region Encoding using Scalable High Efficiency Video Coding for Surgical Telementoring Applications

Interlayer Restoration Deep Neural Network for Scalable High Efficiency Video Coding

This paper applies an interlayer restoration deep neural network (IRDNN) for scalable high efficiency video coding (SHVC) to improve visual quality and coding efficiency. It is the first time to combine deep neural network (DNN) and SHVC. Considering the coding architecture of SHVC, we elaborate a multi-frame and multi-layer neural network to restore the interlayer of SHVC by utilizing both the adjacent reconstructed frames of the base layer (BL) and enhancement layer (EL). Moreover, we analyze the temporal motion relationship of frames in one layer and the compression degradation relationship of frames between different layers, and propose the synergistic mechanism of motion restoration and compression restoration in our IRDNN. The network can generate an interlayer with higher quality serving for the EL coding and thus enhance the coding efficiency. A large-scale and various-quality-degradation dataset is self-made for the task of interlayer restoration of SHVC. The experimental results show that with our implementation on SHVC, the EL Bj<inline-formula> <tex-math notation="LaTeX">$\phi $ </tex-math></inline-formula>ntegaard delta bit-rate (BD-BR) reduction is 9.291&#x0025; and 6.007&#x0025; in signal-to-noise ratio scalability and spatial scalability, respectively. The code is available at <i><uri>https://github.com/icecherylXuli/IRDNN</uri></i>.

Probability-Based Fast Intra Prediction Algorithm for Spatial SHVC

Due to multi-layer encoding and Inter-layer prediction, Scalable High Efficiency Video Coding (SHVC) has extremely high coding complexity. It is very crucial to improve its coding speed so as to promote widespread and cost-effective SHVC applications. In this paper, we propose a new probability-based fast Intra prediction algorithm for spatial SHVC. More specifically, first, we integrate depth probabilities with textural based all-zero blocks and all-nonzero blocks through Lagrange Interpolation Polynomial (LIP) to derive thresholds to early skip unlikely depths and early terminate depth selection. Second, to early skip Intra mode prediction, we combine mode probabilities with Jarque-Bera test through LIP to test the residual coefficients of Inter-layer reference (ILR) mode. Third, we adopt the difference of Hadamard Costs (HCs) of horizontal, vertical and diagonal directional modes to predict candidate directional modes (DMs), and then exploit both the percentages of gradient amplitudes and HCs for DM early termination. Experimental results demonstrate that the proposed algorithm can achieve a speed up gain of more than 61&#x0025; with 0.03&#x0025; decrease in Bj&#x00F8;ntegaard Delta Bit Rate on average.

Social-Content-Aware Scalable Video Streaming in Internet of Video Things

The Internet of Things (IoT) is evolving into the Internet of Video Things (IoVT) that supports massive smart devices and multiple video applications. However, how to effectively control massive devices and transmit large-volume video data have become challenges in the current IoVT. Inspired by device-to-device (D2D) communications and coalitional game, this article constructs a self-organized D2D collaborative video content sharing framework for the IoVT. Specifically, we first propose a collaboration mechanism by introducing the social attributes of IoVT devices and their owners. In this mechanism, D2D collaborative coalitions are automatically formed among IoVT devices and video content is shared in the coalitions through D2D links. In this way, the burden of controlling massive IoVT devices and video data traffic are offloaded. Then, by integrating the scalability of the scalable-high-efficiency-video-coding (SHVC) streams and the flexibility of D2D networking, a collaborative video streaming strategy is developed. It takes advantage of provider set arrangement and transmission scheduling to reduce the impact of network instability on video services. Simulation results verify the effectiveness of the proposed mechanism and strategy.

Inter-layer correlation-based adaptive bit allocation for enhancement layer in scalable high efficiency video coding

To facilitate the diversity of network and end-user devices, bit allocation technology must be combined with scalable high efficiency video coding (SHVC) to achieve continuous bitrate variation. However, to date, the multilayer coding feature in SHVC has not been fully utilized. In this paper, a bit allocation algorithm is proposed for the enhancement layer in SHVC. The algorithm includes two parts: a bit allocation method for the initial frame and another for subsequent frames. The bit allocation method for the initial frame consists of a bit allocation factor model constructed based on investigation of the influence of the initial target bitrate on the overall coding performance. For the bit allocation of the subsequent frames, alternate GOP coding is designed and implemented to fully exploit the inter layer correlation. In addition, an adaptive frame layer bit allocation ratio model is deduced according to the rate distortion optimization theory. Experimental results show that the proposed algorithm can improve PSNR by 0.44 dB and 0.41 dB under low delay and random-access configurations, respectively, and achieve high bitrate control accuracy. The algorithm also outperforms the state-of-the-art algorithms in term of PSNR improvement.

Bitrate Estimation for Spatial Scalable Videos

Spatial scalable video service has surged in the time of multiple screens. Existing bitrate allocation methods are principled by rate-distortion theory and characterized by iterative encoding, which is accurate yet complex. However, the quasi-quantitative description is preferred in practice of broadcasting. In this paper, we propose a task of bitrate estimation for scalable videos concerning the content, aiming at a more efficient model at the cost of precision. First, we exhibit necessity to build a model for Scalable High Efficiency Video Coding (SHVC) and quantitative relation between video content and bitrate using different encoders. Then, a scalable-video dataset is prepared. It covers various types of content to offer diversity for model training. In the end, multi-linear regression is utilized to estimate the bitrate of scalable videos, with spatial and temporal indices as explanatory variables. Our statistical experiments show the model is able to estimate bitrate after trained on the self-built dataset.

Fast intra- and inter-coding algorithms for the spatially scalable extension of H.265/HEVC

High Efficiency Video Coding (HEVC) is the latest international video coding standard. As an extension of HEVC, scalable HEVC (SHVC) achieves high compression efficiency at the cost of dramatically increased computational complexity. In this paper, a fast Coding Unit (CU) size decision algorithm is suggested for spatial SHVC intra-coding to early terminate the CU size decision process by jointly utilising the inter-layer and spatiotemporal correlations. Furthermore, in order to realise a fast Prediction Unit (PU) intra-mode decision, the mode dependency between the Base Layer (BL) and the Enhancement Layer (EL) is effectively used to remove unlikely coding modes. A fast PU inter-mode decision method is also developed for spatial SHVC inter-coding, in which the degree of motion activity and the PU mode dependency are exploited. Simulation results show that the proposed fast intra-coding algorithm produces a saving in encoding time of up to 72% and the proposed fast inter-coding scheme reduces the encoding time by up to 50% compared with the original SHVC encoder, and the loss in Rate-Distortion (RD) performance is negligible.

Performance Analysis of HEVC Scalable Extension for Depth Maps

High Efficiency Video Coding (HEVC) is actually the latest video standard that enables significant compression performance improvement in comparison with its predecessors. Scalable High Efficiency Video Coding (SHVC) and 3D High-Efficiency Video Coding (3D-HEVC) are extensions of HEVC hooked on recently emerged applications and issues. 3D-HEVC targets the monolayer compression of the specific multiview plus depth 3D video representation. It consists of multiple views and their associated depth maps that are required for synthesizing intermediate views in advanced free viewpoint applications. SHVC combines the compression efficiency and visual quality of HEVC, with the possibility of encoding different versions of a same video in a single bitstream. It is conceived to cope with the heterogeneity of networks and devices used in the transmission chain of texture videos. With the ongoing progess in equipments and technologies, the scalable multilayer coding is also solicited, not to say required, for depth maps. In comparison with 3D-HEVC, this paper shows how SHVC works for quality scalable depth maps compression over different base and enhancement layers quantization scenarios. Besides, this paper analyzes the SHVC efficiency for intermediate views that are synthesized from encoded depth maps. Thus we aim to provide support for upcoming works on depth map quality scalable encoding.

SLEPX: An Efficient Lightweight Cipher for Visual Protection of Scalable HEVC Extension

This paper proposes a lightweight cipher scheme aimed at the scalable extension of the High Efficiency Video Coding (HEVC) codec, referred to as the Scalable HEVC (SHVC) standard. This stream cipher, Symmetric Cipher for Lightweight Encryption based on Permutation and EXlusive OR (SLEPX), applies Selective Encryption (SE) over suitable coding syntax elements in the SHVC layers. This is achieved minimal computational complexity and delay. The algorithm also conserves most SHVC functionalities, i.e. preservation of bit-length, decoder format-compliance, and error resilience. For comparative analysis, results were taken and compared with other state-of-art ciphers i.e. Exclusive-OR (XOR) and the Advanced Encryption Standard (AES). The performance of SLEPX is also compared with existing video SE solutions to confirm the efficiency of the adopted scheme. The experimental results demonstrate that SLEPX is as secure as AES in terms of visual protection, while computationally efficient comparable with a basic XOR cipher. Visual quality assessment, security analysis and extensive cryptanalysis (based on numerical values of selected binstrings) also showed the effectiveness of SLEPX's visual protection scheme for SHVC compared to previously-employed cryptographic techniques.

Fast Depth and Mode Decision in Intra Prediction for Quality SHVC.

Scalable High Efficiency Video Coding (SHVC) is the extension of High Efficiency Video Coding (HEVC). In intra prediction for quality SHVC, a Coding Unit (CU) is recursively divided into a quadtree-based structure from the largest 64×64 CU to the smallest 8×8 CU, in which 35 intra prediction modes and Inter-Layer Reference (ILR) mode are checked to determine the best possible mode. This leads to very high coding efficiency but also results in an extremely high coding complexity. To improve coding speed while maintaining coding efficiency, in this paper, we propose a new efficient algorithm for fast intra prediction for enhancement layer in SHVC. First, temporal and spatial correlations, as well as their correlation degrees, are combined in a Naive Bayes classifier to predict depth probabilities and skip depths with low likelihood. Second, for a given depth candidate, we combine ILR mode probability with Partial Zero Blocks (PZBs) based on the Sum of Squared Differences (SSD) to determine whether the ILR mode is the best one. In that case, we can skip intra prediction, which requires very high complexity. Third, initial Intra Modes (IMs) are obtained through Sobel operator, and are combined with the relationship between IMs and their corresponding Hadamard Cost (HC) values to predict candidate IMs in Rough Mode Decision (RMD). Then, an analytical criterion of early termination is developed based on the HC values of two neighboring IMs in the Rate-Distortion Optimization (RDO) process. Finally, we combine depth probabilities and the distribution of residual coefficients at the current depth to early terminate depth selection. The proposed scheme can significantly decrease the complexity of depth determination while reducing the complexity of mode decision for a depth candidate. Our experimental results demonstrate that the proposed scheme can achieve a speed up gain of more than 80% in average, while maintaining coding efficiency.

Versatile Video Coding Based Quality Scalability With Joint Layer Reference

Scalability is an essential coding feature for adaptive video streaming applications, notably considering the growing heterogeneity of the transmission, display and consumption environments. Versatile video coding (VVC) is the emerging video coding standard, targeting offering higher compression efficiency regarding previous standards to further facilitate already available and novel video applications, notably at higher spatial resolutions. In this context, this letter proposes the first VVC-based quality scalability extension, targeting to offer higher compression efficiency than the native VVC quality scalability solution. The proposed Quality Scalable Versatile Video Coding (QS-VVC) solution is designed based on a layered coding approach with one base layer (BL) and one or more enhancement layers (EL). To achieve higher compression performance, a novel joint layer referencing approach is proposed where the base and enhancement layers decoded information are jointly exploited to create a new EL coding reference. Experimental results shown that the proposed QS-VVC codec outperforms the most relevant benchmarks, notably VVC-based simulcasting, native VVC quality scalability, and the previous Scalable High Efficiency Video Coding (SHVC) standard.

VRFCNN: Virtual Reference Frame Generation Network for Quality SHVC

For more efficient inter prediction in quality scalable high efficiency video coding (SHVC), a learning-based framework for generating a virtual reference frame (VRF) is proposed in this letter. In our method, reconstructed base layer (BL) and enhancement layer (EL) frames are employed to make the generated VRF and current EL frame as same as possible. To this end, this letter proposes a novel VRF generation convolutional neural network (VRFCNN) to jointly handle enhancement of corresponding BL frame and compensation of previous EL frame. Specifically, the VRFCNN consists of BL enhancement, EL compensation and feature fusion subnets. Previous EL frames are firstly compensated with the learned coarse flow between two adjacent BL frames and then adopted to provide interlayer information for corresponding BL enhancement. The learned finer flow between the EL and enhanced BL features is adopted to provide temporal information for previous EL compensation. For efficiently handling slow- and fast-motion videos, the enhanced BL and compensated EL features are fused to generate a VRF. Experimental results show that VRFCNN averagely achieves 11.8% BD-rate reduction under low delay P configuration, which outperforms other methods. The code of our VRFCNN approach is available at https://github.com/dq0309/VRFCNN.

SHVC CU Processing Aided by a Feedforward Neural Network

The development of multimedia and hardware technologies has led to a great number of industrial video applications, such as virtual reality, high-definition video surveillance, and remote monitoring. As complex communication environments and heterogeneous networks are common in industrial applications, industrial videos are required to support a diverse range of display resolutions and transmission channel capacities. Scalable high-efficiency video coding (SHVC) standards provide the tools to meet this requirement. However, it is highly computationally expensive. Coding complexity has a great impact on SHVC performance in industrial applications. Many of these applications are sensitive to time delay and have limited power. Thus, improvements are required to ensure the practical usability of SHVC encoders. In SHVC encoders, intra/interprediction of variable coding unit (CU) sizes is independently performed for the base and enhancement layers (ELs). There are many interlayer similarities that can be exploited to speed up the procedure for EL coding. In this paper, we propose a feedforward neural network aided model for CU size and mode decisions for SHVC, which utilizes base layer coding information and the coding data of spatiotemporal neighboring CUs to decide which CU sizes or prediction modes can be bypassed for certain EL CUs. Two feedforward neural network based learning models are built for CU classification, which are introduced in the procedures for CU size and mode decisions, respectively. According to the analysis from a large number of video sequences, the representative features are directly extracted from the coding information of previously coded neighboring CUs to avoid computational overheads. After the training is finished, these two models are designed and integrated to build classifiers. Then, two online classification approaches are designed for the CU size and mode decision procedures to classify each CU's type. Finally, different candidate CU sizes and prediction modes are adaptively assigned for each type of CU. This approach outperforms the state-of-the-art fast SHVC/high-efficiency video coding (HEVC) algorithms with approximately 19–42% coding time savings or better compression efficiency, which will be beneficial for the realization of real-time scalable video coding.

Joint Layer Prediction for Improving SHVC Compression Performance and Error Concealment

Scalable high efficiency video coding (SHVC) standard is expected to play a more important role in the heterogeneous landscape of broadcasting, multimedia, networks, and various services applications as it is specified as a layered coding technique in the advanced television systems committee 3.0. However, its block-based structure of temporal and spatial prediction makes it sensitive to information loss and error propagation due to transmission errors. In this context, we propose an improved SHVC with a joint layer prediction (JLP) solution which adaptively combines the decoded information from the base and the enhancement layers to create an additional reference for the SHVC enhancement encoder. To optimize the quality of the joint prediction, the minimum mean square error estimation is executed in computing a combination factor which gives weights to each contribution of the decoded information from the layers. In addition, the proposed JLP is integrated into the SHVC decoder to work as an error concealment solution to mitigate the error propagation happening inevitably in practical video transmission. Experiments have shown that the proposed SHVC framework significantly outperforms its relevant benchmarks, notably by up to 14.8% in bitrate reduction with respect to the standard SHVC codec. The proposed SHVC error concealment strategy also greatly improves the concealed picture quality as well as reducing the problem of error propagation when compared to conventional error concealment approaches.

Fast Depth and Inter Mode Prediction for Quality Scalable High Efficiency Video Coding

The scalable high efficiency video coding (SHVC) is an extension of high efficiency video coding (HEVC). It introduces multiple layers and inter-layer prediction, thus significantly increases the coding complexity on top of the already complicated HEVC encoder. In inter prediction for quality SHVC, in order to determine the best possible mode at each depth level, a coding tree unit can be recursively split into four depth levels, including merge mode, inter2N×2N, inter2N×N, interN×2N, interN×N, inter2N×nU, inter2N×nD, internL×2N and internRx×2N, intra modes and inter-layer reference (ILR) mode. This can obtain the highest coding efficiency, but also result in very high coding complexity. Therefore, it is crucial to improve coding speed while maintaining coding efficiency. In this research, we have proposed a new depth level and inter mode prediction algorithm for quality SHVC. First, the depth level candidates are predicted based on inter-layer correlation, spatial correlation and its correlation degree. Second, for a given depth candidate, we divide mode prediction into square and non-square mode predictions respectively. Third, in the square mode prediction, ILR and merge modes are predicted according to depth correlation, and early terminated whether residual distribution follows a Gaussian distribution. Moreover, ILR mode, merge mode and inter2N×2N are early terminated based on significant differences in Rate Distortion (RD) costs. Fourth, if the early termination condition cannot be satisfied, non-square modes are further predicted based on significant differences in expected values of residual coefficients. Finally, inter-layer and spatial correlations are combined with residual distribution to examine whether to early terminate depth selection. Experimental results have demonstrated that, on average, the proposed algorithm can achieve a time saving of 71.14%, with a bit rate increase of 1.27%.

SHVC performance enhancement using superior step search algorithm

Abstract The SHVC (Scalable High Efficiency Video Coding) standard developed as an extension to H.265 HEVC (High Efficiency Video Coding) that focus on data transmission over heterogeneous network with wide scalability. In this paper a Superior Step Search (SSS) algorithm is proposed to enhance the PSNR performance along with coding efficiency without much increase in the computational complexity. The proposed SSS algorithm performs dynamic search to predict the best matching block using a fine search pattern. Simulation results show 0.5% average increased performance in PSNR and 32% coding efficiency in Bit rate when compared with the SHM reference software.