Scalable High Efficiency Video Coding Research Articles

4K Real Time Software Solution of Scalable HEVC for Broadcast Video Application

Scalable high-efficiency video coding (SHVC) is the scalable extension of the high-efficiency video coding (HEVC) standard. SHVC enables spatial, quality, bit-depth, color gamut, and codec scalability. The architecture of the SHVC encoder is based on multiple instances of the HEVC encoder where each instance encodes one video layer. This architecture offers several advantages of being modular and close to the native HEVC coding block scheme. However, the close-loop SHVC architecture requires the complete decoding of the reference lower layer frames to decode a higher quality layer, which considerably increases the complexity of both encoder and decoder processes. In this paper, we propose an end-to-end 4K real-time SHVC solution, including both software encoder and decoder, for video broadcast applications. The SHVC codec relies on low-level optimizations for specific Intel ×86 platform and parallel processing to speed up the encoding and decoding processes. The proposed encoder enables real-time processing of 4Kp30 video in 2× spatial scalabilities on the 4 × 10 cores Intel Xeon processor (E5-4627V3) running at 2.6 GHz. In addition, the SHVC decoder enables to decode, respectively, the lower quality layer in full HD (1920 × 1080p30) resolution, an advanced RISC machine (ARM) Neon mobile platform, and the enhancement layer in UHD (3840 × 2160p30), on a fitted laptop, with 4 cores Intel i7 processor running at 2.7 GHz. Finally, experimental results have shown that the proposed solution can reach a high rate-distortion performance close to the reference SHVC reference software model (SHM) with a speedup of 37 and 66 in intra and inter coding configurations.

Fast Intra Coding of High Dynamic Range Videos in SHVC

Compared with the conventional standard dynamic range (SDR) content, high dynamic range (HDR) content supplies viewers with more immersive experience by offering a much higher range of luminance. Most of current consumer devices cannot afford to this emerging technology, and content providers decide to create both an HDR version and an SDR version of the same video. In this letter, scalable high efficiency video coding (HEVC) scalable extension of HEVC (SHVC) serves as the coding framework where the base layer (BL) is an 8-b SDR version and the enhancement layer (EL) is a 12-b HDR version. Recently, many fast coding algorithms for SDR videos are proposed, and there is an urgent demand for fast coding algorithms for EL HDR videos. With the coding information of the BL SDR videos, this letter proposes a fast algorithm to reduce the complexity of intra coding for EL HDR videos. First, depth information of neighboring coding tree units (CTUs) in the HDR version and the colocated CTU in the SDR version is used for early coding unit (CU) depth determination. Moreover, four classifiers are trained to predict the CTU depth range. Two classifiers are trained for CTUs in frames with a high average luma, and another two classifiers are used for CTUs in frames with a low average luma. Experimental results show that the proposed algorithm achieves 43% encoding time saving on average, with only a 0.54% Bjøntegaard delta bit rate (BDBR) increase compared to the original SHVC test model.

Efficient early termination and bypass scheme for CU size decision in spatially scalable SHVC intra coding

As an extension of the High Efficiency Video Coding standard, Scalable High-efficiency Video Coding (SHVC) has been developed to satisfy the challenges posed by heterogeneous environments. A layer-based scheme is employed to support various forms of scalability, but this results in significantly increased computational complexity. A fast Coding Unit (CU) size decision algorithm is proposed to alleviate the additional computational requirement of the Enhancement Layers (ELs). The inter-layer dependency and the spatial correlation are jointly used to skip and early terminate the CU size decision process. In addition, the quantitive relationship between the texture homogeneity and the AC coefficient energy is investigated to further narrow the depth range to be evaluated. Experimental results show that the proposed scheme saves encoding time by up to 72% compared with the SHM 12.0 implementation, with negligible degradation in Rate-Distortion performance.

A 120 fps High Frame Rate Real-time HEVC Video Encoder with Parallel Configuration Scalable to 4K

This paper describes a new 120 fps (frames per second) real-time HEVC (High Efficiency Video Coding) encoder for HFR (high frame rate) video encoding and transmission. HFR provides more immersive viewing experience features by solving the problems created by fast moving scenes. Temporally scalable encoding with backward compatibility for legacy non-HFR systems is suitable for the rapid spread of HFR content delivery, avoiding the need to distribute multiple bitstreams of the same video with different frame rates. Such temporal scalability requires flexible encoder control functionalities to support newly-customized reference picture structures and dual-stream bitrate control. In this paper, modification in the customizable software architecture of encoder LSIs makes it possible to achieve 120 fps temporally scalable HEVC encoding for existing 60 fps-based systems. The encoder also achieves ${4\mathrm{K}/ 120\;\mathrm{fps}}$ video encoding in real time through the synchronized operation of multiple ${2\mathrm{K}/ 120\;\mathrm{fps}}$ encoders working in parallel. Our evaluations show that the bitrate increase rate from 60 fps to 120 fps under the same objective image quality condition are within the range of less than 57.2 percent in all video sequences and its average value is 53.8 percent. Both values are lower than that of the HM (HEVC reference software). The proposed encoder systems will open the door to the next generation high frame rate UHDTV (ultra high definition television) services.

Fast CU size decision and PU mode decision algorithm for quality SHVC inter coding

As a scalable extension of the High Efficiency Video Coding (HEVC), the Scalable High Efficiency Video Coding (SHVC) encoder needs to encode multiple HEVC layers with Inter-layer predictions, which causes the significant increase in coding complexity. In this paper, we proposed a novel Inter prediction scheme to effectively reduce computational complexity in Quality SHVC. The new features of the proposed algorithm include: First, spatial and Inter-layer depth correlations are used to predict the most possible coding unit (CU) depth level candidates. Second, a statistical test method on the current CU depth level is introduced to examine whether the residual coefficients within its block present similar distribution to terminate depth selection early. Finally, during Inter prediction selection from 8 Prediction Unit (PU) sizes, spatial and Inter-layer correlations are combined with residual coefficients distribution to determine the PU partitioning mode is Symmetric Motion Partitioning (SMP) or Asymmetric Motion Partitioning (AMP). Experimental results demonstrate that the proposed algorithm can save an average of 65.33% coding time of enhancement layer (EL) while achieving a better rate distortion (RD) performance over other state-of-the-art work.

Fast inter-prediction algorithms for spatial Scalable High efficiency Video Coding SHVC

Scalable High Efficiency Video Coding (SHVC) has aggregated an exhaustive algorithm for mode decision based on a recursive quad-tree structure coding unit. However, this comes at the cost of high computational complexity. Therefore, in order to improve the SHVC in terms of encoding time, fast inter-mode decision algorithms for spatial scalability were suggested in this paper. This study focuses on using statistical analysis and the correlation between base layer and enhanced layer modes to speed up the mode decision algorithm for inter-mode. Consequently, the proposed method concerns only the inter-prediction module. It is devoted to three principal levels: CUs level, PUs, and skip mode. Experimental results for different proposed methods showed an important improvement in terms of encoding time which is the most important constraint for any real-time implementation. Moreover, the combination of these three methods could achieve up to 48% of time reduction with insignificant loss in video quality.

Fuzzy logic-based scalable video rate control algorithm for high-delay applications of scalable high-efficiency video coding

The second version of the high-efficiency video coding standard introduced a scalable extension, which is called scalable high-efficiency video coding. It is necessary to design appropriate rate controllers for this extension based on the specific application requirements. According to delay and buffering constraints, rate control algorithms are divided into constant bit rate and variable bit rate (VBR) algorithms. Our focus is on high-delay applications like streaming and broadcast for which VBR controllers are suitable. Our proposed VBR controller utilizes a fuzzy logic-based algorithm to control long-term average bit rate while preventing buffer overflow or underflow. Targeting perceptual quality, the proposed algorithm has been implemented at the group of pictures level to minimize unnecessary fluctuations of quantization parameter and consequently minimizing video quality fluctuations. Experimental results show a high performance in terms of rate control and quality of the compressed video for the proposed algorithm.

FPGA Implementation of the IEEE 802.16 Physical Layer for SHVC Video Transmission

This paper presents a video transmission system for scalable High-Efficiency Video Coding (HEVC) videos using a 4G standard’s physical layer. SHVC, the scalable HEVC is used to compress the different layers of videos into binary files. The resultant binary files are easily transportable over any network thus solving many issues mainly related to videos with high resolutions. Three scenarios are studied and simulated at first, namely, Single Input Single Output (SISO), Multiple Input Single Output (MISO) and MIMO. Since the MIMO scenario offers the best results, it is considered in the implementation of the system on Field Programmable Gate Array (FPGA) using Xilinx System Generator (XSG). A Simulink model is developed under Matlab to simulate the video transmission scenarios using the WIMAX physical layer. Then, the MIMO system is implemented using a Zed-Board to co-simulate the video transmission in real-time and which allows a successful reception of the video sequences.

Efficient Transmission of Multiple Broadcasting Services Using LDM and SHVC

In this paper, we present an efficient transmission method of multiple broadcasting services using layered division multiplexing (LDM) and scalable high efficiency video coding (SHVC) based on the next generation terrestrial digital broadcasting standard, advanced television systems committee 3.0. In a two-layer LDM system, each layer forms a physical layer pipe that carries base layer and enhancement layer video streams of SHVC, respectively. This combination of physical and presentation layers technologies can maximize channel utilization when multiple broadcasting services are delivered in a single radio frequency channel with different robustness and reception conditions. An intensive performance analysis is provided when the proposed combination of LDM and SHVC is compared with other approaches, such as time division multiplexing and/or high efficiency video coding simulcast broadcasting. Several performance measurements are also introduced, such as constant data rate measurement and constant quality measurement, which are used to evaluate the performance of video codecs in conjunction with different physical layer system parameters. A prototype hardware system with LDM and SHVC capable of selecting different parameter combinations is tested in the laboratory and in real field environments to verify the performance and feasibility of the proposed LDM and SHVC combination. Results show that the proposed LDM and SHVC combination provides significant gains on video service quality as well as reception robustness.

Boundary correlation-based intracoding for SHVC algorithm and its efficient VLSI architecture

Scalable high-efficiency video coding (SHVC) can provide variable video quality according to terminal devices. However, a computational complexity of SHVC is increased by introducing new techniques based on high-efficiency video coding (HEVC). In this paper, a hardware-oriented low complexity algorithm is proposed for the reference software of SHVC (SHM11.0). In our proposed algorithm, an optimal coding unit depth is determined by analyzing the boundary correlation in a coding tree unit before encoding starts. Simulation results show that the proposed algorithm can achieve over 62% computation complexity reduction comparing to the original SHM11.0. Compared with other related work, over 11% time saving has been achieved without PSNR loss. Moreover, to confirm the efficacy of the proposed algorithm, a hardware architecture is designed targeting on the CU depth decision algorithm. Synthesis results show that the hardware cost is about 1.8K gate and achieve a scalable working clock frequency in the case of FPGA (CycloneV) implementation.

A Joint Framework for QoS and QoE for Video Transmission over Wireless Multimedia Sensor Networks

With the emergence of Wireless Multimedia Sensor Networks (WMSNs), the distribution of multimedia contents have now become a reality. Without proper management, the transmission of multimedia data over WMSNs affects the performance of networks due to excessive packet-drop. The existing studies on Quality of Service (QoS) mostly deal with simple Wireless Sensor Networks (WSNs) and as such do not account for an increasing number of sensor nodes and an increasing volume of data. In this paper, we propose a novel framework to support QoS in WMSNs along with a light-weight Error Concealment (EC) scheme. The EC schemes play a vital role to enhance Quality of Experience (QoE) by maintaining an acceptable quality at the receiving ends. The main objectives of the proposed framework are to maximize the network throughput and to cover-up the effects produced by dropped video packets. To control the data-rate, Scalable High efficiency Video Coding (SHVC) is applied at multimedia sensor nodes with variable Quantization Parameters (QPs). Multi-path routing is exploited to support real-time video transmission. Experimental results show that the proposed framework can efficiently adjust large volumes of video data under certain network distortions and can effectively conceal lost video frames by producing better objective measurements.

Scalable Video Coding Based on User’s View for Real-Time Virtual Reality Applications

The transmission of virtual reality (VR) videos requires huge bandwidth, which brings great challenges for the system to implement real-time applications. This letter proposes a scalable full-panorama video coding method to adapt the insufficient bandwidth, in which user’s movement information is utilized as a feedback from the VR device to the video encoder. The regions that the user is interested in are first mapped and then coded in high quality, while the others in low quality. This different-quality coding is achieved through scalable high efficiency video coding (SHVC), which only makes limited modifications on the SHVC encoder and no modification on the SHVC decoder. In addition, full-panorama coding also avoids the reference problem in inter prediction and relieves the delay-sensitive problem. Experiments results show that our proposed method reduces approximately 87% bit rate with no significant decrease in quality of the region of interest in panorama.

Computing Offloading Over mmWave for Mobile VR: Make 360 Video Streaming Alive

Compared with previous video streaming technologies, 360 video streaming is an emerging technology in commercial platforms that promise greater immersive video service. The use of 360 video streaming requires larger bandwidth and rapid responsiveness to users. In addition, mobile virtual-reality (VR) devices locally process video decoding, post-processing, and rendering. However, its performance is insufficient to stream high-resolution videos such as 4k–8k. Therefore, we propose an adaptive computing offloading scheme using millimeter wave (mmWave) communication. This offloading scheme helps mobile device sharing video decoding tasks to a powerful GPU-based PC. It allows the mobile VR device enhancing the capability of playing high-resolution videos. The mmWave 802.11ad wireless technology promises the use of high-bandwidth wireless communication to improve the capacity of multimedia systems. However, the current studies on video transmission over 802.11ad mmWave are specifically focused on supporting the outdoor environment. Furthermore, synchronization is an essential issue for ensuring the quality of service (QoS) of 360 video transmission. Thus, the proposed scheme concentrates on two parts: 1) 360 video streaming in an indoor environment, and offloading mechanism using mmWave 802.11ad 60-GHz wireless communication to offload video coding tasks and 2) adaptive video synchronization mechanisms, and high-efficiency video coding (HEVC) using scalable HEVC extension for 360 videos with equirectangular projection. The experiments prove that the proposed scheme provides high performance 360 video streaming in QoS-sensitive video streaming applications.

Performance Evaluation of Frame-loss Error-concealment Solutions for the SHVC Standard

Scalable High Efficiency Video Coding (SHVC) has been emerging as an efficient video coding solution for adaptive video streaming and conferencing. SHVC is designed with a layered coding structure in which High Efficiency Video Coding (HEVC) tools are employed as core elements. However, compression efficiency and error sensitivity associated with efficient HEVC tools make this scalable codec less attractive for practical video transmission, especially when the loss of packets or video frames occurs. This problem severely impacts the display quality of received video. To address this problem, we propose efficient error concealment (EC) methods that compensate for a whole frame that is lost and that mitigate the error propagation problem occurring in practical video transmissions using SHVC. The presented EC methods mainly rely on the decoded information; thus, it is easily integrated into the SHVC decoder as a post-processing component. The experimental results obtained for both subjective and objective quality assessments show that the inter-layer correlation-based EC approach typically provides the highest concealed frame quality; thus, it is highly recommended for practical video transmission.

Efficient Multi-Strategy Intra Prediction for Quality Scalable High Efficiency Video Coding.

As an extension of High Efficiency Video Coding (HEVC), the Scalable High Efficiency Video Coding (SHVC) introduces multiple layers with inter-layer predictions, which greatly increases the complexity on top of the already complicated HEVC encoder. In Intra prediction for Quality SHVC, Coding Tree Unit (CTU) allows recursive splitting into four depth levels, which considers 35 Intra prediction modes and interlayer reference (ILR) mode to determine the best possible mode at each depth level. This achieves the highest coding efficiency but incurs a substantially high computational complexity. In this paper, we propose a novel Intra prediction scheme to effectively speed up the enhancement layer Intra-coding in Quality SHVC. The new features of the proposed framework include: First, spatial correlation and its correlation degree are combined to predict most probable depth level candidates. Second, for a given depth candidate, based on the probabilities of the ILR mode, we check the ILR mode by examining the residual distribution based on skewness and kurtosis to determine whether the residuals follow a Gaussian distribution. In that case, the Intra prediction comparisons, which require a high complexity, are skipped. Third, during Intra prediction selection from 35 Intra prediction modes, spatial and inter-layer correlations are combined with the local monotonicity of the Hadamard costs associated with the modes in a small neighborhood, to examine only a portion of Intra prediction modes. Finally, a hypothesis testing on the currently selected depth level is performed to examine whether the residuals present significant differences within their block to early terminate depth selection. The proposed multi-step multistrategy scheme aims to minimize the number of depth selections while greatly reducing the mode decision complexity for a depth candidate in a hierarchical fashion. Our experimental results demonstrate that the proposed scheme can achieve a speedup gain of more than 75% in average on the test video sequences, while maintaining almost the same coding efficiency. .

Adaptive Scalable Video Coding: An HEVC-Based Framework Combining the Predictive and Distributed Paradigms

The emerging scalable High Efficiency Video Coding (SHVC) video coding standard provides an efficient solution for transmission of video over heterogeneous and time dynamic networks, terminals, and usage environments. The encoding complexity and the error sensitivity associated with the efficient HEVC coding tools adopted in SHVC make this scalable codec less attractive to some emerging applications such as video surveillance, visual sensor network, and remote space transmission where these requirements are critical. To address the requirements of these application scenarios including scalability, this paper proposes a novel HEVC-based framework offering quality scalability on top of an HEVC compliant base layer while appropriately combining the predictive and distributed coding paradigms. To achieve the best enhancement layer compression efficiency, two novel coding tools are proposed, notably a machine learning-based side information creation mechanism and an adaptive correlation modeling process. The experimental results reveal that the rate-distortion performance of the proposed distributed scalable video coding-HEVC solution outperforms the relevant alternative coding solutions, notably by up to 52.9% and 23.7% BD-rate gains regarding the HEVC-Simulcast and SHVC standard solutions, respectively, for an equivalent prediction configuration, while achieving a lower encoding complexity.

Adaptive Bit Allocation for Consistent Video Quality in Scalable High Efficiency Video Coding

Scalable video coding (SVC) is a coding paradigm that allows once-encoded video content to be used in diverse scenarios. SVC-coded videos can be transmitted and rendered at specified bitrates according to network bandwidth and end device requirements. In this paper, an adaptive bit allocation algorithm is proposed for the emerging scalable High Efficiency Video Coding (SHVC) standard. The bit budget at the group-of-pictures level is allocated according to buffer occupancy. Picture complexity, measured using the predicted mean absolute difference (MAD), buffer occupancy, and hierarchical level, is proposed for regulating the bitrate at the picture level. The MAD of the current picture is predicted using a novel mean prediction error (MPE) model, which is obtained from the advanced motion vector prediction, and the test zone search specified in SHVC and the associated reference software of SHVC. Moreover, MPE is used to determine the number of assigned bits at the coding-tree-unit level. The bit budget of each level is incorporated with the $R-\lambda $ model for computing the required quantization parameter. Experimental results reveal that the proposed method achieves accurate bitrates with enhanced and consistent visual quality and more satisfactorily controls buffer occupancy compared with the state-of-the-art approaches reported in the literature.

Real-time selective video encryption based on the chaos system in scalable HEVC extension

In this paper we propose a real-time selective video encryption solution in the scalable extension of High Efficiency Video Coding (HEVC) standard, referred to as SHVC. The proposed scheme encrypts a set of sensitive SHVC parameters with a minimum delay and complexity overheads. The encryption process is performed at the CABAC binstring level and fulfills both constant bitrate and format compliant video encryption requirements. In addition, it preserves all SHVC functionalities, including bitstream extraction for mid-network adaptation and error resilience.We compare the performance of three selective SHVC encryption schemes: the first scheme encrypts only the lowest SHVC layer, the second encrypts all layers and the last scheme encrypts only the highest layer. The performance of the proposed schemes is assessed over different video encryption criteria, at different scalability configurations and various High Definition (HD) video sequences. Experimental results showed that encrypt only the lowest layer or all layers enables a high security level, while encrypting only the highest layer leads to a perceptual encryption solution, by slightly decreasing the highest layer quality. Moreover, the processing complexity of the proposed solution is assessed in the context of a real-time SHVC decoder. The complexity overhead remains low and does not exceed 6% of the real-time decoding of SHVC video sequences.

Efficient temporal and interlayer parameter prediction for weighted prediction in scalable high efficiency video coding

Weighted prediction (WP) is an efficient video coding tool that was introduced since the establishment of the H.264/AVC video coding standard, for compensating the temporal illumination change in motion estimation and compensation. WP parameters, including a multiplicative weight and an additive offset for each reference frame, are required to be estimated and transmitted to the decoder by slice header. These parameters cause extra bits in the coded video bitstream. High efficiency video coding (HEVC) provides WP parameter prediction to reduce the overhead. Therefore, WP parameter prediction is crucial to research works or applications, which are related to WP. Prior art has been suggested to further improve the WP parameter prediction by implicit prediction of image characteristics and derivation of parameters. By exploiting both temporal and interlayer redundancies, we propose three WP parameter prediction algorithms, enhanced implicit WP parameter, enhanced direct WP parameter derivation, and interlayer WP parameter, to further improve the coding efficiency of HEVC. Results show that our proposed algorithms can achieve up to 5.83% and 5.23% bitrate reduction compared to the conventional scalable HEVC in the base layer for SNR scalability and 2× spatial scalability, respectively.

Depth Partitioning and Coding Mode Selection Statistical Analysis for SHVC

The Scalable High Efficiency Video Coding (SHVC) has been proposed to improve the coding efficiency. However, this additional extension generally results an important coding complexity. Several studies were performed to overcome the complexity through algorithmic optimizations that led to an encoding time reduction. In fact, mode decision analysis is imperatively important in order to have an idea about the partitioning modes based on two parameters, such as prediction unit size and frame type. This paper presents statistical observations at two levels: coding units (CUs) and prediction units (PUs) selected by the encoder. Analysis was performed for several test sequences with different motion and texture characteristics. The experimental results show that the percentage of choosing coding or prediction unit size and type depends on sequence parameters, frame type, and temporal level.