High Efficiency Video Coding Standard Research Articles

Efficient inter partitioning of versatile video coding based on supervised contrastive learning

Nowadays, Versatile Video Coding (VVC) has achieved a superior performance than previous video coding standard (High Efficiency Video Coding). The Quadtree with Nested Multi-Type Tree (QTMT) coding block structure can enhance the coding performance. Nevertheless, this technique also leads to the significantly increasing complexity of VVC inter coding. Therefore, complexity optimization is an urgent problem to be optimized in the market application of VVC. To solve this issue, we propose a Supervised-Contrastive-Learning-based Inter Partitioning (SCLIP) method in this paper. Firstly, we define the above complexity optimization problem as a supervised classification task. Next, we develop a SCLIP Estimation Network (SCLIPEst-Net) with a supervised contrastive learning module and a classification module. After training on a newly established dataset, the SCLIPEst-Net can reasonably predict the mode partitioning. Finally, we propose an overall SCLIP algorithm that effectively determines the inter partitions of VVC with a low computational overhead. Experimental results indicate that our method achieves 45.14% average Time Saving (TS) with a 2.40% Bjøntegaard Delta Bit Rate (BDBR) in Random Access (RA), outperforming the benchmarks.

High Performance VLSI Architecture for DC and Planar Modes of Intra Prediction in HEVC

Abstract: High Efficiency Video Coding (HEVC) achieves improved compression efficiency, but it introduces higher computational complexity due to intricate partitioning and increased angular modes in intra prediction. In this study, we propose a hardware architecture for handling the direct current (DC) and planar modes of intra prediction in the HEVC standard. To evaluate our proposed architecture, we performed synthesis using both TSMC 180nm and TSMC 90nm technologies. We later conducted physical implementation using Physical Cell 90nm technology. The results show significant improvements when moving from TSMC 180nm to TSMC 90nm technology, with the chip area reducing by approximately three times and power consumption decreasing by over six times. After completing the physical layout phase, we obtained a chip area of 112.19 mm2 and a power consumption of 3.88 mW. Comparing the synthesis results with the physical design phase, we observed a slight increase in chip area by around 1.5 times, while the power consumption decreased by approximately 0.4 times. The proposed architecture achieves a throughput of 27 pixels per clock cycle and supports a block size of 16x16, with the potential for further extension

Effects on Long-Range Dependence and Multifractality in Temporal Resolution Recovery of High Frame Rate HEVC Compressed Content

In recent years, video research has dealt with high-frame-rate (HFR) content. Even though low or standard frame rates (SFR) that correspond to values less than 60 frames per second (fps) are still covered. Temporal conversions are applied accompanied with video compression and, thus, it is of importance to observe and detect possible effects of typical compressed video manipulations over HFR (60 fps+) content. This paper addresses ultra-high-definition HFR content via Hurst index as a measure of long-range dependency (LRD), as well as using Legendre multifractal spectrum, having in mind standard high-efficiency video coding (HEVC) format and temporal resolution recovery (TRR), meaning frame upconversion after temporal filtering of compressed content. LRD and multifractals-based studies using video traces have been performed for characterization of compressed video, and they are mostly presented for advanced video coding (AVC). Moreover, recent studies have shown that it is possible to perform TRR detection for SFR data compressed with standards developed before HEVC. In order to address HEVC HFR data, video traces are analyzed using LRD and multifractals, and a novel TRR detection model is proposed based on a weighted k-nearest neighbors (WkNN) classifier and multifractals. Firstly, HFR video traces are gathered using six constant rate factors (crfs), where Hurst indices and multifractal spectra are calculated. According to TRR and original spectra comparison, a novel detection model is proposed based on new multifractal features. Also, five-fold cross-validation using the proposed TRR detection model gave high-accuracy results of around 98%. The obtained results show the effects on LRD and multifractality and their significance in understanding changes in typical video manipulation. The proposed model can be valuable in video credibility and quality assessments of HFR HEVC compressed content.

Low-Complexity Fast CU Classification Decision Method Based on LGBM Classifier

At present, the latest video coding standard is Versatile Video Coding (VVC). Although the coding efficiency of VVC is significantly improved compared to the previous generation, standard High-Efficiency Video Coding (HEVC), it also leads to a sharp increase in coding complexity. VVC significantly improves HEVC by adopting the quadtree with nested multi-type tree (QTMT) partition structure, which has been proven to be very effective. This paper proposes a low-complexity fast coding unit (CU) partition decision method based on the light gradient boosting machine (LGBM) classifier. Representative features were extracted to train a classifier matching the framework. Secondly, a new fast CU decision framework was designed for the new features of VVC, which could predict in advance whether the CU was divided, whether it was divided by quadtree (QT), and whether it was divided horizontally or vertically. To solve the multi-classification problem, the technique of creating multiple binary classification problems was used. Subsequently, a multi-threshold decision-making scheme consisting of four threshold points was proposed, which achieved a good balance between time savings and coding efficiency. According to the experimental results, our method achieved a significant reduction in encoding time, ranging from 47.93% to 54.27%, but only improved the Bjøntegaard delta bit-rate (BDBR) by 1.07%~1.57%. Our method showed good performance in terms of both encoding time reduction and efficiency.

Apron surveillance video coding based on compositing virtual reference frame with object library

AbstractA video coding framework for the apron surveillance scene has been proposed in this paper, which aims to improve coding efficiency by eliminating long‐term redundancy at the object level. To achieve this goal, this study first develops an existing block‐based hybrid video coding framework by exploiting the video redundancy on the object level to perform video coding. Second, an object‐library mechanism is designed to collect the representative object images as coding references on larger temporal and spatial scales. Finally, a virtual reference frame, which blends background and foreground references from the object library, is adaptively composited according to the video content to improve the inter‐prediction performance. Preliminary experimental results demonstrate that the proposed method achieves a high BD rate reduction of up to 23.97% in apron surveillance video sequences, compared to the standard high efficiency video coding (HEVC).

HEVC extension for phase hologram compression.

Compressing digital holograms have growing attention nowadays due to their huge amount of original data sizes. Although many progresses have been reported for full-complex holograms, the coding performance for phase-only holograms (POHs) has been quite limited so far. In this paper, we present a very efficient compression method for POHs. It extends the conventional video coding standard HEVC (High Efficiency Video Coding) in such a way that the standard can be able to compress not only the natural images but also the phase images effectively. First, we suggest a proper way to calculate differences, distances and clipped values for phase signals by considering the inherent periodicity of phases. Then, some of the HEVC encoding and decoding processes are modified accordingly. The experimental results show that the proposed extension significantly outperforms the original HEVC for POH video sequences; specifically, average BD-rate reductions of 63.3% and 65.5% are achieved in phase domain and numerical reconstruction domain, respectively. It is worth mentioning that the modified encoding & decoding processes are very minimal and also applicable to the VVC (Versatile Video Coding), which is a successor of the HEVC.

Tree Social Relations Optimization-Based ReLU-BiLSTM Framework for Improving Video Quality in Video Compression

High-Efficiency Video Coding (HEVC) has a higher coding efficiency, its encoding performance must be increased to keep up with the expanding number of multimedia applications. Therefore, this paper proposes a novel Rectified Linear Unit-Bidirectional Long Short-Term Memory-based Tree Social Relations Optimization (ReLU-BiLSTM-based TSRO) method to enhance the quality of video transmission. The significant objective of our proposed method aims in enhancing the standards of entropy encoding process in HEVC. Here, context-adaptive binary arithmetic coding (CABAC) framework which is prevalent and an improved form of entropy coding model is utilized in HEVC standards. In addition to this, the performances of the proposed method are determined by evaluating various measures such as mean square error, cumulative distribution factor, compression ratio, peak signal-to-noise ratio (PSNR) and bit error rate. Finally, the proposed method is examined with five different sequences of video from football, tennis, garden, mobile and coastguard. The performances of the proposed method are compared with various approaches, and the result analysis shows that the proposed method attained minimum mean square error (MSE) loss with maximum PSNR rate.

LOW COMPLEXITY HEVC SCALABLE ENCODER BASED ON FSS ALGORITHM

High efficiency video coding (HEVC) provides compressed video bit streams with good quality video that offers an extension of its kind-Scalable extension of high efficiency video coding (SHVC). SHVC offers delivery of compressed video bit streams over various types of networks with broader scalability. Mode decision and motion search in standard HEVC is more exhaustive and complex, and it is even more complex in the HEVC scalable extension. The encoder time increases due to the computational complexity in SHVC to find the best mode in the search algorithm. To overcome this, a forward-looking step search (FSS) algorithm is proposed to offer less computational complexity with acceptable coding efficiency. The FSS algorithm searches for the optimal matching block by fixing nine points around the assumed center point. The optimal block is determined by forward-looking each point that has minimal rate distortion cost (RDC). The simulation results show that the algorithm works better with a 0.5% increase in the peak signal-to-noise ratio (PSNR) and an encoder time savings of 21.16% with the standard SHVC SHM software.

Fast CU Partition Decision Algorithm for VVC Intra Coding Using an MET-CNN

The newest video coding standard, the versatile video coding standard (VVC/H.266), came into effect in November 2020. Different from the previous generation standard—high-efficiency video coding (HEVC/H.265)—VVC adopts a more flexible block division structure, the quad-tree with nested multi-type tree (QTMT) structure, which improves its coding performance by 24%. However, it also causes a substantial increase in computational complexity. Therefore, this paper first proposes the concept of a stage grid map, which divides the overall division of a 32 × 32 coding unit (CU) into four stages and represents it as a structured output. Second, a multi-stage early termination convolutional neural network (MET-CNN) model is devised to predict the full partition information of a CU with a size of 32 × 32. Finally, a fast CU partition decision algorithm for VVC intra coding based on an MET-CNN is proposed. The algorithm can predict all partition information of a CU with a size of 32 × 32 and its sub-CUs in one run, completely replacing the complex rate-distortion optimization (RDO) process. It also has an early exit mechanism, thereby greatly reducing the encoding time. The experimental results illustrate that the scheme proposed in this paper reduces the encoding time by 49.24% on average, while the Bjøntegaard Delta Bit Rate (BDBR) only increases by 0.97%.

Saliency-Enabled Coding Unit Partitioning and Quantization Control for Versatile Video Coding

The latest video coding standard, versatile video coding (VVC), has greatly improved coding efficiency over its predecessor standard high efficiency video coding (HEVC), but at the expense of sharply increased complexity. In the context of perceptual video coding (PVC), the visual saliency model that utilizes the characteristics of the human visual system to improve coding efficiency has become a reliable method due to advances in computer performance and visual algorithms. In this paper, a novel VVC optimization scheme compliant PVC framework is proposed, which consists of fast coding unit (CU) partition algorithm and quantization control algorithm. Firstly, based on the visual saliency model, we proposed a fast CU division scheme, including the redetermination of the CU division depth by calculating Scharr operator and variance, as well as the executive decision for intra sub-partitions (ISP), to reduce the coding complexity. Secondly, a quantization control algorithm is proposed by adjusting the quantization parameter based on multi-level classification of saliency values at the CU level to reduce the bitrate. In comparison with the reference model, experimental results indicate that the proposed method can reduce about 47.19% computational complexity and achieve a bitrate saving of 3.68% on average. Meanwhile, the proposed algorithm has reasonable peak signal-to-noise ratio losses and nearly the same subjective perceptual quality.

Fast 3D‐HEVC inter coding using data mining and machine learning

The Three-Dimensional High Efficiency Video Coding standard is a video compression standard developed based on the two-dimensional video coding standard HEVC and used to encode multi-view plus depth format video. This paper proposes an algorithm based on eXtreme Gradient Boosting to solve the problem of high inter-frame coding complexity in 3D-HEVC. Firstly, explore the correlation between the division depth of the inter-frame coding unit and the texture features in the map, as well as the correlation between the coding unit division structure between each map and each viewpoint. After that, based on the machine learning method, a fast selection mechanism for dividing the depth range of the inter-frame coding tree unit based on the eXtreme Gradient Boosting algorithm is constructed. Experimental results show that, compared with the reference software HTM-16.0, this method can save an average of 35.06% of the coding time, with negligible degradation in terms of coding performance. In addition, the proposed algorithm has achieved different degrees of improvement in coding performance compared with the related works.

Perceptual Quality Assessment of HEVC and VVC Standards for 8K Video

With the growing data consumption of emerging video applications and users’ requirement for higher resolutions, up to 8K, a huge effort has been made in video compression technologies. Recently, versatile video coding (VVC) has been standardized by the moving picture expert group (MPEG), providing a significant improvement in compression performance over its predecessor high efficiency video coding (HEVC). In this paper, we provide a comparative subjective quality evaluation between VVC and HEVC standards for 8K resolution videos. In addition, we evaluate the perceived quality improvement offered by 8K over UHD 4K resolution. The compression performance of both VVC and HEVC standards has been conducted in random access (RA) coding configuration, using their respective reference software, VVC test model (VTM-11) and HEVC test model (HM-16.20). Objective measurements, using PSNR, MS-SSIM and VMAF metrics have shown that the bitrate gains offered by VVC over HEVC for 8K video content are around 31%, 26% and 35%, respectively. Subjectively, VVC offers an average of around 41% of bitrate reduction over HEVC for the same visual quality. A compression gain of 50% has been reached for some tested video sequences regarding a Student’s t-test analysis. In addition, for most tested scenes, a significant visual difference between uncompressed 4K and 8K has been noticed.

Visual saliency guided perceptual adaptive quantization based on HEVC intra-coding for planetary images.

Due to the limited storage space of spacecraft and downlink bandwidth in the data delivery during planetary exploration, an efficient way for image compression onboard is essential to reduce the volume of acquired data. Applicable for planetary images, this study proposes a perceptual adaptive quantization technique based on Convolutional Neural Network (CNN) and High Efficiency Video Coding (HEVC). This technique is used for bitrate reduction while maintaining the subjective visual quality. The proposed algorithm adaptively determines the Coding Tree Unit (CTU) level Quantization Parameter (QP) values in HEVC intra-coding using the high-level features extracted by CNN. A modified model based on the residual network is exploited to extract the saliency map for a given image automatically. Furthermore, based on the saliency map, a CTU level QP adjustment technique combining global saliency contrast and local saliency perception is exploited to realize a flexible and adaptive bit allocation. Several quantitative performance metrics that efficiently correlate with human perception are used for evaluating image quality. The experimental results reveal that the proposed algorithm achieves better visual quality along with a maximum of 7.17% reduction in the bitrate as compared to the standard HEVC coding.

Correction to: An Efficient VLSI Architecture for Fast Motion Estimation Exploiting Zero Motion Prejudgment Technique and a New Quadrant-Based Search Algorithm in HEVC

In this manuscript, new quadrant-based search algorithm with zero motion prejudgment is proposed for motion estimation (ME) in HEVC (High Efficiency Video Coding) standard. The HEVC standard is used to obtain efficient output with low motion estimation time. The proposed quadrant-based search algorithm is a fast block matching algorithm that obtain better block matching amid the current block and reference block. The zero motion prejudgment (ZMP) method is used to find the block, whether it is motion or static and it is used for decreasing the computational complexity (CC) in the proposed quadrant-based search algorithm. The proposed quadrant-based search algorithm with ZMP technique for motion estimation in HEVC is implemented on the FPGA hardware platform. The entire architecture is executed in Verilog HDL with Virtex-5 technology and integrated with Xilinx ISE Design Suite 14.5. The results are integrated into the CIF (352 × 288 pixels) and HD (1280 × 720 pixels) video input sequence. The evaluation metrics like PSNR, Motion estimation time, sum of absolute difference (SAD) value are analyzed with existing method like hexagon, adaptive root pattern algorithm, and diamond search algorithm. Then the hardware parameters like power consumption and maximum operating frequency are measured. The hardware utilization is reduced and the power consumption of the proposed model is diminished to 0.143 W. The maximal operating frequency of the proposed model is 440.470 MHz. The experimental outcomes demonstrate that the proposed motion evaluation approach in HEVC is more effective than existing algorithms.

The Optimal Real-Time Video Encoding Scheme for Multi-Camera Panoramic System Using NVENC

Today, modern, powerful, and relatively available computer technology allows the proposal of real-time camera systems working with several video streams. There is also a permanent tendency to increase image resolution, what can be useful for example in some image processing methods used in surveillance systems. Given this, an important part of a multi-camera real-time system proposal is optimal and efficient video compression. The modern encoding standards support 4K or 8K video content. One of these standards is H.264 or HEVC (High Efficiency Video Coding) and the popular available encoding technology using GPU (Graphics Processing Unit) acceleration is NVENC (Nvidia Encoding) provided by Nvidia. Many papers deal with the comparison of encoding standards in terms of quality or bitrate savings on testing video sequences [1], [2]. However, this paper is aimed primarily at a comparison of several practical implementations of NVENC technology using different presets of H.264 and HEVC standards. The experiments are mainly focused on testing the encoder’s real-time ability to deal with single video input obtained from the industrial camera using different settings. To do that, some system performance parameters for the continuous monitoring during the NVENC encoding process were chosen. Results in tables I and II are compared, discussed and the best NVENC implementation with encoding scheme is chosen in the context of our briefly introduced novel panoramic system. Finally, the system is fully loaded by the 7 cameras, all streams are encoded using an optimal encoding scheme and a performance limit is established for our hardware configuration.

Improved HEVC video compression algorithm using low-complexity frame rate up conversion

The high-efficiency video coding (HEVC) standard can reduce the bit rate of compression video. However, the compression efficiency of HEVC is still challenging current internet bandwidth. This problem can be solved by reducing the number of video frames encoded by HEVC and reconstructing the reduced frames that rely on frame rate up conversion (FRUC). Unfortunately, most existing FRUC methods need to re-estimate the motion information, a step that will lead to a huge increase of video decoder complexity. An improved HEVC video compression algorithm using low-complexity FRUC is proposed based on reusing the motion vector (MV) predicted by HEVC encoding procedure to directly reconstruct the skipped frames. First, an adaptive frame-skipping scheme is used to maximize reducing the number of frames that will encode by HEVC. Second, decoder extracts the MV and residual from the received bit stream. Then an MV reliability checking and rectifying method is proposed according to adaptive residual energy analyzing. Finally, a motion-compensation method and efficient occluded area processing algorithm are proposed to reconstruct skipped frames. These methods all directly utilize the rectified MVs. Experimental results show that due to the reduced frames and decoder that does not need to re-estimate MV, the proposed method can not only significantly decrease the time of all codec procedure when comparing HEVC and other FRUC-based video compression methods but also achieve 9.7022% to 66.0030% reduction in Bjøntegaard delta bit rate at low bit rates when compared to HEVC standard. In addition, the compression of the proposed method is also better than other state-of-the-art methods.

CtuNet: A Deep Learning-based Framework for Fast CTU Partitioning of H265/HEVC Intra- coding

Nowadays, real-time multimedia applications mandate high video quality while maintaining reasonable bitrates. The H.264 coding delivered inexpensive bitrate costs compared to other coding schemes while maintaining high-grade video quality, yet bounded to deliver higher qualities. Later, High-Efficiency Video Coding (HEVC) improved on H.264 by providing higher video qualities with an efficient bitrate. However, such improvement obligates higher computational expenses due to employing superior techniques like quad-tree for coding tree unit (CTU) partitioning. This paper proposes a framework, named CtuNet, for CTU partitioning by approximating its functionality using deep learning techniques. A ResNet18-CNN model is adopted to predict the CTU partition of the HEVC standard. We have baselined our suggestion with state-of-the-art approaches. The results demonstrate the supremacy of the proposed CtuNet over the other approaches. The CtuNet framework maintains near-optimal results by reducing computational complexity up to 63.68% with negligible degradation in bitrate by 1.77% at intra-prediction.

DeepQTMT: A Deep Learning Approach for Fast QTMT-Based CU Partition of Intra-Mode VVC.

Versatile Video Coding (VVC), as the latest standard, significantly improves the coding efficiency over its predecessor standard High Efficiency Video Coding (HEVC), but at the expense of sharply increased complexity. In VVC, the quad-tree plus multi-type tree (QTMT) structure of the coding unit (CU) partition accounts for over 97% of the encoding time, due to the brute-force search for recursive rate-distortion (RD) optimization. Instead of the brute-force QTMT search, this paper proposes a deep learning approach to predict the QTMT-based CU partition, for drastically accelerating the encoding process of intra-mode VVC. First, we establish a large-scale database containing sufficient CU partition patterns with diverse video content, which can facilitate the data-driven VVC complexity reduction. Next, we propose a multi-stage exit CNN (MSE-CNN) model with an early-exit mechanism to determine the CU partition, in accord with the flexible QTMT structure at multiple stages. Then, we design an adaptive loss function for training the MSE-CNN model, synthesizing both the uncertain number of split modes and the target on minimized RD cost. Finally, a multi-threshold decision scheme is developed, achieving a desirable trade-off between complexity and RD performance. The experimental results demonstrate that our approach can reduce the encoding time of VVC by 44.65%~66.88% with a negligible Bjøntegaard delta bit-rate (BD-BR) of 1.322%~3.188%, significantly outperforming other state-of-the-art approaches.

Rate Control for Predictive Transform Screen Content Video Coding Based on RANSAC

In predictive transform video coding, optimal bit allocation and quantization parameter (QP) estimation are important to control the bit rate of blocks, frames and the whole sequence. Common solutions to this problem rely on trained models to approximate the rate-distortion (R-D) characteristics of the video content during coding. Moreover, these solutions are mainly targeted for natural content sequences, whose characteristics differ greatly from those of screen content (SC) sequences. In this article, we depart from such trained R-D models and propose a low-complexity RC method for SC sequences that leverages the availability of information about the R-D characteristics of previously coded blocks within a frame. Namely, our method first allocates bits at the frame- and block-levels based on their motion and texture characteristics. It then approximates the R-D and R-QP curves of each block by a set control points and random sample consensus (RANSAC). Finally, it computes the appropriate block-level QP values to attain a target bit rate with the minimum distortion possible. The proposed RC method is embedded into a standard High-Efficiency Video Coding (H.265/HEVC) encoder and evaluated on several SC sequences. Our results show that our method not only attains better R-D performance than that of H.265/HEVC and other methods designed for SC sequences but also attains a more constant and higher reconstruction quality on all frames.

Adaptive Content Frame Skipping for Wyner–Ziv-Based Light Field Image Compression

Light field (LF) imaging introduces attractive possibilities for digital imaging, such as digital focusing, post-capture changing of the focal plane or view point, and scene depth estimation, by capturing both spatial and angular information of incident light rays. However, LF image compression is still a great challenge, not only due to light field imagery requiring a large amount of storage space and a large transmission bandwidth, but also due to the complexity requirements of various applications. In this paper, we propose a novel LF adaptive content frame skipping compression solution by following a Wyner–Ziv (WZ) coding approach. In the proposed coding approach, the LF image is firstly converted into a four-dimensional LF (4D-LF) data format. To achieve good compression performance, we select an efficient scanning mechanism to generate a 4D-LF pseudo-sequence by analyzing the content of the LF image with different scanning methods. In addition, to further explore the high frame correlation of the 4D-LF pseudo-sequence, we introduce an adaptive frame skipping algorithm followed by decision tree techniques based on the LF characteristics, e.g., the depth of field and angular information. The experimental results show that the proposed WZ-LF coding solution achieves outstanding rate distortion (RD) performance while having less computational complexity. Notably, a bit rate saving of 53% is achieved compared to the standard high-efficiency video coding (HEVC) Intra codec.