Bjontegaard Delta Bit Research Articles

Convex Hull Prediction for Adaptive Video Streaming by Recurrent Learning.

Adaptive video streaming relies on the construction of efficient bitrate ladders to deliver the best possible visual quality to viewers under bandwidth constraints. The traditional method of content dependent bitrate ladder selection requires a video shot to be pre-encoded with multiple encoding parameters to find the optimal operating points given by the convex hull of the resulting rate-quality curves. However, this pre-encoding step is equivalent to an exhaustive search process over the space of possible encoding parameters, which causes significant overhead in terms of both computation and time expenditure. To reduce this overhead, we propose a deep learning based method of content aware convex hull prediction. We employ a recurrent convolutional network (RCN) to implicitly analyze the spatiotemporal complexity of video shots in order to predict their convex hulls. A two-step transfer learning scheme is adopted to train our proposed RCN-Hull model, which ensures sufficient content diversity to analyze scene complexity, while also making it possible to capture the scene statistics of pristine source videos. Our experimental results reveal that our proposed model yields better approximations of the optimal convex hulls, and offers competitive time savings as compared to existing approaches. On average, the pre-encoding time was reduced by 53.8% by our method, while the average Bjøntegaard delta bitrate (BD-rate) of the predicted convex hulls against ground truth was 0.26%, and the mean absolute deviation of the BD-rate distribution was 0.57%.

Learning-Based Early Transform Skip Mode Decision for VVC Screen Content Coding

One of the design goals of the recently published international video coding standard, Versatile Video Coding (VVC/H.266), is efficient coding of computer-generated video content (commonly referred to as screen content) which exhibits different signal characteristics from the usual camera-captured video (commonly referred as natural content). VVC can perform transform in multiple different ways including skipping the transform itself, which demands much computation for its best selection among many combinatory options. In this paper, we investigate designing a machine-learning-based early transform skip mode decision (ML-TSM) which makes a determination whether or not to skip the transform in an early stage by making a simple classification employing key features designed in such a way to reflect the characteristics of TSM blocks well. Compared with the VVC reference software 14.0, the proposed scheme is verified to reduce computational complexity by 11% and 4% with a Bjontegaard delta bitrate (BDBR) increase of 0.34% and 0.23% respectively under all-intra (AI) and random-access (RA) configurations.

Complexity-Efficient Dependent Quantization for Versatile Video Coding

The dependent quantization (DQ) technique in Versatile Video Coding (VVC) provides high compression efficiency, but it suffers from much computation, which is not preferred in practical applications. In this context, we propose a complexity-efficient dependent quantization structure employing a machine learning-based DQ start position prediction and a simplified trellis structure. Firstly, the proposed DQ start position is decided by a binary classifier which finds an appropriate location to start the DQ process. To further reduce the processing time of the binary classification, we also set up a precondition with a learned preset threshold which can reduce the number of coefficients subjected to the classifier. Based on our statistical analysis showing that under certain conditions some branches of the trellis structure are not necessary to check, we also design a branch pruning method and a removal scheme of all-zero coefficient group checking to simplify the DQ trellis structure. Our proposed complexity-efficient DQ has much fewer calculations of rate-distortion costs. Our experimental results show that it can reduce the computational complexity of the DQ process with nearly no performance loss of Bjontegaard delta bit rate (BDBR) irrespective of the all intra or the random access configurations.

An Adaptive CU Split Method for VVC Intra Encoding

In order to reduce the coding complexity and ensure that versatile video coding (H.266/VVC) effectively maintains the equilibrium between coding efficiency and quality in the coding process, an adaptive coding unit (CU) split method for intra encoding is proposed in this paper. First, the coding speed of CU division is accelerated by an iterative algorithm based on the interval relation of texture complexity energy. Then, according to the relationship between texture features and CU segmentation patterns, the Scharr edge gradient operator algorithm is used to describe the texture information and the threshold. Texture uniformity, directionality, and subblock discrepancy are analyzed and discussed, respectively. Finally, the optimal segmentation pattern is determined based on the texture information to adaptively skip unnecessary segmentation modes. The results show that the algorithm can save 45.2% of the overall coding time on average with an average cost of 0.51% Bjontegaard delta bit rate (BDBR). The algorithm ensures the coding quality while reducing the computational complexity.

A Hardware-Friendlyand High-Efficiency H.265/HEVC Encoder for Visual Sensor Networks.

Visual sensor networks (VSNs) have numerous applications in fields such as wildlife observation, object recognition, and smart homes. However, visual sensors generate vastly more data than scalar sensors. Storing and transmitting these data is challenging. High-efficiency video coding (HEVC/H.265) is a widely used video compression standard. Compare to H.264/AVC, HEVC reduces approximately 50% of the bit rate at the same video quality, which can compress the visual data with a high compression ratio but results in high computational complexity. In this study, we propose a hardware-friendly and high-efficiency H.265/HEVC accelerating algorithm to overcome this complexity for visual sensor networks. The proposed method leverages texture direction and complexity to skip redundant processing in CU partition and accelerate intra prediction for intra-frame encoding. Experimental results revealed that the proposed method could reduce encoding time by 45.33% and increase the Bjontegaard delta bit rate (BDBR) by only 1.07% as compared to HM16.22 under all-intra configuration. Moreover, the proposed method reduced the encoding time for six visual sensor video sequences by 53.72%. These results confirm that the proposed method achieves high efficiency and a favorable balance between the BDBR and encoding time reduction.

Plenoptic Point Cloud Compression Using Multiview Extension of High Efficiency Video Coding

Plenoptic point clouds are more complete representations of three-dimensional (3-D) objects than single-color point clouds, as they can have multiple colors per spatial point, representing colors of each point as seen from different view angles. They are more realistic but also involve a larger volume of data in need of compression. Therefore, in this paper, a multiview-video-based framework is proposed to better exploit correlations in color across different viewpoints. To the best of the authors' knowledge, this is the first work to exploit correlations in color across different viewpoints using a multiview-video-based framework. In addition, it is observed that some unoccupied pixels, which do not have corresponding points in plenoptic point clouds and are of no use to the quality of the reconstructed plenoptic point cloud colors, may cost many bits. To address this problem, a block-based group smoothing and a combined occupancy-map-based rate distortion optimization and four-neighbor average residual padding are further proposed to reduce the bit cost of unoccupied color pixels. The proposed algorithms are implemented in the moving pictures experts group (MPEG) video-based point cloud compression (V-PCC) and multiview extension of High Efficiency Video Coding (MV-HEVC) reference software. The experimental results show that the proposed algorithms can lead to a Bjontegaard Delta bitrate (BD-rate) reduction of 40% compared with the RAHT-KLT. Compared with the V-PCC independently applied to each view direction, the proposed algorithms can provide a BD-rate reduction of over 70%.

Unified Cross-Component Linear Model in VVC Based on a Subset of Neighboring Samples

To compress industrial video content efficiently, H.266/Versatile Video Coding (VVC) introduces cross-component linear model (CCLM) prediction as a new coding tool, in which chroma components are predicted from the luma component based on a linear model. In this article, we propose a subset-based CCLM (S-CCLM), in which the model parameters are derived based on a subset of neighboring samples. To choose the most proper subset, we build the relationship between the prediction error and the geometric distance and resolve the optimal subset construction problem by minimizing the geometric distance. With the well-designed subset, a weight-guided parameter derivation algorithm is further proposed to improve the accuracy of the model parameters. The experimental results show that the proposed S-CCLM can achieve Bjontegaard delta bitrate (BD-rate) reductions of 0.14%, 0.64%, and 0.75% for the Y, Cb, and Cr components, respectively, when the number of samples in the subset, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> , is 4 and BD-rate reductions of 0.22%, 0.80%, and 0.95% when <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> is 8. Given a small fixed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> , fewer memory access operations are needed during the CCLM calculation, and a unified CCLM process can be achieved for coding blocks with different sizes and different modes. Due to its hardware-friendly architecture, the S-CCLM has been partially adopted by H.266/VVC.

QTMT‐LNN: A fast intra CU partition using lightweight neural network for 360‐degree video coding on VVC

In the newest generation of video coding standard, Versatile Video Coding (VVC), a new technique called Quad Tree with nested Multi-type Tree (QTMT) structure is introduced. QTMT significantly improves the coding efficiency, but the improvement in compression performance comes at the cost of drastically increased complexity. This paper proposes a fast intra partition algorithm using Lightweight Neural Network (LNN) to skip QTMT partition steps which are unlikely to be chosen as the best split modes. Specifically, five LNNs (for five QTMT split modes) are trained, using features that consider the characteristic of 360-degree videos. The experimental results demonstrate that the proposed QTMT-LNN can reduce the encoding time from 52.28% to 72.17% on average, with coding efficiency losses ranging from 0.71% to 1.78%. Compared to other fast algorithms for intra coding unit (CU) partition, the method outperforms the related works in terms of Bjontegaard delta bit-rate (BDBR) and encoding time reduction.

Temporal Prediction Model-Based Fast Inter CU Partition for Versatile Video Coding.

Versatile video coding (VVC) adopts an advanced quad-tree plus multi-type tree (QTMT) coding structure to obtain higher compression efficiency, but it comes at the cost of a considerable increase in coding complexity. To effectively reduce the coding complexity of the QTMT-based coding unit (CU) partition, we propose a fast inter CU partition method based on a temporal prediction model, which includes early termination QTMT partition and early skipping multi-type tree (MT) partition. Firstly, according to the position of the current CU, we extract the optimal CU partition information of the position corresponding to the previously coded frames. We then establish a temporal prediction model based on temporal CU partition information to predict the current CU partition. Finally, to reduce the cumulative of errors of the temporal prediction model, we further extract the motion vector difference (MVD) of the CU to determine whether the QTMT partition can be terminated early. The experimental results show that the proposed method can reduce the inter coding complexity of VVC by 23.19% on average, while the Bjontegaard delta bit rate (BDBR) is only increased by 0.97% on average under the Random Access (RA) configuration.

Fast Decision Algorithm of CU Size for HEVC Intra-Prediction Based on a Kernel Fuzzy SVM Classifier

High Efficiency Video Coding (HEVC) achieves a significant improvement in compression efficiency at the cost of extremely high computational complexity. Therefore, large-scale and wide deployment applications, especially mobile real-time video applications under low-latency and power-constrained conditions, are more challenging. In order to solve the above problems, a fast decision method for intra-coding unit size based on a new fuzzy support vector machine classifier is proposed in this paper. The relationship between the depth levels of coding units is accurately expressed by defining the cost evaluation criteria of texture and non-texture rate-distortion cost. The fuzzy support vector machine is improved by using the information entropy measure to solve the negative impact of data noise and the outliers problem. The proposed method includes three stages: the optimal coded depth level “0” early decision, coding unit depth early skip, and optimal coding unit early terminate. In order to further improve the rate-distortion complexity optimization performance, more feature vectors are introduced, including features such as space complexity, the relationship between coding unit depths, and rate-distortion cost. The experimental results showed that, compared with the HEVC reference test model HM16.5, the proposed algorithm can reduce the encoding time of various test video sequences by more than 53.24% on average, while the Bjontegaard Delta Bit Rate (BDBR) only increases by 0.82%. In addition, the proposed algorithm is better than the existing algorithms in terms of comprehensively reducing the computational complexity and maintaining the rate-distortion performance.

HEVC’s intra mode process expedited using Histogram of Oriented Gradients

The brute-force behavior of High Efficiency Video Coding (HEVC) is the biggest hurdle in the communication of multimedia content. Therefore, two novel methods will be presented here to expedite the intra mode decision process of HEVC. In the first algorithm, the feasibility of Histogram of Oriented Gradients (HOG) for early intra mode decision is presented by using statistical evidence. Then, HOG of the current block and 35 intra predictions are obtained. The intra-prediction that gives the least sum of absolute difference (SAD) with the HOG of the current block is selected as the termination point. In the second algorithm, the difference between the Hardmard-cost of intra modes is modeled to achieve fast intra mode decision. The proposed algorithms accelerated the encoding process of the HEVC by 5% and 35.57%, while their Bjontegaard Delta Bit Rate (BD-BR) is 1.09% and 1.61%, respectively.

Fast coding unit partitioning algorithms for versatile video coding intra coding

Versatile video coding (VVC) is the newest video compression standard. It adopts quadtree with nested multi-type tree (QT-MTT) to encode square or rectangular coding units (CUs). The QT-MTT coding structure is more flexible for encoding video texture, but it is also accompanied by many time-consuming algorithms. So, this work proposes fast algorithms to determine horizontal or vertical split for binary or ternary partition of a 32 × 32 CU in the VVC intra coding to replace the rate-distortion optimization (RDO) process, which is time-consuming. The proposed fast algorithms are actually a two-step algorithm, including feature analysis method and deep learning method. The feature analysis method is based on variances of pixels, and the deep learning method applies the convolution neural networks (CNNs) for classification. Experimental results show that the proposed method can reduce encoding time by 28.94% on average but increase Bjontegaard delta bit rate (BDBR) by about 0.83%.

Accelerating Transform Algorithm Implementation for Efficient Intra Coding of 8K UHD Videos

Real-time ultra-high-definition (UHD) video applications have attracted much attention, where the encoder side urgently demands the high-throughput two-dimensional (2D) transform hardware implementation for the latest video coding standards. This article proposes an effective acceleration method for transform algorithm in UHD intra coding based on the third generation of audio video coding standard (AVS3). First, by conducting detailed statistical analysis, we devise an efficient hardware-friendly transform algorithm that can reduce running cycles and resource consumption remarkably. Second, to implement multiplierless computation for saving resources and power, a series of shift-and-add unit (SAU) hardwares are investigated to have much less adoptions of shifters and adders than the existing methods. Third, different types of hardware acceleration methods, including calculation pipelining, logical-loop unrolling, and module-level parallelism, are designed to efficaciously support the data-intensive high frame-rate 8K UHD video coding. Finally, due to the scarcity of 8K video sources, we also provide a new dataset for the performance verification. Experimental results demonstrate that our proposed method can effectively fulfill the real-time 8K intra encoding at beyond 60 fps, with very negligible loss on rate-distortion (R-D) performance, which is averagely 0.98% Bjontegaard-Delta Bit-Rate (BD-BR).

Tunable early CU size decision for depth map intra coding in 3D-HEVC using unsupervised learning

To further improve coding gain for depth map in 3D extension of High Efficiency Video Coding (3D-HEVC), many new coding techniques are introduced, which drastically increase the coding complexity. In this work, an early intra coding unit (CU) size decision scheme is proposed for 3D-HEVC intra depth map coding based on unsupervised learning approach. First, we treat the early CU size decision as a clustering problem. Then, three clustering models are developed for those CUs with the sizes of 64×64, 32×32 and 16×16 to early determine whether they be further split or not. Among them, the center of the clustering model is obtained by using intra learning method. Finally, in order to meet the user's specific application preference, the similarity distance is introduced into the early CU size decision. By adjusting the similarity distance, a tunable early CU size decision is achieved to obtain different levels of coding complexity reduction. Experimental results show that the proposed scheme achieves encoding time reduction ranging from 54.0% to 68.4% on average for depth map intra coding, while the Bjontegaard delta bit rate (BDBR) of synthesized views only increased by 0.01% to 0.73%, it outperforms the state-of-the-art works in term of coding complexity reduction and BDBR increase.

High Efficiency Intra CU Partition and Mode Decision Method for VVC

Versatile video coding (VVC/H.266) is the newest video compression standard, which is developed by the Joint Video Experts Team. Compared with previous encoding schemes, VVC achieves higher compression efficiency by introducing a new partition structure and additional intra prediction modes but results in high computational complexity. To efficiently solve this problem of redundant processing in quad-tree with nested multi-type tree structures and intra mode prediction, we propose a texture analysis&#x2013;based ternary tree (TT) and binary tree (BT) partition strategy, and a gradient-based intra mode decision method to accelerate TT and BT partition and intra mode prediction, separately. The texture complexity and prediction direction of coding unit (CU) is calculated by texture detection method. A texture analysis&#x2013;based TT and BT partition strategy is established by using the regression method based on analyzing the texture complexity of the CU. Then, a texture analysis&#x2013;based TT and BT partition strategy is applied to reduce the redundant partition for each CU. By using the prediction direction of CU, a gradient-based intra mode decision method is established for skipping the impossible modes for each CU. Experimental results revealed that the proposed method could save 49.49% in encoding time and increase the Bjontegaard delta bit rate (BDBR) by only 0.56%. It confirms that the proposed method achieved high efficiency and a good balance between the BDBR and time saving.

Notice of Violation of IEEE Publication Principles: VVC/H.266 Intra Mode QTMT Based CU Partition Using CNN

Notice of Violation of IEEE Publication Principles <br><br>“VVC/H. 266 Intra Mode QTMT Based CU Partition Using CNN” <br>by Sameena Javaid, Safdar Rizvi, Muhammad Talha Ubaid, Abdullah Tariq <br>in IEEE Access, April 2022 <br><br>After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE’s Publication Principles. <br><br>This paper copied content from the article below. The original text was copied without attribution (including appropriate references to the original author(s) and/or paper title) and without permission. <br><br> “DeepQTMT: A Deep Learning Approach for Fast QTMT-Based CU Partition of Intra-Mode VVC” <br> by Tianyi Li, Mai Xu, Runzhi Tang, Ying Chen, Qunliang Xing <br>in IEEE Transactions on Image Processing, Vol 30, 2021 <br><br> <br/> The latest standard for video coding is versatile video coding (VVC) / H.266 which is developed by the joint video exploration team (JVET). Its coding structure is a multi-type tree (MTT) structure, which consists of two types of trees that are Ternary Tree (TT) and Binary Tree (BT). Due to the use of brute force quest for residual rate-distortion the quad-tree and multi-type tree (QTMT) structure of the coding unit (CU) split and contributes over 98% of the encoding time. This structure is efficient in coding, however, increases computational complexity. The current paper proposes a deep learning technique to predict the QTMT based CU split rather than just the brute-force QTMT method to substantially speed up the time of the encoding process for VVC/H.266 intra mode. In the first phase, we developed an extensive database containing ample CU splitting patterns and various streaming videos. In the second phase, we suggest a multi-level exit CNN (MLE-CNN) model with a redundancy removal mechanism at different levels to determine the CU partition. In the third phase, for the training of MLECNN model we have established the adaptive loss function and analyzing the both unknown number of partition modes and the focus on RD cost minimization. Finally, a variable threshold decision system is established to achieve the targeted low complexity and RD performance. Ultimately experimental findings show that VVC/H.266 encoding time has reduced to 69.11% from 47.91% with insignificant bjontegaard delta bit rate (BDBR) to 2.919% from 1.023% which performs better than the existing futuristic and modern approaches.

Fast Intra Mode Selection in HEVC Using Statistical Model

Comprehension algorithms like High Efficiency Video Coding (HEVC) facilitates fast and efficient handling of multimedia contents. Such algorithms involve various computation modules that help to reduce the size of content but preserve the same subjective viewing quality. However, the brute-force behavior of HEVC is the biggest hurdle in the communication of multimedia content. Therefore, a novel method will be presented here to accelerate the encoding process of HEVC by making early intra mode decisions for the block. Normally, the HEVC applies 35 intra modes to every block of the frame and selects the best among them based on the RD-cost (rate-distortion). Firstly, the proposed work utilizes neighboring blocks to extract available information for the current block. Then this information is converted to the probability that tells which intra mode might be best in the current situation. The proposed model has a strong foundation as it is based on the probability rule-2 which says that the sum of probabilities of all outcomes should be 1. Moreover, it is also based on optimal stopping theory (OST). Therefore, the proposed model performs better than many existing OST and classical secretary-based models. The proposed algorithms expedited the encoding process by 30.22% of the HEVC with 1.35% Bjontegaard Delta Bit Rate (BD-BR).

Decision tree accelerated CTU partition algorithm for intra prediction in versatile video coding.

Versatile video coding (VVC) achieves enormous improvement over the advanced high efficiency video coding (HEVC) standard due to the adoption of the quadtree with nested multi-type tree (QTMT) partition structure and other coding tools. However, the computational complexity increases dramatically as well. To tackle this problem, we propose a decision tree accelerated coding tree units (CTU) partition algorithm for intra prediction in VVC. Firstly, specially designated image features are extracted to characterize the coding unit (CU) complexity. Then, the trained decision tree is employed to predict the partition results. Finally, based on our newly designed intra prediction framework, the partition process is early terminated or redundant partition modes are screened out. The experimental results show that the proposed algorithm could achieve around 52% encoding time reduction for various test video sequences on average with only 1.75% Bjontegaard delta bit rate increase compared with the reference test model VTM9.0 of VVC.

HEVC’s Intra Mode Selection Using Odds Algorithm

The brute-force behaviour of High-Efficiency Video Coding (HEVC) is the biggest hurdle in the communication of the multimedia contents. Therefore, a novel method will be presented here to expedite the intra mode decision process of HEVC. In the first step, the feasibility of the odds-algorithm for the early intra mode decision is presented by using statistical evidence. Then, various elements of odds algorithm are analyzed and then mapped to the intra mode process (elements) of HEVC. Finally, the probability required by the odds algorithm is obtained by utilizing the correlation between the current and the neighboring blocks. The proposed algorithm accelerated the encoding process of the HEVC by 25% to 35%, while the Bjontegaard Delta Bit Rate (BD-BR) is 0.95% to 1.84%.

Fast CU Partition Decision Method Based on Bayes and Improved De-Blocking Filter for H.266/VVC

The H.266/Versatile Video Coding (VVC) standard is the latest video coding standards released by the Joint Video Exploration Group (JVET). These new technologies have brought more than 40% increase in compression rate and brought huge coding complexity. In order to reduce the coding complexity of VVC, we propose a fast Coding Unit (CU) partition decision algorithm combining Bayes algorithm and Improved De-blocking Filter (IDBF). The algorithm includes two stages. In the parameter update stage, the original VVC Test Model (VTM) is used to encode every certain video frame, and the Rough Mode Decision evaluation cost (JRMD) generated during the encoding process is used to update the parameters of the Bayes algorithm. In the fast coding stage, the video frame between two parameter update frames is encoded. First, the JRMD of the current CU is calculated and compares it with the split threshold to decide whether the current CU continues to split. If the current CU continues to be split, the current CU is divided into the same 16 simulation blocks, and the IDBF is used to calculate the texture direction and texture complexity of each simulation block. According to the texture features contained in the simulation block, the texture distribution position and texture complexity in the current CU can be judged, so that some bad candidate splitting modes can be discarded. Experimental results show that the proposed method can reduce the complexity of VVC about 56.08%, while Bjontegaard delta bit rate (BDBR) only increases by 1.30%.