RD Performance Research Articles

A CTU-Level Screen Content Rate Control for Low-Delay Versatile Video Coding

In this paper, a rate control scheme for screen content video coding is proposed for the Versatile Video Coding (VVC) standard. In view of the critical challenges arising from the spatial and temporal unnaturalness of screen content sequences, the proposed method relies on the specifically designed pre-analysis such that the content information regarding the scene complexity can be obtained. As such, the estimated residual complexity is then incorporated into the proposed complexity-aware rate models and distortion models, leading to the optimal bit allocations for each frame and coding tree unit (CTU). In particular, the optimization problem can be analytically solved with the proposed models, and the coding parameters such as Lagrangian multiplier λ and quantization parameter of each frame and CTU could be delicately derived according to the allocated bits through the proposed analytical models. Extensive experiments have been conducted to evaluate the effectiveness of the proposed method. Compared to the default hierarchical λ-domain rate control and other screen content rate control algorithms, the proposed method could achieve obvious RD performance gain, and the bit-rate accuracy could be improved.

Current trends and prospects for the development of the university sector of Russian science

This study is aimed at identifying the features and prospects for the development of the university sector of Russian science, including with account for modern foreign trends. Some topical aspects in the development of university science are analyzed in detail proceeding from statistical data and the results of sociological surveys on the degree of involvement of universities and their units in RD performance, as well as on the cost structure, types and sources of research funding, and the dynamics of the development of HR potential and certain indicators of the postgraduate system of training highly qualified personnel. Based on the international comparisons presented in this article, the authors consider the expediency and validity of the broad participation of universities in the implementation of RD and, ultimately, the ability to provide a high level of graduate training under such conditions. Since Russia lacks higher education establishments specializing mainly in basic research, it is concluded that it is necessary to create research universities on the basis of divisions and scientific centers of the Russian Academy of Sciences, endowed with the right to develop standards and programs for training highly qualified scientific personnel independently. It is argued that to implement the relevant tasks of the country’s socioeconomic development, it is advisable to develop alliances and consortia of scientific, educational, and business organizations.

A Method to Reduce the Intra-Frame Prediction Complexity of HEVC Based on D-CNN

Among a series of video coding standards jointly developed by ITU-T, VCEG, and MPEG, high-efficiency video coding (HEVC) is one of the most widely used video coding standards today. Therefore, it is still necessary to further reduce the coding complexity of HEVC. In the HEVC standard, a flexible partitioning procedure entitled “quad-tree partition” is proposed to significantly improve the coding efficiency, which, however, leads to high coding complexity. To reduce the coding complexity of the intra-frame prediction, this paper proposes a scheme based on a densely connected convolution neural network (D-CNN) to predict the partition of coding units (CUs). Firstly, a densely connected block was designed to improve the efficiency of the CU partition by fully extracting the pixel features of CTU. Then, efficient channel attention (ECA) and adaptive convolution kernel size were applied to a fast CU partition for the first time to capture the information of the D-CNN convolution channels. Finally, a threshold optimization strategy was formulated to select the best threshold for each depth to further balance the computation complexity of video coding and the performance of RD. The experimental results show that the proposed method reduces the encoding time of HEVC by 60.14%, with a negligible reduction in RD performance, which is better than the existing fast partitioning methods.

Fast CU Partitioning Algorithm for VVC Based on Multi-Stage Framework and Binary Subnets

VVC is the latest video compression technology available, and while the coding efficiency has improved significantly over the previous generation of standards, it has also led to a dramatic increase in coding complexity. As VVC uses a QTMT division structure, the more flexible division structure also allows for a significant increase in coding time. We have built a multi-stage network framework to solve the above problem by dividing the CU into different stages according to the size of the blocks. The desired features are extracted by dynamically adjusting to the size of the input CU. Secondly, we construct a binary classification subnet to perform the classification task at each stage and can determine the QT and MT division decisions. Finally, the resulting experimental results can demonstrate that our novel two-threshold decision scheme can achieve a balance between RD performance and TS. Our method succeeds in reducing the coding time by 49.08% to 52.56%, while the complexity of the negligible BD-BR increases by only 1.10% to 1.42%.

PTR-CNN for in-loop filtering in video coding

A deep learning method called PTR-CNN (Predicted frame with Transform unit partition and prediction Residual aided CNN) is proposed for in-loop filtering in video compression. To reduce the computational complexity of an end-to-end CNN in-loop filter, a non-learning method of reference frame selection is designed to select the highest quality frame based on the frame’s blurriness and smoothiness scores. The transform unit (TU) partition and the prediction residual (PR) of the current frame are used as extra inputs to the neural network as the filtering guidance. The selected similar and high quality reference frame (RF) and the current unfiltered frame (CUF) are input to a CNN based motion compensation module to generate a predicted frame (PF). Finally input the PF, the CUF, the CUF’s TU partition and the CUF’s PR into the main CNN to reconstruct the filtered frame. The model is implemented in Tensorflow and tested in HEVC and AV1. Experimental results show that the complexity of proposed PTR-CNN is less than SOTA CNN-based reference aided in-loop filtering methods and slightly outperforms their RD performance. The scheme introduces a complexity overhead of 7% on the encoder. In particular, for random access, the proposed model achieves 11.78% coding gain over HEVC with DBF/SAO off, while has a gain of 4.76% over HEVC with DBF/SAO on. Ablation study demonstrates that the RF contributes about 10% of the total gain, and the TU and PR contribute over 4% of the total one, proving the effectiveness of each module. Moreover, it is observed that the proposed method can restore detailed structures and textures and hence improve the subjective quality.

Intra Complexity Control Algorithm for VVC

Versatile Video Coding (VVC) has advantages over High Efficiency Video Coding (HEVC); it can save nearly half of the bit rate and significantly improve the compression efficiency, but VVC’s coding complexity is extremely high. Therefore, VVC encoders are difficult to implement in video devices with different computing capabilities and power constraints. In this paper, we apply texture information and propose a VVC intra complexity control algorithm. The algorithm assigns a different encoding time to each CU based on the corresponding texture entropy. Besides, the complexity reduction strategy at the CU level is designed to balance the complexity control while taking rate-distortion performance into consideration. Experiments in our paper show that the coding complexity can be accurately controlled from 90% to 70% with a slight loss of RD performance.

Sensorimotor and working memory systems jointly support development of perceptual rhythm processing.

We studied the role of sensorimotor and working memory systems in supporting development of perceptual rhythm processing with 119 participants aged 7-12years. Children were assessed for their abilities in sensorimotor synchronization (SMS; beat tapping), auditory working memory (AWM; digit span), and rhythm discrimination (RD; same/different judgment on a pair of musical rhythm sequences). Multiple regression analysis revealed that children's RD performance was independently predicted by higher beat tapping consistency and greater digit span score, with all other demographic variables (age, sex, socioeconomic status, music training) controlled. The association between RD and SMS was more robust in the slower tempos (60 and 100 beats-per-minute (BPM)) than faster ones (120 and 180BPM). Critically, the relation of SMS to RD was moderated by age in that RD performance was predicted by beat tapping consistency in younger children (age: 7-9 years), but not in older children (age: 10-12years). AWM was the only predictor of RD in older children. Together, the current findings demonstrate that the sensorimotor and working memory systems jointly support RD processing during middle-to-late childhood and that the degree of association between the two systems and perceptual rhythm processing is shifted before entering into early adolescence.

Notice of Violation of IEEE Publication Principles: VVC/H.266 Intra Mode QTMT Based CU Partition Using CNN

Notice of Violation of IEEE Publication Principles <br><br>“VVC/H. 266 Intra Mode QTMT Based CU Partition Using CNN” <br>by Sameena Javaid, Safdar Rizvi, Muhammad Talha Ubaid, Abdullah Tariq <br>in IEEE Access, April 2022 <br><br>After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE’s Publication Principles. <br><br>This paper copied content from the article below. The original text was copied without attribution (including appropriate references to the original author(s) and/or paper title) and without permission. <br><br> “DeepQTMT: A Deep Learning Approach for Fast QTMT-Based CU Partition of Intra-Mode VVC” <br> by Tianyi Li, Mai Xu, Runzhi Tang, Ying Chen, Qunliang Xing <br>in IEEE Transactions on Image Processing, Vol 30, 2021 <br><br> <br/> The latest standard for video coding is versatile video coding (VVC) / H.266 which is developed by the joint video exploration team (JVET). Its coding structure is a multi-type tree (MTT) structure, which consists of two types of trees that are Ternary Tree (TT) and Binary Tree (BT). Due to the use of brute force quest for residual rate-distortion the quad-tree and multi-type tree (QTMT) structure of the coding unit (CU) split and contributes over 98% of the encoding time. This structure is efficient in coding, however, increases computational complexity. The current paper proposes a deep learning technique to predict the QTMT based CU split rather than just the brute-force QTMT method to substantially speed up the time of the encoding process for VVC/H.266 intra mode. In the first phase, we developed an extensive database containing ample CU splitting patterns and various streaming videos. In the second phase, we suggest a multi-level exit CNN (MLE-CNN) model with a redundancy removal mechanism at different levels to determine the CU partition. In the third phase, for the training of MLECNN model we have established the adaptive loss function and analyzing the both unknown number of partition modes and the focus on RD cost minimization. Finally, a variable threshold decision system is established to achieve the targeted low complexity and RD performance. Ultimately experimental findings show that VVC/H.266 encoding time has reduced to 69.11% from 47.91% with insignificant bjontegaard delta bit rate (BDBR) to 2.919% from 1.023% which performs better than the existing futuristic and modern approaches.

Low-Complexity Rate-Distortion Optimization for HEVC Encoders

Typical high efficiency video coding (HEVC) encoders use rate-distortion optimization (RDO) to select the best coding parameters among numerous candidates to achieve high compression efficiency. However, calculating the rate-distortion cost for each coding parameter entails high computational complexity, which limits the real-time application of these encoders. In this study, time-consuming processes, such as reconstruction and entropy coding, are removed during the calculation of the full RD cost to reduce the complexity of the encoder. Instead, RD costs are estimated for transformed coefficients by using the RD cost model, which is made up of a distortion model based on the distribution features of the transformed coefficients, a residual rate model based on the characteristics of the entropy coding module, and a simple and convenient header rate model. In addition to the full RD cost, typical encoders also adopt fast RD cost, which is composed of the sum of absolute transform differences (SATD) and header bit. To further reduce encoding complexity, three simplified SATD (SSATD) methods based on DCT and DST features are proposed to serve as substitutes for SATD in fast RD cost, thus reducing the computational complexity of SATD calculation. Experimental results show that 35.53%, 14.57%, and 14.37% of coding time can be reduced respectively under AI, RA and LDP configurations with negligible RD performance loss when applying the proposed RD cost model and the SSATD methods to HM16.7. Besides, the proposed methods also can reduce 4%-9% of encoding time and exhibits the same RD performance as state-of-the-art methods.

Parallel spiral search algorithm applied to integer motion estimation

Thanks to its flexible coding structure, high-efficiency video coding (HEVC) can save more coding bit rates than the previous standard, H.264. However, it also increases the complexity of integer-pixel motion estimation (IME). To speed up the encoding process, we propose a parallel spiral search (PSS) algorithm, which features the following characteristics and advantages. First, the proposed algorithm is hardware-friendly. PSS has both a fix search order that cuts the correlation between search points and a high data reuse level that facilitates the pipeline application in hardware implementation. Second, the PSS algorithm processes all prediction units (PU) blocks in parallel, which speeds up the RD calculation. Finally, the early termination strategy is proposed to end the search for unnecessary search points and further reduce search time. Experimental results show that the proposed algorithm outperforms other popular hardware-oriented IME algorithms in terms of coding speed, with the same loss of RD performance. Compared with the default full search algorithm (FSA) in the HEVC test model HM-16.7, the proposed algorithm achieves average time saving ratio of up to 92.55%, with BD-PSNR loss of 0.056 dB and an increase by 1.38% in terms of BD-BR.

CNN-LSTM Learning Approach-Based Complexity Reduction for High-Efficiency Video Coding Standard

High-Efficiency Video Coding provides a better compression ratio compared to earlier standard, H.264/Advanced Video Coding. In fact, HEVC saves 50% bit rate compared to H.264/AVC for the same subjective quality. This improvement is notably obtained through the hierarchical quadtree structured Coding Unit. However, the computational complexity significantly increases due to the full search Rate-Distortion Optimization, which allows reaching the optimal Coding Tree Unit partition. Despite the many speedup algorithms developed in the literature, the HEVC encoding complexity still remains a crucial problem in video coding field. Towards this goal, we propose in this paper a deep learning model-based fast mode decision algorithm for HEVC intermode. Firstly, we provide a deep insight overview of the proposed CNN-LSTM, which plays a kernel and pivotal role in this contribution, thus predicting the CU splitting and reducing the HEVC encoding complexity. Secondly, a large training and inference dataset for HEVC intercoding was investigated to train and test the proposed deep framework. Based on this framework, the temporal correlation of the CU partition for each video frame is solved by the LSTM network. Numerical results prove that the proposed CNN-LSTM scheme reduces the encoding complexity by 58.60% with an increase in the BD rate of 1.78% and a decrease in the BD-PSNR of -0.053 dB. Compared to the related works, the proposed scheme has achieved a best compromise between RD performance and complexity reduction, as proven by experimental results.

A Bit Allocation Method Based on Inter-View Dependency and Spatio-Temporal Correlation for Multi-View Texture Video Coding

Due to the limited storage capacity and network bandwidth, an efficient rate control (RC) algorithm becomes more and more critical for multi-view video coding (MVC). Based on inter-view dependency and spatio-temporal correlation, a novel bit allocation method for multi-view texture video coding is proposed in this article. Firstly, considering that the distortion in the base view (BV) is directly transmitted to the dependent view (DV) by inter-view skip mode, a joint multi-view RD model is built based on the inter-view dependency. Based on the proposed joint multi-view RD model, a precise power model is derived to represent the target bitrates relationship between the BV and the DV. Secondly, since the P frame in the DV (P-DV) is mainly predicted from the corresponding I frame in the BV (I-BV) by disparity compensated prediction (DCP), the constant proportional relationship between the ratio of the average bitrates of the P-DV to the corresponding I-BV and the ratio of the total bitrates of the DV to the BV is discovered. Based on this discovery, a novel linear model is developed to assign the target bitrates of the P-DV. Finally, considering the spatio-temporal correlation, a new parameter prediction method is proposed for the R-$\lambda $ model in coding tree unit (CTU) level. Extensive experimental results show that the proposed overall method outperforms other state-of-the-art algorithms in terms of RD performance.

Convolutional neural network based low complexity HEVC intra encoder

Video coding is one of the key technologies of visual sensors. As the state-of-art video coding standard, High Efficiency Video Coding (HEVC) achieves a significant high compression ratio for video. However, it also introduces heavy computational complexity, leading to challenges in application of visual sensors. To reduce the complexity of HEVC intra encoder, this paper proposed a one-stage decision method of CU/PU partition and prediction mode for intra coding. First, the potential factors that may related to the corresponding decisions in CU/PU are explored. Based on this, a one-stage decision network (OSDN) structure is specially designed to determine these decisions. Consequently, the complexity of HEVC intra coding can be drastically reduced by avoiding the brute-force search. Then, OSDN is embedded into the HEVC reference software HM 15.0. Thresholds are set to let the encoder switch between OSDN and the original implementation in HEVC to obtain the final decisions. The experimental results show that the proposed method can reduce 73.69% intra encoding time with 0.1673 dB BD-PSNR loss on average. In addition, the trade-off between RD performance degradation and complexity reduction can be controlled by thresholds.

Study Regarding Research and Development Activity in Korea: Centered on RD Performance Rate

Study Regarding Research and Development Activity in Korea: Centered on RD Performance Rate

Viewport-adaptive 360-degree video coding

360-degree videos contain an omnidirectional view with ultra-high resolution, which will lead to the bandwidth-hungry issue in virtual reality (VR) applications. However, only a part of a 360-degree video is displayed on the head-mounted displays (HMDs). Thus, we propose a viewport-adaptive 360-degree video coding approach based on a novel viewport prediction strategy. Specifically, we firstly introduce a novel viewport prediction model based on deep 3-dimensional convolutional neural networks. In this model, a video saliency encoder and a trajectory encoder are trained to extract the features of video content and the history view path. With the outputs of the two encoders, a video prior analysis network is trained to adaptively determine the best fusion weight to generate the final feature. Moreover, benefiting from the viewport prediction model, a viewport-adaptive rate-distortion optimization (RDO) method is presented to decrease the bitrate and ensure an immersive experience. In addition, we also consider the scaling factor of the area from rectangular plane to spherical surface. Therefore, the Lagrange multiplier and quantization parameter are adaptively adjusted based on the weight of each coding tree unit. The experiments have demonstrated that the proposed RDO method gains considerably better RD performance than the traditional RDO method.

Light Field Compression Using Global Multiplane Representation and Two-Step Prediction

Due to its spatio-angular structure, light field image allows for a wealth of post-processing techniques like digital refocusing and depth estimation. In order to compress the data of the two domains, the current proposal intends to embed the disparity-based view synthesis method into the decoder. However, predicting each view separately or in local groups means bringing more computational burden to the decoder and destroying the light field structure. Since disparity contains the relationship between all light rays in the light field, the proposed solution is to predict a disparity-based global representation as the first step. In the second step, all the views can be predicted easily based on this representation. In this letter, we use the recently proposed multiplane as the form of this global representation. The experimental results show the effectiveness of the proposed solution, and the better RD performance compared to other schemes especially under low bitrates.

Efficient In-loop Filtering Based on Enhanced Deep Convolutional Neural Networks for HEVC.

The raw video data can be compressed much by the latest video coding standard, high efficiency video coding (HEVC). However, the block-based hybrid coding used in HEVC will incur lots of artifacts in compressed videos, the video quality will be severely influenced. To settle this problem, the in-loop filtering is used in HEVC to eliminate artifacts. Inspired by the success of deep learning, we propose an efficient in-loop filtering algorithm based on the enhanced deep convolutional neural networks (EDCNN) for significantly improving the performance of in-loop filtering in HEVC. Firstly, the problems of traditional convolutional neural networks models, including the normalization method, network learning ability, and loss function, are analyzed. Then, based on the statistical analyses, the EDCNN is proposed for efficiently eliminating the artifacts, which adopts three solutions, including a weighted normalization method, a feature information fusion block, and a precise loss function. Finally, the PSNR enhancement, PSNR smoothness, RD performance, subjective test, and computational complexity/GPU memory consumption are employed as the evaluation criteria, and experimental results show that when compared with the filter in HM16.9, the proposed in-loop filtering algorithm achieves an average of 6.45% BDBR reduction and 0.238 dB BDPSNR gains.

Low complexity high efficiency coding of light fields using ensemble classifiers

Light field images can be efficiently compressed using standard video codecs, such as the High Efficiency Video Coding (HEVC). However, the huge amount of data combined with the high computational complexity of HEVC, poses limitations on high-speed light field capturing and storage. This paper presents a contribution for low complexity encoding of light fields, in different formats using HEVC, based on a Random Forests ensemble classifier. Optimal features for training the classifier are found through a score fusion based approach. Using the HEVC still image profile, the proposed method gives speed-up of 56.23% for sub-aperture images. For pseudo video format, the proposed method outperforms others available in the literature, yielding an average speed-up of 62.18%, 56.54% and 44.73% for Random Access, Low-delay Main and All-Intra profiles respectively, with negligible decrease in RD performance. These are novel results in fast coding of light fields, which are useful for further research and benchmarking.

Successive refinement of side information frames in distributed video coding

In distributed video coding, generation of high-quality side information is required for reliable soft-input information for decoding each DCT band of a Wyner-Ziv (WZ) frame in the decoder, which in turn leads to a more efficient decoding. Consequently, less error-correcting bits need to be transmitted from the encoder to the decoder to decode the bitplanes of each DCT band, leading to a better compression efficiency and rate-distortion performance. In this paper, we investigate the problem of successive improvements in the quality of side information frames in distributed video coding in order to improve the rate-distortion performance. A new algorithm for the successive refinement of the side information is proposed to refine the initial side information frame using the additional information obtained after decoding the previous DCT bands of a WZ frame. As more information about the WZ frame becomes available after the decoding of each DCT band of the WZ frame, the corresponding side information frame is refined and then employed to decode the next DCT band of the WZ frame. Simulations are carried out demonstrating that the proposed algorithm for refinement of side information frame results in a considerable improvement in the RD performance of distributed video coding.

Hybrid All Zero Soft Quantized Block Detection for HEVC.

Transform and quantization account for a considerable amount of computation time in video encoding process. However, there are a large number of discrete cosine transform coefficients which are finally quantized into zeros. In essence, blocks with all zero quantized coefficients do not transmit any information, but still occupy substantial unnecessary computational resources. As such, detecting all-zero block (AZB) before transform and quantization has been recognized to be an efficient approach to speed up the encoding process. Instead of considering the hard-decision quantization (HDQ) only, in this paper, we incorporate the properties of soft-decision quantization into the AZB detection. In particular, we categorize the AZB blocks into genuine AZBs (G-AZB) and pseudo AZBs (P-AZBs) to distinguish their origins. For G-AZBs directly generated from HDQ, the sum of absolute transformed difference-based approach is adopted for early termination. Regarding the classification of P-AZBs which are generated in the sense of rate-distortion optimization, the rate-distortion models established based on transform coefficients together with the adaptive searching of the maximum transform coefficient are jointly employed for the discrimination. Experimental results show that our algorithm can achieve up to 24.16% transform and quantization time-savings with less than 0.06% RD performance loss. The total encoder time saving is about 5.18% on average with the maximum value up to 9.12%. Moreover, the detection accuracy of larger TU sizes, such as and can reach to 95% on average.