All Intra Research Articles

Block Partitioning Information-Based CNN Post-Filtering for EVC Baseline Profile.

The need for efficient video coding technology is more important than ever in the current scenario where video applications are increasing worldwide, and Internet of Things (IoT) devices are becoming widespread. In this context, it is necessary to carefully review the recently completed MPEG-5 Essential Video Coding (EVC) standard because the EVC Baseline profile is customized to meet the specific requirements needed to process IoT video data in terms of low complexity. Nevertheless, the EVC Baseline profile has a notable disadvantage. Since it is a codec composed only of simple tools developed over 20 years, it tends to represent numerous coding artifacts. In particular, the presence of blocking artifacts at the block boundary is regarded as a critical issue that must be addressed. To address this, this paper proposes a post-filter using a block partitioning information-based Convolutional Neural Network (CNN). The proposed method in the experimental results objectively shows an approximately 0.57 dB for All-Intra (AI) and 0.37 dB for Low-Delay (LD) improvements in each configuration by the proposed method when compared to the pre-post-filter video, and the enhanced PSNR results in an overall bitrate reduction of 11.62% for AI and 10.91% for LD in the Luma and Chroma components, respectively. Due to the huge improvement in the PSNR, the proposed method significantly improved the visual quality subjectively, particularly in blocking artifacts at the coding block boundary.

Fast intra coding in AVS3 based on direct non-first pre-coding skip

The third generation of Audio Video Coding Standard (AVS3) adopts sophisticated block partitioning structure with a multi-type tree, which includes Quad-Tree (QT), Binary-Tree (BT), and Extend Quad-Tree (EQT). During multi-type tree recursive block partitioning and pre-coding process of encoding, each Coding Unit (CU) of each Size at each Location (CUeSeL) of the picture is repeatedly pre-coded many times when visiting each branch of each tree. Although there are early termination mechanism, the number of visits for each CUeSeL is still very high and up to more than fifty. This process brings very high coding complexity. In order to reduce the coding complexity of AVS3, this paper proposes a method named Direct Non-First Pre-coding Skip (DNFPS) for a CUeSeL to directly and completely skip all non-first pre-coding under specific conditions. The conditions include, but are not limited to, those related to the first pre-coding. The contributions of the paper are as follows. In I pictures, for a CUeSeL with top-left location (x, y), if both x and y are multiples of 64 or if a set of conditions related to first pre-coding is satisfied, then each non-first pre-coding is skipped, and the coding result of the first pre-coding is reused for each non-first pre-coding skipped. In non-I pictures, each non-first pre-coding is skipped, and the coding result of the first pre-coding is reused for each non-first pre-coding skipped, provided a set of conditions related to first pre-coding is satisfied. The experimental results demonstrate that DNFPS has a negligible impact on coding efficiency under All Intra (AI) configurations, and coding runtime is reduced by 16 %.

Lightweight fully connected network-based fast CU size decision for video-based point cloud compression

Video-based point cloud compression (V-PCC) utilizes high efficiency video coding (HEVC) to compress geometry and attribute videos generated from dynamic point cloud projection. However, the HEVC exhaustive coding unit (CU) size decision process is complex and hinders the real-time application of V-PCC. To reduce the coding complexity of V-PCC, this paper proposes a method that combined hand-crafted features and lightweight neural network to accurately predict the best CU partition in advance. First, we extract hand-crafted features, including direct features (DFs) and indirect feature (IF), as mixed features. DFs are simple and require no additional calculation, while IF is obtained indirectly by transforming the global and local distortions of the CU extracted before size decision determination. Second, we propose a lightweight fully connected network (LFCN) as the backbone network, two feature types are used as inputs to the LFCN to predict whether the CU should be split into sub-CUs, and the LFCN can be fully integrated into the encoder with only about 1.58KB of additional parameters. Experimental results show that the proposed method reduces coding complexity by an average of 51.2% while Luma’s BD-TotalRate only increases by 0.1% on average under the All Intra (AI) configuration.

Learning-Based Early Transform Skip Mode Decision for VVC Screen Content Coding

One of the design goals of the recently published international video coding standard, Versatile Video Coding (VVC/H.266), is efficient coding of computer-generated video content (commonly referred to as screen content) which exhibits different signal characteristics from the usual camera-captured video (commonly referred as natural content). VVC can perform transform in multiple different ways including skipping the transform itself, which demands much computation for its best selection among many combinatory options. In this paper, we investigate designing a machine-learning-based early transform skip mode decision (ML-TSM) which makes a determination whether or not to skip the transform in an early stage by making a simple classification employing key features designed in such a way to reflect the characteristics of TSM blocks well. Compared with the VVC reference software 14.0, the proposed scheme is verified to reduce computational complexity by 11% and 4% with a Bjontegaard delta bitrate (BDBR) increase of 0.34% and 0.23% respectively under all-intra (AI) and random-access (RA) configurations.

Multisensor Collaboration Network for Video Compression Based on Wavelet Decomposition

Multi-sensor data has significant advantages over single sensor data due to the unique property of each sensor. We discover that multi-sensor collaboration can save bits and achieve stable compression performance according to quantization parameter (QP). In this paper, we propose a multi-sensor collaboration network for video compression based on wavelet decomposition, called MSCN. We introduce MSCN into 3D video coding based on color and depth sensors. The images acquired by a color sensor represent color and texture of the scene, while the images obtained by a depth sensor represent 3D geometric shape of the scene objects. Two sensor data are complementary, and color images help to reconstruct their corresponding depth images. First, we perform uniform sampling on the input depth video. Then, we compress the color and downsampled depth videos using 3D-HEVC codec. Finally, we reconstruct the depth video from the decoded color and depth videos by color guided depth super-resolution (SR). Experimental results show that MSCN achieves average BD-rate reductions of {−9.3%, −65.6%, −66.3%} and {−6.2%, −67.7%, and −69.2%} on 3D-HEVC test datasets for sampling factors 1, 2 and 4 in Random Access (RA) and All Intra (AI) configurations, respectively. Moreover, they verify that multi-sensor collaboration remarkably saves bits in video compression.

Region-based Template Matching Prediction for Intra Coding.

Copy prediction is a renowned category of prediction techniques in video coding where the current block is predicted by copying the samples from a similar block that is present somewhere in the already decoded stream of samples. Motion-compensated prediction, intra block copy, template matching prediction etc. are examples. While the displacement information of the similar block is transmitted to the decoder in the bit-stream in the first two approaches, it is derived at the decoder in the last one by repeating the same search algorithm which was carried out at the encoder. Region-based template matching is a recently developed prediction algorithm that is an advanced form of standard template matching. In this method, the reference area is partitioned into multiple regions and the region to be searched for the similar block(s) is conveyed to the decoder in the bit-stream. Further, its final prediction signal is a linear combination of already decoded similar blocks from the given region. It was demonstrated in previous publications that region-based template matching is capable of achieving coding efficiency improvements for intra as well as inter-picture coding with considerably less decoder complexity than conventional template matching. In this paper, a theoretical justification for region-based template matching prediction subject to experimental data is presented. Additionally, the test results of the aforementioned method on the latest H.266/Versatile Video Coding (VVC) test model (version VTM-14.0) yield an average Bjøntegaard-Delta (BD) bit-rate savings of -0.75% using all intra (AI) configuration with 130% encoder run-time and 104% decoder run-time for a particular parameter selection.

High Quality Video Frames From VVC: A Deep Neural Network Approach

In recent years, video content has become a significant contributor to Internet traffic, prompting the development of efficient codecs, such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC), to reduce bandwidth usage and storage requirements. However, these video coding standards still exhibit quality degradation and artifacts in the decoded frames. To address this issue, researchers have introduced several network architectures based on deep-learning algorithms; however, most of them focus on in-loop filtering, which requires additional bits to transmit filter information from the encoder to the decoder under a video-coding framework. In this paper, we propose a neural-network-based post-processing method to enhance the decoded frames. In the experimental result, the proposed model achieves a significant bitrate reduction, as measured by Bjøntegaard Delta of 4.54%, 4.13%, and 5.21% for random access (RA), low-delay (LD), and all-intra (AI) configurations, respectively, while also improving peak signal-to-noise ratio (PSNR).

Dynamic convolutional capsule network for In‐loop filtering in HEVC video codec

Recently, several in-loop filtering algorithms based on convolutional neural network (CNN) have been proposed to improve the efficiency of HEVC (High Efficiency Video Coding). Conventional CNN-based filters only apply a single model to the whole image, which cannot adapt well to all local features from the image. To solve this problem, an in-loop filtering algorithm based on a dynamic convolutional capsule network (DCC-net) is proposed, which embeds localized dynamic routing and dynamic segmentation algorithms into capsule network, and integrate them into the HEVC hybrid video coding framework as a new in-loop filter. The proposed method brings average 7.9% and 5.9% BD-BR reductions under all intra (AI) and random access (RA) configurations, respectively, as well as, 0.4 dB and 0.2 dB BD-PSNR gains, respectively. In addition, the proposed algorithm has an outstanding performance in terms of time efficiency.

Editorial Real-Time Implementation of VVC Standard for Consumer Electronic Devices

We are pleased to present this Special Issue on this timely and important topic. The next generation ISO/IEC MPEG and ITU-T VCEG video coding standard called versatile video coding (VVC) was released in July 2020 <xref ref-type="bibr" rid="ref1">[1]</xref>, <xref ref-type="bibr" rid="ref2">[2]</xref>. Gls vvc includes some coding tools enabling an important coding gains estimated to 35&#x0025; of bit-rate reductions for the same quality in terms of peak signal to noise ration (PSNR) compared with high efficiency video coding (HEVC) in random access (RA) coding configurations. For all intra (AI) configuration the improvement is a bit lower and estimated in a 25&#x0025;. Moreover, subjective comparison recently conducted between HEVC and the VVC reference software, has shown that this gain is even higher reaching up to 50&#x0025; of bit-rate reduction for the same perceived quality <xref ref-type="bibr" rid="ref3">[3]</xref> - <xref ref-type="bibr" rid="ref4">[4]</xref>. Unfortunately the coding gain is associated to an increase in the algorithm complexity at both encoder and decoder sides. The VVC encoder is estimated to be <inline-formula> <tex-math notation="LaTeX">$8\times $ </tex-math></inline-formula> more complex than HEVC in RA configuration and <inline-formula> <tex-math notation="LaTeX">$25\times $ </tex-math></inline-formula>. The complexity of the VVC decoder is doubled compared to HEVC. Therefore, complexity reduction and optimization of VVC is critical to design of software and hardware real time VVC codecs for successful deployment on video consumer devices including among others TVs, mobile phones and tablets.

Object Detection-Based Video Compression

Video compression is designed to provide good subjective image quality, even at a high-compression ratio. In addition, video quality metrics have been used to show the results can maintain a high Peak Signal-to-Noise Ratio (PSNR), even at high compression. However, there are many difficulties in object recognition on the decoder side due to the low image quality caused by high compression. Accordingly, providing good image quality for the detected objects is necessary for the given total bitrate for utilizing object detection in a video decoder. In this paper, object detection-based video compression by the encoder and decoder is proposed that allocates lower quantization parameters to the detected-object regions and higher quantization parameters to the background. Therefore, better image quality is obtained for the detected objects on the decoder side. Object detection-based video compression consists of two types: Versatile Video Coding (VVC) and object detection. In this paper, the decoder performs the decompression process by receiving the bitstreams in the object-detection decoder and the VVC decoder. In the proposed method, the VVC encoder and decoder are processed based on the information obtained from object detection. In a random access (RA) configuration, the average Bjøntegaard Delta (BD)-rates of Y, Cb, and Cr increased by 2.33%, 2.67%, and 2.78%, respectively. In an All Intra (AI) configuration, the average BD-rates of Y, Cb, and Cr increased by 0.59%, 1.66%, and 1.42%, respectively. In an RA configuration, the averages of ΔY-PSNR, ΔCb-PSNR, and ΔCr-PSNR for the object-detected areas improved to 0.17%, 0.23%, and 0.04%, respectively. In an AI configuration, the averages of ΔY-PSNR, ΔCb-PSNR, and ΔCr-PSNR for the object-detected areas improved to 0.71%, 0.30%, and 0.30%, respectively. Subjective image quality was also improved in the object-detected areas.

Inverse Transform Using Linearity for Video Coding

In hybrid block-based video coding, transform plays an important role in energy compaction. Transform coding converts residual data in the spatial domain into frequency domain data, thereby concentrating energy in a lower frequency band. In VVC (versatile video coding), the primary transform is performed using DCT-II (discrete cosine transform type 2), DST-VII (discrete sine transform type 7), and DCT-VIII (discrete cosine transform type 8). Considering that DCT-II, DST-VII, and DCT-VIII are all linear transforms, inverse transform is proposed to reduce the number of computations by using the linearity of transform. When the proposed inverse transform using linearity is applied to the VVC encoder and decoder, run-time savings can be achieved without decreasing the coding performance relative to the VVC decoder. It is shown that, under VVC common-test conditions (CTC), average decoding time savings values of 4% and 10% are achieved for all intra (AI) and random access (RA) configurations, respectively.

Joint Local and Nonlocal Progressive Prediction for Versatile Video Coding.

In the latest video coding standard, namely Versatile Video Coding (VVC), more directional intra modes and reference lines have been utilized to improve prediction efficiency. However, complex content still cannot be predicted well with only the adjacent reference samples. Although nonlocal prediction has been proposed to further improve the prediction efficiency in existing algorithms, explicit signalling or matching error potentially limits the coding efficiency. To address these issues, we propose a joint local and nonlocal progressive prediction scheme, targeting at improving nonlocal prediction accuracy without additional signalling. Specifically, template matching based prediction (TMP) is conducted firstly to derive an initial nonlocal predictor. Based on the first prediction and previously decoded reconstruction information, a local template, including inner textures and neighboring reconstruction, is carefully designed. With the local template involved in nonlocal matching process, a more accurate nonlocal predictor can be found progressively in the second prediction. Finally, the coefficients from the two predictions are fused and transmitted in bitstreams. In this way, more accurate nonlocal predictor can be derived implicitly with local information instead of being explicitly signalled. Experimental results on the reference software VTM-9.0 of VVC show that the method achieves 1.02% BD-Rate reduction for natural sequences and 2.31% BD-Rate reduction for screen content videos under all intra (AI) configuration.

Enhanced Intra String Copy for Screen Content Coding in AVS3

Driven by growing applications that use computer screens as interfaces for daily remote interactions, almost all current video coding standards have included screen content coding (SCC) tools. Recently, an efficient SCC tool called intra string copy (ISC) has been adopted in the third-generation of audio video coding standard in China (AVS3). ISC has two coding unit (CU) level sub-modes: fully-matching-string and partially-matching-string based string prediction (FPSP) sub-mode and equal-value-string, unit-basis-vector-string, and unmatched-pixel-string based string prediction (EUSP) sub-mode. To further improve the coding efficiency of SCC, this paper proposes four enhancement techniques of ISC (EISC), including CU partition improvements, point vector (PV) relocation and reactivation, line-based overlapping string prediction, and an optimized coding method for string length in the EUSP sub-mode. Compared with the latest AVS3 reference software HPM with EISC disabled, using AVS3 SCC common test condition and YUV test sequences in text and graphics with motion and mixed content categories, the proposed technique achieves an average Y BD-rate reduction of 2.39% and 1.49% for all intra (AI) and low-delay B (LDB) configurations, respectively, with low additional encoding complexity and almost no additional decoding complexity. All proposed ISC enhancement techniques have been adopted in AVS3.

QA-Filter: A QP-Adaptive Convolutional Neural Network Filter for Video Coding.

Convolutional neural network (CNN)-based filters have achieved great success in video coding. However, in most previous works, individual models were needed for each quantization parameter (QP) band, which is impractical due to limited storage resources. To explore this, our work consists of two parts. First, we propose a frequency and spatial QP-adaptive mechanism (FSQAM), which can be directly applied to the (vanilla) convolution to help any CNN filter handle different quantization noise. From the frequency domain, a FQAM that introduces the quantization step (Qstep) into the convolution is proposed. When the quantization noise increases, the ability of the CNN filter to suppress noise improves. Moreover, SQAM is further designed to compensate for the FQAM from the spatial domain. Second, based on FSQAM, a QP-adaptive CNN filter called QA-Filter that can be used under a wide range of QP is proposed. By factorizing the mixed features to high-frequency and low-frequency parts with the pair of pooling and upsampling operations, the QA-Filter and FQAM can promote each other to obtain better performance. Compared to the H.266/VVC baseline, average 5.25% and 3.84% BD-rate reductions for luma are achieved by QA-Filter with default all-intra (AI) and random-access (RA) configurations, respectively. Additionally, an up to 9.16% BD-rate reduction is achieved on the luma of sequence BasketballDrill. Besides, FSQAM achieves measurably better BD-rate performance compared with the previous QP map method.

An Analytic Transform Kernel Derivation Method for Video Codecs

In the standardization of versatile video coding (VVC), discrete cosine transform (DCT)-2, discrete sine transform (DST)-7, and DCT-8 are regarded as the primary transform kernels. However, DST-4 and DCT-4 can also be considered as the transform kernels instead of using DST-7 and DCT-8 owing to their effectiveness in smaller resolution test sequences. To implement these different block size transform kernels, a considerable amount of memory has to be allocated. Moreover, memory consumption to store different block size transform kernels is regarded as a major issue in video coding standardization. To address this problem, a common sparse unified matrix concept is introduced in this study, where any block size transform kernel matrix can be obtained after some mathematical operations. The proposed common sparse unified matrix saves approximately 80% of the static memory by storing only a few transform kernel elements for DCT-2, DST-7, and DCT-8. Full-required transform kernels are derived using the stored transform kernels and generated unit-element matrices and a permutation matrix. The static memory required is only for 1648 elements instead of 8180 elements, each with 8-bit precision. The defined common sparse unified matrix is composed of two parts: a unified DST-3 matrix and a grouped DST-7 matrix. The unified DST-3 matrix is used to derive different points of DCT-2 transform kernels, and the grouped DST-7 matrix is used to derive different points of DST-7 and DCT-8 transform kernels. The new technique of grouping concept is introduced, which shows the relationship between different rows of DST-7 transform kernels with various block sizes. The proposed grouping concept supports the fast algorithm of DST-7 by implementing the proposed method of the “one group one feature” principle. The simulation was conducted using the VTM-3.0 reference software under common test conditions. The simulation result of the all intra (AI) configuration is Y = 0.00%, U = −0.02%, V = 0.00% with an encoding time of 100%, and a decoding time of 100%. Similarly, the simulation results of random access (RA) configuration are Y = −0.01%, U = 0.09%, V = 0.06%, and the encoding and decoding times are 101% and 100%, respectively. The simulation result of the low delay B (LDB) configuration is Y = 0.01%, U = 0.08%, and V = −0.27%, for encoding and decoding times of 101% and 100%, respectively.

Fast ISP coding mode optimization algorithm based on CU texture complexity for VVC

In lately published video coding standard Versatile Video Coding (VVC/ H.266), the intra sub-partitions (ISP) coding mode is proposed. It is efficient for frames with rich texture, but less efficient for frames that are very flat or constant. In this paper, by comparing and analyzing the rate distortion cost (RD-cost) of coding unit (CU) with different texture features for using and not using ISP(No-ISP) coding mode, it is observed that CUs with simple texture can skip ISP coding mode. Based on this observation, a fast ISP coding mode optimization algorithm based on CU texture complexity is proposed, which aims to determine whether a CU needs to use ISP coding mode in advance by calculating CU texture complexity, so as to reduce the computation complexity of ISP. The experimental results show that under All Intra (AI) configuration, the coding time can be reduced by 7%, while the BD rate only increase by 0.09%.

A fast algorithm based on gray level co-occurrence matrix and Gabor feature for HEVC screen content coding

To reduce the computational complexity of screen content video coding (SCC), a fast algorithm based on gray level co-occurrence matrix and Gabor feature model for HEVC-SCC, denoted as GGM, is proposed in this paper. By studying the correlation of non-zero number in gray level co-occurrence matrix with different partitioning depth, the coding unit (CU) size of intra coding can be prejudged, which selectively skips the intra prediction process of CU in other depth. With Gabor filter, the edge information reflecting the features of screen content images to the human visual system (HVS) are extracted. According to Gabor feature, CUs are classified into natural content CUs (NCCUs), smooth screen content CUs (SSCUs) and complex screen content CUs (CSCUs), with which, the calculation and judgment of unnecessary intra prediction modes are skipped. Under all-intra (AI) configuration, experimental results show that the proposed algorithm GGM can achieve encoding time saving by 42.13% compared with SCM-8.3, and with only 1.85% bit-rate increasement.

Complexity reduction of versatile video coding standard: a deep learning approach

The new video coding standard, known as versatile video coding (VVC) is projected to be concluded by the end of 2020. This standard is conducted mainly to address 8k videos and emerging applications such as 360 deg and high dynamic range. Intraprediction is a part of the prediction step in the video coding that exploits spatial redundancy. This module has been improved, compared to the high-efficiency video coding (HEVC), by increasing the set of angular intraprediction modes (IPM) from 33 to 65 to model directional textures more accurately. Moreover, a quadtree plus binary tree (QTBT) structure replaced the QT of the HEVC. These improvements targeting at enhancing the coding efficiency resulted in significant coding complexity, especially in terms of encoding time. This paper fits into this context. It evokes the optimizations of the intramode and coding unit size decisions using statistical methods of fast decision and deep learning. A fast intramode decision algorithm is proposed for the different binary depths of the QTBT structure. Thus, an optimization by deep learning for square blocks is also included. Results show that the combinations of these two approaches can significantly reduce the complexity of the VVC encoder. Under the all intra (AI) configuration, a reduction of about 61.04% of the intraencoding time is achieved while maintaining an acceptable rate-distortion performance.

Fast CU Partition Decision Algorithm for AVS3 Intra Coding

The third generation of Audio Video coding Standard (AVS3) is an emerging video coding standard that surpasses High Efficiency Video Coding (HEVC). AVS3 allows flexible block subdivision by applying Quad-Tree (QT), Binary-Tree (BT) plus Extend Quad-Tree (EQT) partition structure, while the increased flexibility comes at the cost of enormous coding complexity. In this paper, an ingenious early termination mechanism is proposed to skip unnecessary exhaustive searches of the whole tree branches. We carry out a series of restrictive measures based on Coding Unit (CU) size and iteration status to make the sophisticated EQT split mode concentrate more on small CUs with complex texture structure. Historical QTBT partition information is also adopted as an important factor to skip EQT split mode in advance. Meanwhile, a fast CU partition algorithm based on the gradient is proposed to skip horizontal or vertical BT/EQT partition of CUs with prominent texture structure in another direction, in which early termination can be directly conducted in homogenous areas at the same time. Extensive experiments demonstrate that the proposed method can save 43% encoding time with only 0.53% BDBR increase on average under All Intra(AI) configuration, which outperforms the preexisting fast algorithms.

Fast Grid-Based Refining Segmentation Method in Video-Based Point Cloud Compression

The video-based point cloud compression (V-PCC, ISO/IEC 23090-5) is the state-of-the-art international standard for compressing dynamic point clouds developed by the moving picture experts group (MPEG). It has been achieved good rate-distortion (RD) performance by employing the 2D-based dynamic point cloud compression. As a brief look, V-PCC first converts the 3D input point cloud into a set of 2D patches followed by a packing process. The packing process then maps the patches into a 2D grid. Such a way allows compressing the patches utilizing the existing video coding standards. Besides the RD performance, complexity is another vital factor to consider in performance evaluations. In the V-PCC encoder, the self-time accounts for on average 15.9% and a maximum of 48.2% of the total-time, which can be a hindrance to realizing real-time V-PCC applications. One of the most computationally intensive modules of V-PCC is the grid-based refining segmentation (G-RS). Thus this paper proposes a fast G-RS method that can adaptively select voxels that need the refining segmentation. More concretely, the proposed method classifies the voxels based on the projection plane indices of 3D points and only applies the refining process to the selected voxels. Experimental results demonstrate that the proposed method reduces the complexity of the refining steps in G-RS, on average, by 60.7% and 62.5% without coding efficiency loss compared to the test model for category 2 (TMC2) version 12.0 reference software under the random access (RA) and all-intra (AI) configurations, respectively.