Versatile Video Coding Research Articles

Multi-feature fusion for efficient inter prediction in versatile video coding

Multi-feature fusion for efficient inter prediction in versatile video coding

OptiRet-Net: An Optimized Low-Light Image Enhancement Technique for CV-based Applications in Resource-Constrained Environments

The illumination of images can significantly impact computer-vision applications such as image classification, multiple object detection, and tracking, leading to a significant decline in detection and tracking accuracy. Recent advancements in deep learning techniques have been applied to low-light image enhancement (LLIE) to combat this issue. Retinex theory-based methods following a decomposition-adjustment pipeline for LLIE have performed well in various aspects. Despite their success, current research on Retinex-based deep learning still needs to improve in terms of optimization techniques and complicated convolution connections, which can be computationally intensive for end-device deployment. We propose an Optimized Retinex-based CNN (OptiRet-Net) deep-learning framework to address these challenges for the low-light image enhancement problem. Our results demonstrate that the proposed method outperforms existing state-of-the-art models in terms of full reference metrics with a PSNR of 21.87, SSIM of 0.80, LPIPS of 0.16, and zero reference metrics with a NIQE of 3.4 and PIQE of 56.6. Additionally, we validate our approach using a comprehensive evaluation comprising five datasets and nine prior methods. Furthermore, we assess the efficacy of our proposed model combining low-light multiple object tracking applications using YOLOX and ByteTrack in versatile video coding (VVC/H.266) across various quantization parameters. Our findings reveal that LLIE-enhanced frames surpass their tracking results with a MOTA of 80.6% and a remarkable precision rate of 96%. Our model also achieves minimal file sizes by effectively compressing the enhanced low-light images while maintaining their quality, making it suitable for resource-constrained environments where storage or bandwidth limitations are a concern.

Lightweight macro-pixel quality enhancement network for light field images compressed by versatile video coding

Previous research demonstrated that filtering Macro-Pixels (MPs) in a decoded Light Field Image (LFI) sequence can effectively enhances the quality of the corresponding Sub-Aperture Images (SAIs). In this paper, we propose a deep-learning-based quality enhancement model following the MP-wise processing approach tailored to LFIs encoded by the Versatile Video Coding (VVC) standard. The proposed novel Res2Net Quality Enhancement Convolutional Neural Network (R2NQE-CNN) architecture is both lightweight and powerful, in which the Res2Net modules are employed to perform LFI filtering for the first time, and are implemented with a novel improved 3D-feature-processing structure. The proposed method incorporates only 205K model parameters and achieves significant Y-BD-rate reductions over VVC of up to 32%, representing a relative improvement of up to 33% compared to the state-of-the-art method, which has more than three times the number of parameters of our proposed model.

Fast CU decision method based on texture characteristics and decision tree for depth map intra-coding

As the demand for higher quality 3D videos continues to grow, the 3D extensions of the Efficient Video Coding (3D-HEVC) standard are gradually unable to meet the needs of users. Versatile Video Coding Standard (H. 266/VVC), as an advanced video coding standard, adopts the nested Multi-Type Tree quadtree (QTMT) partitioning structure that current fast depth map coding unit (CU) partitioning methods cannot apply. Therefore, we have designed a fast intra-frame CU partitioning algorithm for VVC 3D video depth maps. Our proposed algorithm in this article consists of two steps, including two sub-algorithms. First, by analyzing the relationship between image entropy and variance and depth map CU division, we establish a bi-criterion decision algorithm to determine whether the texture complexity of the current CU is low enough to terminate its partitioning process. Then, for CUs that have been determined by the first algorithm to need further partitioning, we use a decision tree model based on Light Gradient Boosting Machine (LGBM) to predict which direction of Rate-Distortion Optimization (RDO) calculation can be skipped, which can avoid some unnecessary RDO calculations in a certain direction. The final experiment demonstrated the effect of the proposed algorithm, which can reduce 47.65% complexity of VVC 3D video intra-coding with negligible 0.23% Bjøntegaard Delta Bitrate (BDBR) increase, superior to other advanced methods.

Impact of Wireless Network Packet Loss on Real-Time Video Streaming Application: A Comparative Study of H.265 and H.266 Codecs

The transmission of real-time videos over wireless networks is prone to the negative consequences of packet loss and delay, which can have a potential effect on the video quality during streaming. These impairments can lead to interruptions, buffering, and degradation of visual and auditory elements, resulting in an unsatisfactory user experience. In this paper, we aim to address the challenges associated with packet loss and delay parameters in wireless networks, and propose an approach to alleviate their impact on real-time video transmission. The proposed approach involves utilizing the H.265/H.266 video coding standards. For Versatile Video Coding (VVC), a patch support for VVdeC and VVenC to FFmpeg (Fast Forward Moving Picture Expert Group) is added. As a result, FFmpeg is used to encode, stream and decode all videos. Raw videos of 2K qualities are encoded based on the adaptive quantization (QP) for the above-mentioned codecs. By selecting optimal transmission data based on various network conditions, this approach enhances the Quality of Experience (QoE) for end-users while minimizing resource usage in the wireless network. Furthermore, the proposed approach selects the codec standards according to their bitrates and frame rates. Simulation results indicate that the proposed approach has a significant improvement for real-time video streaming over wireless networks to satisfy the end user experience. The approach also outperforms other related work by gaining a PSNR of +12 dB for H.265 and +13 dB for H.266 when the network packet loss is 1%.

Learning-based CU partition prediction for fast panoramic video intra coding

In recent years, Virtual Reality (VR) technology has been applied in various aspects of life. As an indispensable component of VR, panoramic video offers viewers immersive experiences, thereby attracting widespread attention. The high resolution of panoramic video poses significant challenges for its storage and transmission, requiring efficient video compression. The Versatile Video Coding (VVC) exhibits superior coding efficiency, albeit at the expense of a several dozen-fold increase in encoding complexity. In this paper, we propose a fast Coding Unit (CU) partition algorithm for intra-coding based on Support Vector Machine (SVM), which selects appropriate features for online training of multiple classifiers based on frame texture characteristics. Additionally, in response to distortion issues in panoramic video coding, we divide video frames into different regions and adjust the Quantization Parameter (QP) values of CUs within each region to improve the encoding efficiency. Experimental results show that the proposed method achieves an average encoding time reduction of 46.21%, with an average BDBR increase of only 1.01%, achieving a good trade-off between time-saving and encoding efficiency.

NN-VVC: A Hybrid Learned-Conventional Video Codec Targeting Humans and Machines

Advancements in artificial intelligence have significantly increased the use of images and videos in machine analysis algorithms, predominantly neural networks. However, the traditional methods of compressing, storing and transmitting media have been optimized for human viewers rather than machines. Current research in coding images and videos for machine analysis has evolved in two distinct paths. The first is characterized by End-to-End (E2E) learned codes, which show promising results in image coding but have yet to match the performance of leading Conventional Video Codecs (CVC) and suffer from a lack of interoperability. The second path optimizes CVC, such as the Versatile Video Coding (VVC) standard, for machine-oriented reconstruction. Although CVC-based approaches enjoy widespread hardware and software compatibility and interoperability, they often fall short in machine task performance, especially at lower bitrates. This paper proposes a novel hybrid codec for machines named NN-VVC, which combines the advantages of an E2E-learned image codec and a CVC to achieve high performance in both image and video coding for machines. Our experiments show that the proposed system achieved up to − 43.20% and − 26.8% Bjøntegaard Delta rate reduction over VVC for image and video data, respectively, when evaluated on multiple different datasets and machine vision tasks according to the common test conditions designed by the VCM study group in MPEG standardization activities. Furthermore, to improve reconstruction quality, we introduce a human-focused branch into our codec, enhancing the visual appeal of reconstructions intended for human supervision of the machine-oriented main branch.

Complexity analysis of the VVenC versus VVC encoder

The joint video experts team (JVET) has recently finalized a next-generation open-source video codec, called versatile video coding (VVC). The new standard presents a higher gain in the run time and rate compressions. Based on the reference software (VTM, VVC Test Model) of VVC, an optimized encoder was developed these last years resulting in a fast and efficient encoder, called Fraunhofer versatile video encoder (VVenC). Based on the Bjontegaard methodology and the GPROF profiling tool, this paper presents a technical complexity analysis and comparison of both VVC and VVenC. The appropriate comparisons cover the percentage taken by each block in terms of processing time and the resulting whole encoding time. The peak signal-to-noise ratio (PSNR) and bit rate between VVC and VVenC encoders based on the common test conditions manual are also analyzed and compared. The profiling results show that the VVenC encoder presents a maximum gain of runtime and bit rate of 90% and 20% respectively in classes A and D test sequences, compared to the VVC encoder.

Decoder-ROI based Versatile Video Coding for Multi-Object Tracking Vision Task

The video encoding standards High Efficiency Video Coding (HEVC) and, more recently, Versatile Video Coding (VVC) have introduced significant advancements in multimedia communication applications, such as video conferencing, broadcasting, and notably, E-learning. However, recent developments in artificial intelligence (AI) and big data have given rise to an urgent need for a specialized video encoding model designed specifically for image and video analysis applications using machine vision. In this paper, we propose a novel video encoding approach that effectively combines the ROI Coding algorithm and the VVC encoding model. The proposed method identifies regions of interest within video frames through fundamental and deep features. Based on this, we propose an adaptive compression method for each frame block, ensuring both the execution performance of machine learning applications and minimal data encoding requirements. To achieve new coding scheme without adding bitrate, New feature extraction approach are utilizing only decoded information (Decoder-ROI). The results demonstrate that the Decoder-ROI achieved significant compression rate improvement when compared to standard and relevant VCM schemes. Furthermore, ROI exploitation contributes to a 3.25\% reduction in encoding time compared to the baseline VVC encoding standard.

Efficient 2D transform hardware architecture for the versatile video coding standard

As the latest generation of video coding standards, versatile video coding (VVC) introduces several new coding tools in transform coding to concentrate the energy of residual blocks. In this work, we propose a regular multiplier (RM) based hardware architecture for 2D transform that can process different transform sizes and types using a unified architecture. This architecture consists of a 1D row transform, a transpose memory, and a 1D column transform, allowing for processing 16 coefficients per cycle. For the 1D row/column transform, the unified calculation structure consists of 512 regular multipliers supporting various transform sizes and types. For the transpose memory, we design an SRAM-based diagonal storage approach, along with a FIFO for storing the block information. Compared with the state-of-the-art MCM-based design, the proposed RM-based design shows a 30.9% reduction in area while can operate at a higher frequency.

DATRA-MIV: Decoder-Adaptive Tiling and Rate Allocation for MPEG Immersive Video

The emerging immersive video coding standard moving picture experts group (MPEG) immersive video (MIV), which is ongoing standardization by MPEG-Immersive (MPEG-I) group, enables six degrees of freedom in a virtual reality environment that represents both natural and computer-generated scenes using multi-view video compression. The MIV eliminates the redundancy between multi-view videos and merges the residuals into multiple pictures, called an atlas. Thus, bitstreams with encoded atlases are generated and corresponding number of decoders are needed, which is challenging for the lightweight device with a single decoder. This article proposes a decoder-adaptive tiling and rate allocation method for MIV to overcome the challenge. First, the proposed method divides atlases into subpictures considering two aspects: (i) subpicture bitstream extracting and merging into one bitstream to use a single decoder and (ii) separation of each source view from the atlases for rate allocation. Second, the atlases are encoded by versatile video coding (VVC), using an extractable subpicture to divide the atlases into subpictures. Third, each subpicture bitstream is extracted, and asymmetric quality allocation for each subpictures is conducted by considering the residuals in the subpicture. Fourth, mixed-quality subpictures were merged by using the proposed bitstream merger. Fifth, the merged bitstream is decoded by using a single decoder. Finally, the viewing area of the user is synthesized by using the reconstructed atlases. Experimental results with the VVC test model (VTM) show that the proposed method achieves a 21.37% Bjøntegaard delta rate saving for immersive video peak signal-to-noise ratio and a 26.76% decoding runtime saving compared to the VTM anchor configuration. Moreover, it supports bitstreams for multiple decoders and single decoder without re-encoding, transcoding, or a substantial increase of the server-side storage.

Efficient inter partitioning of versatile video coding based on supervised contrastive learning

Nowadays, Versatile Video Coding (VVC) has achieved a superior performance than previous video coding standard (High Efficiency Video Coding). The Quadtree with Nested Multi-Type Tree (QTMT) coding block structure can enhance the coding performance. Nevertheless, this technique also leads to the significantly increasing complexity of VVC inter coding. Therefore, complexity optimization is an urgent problem to be optimized in the market application of VVC. To solve this issue, we propose a Supervised-Contrastive-Learning-based Inter Partitioning (SCLIP) method in this paper. Firstly, we define the above complexity optimization problem as a supervised classification task. Next, we develop a SCLIP Estimation Network (SCLIPEst-Net) with a supervised contrastive learning module and a classification module. After training on a newly established dataset, the SCLIPEst-Net can reasonably predict the mode partitioning. Finally, we propose an overall SCLIP algorithm that effectively determines the inter partitions of VVC with a low computational overhead. Experimental results indicate that our method achieves 45.14% average Time Saving (TS) with a 2.40% Bjøntegaard Delta Bit Rate (BDBR) in Random Access (RA), outperforming the benchmarks.

Towards real-time practical image compression with lightweight attention

The rate–distortion (RD) performance of learning-based image compression (LIC) has already surpassed that of traditional Versatile Video Coding (VVC) intra-coding. However, the gain of the compression efficiency is at the cost of high computational complexity, which is prohibitively expensive and becomes a bottleneck for practical applications. To this end, we proposed an efficient LIC method with lightweight designs for real-time practical applications, which achieves a better trade-off between compression performance and computational complexity. Specifically, residual connected lightweight attention units (RLAUs) are stacked for feature extraction, which achieves effective global spatial information embedding while keeping relatively low complexity. Meanwhile, a trainable channel-gained adaptive module (CGAM) is introduced into the nonlinear transform network and multi-stage context model. This module re-distributes the importance of different channels, further improving compression efficiency. The experimental results demonstrate that the proposed method achieves better compression efficiency than VVC intra-coding. Furthermore, compared to other state-of-the-art LIC approaches, the proposed method significantly reduces the coding time while maintaining comparable coding efficiency. The source codes are available at https://github.com/llsurreal919/LightweightLIC.

HDRC: a subjective quality assessment database for compressed high dynamic range image

Evaluating the quality of high dynamic range (HDR) images has emerged as a challenging and contemporary topic with the proliferation of HDR content. In this work, a new HDR compression (HDRC) database is proposed, aiming to provide a benchmark for the development of full-reference HDR image quality assessment (IQA) algorithms when facing the latest HDR compression distortions. In particular, the proposed HDRC database is the first HDR-IQA database to incorporate Versatile Video Coding (VVC) compression distortions, closely associated with practical application scenarios. Furthermore, the proposed HDRC database is currently the largest HDR-IQA database, including 80 reference images and 400 distorted images. Extensive experiments are conducted by studying the performance compared to existing HDR-IQA databases when evaluating three HDR-specific IQA models and nine IQA models prevalent for low dynamic range (LDR) content, revealing the challenges the proposed HDRC database brings. The results indicate that the existing IQA models demonstrate noticeable decreases in accuracy when assessing new compression distortions, underscoring the need for the development of novel HDR-IQA models. Consequently, the suggested HDRC database can serve as a potential database for HDR-IQA research, fostering a comprehensive exploration of the associated fields.

Transform coefficient distribution modeling based frame-level adaptive rate control for VVC

Rate control is a key issue in the research of video coding optimization. In this work, we propose a statistical modeling based frame-level distortion estimation model and a distortion dependency based rate control optimization algorithm for Versatile Video Coding (VVC). VVC has incorporated into more advanced coding tools, resulting in the distribution of transform coefficient more sharper. In response to this phenomenon, the TCD of VVC is modeled by the probability density function (PDF) of Cauchy distribution and generalized Gaussian distribution (GGD) individually. Then, a sliding window-based frame-level distortion model is proposed, which is used for derivation of inter-frame distortion dependency factor. According to the derived distortion dependency factor, a frame-level rate control optimization algorithm is presented to update the coding parameters in practical coding process. Experiments have shown that the proposed method could effectively improve rate–distortion performance, achieving an average rate distortion performance improvement of 0.72% in sequences blow 4K resolution, and 1.47% in 4K sequences in low delay P frame (LDP) configuration.

Fast Decision-Tree-Based Series Partitioning and Mode Prediction Termination Algorithm for H.266/VVC

With the advancement of network technology, multimedia videos have emerged as a crucial channel for individuals to access external information, owing to their realistic and intuitive effects. In the presence of high frame rate and high dynamic range videos, the coding efficiency of high-efficiency video coding (HEVC) falls short of meeting the storage and transmission demands of the video content. Therefore, versatile video coding (VVC) introduces a nested quadtree plus multi-type tree (QTMT) segmentation structure based on the HEVC standard, while also expanding the intra-prediction modes from 35 to 67. While the new technology introduced by VVC has enhanced compression performance, it concurrently introduces a higher level of computational complexity. To enhance coding efficiency and diminish computational complexity, this paper explores two key aspects: coding unit (CU) partition decision-making and intra-frame mode selection. Firstly, to address the flexible partitioning structure of QTMT, we propose a decision-tree-based series partitioning decision algorithm for partitioning decisions. Through concatenating the quadtree (QT) partition division decision with the multi-type tree (MT) division decision, a strategy is implemented to determine whether to skip the MT division decision based on texture characteristics. If the MT partition decision is used, four decision tree classifiers are used to judge different partition types. Secondly, for intra-frame mode selection, this paper proposes an ensemble-learning-based algorithm for mode prediction termination. Through the reordering of complete candidate modes and the assessment of prediction accuracy, the termination of redundant candidate modes is accomplished. Experimental results show that compared with the VVC test model (VTM), the algorithm proposed in this paper achieves an average time saving of 54.74%, while the BDBR only increases by 1.61%.

Adaptive quadtree splitting parallelization (AQSP) algorithm for the VVC standard

The Versatile Video Coding (VVC) standard, also known as H.266, was released in 2020 as the natural successor to the High Efficiency Video Coding (HEVC) standard. Among its innovative coding tools, VVC extended the concept of quadtree (QT) splitting to the multi-type tree (MTT) structure, introducing binary and ternary partitions to enhance HEVC’s coding efficiency. While this brought significant compression improvements, it also resulted in a substantial increase in encoding time, primarily due to VVC’s larger Coding Tree Unit (CTU) size of 128×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document}128 pixels. To mitigate this, this work introduces a flexible parallel approach for the QT traversal and splitting scheme of the VVC encoder, called adaptive quadtree splitting parallelization (AQSP) algorithm. This approach is based on the distribution of coding units (CUs) among different threads using the current depth level of the QT as a basis to minimize the number of broken dependencies. In this way, the algorithm achieves a good trade-off between time savings and coding efficiency. Experimental results show that, when compared with the original VVC encoder, AQSP achieves an acceleration factor of 2.04×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes $$\\end{document} with 4 threads at the expense of a low impact in terms of BD rate. These outcomes position AQSP competitively in comparison with other state-of-the-art approaches.

Fast CU size decision and intra-prediction mode decision method for H.266/VVC

AbstractH.266/Versatile Video Coding (VVC) is the most recent video coding standard developed by the Joint Video Experts Team (JVET). The quad-tree with nested multi-type tree (QTMT) architecture that improves the compression performance of H.266/VVC is introduced. Moreover, H.266/VVC contains a greater number of intra-prediction modes than H.265/High Efficiency Video Coding (HEVC), totalling 67. However, these lead to extremely the coding computational complexity. To cope with the above issues, a fast intra-coding unit (CU) size decision method and a fast intra-prediction mode decision method are proposed in this paper. Specifically, the trained Support Vector Machine (SVM) classifier models are utilized for determining CU partition mode in a fast CU size decision scheme. Furthermore, the quantity of intra-prediction modes added to the RDO mode set decreases in a fast intra-prediction mode decision scheme based on the improved search step. Simulation results illustrate that the proposed overall algorithm can decrease 55.24% encoding runtime with negligible BDBR.

Suboptimal video coding for machines method based on selective activation of in‐loop filter

AbstractA conventional codec aims to increase the compression efficiency for transmission and storage while maintaining video quality. However, as the number of platforms using machine vision rapidly increases, a codec that increases the compression efficiency and maintains the accuracy of machine vision tasks must be devised. Hence, the Moving Picture Experts Group created a standardization process for video coding for machines (VCM) to reduce bitrates while maintaining the accuracy of machine vision tasks. In particular, in‐loop filters have been developed for improving the subjective quality and machine vision task accuracy. However, the high computational complexity of in‐loop filters limits the development of a high‐performance VCM architecture. We analyze the effect of an in‐loop filter on the VCM performance and propose a suboptimal VCM method based on the selective activation of in‐loop filters. The proposed method reduces the computation time for video coding by approximately 5% when using the enhanced compression model and 2% when employing a Versatile Video Coding test model while maintaining the machine vision accuracy and compression efficiency of the VCM architecture.

Sample-Based Gradient Edge and Angular Prediction for VVC Lossless Intra-Coding

Lossless coding is a compression method in the Versatile Video Coding (VVC) standard, which can compress video without distortion. Lossless coding has great application prospects in fields with high requirements for video quality. Since the current VVC standard is mainly designed for lossy coding, the compression efficiency of VVC lossless coding makes it hard to meet people’s needs. In order to improve the performance of VVC lossless coding, this paper proposes a sample-based intra-gradient edge detection and angular prediction (SGAP) method. SGAP utilizes the characteristics of lossless intra-coding to employ samples adjacent to the current sample as reference samples and performs prediction through sample iteration. SGAP aims to improve the prediction accuracy for edge regions, smooth regions and directional texture regions in images. Experimental results on the VVC Test Model (VTM) 12.3 reveal that SGAP achieves 7.31% bit-rate savings on average in VVC lossless intra-coding, while the encoding time is only increased by 5.4%. Compared with existing advanced sample-based intra-prediction methods, SGAP can provide significantly higher coding performance gain.