In-loop Filter Research Articles

Adaptive Loop Filter Hardware Design for 4K ASIC VVC Decoders

Versatile video coding (VVC) is the next generation video coding standard released in July 2020. VVC introduces new coding tools enhancing the coding efficiency compared to its predecessor high efficiency video coding (HEVC). These new tools have a significant impact on the VVC software decoder with a complexity estimated to two times HEVC decoder complexity. In particular, the adaptive loop filter (ALF) introduced in VVC as an in-loop filter increases both the decoding complexity and memory usage. These concerns need to be carefully addressed regarding the design of an efficient hardware implementation of a VVC decoder. In this paper, we present an efficient hardware implementation of the ALF tool for VVC decoder. The proposed solution establishes a novel scanning order between Luma and Chroma components that reduces significantly the ALF memory. The design takes advantage of all ALF features and establishes an unified hardware module for all ALF filters. The design uses 26 regular multipliers in a pipelined architecture with a fixed throughput of 2 pixels/cycle and fixed system latency regardless of the selected filter. This design operates at 600 MHz frequency enabling to decode on ASIC platform a 4K video at 30 frames per second in 4:2:2 chroma sub-sampling format.

A nonlocal HEVC in-loop filter using CNN-based compression noise estimation

A nonlocal HEVC in-loop filter using CNN-based compression noise estimation

One-for-All: An Efficient Variable Convolution Neural Network for In-Loop Filter of VVC

Recently, many researches on convolution neural network (CNN) based in-loop filters have been proposed to improve coding efficiency. However, most existing CNN based filters tend to train and deploy multiple networks for various quantization parameters (QP) and frame types (FT), which drastically increases resources in training these models and the memory burdens for video codec. In this paper, we propose a novel variable CNN (VCNN) based in-loop filter for VVC, which can effectively handle the compressed videos with different QPs and FTs via a single model. Specifically, an efficient and flexible attention module is developed to recalibrate features according to QPs or FTs. Then we embed the module into the residual block so that these informative features can be continuously utilized in the residual learning process. To minimize the information loss in the learning process of the entire network, we utilize a residual feature aggregation module (RFA) for more efficient feature extraction. Based on it, an efficient network architecture VCNN is designed that can not only effectively reduce compression artifacts, but also can be adaptive to various QPs and FTs. To address training data imbalance on various QPs and FTs and improve the robustness of the model, a focal mean square error loss function is employed to train the proposed network. Then we integrate the VCNN into VVC as an additional tool of in-loop filters after the deblocking filter. Extensive experimental results show that our VCNN approach obtains on average 3.63%, 4.36%, 4.23%, 3.56% under all intra, low-delay P, low-delay, and random access configurations, respectively, which is even better than QP-Separate models.

Deformable Wiener Filter for Future Video Coding.

In-loop filters have attracted increasing attention due to the remarkable noise-reduction capability in the hybrid video coding framework. However, the existing in-loop filters in Versatile Video Coding (VVC) mainly take advantage of the image local similarity. Although some non-local based in-loop filters can make up for this shortcoming, the widely-used unsupervised parameter estimation method by non-local filters limits the performance. In view of this, we propose a deformable Wiener Filter (DWF). It combines the local and non-local characteristics and supervisedly trains the filter coefficients based on the Wiener Filter theory. In the filtering process, local adjacent samples and non-local similar samples are first derived for each sample of interest. Then the to-be-filtered samples are classified into specific groups based on the patch-level noise and sample-level characteristics. Samples in each group share the same filter coefficients. After that, the local and non-local reference samples are adaptively fused based on the classification results. Finally, the filtering operation with outlier data constraints is conducted for each to-be-filtered sample. Moreover, the performance of the proposed DWF is analyzed with different reference sample derivation schemes in detail. Simulation results show that the proposed approach achieves 1.16%, 1.92%, and 2.67% bit-rate savings on average compared to the VTM-11.0 for All Intra, Random Access, and Low-Delay B configurations, respectively.

In-Loop Adaptive Filters to Improve the Power Quality of Switched Reluctance Generator in WECS

In this work, a proposal is presented to mitigate the voltage ripple effect generated by switched reluctance generator (SRG) in wind energy conversion systems (WECS). Voltage and current oscillations of switched reluctance generators are an expected consequence of the machine’s natural switching operation. This switching operation creates a known frequency component, named stroke frequency. When the SRG is connected to the AC utility grid in WECS, these oscillations may compromise the generated power quality by introducing stroke frequency components in the grid electric current. To solve such issue, this work has as an original contribution the proposal of in-loop adaptive filters applied to mitigate the propagation of the SRG stroke frequency content in the DC-bus voltage of a grid-connected WECS. The notch cascading and moving average filters are implemented at the voltage loop of the voltage source converter in an adaptive form, as function of the SRG speed. Simulation and experimental results show that the waveform distortion of the injected electric current is greatly reduced compared to the conventional non-filtered SRG connection.

BVI-DVC: A Training Database for Deep Video Compression

Deep learning methods are increasingly being applied in the optimisation of video compression algorithms and can achieve significantly enhanced coding gains, compared to conventional approaches. Such approaches often employ Convolutional Neural Networks (CNNs) which are trained on databases with relatively limited content coverage. In this paper, a new extensive and representative video database, BVI-DVC, is presented for training CNN-based video compression systems, with specific emphasis on machine learning tools that enhance conventional coding architectures, including spatial resolution and bit depth up-sampling, post-processing and in-loop filtering. BVI-DVC contains 800 sequences at various spatial resolutions from 270p to 2160p and has been evaluated on ten existing network architectures for four different coding tools. Experimental results show that this database produces significant improvements in terms of coding gains over three existing (commonly used) image/video training databases under the same training and evaluation configurations. The overall additional coding improvements by using the proposed database for all tested coding modules and CNN architectures are up to 10.3% based on the assessment of PSNR and 8.1% based on VMAF.

Explicit Impedance Modeling and Shaping of Grid-Connected Converters via an Enhanced PLL for Stabilizing the Weak Grid Connection

The voltage source converters (VSCs) are used to interface and control the renewable energy resources that are integrated into the power grids. However, a weak grid connection raises a stability problem for grid synchronization of the grid-feeding current-controlled VSCs (CCVSCs). This paper handles modeling and controller design for stabilizing the weak grid connection of CCVSCs. The impedance modeling/shaping of the VSC has been recognized as an effective tool to analyze and design the dynamics of the VSC. But an explicit and accurate impedance model of the CCVSC is required. Particularly, modeling the impact of the phase-locked loop (PLL) synchronization unit on the impedance model of the CCVSC is complex. Therefore, at first, an efficient impedance model of the CCVSC is developed while the impact of the PLL is rigorously considered through a complex procedure that results in an explicit/accurate impedance model. The developed impedance model is used to conduct passivity/weak grid connection stability analysis to clarify the underlying causes of instability problems. Then, a novel PLL, with an in-loop low-pass filter (LPF) to account for harmonics/asymmetries, is proposed with enhanced characteristics using the state feedback control. The impedance shaping method (with the aid of the impedance model) is utilized to design the state feedback gains. The state feedback loops provide degrees of freedom for bandwidth design of the PLL considering the current/power control loops and stabilize the system under weak grid connection/distorted conditions. Simulation results prove the accuracy and effectiveness of the models.

VVC In-Loop Filters

This paper presents an overview of the technologies for in-loop processing and filtering in the Versatile Video Coding (VVC) standard. These processes comprise luma mapping with chroma scaling, deblocking filter, sample adaptive offset, adaptive loop filter and cross-component adaptive loop filter. They are qualified as “in-loop” because they are applied inside the encoding and decoding loops, before storing the pictures in the decoded picture buffer. The filters are complementary and address different purposes. Luma mapping with chroma scaling aims at adaptively modifying the coded samples distribution for improved coding efficiency. The deblocking filter aims at reducing blocking discontinuities. Sample adaptive offset mostly aims at reducing artifacts resulting from the quantization of transform coefficients. Adaptive loop filter and cross-component adaptive loop filter are adaptive filters enabling to enhance the reconstructed signal, using for instance Wiener-filter encoding approaches. The paper provides an overview of the in-loop filtering process and a detailed description of the filtering algorithms. Objective compression efficiency results are provided for each filter, with indication of cumulative coding gains. Subjective benefits are illustrated. Implementation issues considered during the design of the VVC in-loop filters are also discussed.

An Effective Solution to Eliminate DC-Offset for Extracting the Phase and Frequency of Grid Voltage

Recently, several approaches with the ability to reject the DC-offset in phase locked loop (PLL) methods have been developed. These approaches include different filtering structures which can be classified into two categories: prefiltering before the PLL input and in-loop filtering in the PLL control loop. As highlighted in the literature, the DC-offset rejection methods based on in-loop filtering have received less attention due to their slow dynamic performance. Therefore, this paper proposes an alternative DC-offset rejection technique as in-loop filtering of the PLL. The effectiveness of the proposed PLL is confirmed by simulation and experimental results.

Deep learning based HEVC in-loop filter and noise reduction

The lossy compression techniques at low bit rate often create ringing and contouring effects on the output images and introduce various blurring and distortion at block bounders. To overcome those compression artifacts different neural network based post-processing techniques have been experimented with over the last few years. The traditional loop-filter methods in the HEVC frame-work support two post-processing operations namely a de-blocking filter followed by a sample adaptive offset (SAO) filter. These operations usually introduce extra signaling bits and become overhead to the network with high-resolution video processing. In this study, we came up with a new deep learning-based algorithm for SAO filtering operations and substantiated the merits of the proposed method. We introduced a variable filter size sub-layered dense CNN (SDCNN) to improve the denoising operation and incorporated large stride deconvolution layers for further computation improvement. We demonstrate that our deconvolution model can effectively be trained by leveraging the high-frequency edge features learned in a shallow network using residual learning and data augmentation techniques. Extensive experiments show that our approach outperformed other state-of-the-art approaches in terms of SSIM, Bjøntegaard delta bit-rate (BD-BR), BD-PSNR measurements on the standard video test set and achieves an average of 8.73 % bit rate saving compared to HEVC baseline.

A fast integrated deblocking filter and sample-adaptive-offset parameter estimation architecture for HEVC

Low power hardware acceleration cores for integration into real-time High Efficiency Video Coding (HEVC) codec for smartphones, tablets, camcorders, and televisions are in great demand. This demand motivates one for an efficient approximation of important power-consuming modules of HEVC including in-loop filters. This paper presents a hardware-efficient implementation of integrated deblocking filter (DBF) and sample adaptive offset (SAO) parameter estimation architecture for 16×16, 32×32, and 64×64 coding tree units (CTU) in HEVC. When the architecture is extended to HEVC-Test-Model (HM) Software, the luminance peak-signal-to-noise-ratio gets increased by 0.02 decibel, and the execution time of DBF and SAO gets decreased by at most 35% and 38%, respectively while compared to the reference algorithm. Moreover, it delivers rate–distortion performance comparable to the HEVC standard and reports mean-squared-error, structural-similarity (SSIM) index, and multi-scale SSIM (MS-SSIM) index of values 0.15, 0.9984, and 1, respectively for 4K video sequences. The architecture consumes minimum power, area, and energy equal to 9.83 milliwatts, 162 kilo-gate-equivalents, and 44 picojoules, respectively while supporting up to forty-six 8K frames per second. Additionally, it reports 78% smaller area and requires 75% less clock-cycles-per-largest-coding-unit as compared to the separate implementation of DBF and SAO with reference to HEVC. Such designs with low power, area, and energy can be integrated into a real-time HEVC codec for portable HEVC-compliant consumer electronic devices.

Group-normalized deep CNN-based in-loop filter for HEVC scalable extension

Group-normalized deep CNN-based in-loop filter for HEVC scalable extension

An explicit self-attention-based multimodality CNN in-loop filter for versatile video coding

An explicit self-attention-based multimodality CNN in-loop filter for versatile video coding

User-Priority Based AV1 Coding Tool Selection

AV1 is an open-source video coding technique developed by Alliance for Open Media (AOMedia). Many powerful coding tools introduced to AV1 has made its encoding time significantly increased. In this paper, we aim at simplifying the AV1 encoder by facilitating selection of some coding tools depending on users' specific application preference. We firstly design a coding tool OFF-test to evaluate coding performance of intra, inter, and in-loop filter coding tools in AV1 encoder in order to define a criterion to evaluate the importance of each coding tool by measuring the Bjøntegaard delta bit rate (BDBR) loss and time saving (TS) when each coding tool is turned OFF. Furthermore, the importance of each coding tool is analyzed based on the predefined criterion. Lastly, we suggest two intra coding tool selection methods, three inter coding tool selection methods, and three overall coding tool selection methods to simplify AV1 encoder based on users' preference of quality-priority or low-complexity-priority. Experimental results show that our proposed low-complexity-priority selection method saves 30.72% of encoding time with only 0.91% loss in BDBR sense, and the quality-priority selection method saves 4.64% of encoding time with 0.02% BDBR loss.

A progressive CNN in-loop filtering approach for inter frame coding

Convolutional Neural Network (CNN) structures have been designed for in-loop filtering to improve video coding performance. These CNN models are usually trained through learning the correlations between the reconstructed and the original frames, which are then applied to every single reconstructed frame to improve the overall video quality. This direct model training and deployment strategy is effective for intra coding since a locally optimal model is sufficient. However, when applied to inter coding, it causes over-filtering because the intertwined reference dependencies across inter frames are not taken into consideration. To address this issue, existing methods usually resort to the Rate–Distortion Optimization (RDO) to selectively apply the CNN model, but fail to address the limitation of using a local CNN model. In this paper, we propose a progressive approach to train and incorporate the CNN-based in-loop filters to work seamlessly with video encoders. First, we develop a progressive training method to obtain the inter model. Using transfer learning, reconstructed frames using the CNN model are progressively involved back into the training of the CNN model itself, to simulate the reference dependencies in inter coding. Next, we design a frame-level model selection strategy for the high-bitrate coding where the over-filtering effect is diluted. Experimental results show that the proposed method outperforms the RDO method that utilizes only local model. Proposed approach also achieves comparable coding performance but with less computational complexity when integrating our progressive model into the RDO scheme.

Detection of transcoded HEVC videos based on in-loop filtering and PU partitioning analyses

With the increasing maturity of video editing technology, forgers are more inclined to transcode videos to High Efficiency Video Coding (HEVC) videos, as HEVC not only enables people to enjoy high-definition videos but also allows broadcasters to stream it more efficiently across networks. Therefore, to verify the originality and authenticity, it is of great significance to propose an algorithm for detecting transcoded HEVC videos. In this paper, a theoretical model of video transcoding is first constructed, and a novel transcoding detection algorithm based on In-loop Filtering and Prediction Units (PU) Partition (IFPP) is proposed. By analyzing the statistical characteristics of strong and normal filtering modes in deblocking filtering and calculating offset values in Sample Adaptive Offset (SAO) filtering, the transcoding traces in inter-coded frames can be captured. In addition, PU partition statistics are also extracted to make full use of traces in intra-coded frames. By fusing these subfeatures, the proposed IFPP feature with 17 dimensions can be obtained, which is further fed to the Support Vector Machine (SVM) classifier. Finally, experiments are conducted on datasets with various coding parameters. Results show that the proposed algorithm outperforms existing algorithms and has better robustness.

AV1 and VVC Video Codecs: Overview on Complexity Reduction and Hardware Design

This article presents an extensive review of the state-of-the-art system-level solutions featuring complexity reduction and/or dedicated hardware designs for the AV1 and VVC video coding formats. These formats introduced several novel coding techniques compared to their predecessors to improve the coding efficiency at the cost of a significant computational cost. In this article, we discuss the main novelties of AV1 and VVC in each coding module, including block partitioning, intra and inter prediction, transform, entropy coding, and in-loop filters. Then, we present the main published works focusing on complexity reduction and hardware designs for AV1 and VVC. Most of the complexity reduction solutions target the complex and flexible block partitioning structures of these encoders to provide a better tradeoff between coding efficiency and complexity reduction whereas the hardware designs focus on the challenge of implementing the new coding tools to attend real-time processing of high-definition videos. Even with the presented works reaching impressive results, these research fields remain opened for innovative contributions, as discussed in this article.

A CNN-Based Prediction-Aware Quality Enhancement Framework for VVC

This paper presents a framework for Convolutional Neural Network (CNN)-based quality enhancement task, by taking advantage of coding information in the compressed video signal. The motivation is that normative decisions made by the encoder can significantly impact the type and strength of artifacts in the decoded images. In this paper, the main focus has been put on decisions defining the prediction signal in intra and inter frames. This information has been used in the training phase as well as input to help the process of learning artifacts that are specific to each coding type. Furthermore, to retain a low memory requirement for the proposed method, one model is used for all Quantization Parameters (QPs) with a Quantization Parameter (QP)-map, which is also shared between luma and chroma components. In addition to the Post Processing (PP) approach, the In-Loop Filtering (ILF) codec integration has also been considered, where the characteristics of the Group of Pictures (GoP) are taken into account to boost the performance. The proposed CNN-based Quality Enhancement (QE) framework has been implemented on top of the Versatile Video Coding (VVC) Test Model (VTM-10). Experiments show that the prediction-aware aspect of the proposed method improves the coding efficiency gain of the default CNN-based QE method by 1.52%, in terms of BD-BR, at the same network complexity compared to the default CNN-based QE filter.

Adaptive Deep Reinforcement Learning-Based In-Loop Filter for VVC.

Deep learning-based in-loop filters have recently demonstrated great improvement for both coding efficiency and subjective quality in video coding. However, most existing deep learning-based in-loop filters tend to develop a sophisticated model in exchange for good performance, and they employ a single network structure to all reconstructed samples, which lack sufficient adaptiveness to the various video content, limiting their performances to some extent. In contrast, this paper proposes an adaptive deep reinforcement learning-based in-loop filter (ARLF) for versatile video coding (VVC). Specifically, we treat the filtering as a decision-making process and employ an agent to select an appropriate network by leveraging recent advances in deep reinforcement learning. To this end, we develop a lightweight backbone and utilize it to design a network set S containing networks with different complexities. Then a simple but efficient agent network is designed to predict the optimal network from S , which makes the model adaptive to various video contents. To improve the robustness of our model, a two-stage training scheme is further proposed to train the agent and tune the network set. The coding tree unit (CTU) is seen as the basic unit for the in-loop filtering processing. A CTU level control flag is applied in the sense of rate-distortion optimization (RDO). Extensive experimental results show that our ARLF approach obtains on average 2.17%, 2.65%, 2.58%, 2.51% under all-intra, low-delay P, low-delay, and random access configurations, respectively. Compared with other deep learning-based methods, the proposed approach can achieve better performance with low computation complexity.

Application Map in the Noise Suppression Filter for Video Compression

This paper considers an application map method usage together with the Nose Suppression Filter (NSF) in-loop filter for video compression. The application map method is described in details and simulations results of basic NSF are provided as well as simulation results of NSF powered with the application map method. It is demonstrated that the application map method allows to significantly improve objective performance results of basic NSF and additionally decrease decoder average complexity. As a result, an average bd-rate saving of NSF with the application map reaches 2.6% for luma component in Low Delay P coding configuration in comparing to Versatile Video Coding reference implementation (VTM) version 1.0.