A Reconfigurable Framework for Neural Network Based Video In-Loop Filtering

Abstract

This article proposes a reconfigurable framework for neural network based video in-loop filtering to guide large-scale models for content-aware processing. Specifically, the backbone neural model is decomposed into several convolutional groups and the encoder systematically traverses all candidate configurations combined by these groups to find the best one. The selected configuration index is then encapsulated as side information and passed to the decoder, enabling dynamic model reconfiguration during the decoding stage. The preceding reconfiguration process is only deployed in the inference stage on top of a pre-trained backbone model. Furthermore, we devise WMSPFormer , a wavelet multi-scale Poolformer, as the backbone network structure. WMSPFormer utilizes a wavelet-based multi-scale structure to losslessly decompose the input into multiple scales for spatial-spectral features aggregation. Moreover, it uses multi-scale pooling operations ( MSPoolformer ) instead of complicated matrix calculations to substitute the attention process. We also extend MSPoolformer to a large-scale version using more parameters, referred to as MSPoolformerExt . Extensive experiments demonstrate that the proposed WMSPFormer+Reconfig. and WMSPFormerExt+Reconfig. achieve a remarkable 7.13% and 7.92% BD-Rate reduction over the anchor H.266/VVC, outperforming most existing methods evaluated under the same training and testing conditions. In addition, the low-complexity nature of the WMSPFormer series makes it attractive for practical applications.