Compression Artifact Reduction Research Articles

Stereo Image Restoration via Attention-Guided Correspondence Learning.

Although stereo image restoration has been extensively studied, most existing work focuses on restoring stereo images with limited horizontal parallax due to the binocular symmetry constraint. Stereo images with unlimited parallax (e.g., large ranges and asymmetrical types) are more challenging in real-world applications and have rarely been explored so far. To restore high-quality stereo images with unlimited parallax, this paper proposes an attention-guided correspondence learning method, which learns both self- and cross-views feature correspondence guided by parallax and omnidirectional attention. To learn cross-view feature correspondence, a Selective Parallax Attention Module (SPAM) is proposed to interact with cross-view features under the guidance of parallax attention that adaptively selects receptive fields for different parallax ranges. Furthermore, to handle asymmetrical parallax, we propose a Non-local Omnidirectional Attention Module (NOAM) to learn the non-local correlation of both self- and cross-view contexts, which guides the aggregation of global contextual features. Finally, we propose an Attention-guided Correspondence Learning Restoration Network (ACLRNet) upon SPAMs and NOAMs to restore stereo images by associating the features of two views based on the learned correspondence. Extensive experiments on five benchmark datasets demonstrate the effectiveness and generalization of the proposed method on three stereo image restoration tasks including super-resolution, denoising, and compression artifact reduction.

Compressed-SDR to HDR Video Reconstruction.

The new generation of organic light emitting diode display is designed to enable the high dynamic range (HDR), going beyond the standard dynamic range (SDR) supported by the traditional display devices. However, a large quantity of videos are still of SDR format. Further, most pre-existing videos are compressed at varying degrees for minimizing the storage and traffic flow demands. To enable movie-going experience on new generation devices, converting the compressed SDR videos to the HDR format (i.e., compressed-SDR to HDR conversion) is in great demands. The key challenge with this new problem is how to solve the intrinsic many-to-many mapping issue. However, without constraining the solution space or simply imitating the inverse camera imaging pipeline in stages, existing SDR-to-HDR methods can not formulate the HDR video generation process explicitly. Besides, they ignore the fact that videos are often compressed. To address these challenges, in this work we propose a novel imaging knowledge-inspired parallel networks (termed as KPNet) for compressed-SDR to HDR (CSDR-to-HDR) video reconstruction. KPNet has two key designs: Knowledge-Inspired Block (KIB) and Information Fusion Module (IFM). Concretely, mathematically formulated using some priors with compressed videos, our conversion from a CSDR-to-HDR video reconstruction is conceptually divided into four synergistic parts: reducing compression artifacts, recovering missing details, adjusting imaging parameters, and reducing image noise. We approximate this process by a compact KIB. To capture richer details, we learn HDR representations with a set of KIBs connected in parallel and fused with the IFM. Extensive evaluations show that our KPNet achieves superior performance over the state-of-the-art methods.

Enhanced Video Super-Resolution Network towards Compressed Data

Video super-resolution (VSR) algorithms aim at recovering a temporally consistent high-resolution (HR) video from its corresponding low-resolution (LR) video sequence. Due to the limited bandwidth during video transmission, most available videos on the internet are compressed. Nevertheless, few existing algorithms consider the compression factor in practical applications. In this paper, we propose an enhanced VSR model towards compressed videos, termed as ECVSR, to simultaneously achieve compression artifacts reduction and SR reconstruction end-to-end. ECVSR contains a motion-excited temporal adaption network (METAN) and a multi-frame SR network (SRNet). The METAN takes decoded LR video frames as input and models inter-frame correlations via bidirectional deformable alignment and motion-excited temporal adaption, where temporal differences are calculated as motion prior to excite the motion-sensitive regions of temporal features. In SRNet, cascaded recurrent multi-scale blocks (RMSB) are employed to learn deep spatio-temporal representations from adapted multi-frame features. Then, we build a reconstruction module for spatio-temporal information integration and HR frame reconstruction, which is followed by a detail refinement module for texture and visual quality enhancement. Extensive experimental results on compressed videos demonstrate the superiority of our method for compressed VSR. Code will be available at https://github.com/lifengcs/ECVSR .

Efficient image restoration with style-guided context cluster and interaction

Recently, convolutional neural networks (CNNs) and vision transformers (ViTs) have emerged as powerful tools for image restoration (IR). Nonetheless, they encountered some limitations due to their characteristics, such as CNNs sacrificing global reception and ViTs requiring large memory and graphics resources. To address these limitations and explore an alternative approach for improved IR performance, we propose two clustering-based frameworks for general IR tasks, which are style-guided context cluster U-Net (SCoC-UNet) and style-guided clustered point interaction U-Net (SCPI-UNet). The SCoC-UNet adopts a U-shaped architecture, comprising position embedding, Encoder, Decoder, and reconstruction block. Specifically, the input low-quality image is viewed as a set of unorganized points, each of which is first given location information by the continuous relative position embedding method. These points are then fed into a symmetric Encoder and Decoder which utilize style-guided context cluster (SCoC) blocks to extract potential context features and high-frequency information. Although SCoC-UNet has obtained decent performance for image restoration, its SCoC block can only capture connectivity at points within the same cluster, which may ignore long-range dependencies in different clusters. To address this issue, we further propose a SCPI-UNet based on SCoC-UNet, which leverages a style-guided clustered point interaction (SCPI) block in place of the SCoC block. The SCPI block utilizes a cross-attention mechanism to establish the connections of feature points between different clusters. Extensive experimental results demonstrate that the proposed SCoC-UNet and SCPI-UNet can handle several typical IR tasks (i.e., JPEG compression artifact reduction, image denoising, and super-resolution) and achieve superior quantitative and qualitative performance over some state-of-the-art methods.

Residual Network for Image Compression Artifact Reduction

This paper proposes an image compression algorithm based on Swin Transformer and residual network (STRN), aiming to reduce blurring and distortions in traditionally compressed images. The algorithm utilizes a dual-channel mechanism to remove artifacts from the image, which takes advantage of the complementary features of the transform and residual networks. The Swin Transformer networks address the issue of long-range dependency, leading to an enhanced and improved reconstructed image quality. The residual network is an effective network that mitigates gradient loss and recovers image details during the image compression process. The paper demonstrates that image compression can be achieved by training a convolutional network based on a transformer and residuals network, which significantly reduces artifacts and provides better reconstructed image quality compared to previously used and current mainstream methods based on traditional convolutional neural networks. The proposed approach can remove blocking artifacts by subtracting estimated artifacts from the input image, while still preserving most of the original details. Therefore, our proposed method is highly effective in improving image quality and reducing visual artifacts caused by traditional compression methods. Moreover, this method is useful for enhancing image transmission and storage efficiency in various computer vision systems that employ digital visual codecs.

Enhancing Image Quality by Reducing Compression Artifacts Using Dynamic Window Swin Transformer

Enhancing Image Quality by Reducing Compression Artifacts Using Dynamic Window Swin Transformer

Perceptual Quality Improvement in Videoconferencing Using Keyframes-Based GAN

In the latest years, videoconferencing has taken a fundamental role in interpersonal relations, both for personal and business purposes. Lossy video compression algorithms are the enabling technology for videoconferencing, as they reduce the bandwidth required for real-time video streaming. However, lossy video compression decreases the perceived visual quality. Thus, many techniques for reducing compression artifacts and improving video visual quality have been proposed in recent years. In this work, we propose a novel GAN-based method for compression artifacts reduction in videoconferencing. Given that, in this context, the speaker is typically in front of the camera and remains the same for the entire duration of the transmission, we can maintain a set of reference keyframes of the person from the higher-quality I-frames that are transmitted within the video stream and exploit them to guide the visual quality improvement; a novel aspect of this approach is the update policy that maintains and updates a compact and effective set of reference keyframes. First, we extract multi-scale features from the compressed and reference frames. Then, our architecture combines these features in a progressive manner according to facial landmarks. This allows the restoration of the high-frequency details lost after the video compression. Experiments show that the proposed approach improves visual quality and generates photo-realistic results even with high compression rates. Code and pre-trained networks are publicly available at https://github.com/LorenzoAgnolucci/Keyframes-GAN.

Sensitivity Decouple Learning for Image Compression Artifacts Reduction.

With the benefit of deep learning techniques, recent researches have made significant progress in image compression artifacts reduction. Despite their improved performances, prevailing methods only focus on learning a mapping from the compressed image to the original one but ignore the intrinsic attributes of the given compressed images, which greatly harms the performance of downstream parsing tasks. Different from these methods, we propose to decouple the intrinsic attributes into two complementary features for artifacts reduction, i.e., the compression-insensitive features to regularize the high-level semantic representations during training and the compression-sensitive features to be aware of the compression degree. To achieve this, we first employ adversarial training to regularize the compressed and original encoded features for retaining high-level semantics, and we then develop the compression quality-aware feature encoder for compression-sensitive features. Based on these dual complementary features, we propose a Dual Awareness Guidance Network (DAGN) to utilize these awareness features as transformation guidance during the decoding phase. In our proposed DAGN, we develop a cross-feature fusion module to maintain the consistency of compression-insensitive features by fusing compression-insensitive features into the artifacts reduction baseline. Our method achieves an average 2.06 dB PSNR gains on BSD500, outperforming state-of-the-art methods, and only requires 29.7 ms to process one image on BSD500. Besides, the experimental results on LIVE1 and LIU4K also demonstrate the efficiency, effectiveness, and superiority of the proposed method in terms of quantitative metrics, visual quality, and downstream machine vision tasks.

BARRN: A Blind Image Compression Artifact Reduction Network for Industrial IoT Systems

BARRN: A Blind Image Compression Artifact Reduction Network for Industrial IoT Systems

Pyramid Attention Network for Image Restoration

Self-similarity refers to the image prior widely used in image restoration algorithms that small but similar patterns tend to occur at different locations and scales. However, recent advanced deep convolutional neural network-based methods for image restoration do not take full advantage of self-similarities by relying on self-attention neural modules that only process information at the same scale. To solve this problem, we present a novel Pyramid Attention module for image restoration, which captures long-range feature correspondences from a multi-scale feature pyramid. Inspired by the fact that corruptions, such as noise or compression artifacts, drop drastically at coarser image scales, our attention module is designed to be able to borrow clean signals from their “clean” correspondences at the coarser levels. The proposed pyramid attention module is a generic building block that can be flexibly integrated into various neural architectures. Its effectiveness is validated through extensive experiments on multiple image restoration tasks: image denoising, demosaicing, compression artifact reduction, and super resolution. Without any bells and whistles, our PANet (pyramid attention module with simple network backbones) can produce state-of-the-art results with superior accuracy and visual quality. Our code is available at https://github.com/SHI-Labs/Pyramid-Attention-Networks

EARN: toward efficient and robust JPEG compression artifact reduction

EARN: toward efficient and robust JPEG compression artifact reduction

Deep steerable pyramid wavelet network for unified JPEG compression artifact reduction

Although numerous methods have been proposed to remove blocking artifacts in JPEG-compressed images, one important issue not well addressed so far is the construction of a unified model that requires no prior knowledge of the JPEG encoding parameters to operate effectively on different compression-level images (grayscale/color) while occupying relatively small storage space to save and run. To address this issue, in this paper, we present a unified JPEG compression artifact reduction model called DSPW-Net, which employs (1) the deep steerable pyramid wavelet transform network for Y-channel restoration, and (2) the classic U-Net architecture for CbCr-channel restoration. To enable our model to work effectively on images with a wide range of compression levels, the quality factor (QF) related features extracted by the convolutional layers in the QF-estimation network are incorporated in the two restoration branches. Meanwhile, recursive blocks with shared parameters are utilized to drastically reduce model parameters and shared-source residual learning is employed to avoid the gradient vanishing/explosion problem in training. Extensive quantitative and qualitative results tested on various benchmark datasets demonstrate the effectiveness of our model as compared with other state-of-the-art deblocking methods.

Video Compression Artifact Reduction by Fusing Motion Compensation and Global Context in a Swin-CNN Based Parallel Architecture

Video Compression Artifact Reduction aims to reduce the artifacts caused by video compression algorithms and improve the quality of compressed video frames. The critical challenge in this task is to make use of the redundant high-quality information in compressed frames for compensation as much as possible. Two important possible compensations: Motion compensation and global context, are not comprehensively considered in previous works, leading to inferior results. The key idea of this paper is to fuse the motion compensation and global context together to gain more compensation information to improve the quality of compressed videos. Here, we propose a novel Spatio-Temporal Compensation Fusion (STCF) framework with the Parallel Swin-CNN Fusion (PSCF) block, which can simultaneously learn and merge the motion compensation and global context to reduce the video compression artifacts. Specifically, a temporal self-attention strategy based on shifted windows is developed to capture the global context in an efficient way, for which we use the Swin transformer layer in the PSCF block. Moreover, an additional Ada-CNN layer is applied in the PSCF block to extract the motion compensation. Experimental results demonstrate that our proposed STCF framework outperforms the state-of-the-art methods up to 0.23dB (27% improvement) on the MFQEv2 dataset.

Rectified Wasserstein Generative Adversarial Networks for Perceptual Image Restoration.

Wasserstein generative adversarial network (WGAN) has attracted great attention due to its solid mathematical background, i.e., to minimize the Wasserstein distance between the generated distribution and the distribution of interest. In WGAN, the Wasserstein distance is quantitatively evaluated by the discriminator, also known as the critic. The vanilla WGAN trained the critic with the simple Lipschitz condition, which was later shown less effective for modeling complex distributions, like the distribution of natural images. We try to improve the WGAN training by introducing pairwise constraint on the critic, oriented to image restoration tasks. In principle, pairwise constraint is to suggest the critic assign a higher rating to the original (real) image than to the restored (generated) image, as long as such a pair of images are available. We show that such pairwise constraint may be implemented by rectifying the gradients in WGAN training, which leads to the proposed rectified Wasserstein generative adversarial network (ReWaGAN). In addition, we build interesting connections between ReWaGAN and the perception-distortion tradeoff. We verify ReWaGAN on two representative image restoration tasks: single image super-resolution (4× and 8×) and compression artifact reduction, where our ReWaGAN not only beats the vanilla WGAN consistently, but also outperforms the state-of-the-art perceptual quality-oriented methods significantly. Our code and models are publicly available at https://github.com/mahaichuan/ReWaGAN.

Multi-Scale Inter-Communication Spatio-Temporal Network for Video Compression Artifacts Reduction

Video compression artifacts are widespread in online videos, which greatly affects the quality of the videos. In recent years, the deep learning-based methods for video compression artifacts reduction have made impressive achievements. However, most of existing methods ignore the rich multi-scale information contained in video frames. In this brief, we propose a new multi-scale inter-communication spatio-temporal network (MSICSTN) for video compression artifacts reduction by fully exploring contextual multi-scale information. MSICSTN is constructed on the basis of the multi-scale feature inter-communication module (MSFICM) and the source feature selection enhancement module (SFSEM). Specifically, the MSFICM adaptively generates respective attention weights using channel statistical information after the inter-communication of different scale features, and recalibrates the input multi-scale features to enhance the multi-scale features. The SFSEM utilizes the statistical information of the shallow features of the target frame to fuse the shallow features with the deep features for better dense pixel predictions. The experimental results on test videos demonstrate that our method outperforms the existing methods.

CUR Transformer: A Convolutional Unbiased Regional Transformer for Image Denoising

Image denoising is a fundamental problem in computer vision and multimedia computation. Non-local filters are effective for image denoising. But existing deep learning methods that use non-local computation structures are mostly designed for high-level tasks, and global self-attention is usually adopted. For the task of image denoising, they have high computational complexity and have a lot of redundant computation of uncorrelated pixels. To solve this problem and combine the marvelous advantages of non-local filter and deep learning, we propose a Convolutional Unbiased Regional (CUR) transformer. Based on the prior that, for each pixel, its similar pixels are usually spatially close, our insights are that (1) we partition the image into non-overlapped windows and perform regional self-attention to reduce the search range of each pixel, and (2) we encourage pixels across different windows to communicate with each other. Based on our insights, the CUR transformer is cascaded by a series of convolutional regional self-attention (CRSA) blocks with U-style short connections. In each CRSA block, we use convolutional layers to extract the query, key, and value features, namely Q , K , and V , of the input feature. Then, we partition the Q , K , and V features into local non-overlapped windows and perform regional self-attention within each window to obtain the output feature of this CRSA block. Among different CRSA blocks, we perform the unbiased window partition by changing the partition positions of the windows. Experimental results show that the CUR transformer outperforms the state-of-the-art methods significantly on four low-level vision tasks, including real and synthetic image denoising, JPEG compression artifact reduction, and low-light image enhancement.

Vision Transformers in Image Restoration: A Survey.

The Vision Transformer (ViT) architecture has been remarkably successful in image restoration. For a while, Convolutional Neural Networks (CNN) predominated in most computer vision tasks. Now, both CNN and ViT are efficient approaches that demonstrate powerful capabilities to restore a better version of an image given in a low-quality format. In this study, the efficiency of ViT in image restoration is studied extensively. The ViT architectures are classified for every task of image restoration. Seven image restoration tasks are considered: Image Super-Resolution, Image Denoising, General Image Enhancement, JPEG Compression Artifact Reduction, Image Deblurring, Removing Adverse Weather Conditions, and Image Dehazing. The outcomes, the advantages, the limitations, and the possible areas for future research are detailed. Overall, it is noted that incorporating ViT in the new architectures for image restoration is becoming a rule. This is due to some advantages compared to CNN, such as better efficiency, especially when more data are fed to the network, robustness in feature extraction, and a better feature learning approach that sees better the variances and characteristics of the input. Nevertheless, some drawbacks exist, such as the need for more data to show the benefits of ViT over CNN, the increased computational cost due to the complexity of the self-attention block, a more challenging training process, and the lack of interpretability. These drawbacks represent the future research direction that should be targeted to increase the efficiency of ViT in the image restoration domain.

JPEG Artifact Reduction Based on Deformable Offset Gating Network Controlled by a Variational Autoencoder

For the reduction of JPEG compression artifacts, there have been many methods using deep neural networks. Most of them use the JPEG compression quality factor (QF) as prior knowledge in designing and training the networks. However, since the images we get from the Internet are often recompressed, the given QF is not so informative or misleading. Also, early works validated their methods on low QFs less than 50, while recent smartphones use high QFs larger than or equal to 90. In this paper, we propose a new JPEG artifacts reduction network considering the above-stated problems. Specifically, to extract quality information from the input image itself instead of the QF provided in the header of the JPEG file, we use a variational autoencoder (VAE) and regard its latent vector as quality information. In designing the artifact reduction network, we let the network change flexibly according to the input image quality by employing a deformable offset gating (DOG) network. The gating network and VAE are merged as our overall network, dubbed DOG-VAE, where the information from the VAE is used to adjust the DOG network according to the input quality. The DOG-VAE is trained end-to-end with the QFs in the range of [10,90]. Extensive experiments validate that our method achieves comparable results to the state-of-the-art method for monochrome images and better results for color images. Our codes are available at https://github.com/yunjh410/DOGNet.

Video restoration based on deep learning: a comprehensive survey

Video restoration concerns the recovery of a clean video sequence starting from its degraded version. Different video restoration tasks exist, including denoising, deblurring, super-resolution, and reduction of compression artifacts. In this paper, we provide a comprehensive review of the main features of existing video restoration methods based on deep learning. We focus our attention on the main architectural components, strategies for motion handling, and loss functions. We analyze the standard benchmark datasets and use them to summarize the performance of video restoration methods, both in terms of effectiveness and efficiency. In conclusion, the main challenges and future research directions in video restoration using deep learning are highlighted.

Compression artifact reduction of low bit‐rate videos via deep neural networks using self‐similarity prior

This paper presents a hybrid scheme that integratedly uses self-similarity prior and deep convolutional neural network (CNN) fusion for compression artifact reduction in low bit-rate video applications. Based on the temporal correlation hypothesis, the self-similarity prior is extended to the temporal domain by using as references not only the current decoded frame but also its neighbouring frames. Furthermore, being cognizant of that the bicubic downsampling process can typically improve the perceptual quality of a video coded at low bit-rate, for each small patch in the current frame, we search for similar patches in down-scaled versions of these references, and then form several self-similarity prior based predictions by tiling these similar patches at corresponding positions. To further exploit information flow across scales, a deep CNN model is constructed that contains two sub-networks to estimate the final output. One sub-network takes the self-similarity prior based predictions along with the decoded frame itself; and the other takes the down-scaled versions of these frames as network input. Experimental results demonstrate that the proposed method can remarkably improve, both subjectively and objectively, quality of compressed video sequences of low bit-rates.