Network For Image Restoration Research Articles

Iterative decoupling deconvolution network for image restoration

The iterative decoupled deblurring BM3D (IDDBM3D) (Danielyan et al., 2011) combines the analysis representation and the synthesis representation, where deblurring and denoising operations are decoupled, so that both problems can be easily solved. However, the IDDBM3D has some limitations. First, the analysis transformation and the synthesis transformation are analytical, thus have limited representation ability. Second, it is difficult to effectively remove image noise from threshold transformation. Third, there exists hyper-parameters to be tuned manually, which is difficult and time consuming. In this work, we propose an iterative decoupling deconvolution network(IDDNet), by unrolling the iterative decoupling algorithm of the IDDBM3D. In the proposed IDDNet, the analysis/synthesis transformation are implemented by encoder/decoder modules; the denoising is implemented by convolutional neural network based denoiser; the hyper-parameters are estimated by hyper-parameter module. We apply our models for image deblurring and super-resolution. Experimental results show that the IDDNet significantly outperforms the state-of-the-art unfolding networks.

Graph Convolutional Network for Image Restoration: A Survey

Image restoration technology is a crucial field in image processing and is extensively utilized across various domains. Recently, with advancements in graph convolutional network (GCN) technology, methods based on GCNs have increasingly been applied to image restoration, yielding impressive results. Despite these advancements, there is a gap in comprehensive research that consolidates various image denoising techniques. In this paper, we conduct a comparative study of image restoration techniques using GCNs. We begin by categorizing GCN methods into three primary application areas: image denoising, image super-resolution, and image deblurring. We then delve into the motivations and principles underlying various deep learning approaches. Subsequently, we provide both quantitative and qualitative comparisons of state-of-the-art methods using public denoising datasets. Finally, we discuss potential challenges and future directions, aiming to pave the way for further advancements in this domain. Our key findings include the identification of superior performance of GCN-based methods in capturing long-range dependencies and improving image quality across different restoration tasks, highlighting their potential for future research and applications.

A Global Self-Attention Memristive Neural Network for Image Restoration

A Global Self-Attention Memristive Neural Network for Image Restoration

An airborne radar forward-looking imaging algorithm based on multi-scale U-shaped networks

ABSTRACT Radar forward-looking imaging has gained increasingly significance in diverse applications, including battlefield reconnaissance, target surveillance, and precision guidance. While Synthetic Aperture Radar techniques are widely employed to achieve high azimuth resolution, the domain of forward-looking imaging has its challenges. These challenges arise due to the minimal Doppler frequency changes in forward-looking areas and the symmetrical Doppler history about flight path. These constraints negatively impact the radar’s ability to accurately detect and image targets in forward-looking areas. Consequently, there has been a growing interest in the development of novel approaches to these problems. Due to the application of deep learning, image super-resolution methods based on convolutional neural networks (CNNs) have received extensive attention and demonstrated good performance. This avenue of research provides fresh insights into improving the azimuth resolution of radar imaging. In this letter, we propose a novel convolutional neural network, called MSUS-Net (Multi-scale U-shaped Network), for radar forward-looking imaging. It combines asymmetric feature fusion with a unique connection structure to replace traditional layers. Moreover, different from conventional image restoration networks that use content loss, we use the L1 distance in the frequency domain, restoring missing high-frequency details. Simulated and real radar data validate that this approach offers robust recovery and better imaging performance.

Omni-Kernel Network for Image Restoration

Image restoration aims to reconstruct a high-quality image from a degraded low-quality observation. Recently, Transformer models have achieved promising performance on image restoration tasks due to their powerful ability to model long-range dependencies. However, the quadratically growing complexity with respect to the input size makes them inapplicable to practical applications. In this paper, we develop an efficient convolutional network for image restoration by enhancing multi-scale representation learning. To this end, we propose an omni-kernel module that consists of three branches, i.e., global, large, and local branches, to learn global-to-local feature representations efficiently. Specifically, the global branch achieves a global perceptive field via the dual-domain channel attention and frequency-gated mechanism. Furthermore, to provide multi-grained receptive fields, the large branch is formulated via different shapes of depth-wise convolutions with unusually large kernel sizes. Moreover, we complement local information using a point-wise depth-wise convolution. Finally, the proposed network, dubbed OKNet, is established by inserting the omni-kernel module into the bottleneck position for efficiency. Extensive experiments demonstrate that our network achieves state-of-the-art performance on 11 benchmark datasets for three representative image restoration tasks, including image dehazing, image desnowing, and image defocus deblurring. The code is available at https://github.com/c-yn/OKNet.

Mixup-Based Neural Network for Image Restoration and Structure Prediction From SEM Images

Mixup-Based Neural Network for Image Restoration and Structure Prediction From SEM Images

Revitalizing Convolutional Network for Image Restoration.

Image restoration aims to reconstruct a high-quality image from its corrupted version, playing essential roles in many scenarios. Recent years have witnessed a paradigm shift in image restoration from convolutional neural networks (CNNs) to Transformerbased models due to their powerful ability to model long-range pixel interactions. In this paper, we explore the potential of CNNs for image restoration and show that the proposed simple convolutional network architecture, termed ConvIR, can perform on par with or better than the Transformer counterparts. By re-examing the characteristics of advanced image restoration algorithms, we discover several key factors leading to the performance improvement of restoration models. This motivates us to develop a novel network for image restoration based on cheap convolution operators. Comprehensive experiments demonstrate that our ConvIR delivers state-ofthe- art performance with low computation complexity among 20 benchmark datasets on five representative image restoration tasks, including image dehazing, image motion/defocus deblurring, image deraining, and image desnowing.

Single Stage Adaptive Multi-Attention Network for Image Restoration.

Recently attention-based networks have been successful for image restoration tasks. However, existing methods are either computationally expensive or have limited receptive fields, adding constraints to the model. They are also less resilient in spatial and contextual aspects and lack pixel-to-pixel correspondence, which may degrade feature representations. In this paper, we propose a novel and computationally efficient architecture Single Stage Adaptive Multi-Attention Network (SSAMAN) for image restoration tasks, particularly for image denoising and image deblurring. SSAMAN efficiently addresses computational challenges and expands receptive fields, enhancing robustness in spatial and contextual feature representation. Its Adaptive Multi-Attention Module (AMAM), which consists of Adaptive Pixel Attention Branch (APAB) and an Adaptive Channel Attention Branch (ACAB), uniquely integrates channel and pixel-wise dimensions, significantly improving sensitivity to edges, shapes, and textures. We perform extensive experiments and ablation studies to validate the performance of SSAMAN. Our model shows state-of-the-art results on various benchmarks, for example, on image denoising tasks, SSAMAN achieves a notable 40.08 dB PSNR on SIDD dataset, outperforming Restormer by 0.06 dB PSNR, with 41.02% less computational cost, and achieves a 40.05 dB PSNR on the DND dataset. For image deblurring, SSAMAN achieves 33.53 dB PSNR on GoPro dataset. Code and models are available at Github.

U²-Former: Nested U-Shaped Transformer for Image Restoration via Multi-View Contrastive Learning

While Transformer has achieved remarkable performance in various high-level vision tasks, it is still challenging to exploit the full potential of Transformer in image restoration. The crux lies in the limited depth of applying Transformer in the typical encoder-decoder framework for image restoration, resulting from heavy self-attention computation load and inefficient communications across different depth (scales) of layers. In this paper, we present a deep and effective Transformer-based network for image restoration, termed as U <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -Former, which is able to employ self-attention of Transformer as the core operation for feature learning to perform image restoration in a deep encoding and decoding space. Specifically, it leverages the nested U-shaped structure to facilitate the interactions across different layers with different scales of feature maps. Furthermore, we optimize the computational efficiency for the basic Transformer block by introducing a simple yet effective feature-filtering mechanism to compress the token representation. Apart from the typical supervision ways for image restoration, our U <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -Former also performs multi-view contrastive learning, which constructs positive pairs in various aspects, to learn noise-sensitive but content-irrelevant features and further decouple the noise component from the background image. Extensive experiments on various image restoration tasks, including reflection removal, rain streak removal and dehazing respectively, demonstrate the effectiveness of the proposed U <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -Former.

MWA-MNN: Multi-patch Wavelet Attention Memristive Neural Network for image restoration

Adverse weather conditions can severely reduce the image quality captured by outdoor imaging devices. However, most existing image restoration algorithms are designed for specific tasks, lack generalization capabilities, and are unable to balance spatial details with contextual feature. Moreover, outdoor images are typically captured using edge devices with limited computing resources, rendering it challenging to deploy image restoration algorithms on edge devices. To tackle these problems, we propose a memristive-based image restoration algorithm, named Multi-patch Wavelet Attention Memristive Neural Network (MWA-MNN). Multiple image restoration tasks can be achieved by implementing Convolutional Neural Network (CNN) based on memristor crossbar arrays, as well as specified memristive circuits. Our proposed network comprises three sub-networks with multi-patch design enabling each sub-network to focus on different scales, thereby preserving more spatial details while fusing rich contextual feature. We propose the Discrete Wavelet Attention Block (DWAB) and the Coordinate Channel Attention Block (CCAB) to extract rich contextual information and the Supervised Selective Kernel Block (SSKB) to effectively filter the extracted features. Our proposed memristor-based circuit implementation provides an effective solution for deploying image restoration algorithms on edge devices. Experiment results on multiple benchmark datasets for image deraining, dehazing, and low-light image enhancement show that our proposed method outperforms over 20 state-of-the-art methods.

Pyramid Attention Network for Image Restoration

Self-similarity refers to the image prior widely used in image restoration algorithms that small but similar patterns tend to occur at different locations and scales. However, recent advanced deep convolutional neural network-based methods for image restoration do not take full advantage of self-similarities by relying on self-attention neural modules that only process information at the same scale. To solve this problem, we present a novel Pyramid Attention module for image restoration, which captures long-range feature correspondences from a multi-scale feature pyramid. Inspired by the fact that corruptions, such as noise or compression artifacts, drop drastically at coarser image scales, our attention module is designed to be able to borrow clean signals from their “clean” correspondences at the coarser levels. The proposed pyramid attention module is a generic building block that can be flexibly integrated into various neural architectures. Its effectiveness is validated through extensive experiments on multiple image restoration tasks: image denoising, demosaicing, compression artifact reduction, and super resolution. Without any bells and whistles, our PANet (pyramid attention module with simple network backbones) can produce state-of-the-art results with superior accuracy and visual quality. Our code is available at https://github.com/SHI-Labs/Pyramid-Attention-Networks

A Variational neural network for image restoration based on coupled regularizers

A Variational neural network for image restoration based on coupled regularizers

Local to non-local: Multi-scale progressive attention network for image restoration

Image restoration (IR) tasks aim to form a balance between complex textures and spatial details. To this end, the combination of local and non-local attention mechanisms has been well studied in recent years. However, existing local attention-based modules ignore the interaction between channel and spatial attention, while non-local attention operations solely focus on short-range or long-range dependency. To overcome these problems, a novel multi-scale progressive attention network is proposed in this paper, which is termed as MPANet. The proposed MPANet is composed of two parts, a local multi-scale feature extractor and a window-dilation self-attention module. In the local multi-scale feature extractor, a single-scale feature enhancement strategy is designed to model the correlation between channel and spatial dimensions, and a multi-scale feature fusion strategy is applied to further exchange contextual information across all the scales. Furthermore, the window-dilation self-attention is introduced to establish global representation while preserving local details. Experimental results on four IR tasks demonstrate that the proposed MPANet outperforms the state-of-the-art methods in both quantitative results and visual perception.

On quantization of convolutional neural networks for image signal processor

Recently, many deep learning applications have been used on the mobile platform. To deploy them in the mobile platform, the networks should be quantized. The quantization of computer vision networks has been studied well but there have been few studies for the quantization of image restoration networks. In this paper, we studied the effect of the quantization of activations for deep learning network on image quality following previous study for weight quantization for deep learning network. This study is also about the quantization on raw RGBW image demosaicing for 10 bit image while fixing weight bit as 8 bit. Experimental results show that 11 bit activation quantization can sustain image quality at the similar level with floating-point network. Even though the activations bit-depth can be very small bit in the computer vision applications, but image restoration tasks like demosaicing require much more bits than those applications. 11 bit may not fit the general purpose hardware like NPU, GPU or CPU but for the custom hardware it is very important to reduce its hardware area and power as well as memory size.

Iterative deep neural networks based on proximal gradient descent for image restoration

The algorithm unfolding networks with explainability of algorithms and higher efficiency of Deep Neural Networks (DNN) have received considerable attention in solving ill-posed inverse problems. Under the algorithm unfolding network framework, we propose a novel end-to-end iterative deep neural network and its fast network for image restoration. The first one is designed making use of proximal gradient descent algorithm of variational models, which consists of denoiser and reconstruction sub-networks. The second one is its accelerated version with momentum factors. For sub-network of denoiser, we embed the Convolutional Block Attention Module (CBAM) in previous U-Net for adaptive feature refinement. Experiments on image denoising and deblurring demonstrate that competitive performances in quality and efficiency are gained by compared with several state-of-the-art networks for image restoration. Proposed unfolding DNN can be easily extended to solve other similar image restoration tasks, such as image super-resolution, image demosaicking, etc.

On quantization of convolutional neural networks for image restoration

Recently, commercial vision sensors hit the mobile market. To achieve that, computer vision networks had to be quantized. However, this topic was not studied well for Image Signal processor (ISP) challenging image restoration tasks, being crucially important for hardware implementation, as well as for deployment on hardware accelerators, e.g. Neural Processors Units (NPU). In this paper, we studied the effect of the quantization of deep learning network on image quality. We tried various quantization on raw RGBW image demosaicing. Experimental results show that 12 bit weight quantization can sustain image quality at the similar level with floating-point network. 10 bit quantized network shows slight degradation in objective image quality and mild visual artifacts. If network weight&#xE2;&#x20AC;&#x2122;s bit-depth can be significantly reduced for computer vision tasks, our finding shows that it is not true for raw image restoration tasks: at least 10 bit weights are required to provide sufficient image quality. However, we can save some memory on feature maps bit-depth. We can conclude that network bit depth is critical for raw image restoration.

A Non-Local Enhanced Network for Image Restoration

Non-local modules have been widely studied in image restoration (IR) tasks since they can learn long-range dependencies to enhance local features. However, most existing non-local modules still focus on extracting long-range dependencies within a single image or feature map. On the other hand, most IR methods simply employ a single type of non-local module in the network. A combination of various types of non-local modules to enhance local features can be more effective. In this paper, we propose a batch-wise non-local module to explore richer non-local dependencies within images. Furthermore, we combine various non-local extractors (different attention modules) with the proposed batch-wise non-local module as the Enhanced Batch-wise Non-local Attentive module (EBNA). Besides exploring richer non-local information, we build the Non-local and Local Information extracting Block (NLIB), in which we combine the EBNA with DEformable-Convolution Block (DECB) to utilize richer non-local and adaptive local information. Finally, We embed the NLIB within a U-net-like structure and build the Non-local Enhanced Network (NLENet). Extensive experiments on synthetic image denoising, real image denoising, JPEG artifacts removal, and real image super resolution tasks demonstrate that our proposed network achieves state-of-the-art performance on several IR benchmark datasets.

Local to Non-Local: Multi-Scale Progressive Attention Network for Image Restoration

Local to Non-Local: Multi-Scale Progressive Attention Network for Image Restoration

TAO-Net: Task-Adaptive Operation Network for Image Restoration and Enhancement

Recently, deep learning based models have been extensively applied to image restoration and enhancement tasks. However, existing approaches neglect the potential correlation among these tasks, and do not fully consider the different intensity factors of degradation which affect image quality. To this end, we propose Task-Adaptive Operation Network (TAO-Net) that stacks a series of operation blocks, guided by the supervised attention mechanism to adaptively enable the model to assign corresponding intensity weights for different degradation factors depending on the input signals. To better recover and preserve the accurate details during the repeated operations, we make full use of the image prior to intruduce structure refinement block as auxiliary information for reconstructing high-quality results. Furthermore, feature aggregation block is also adopted to avoid feature interference caused by the direct concatenation between the backbone operation block and the auxiliary refinement block together. Extensive experiments on multiple benchmark datasets demonstrate that our proposed method outperforms previous baselines in image deraining and low-light image enhancement.

Learning Deformable and Attentive Network for image restoration

Image restoration (IR) aims to recover image quality from various degradations. Existing convolutional neural networks (CNN) based IR methods try to improve performance by enlarging the model receptive field with the sacrifice of fine spatial details and extra artifacts. This paper proposes a Deformable and Attentive Network (DANet) to address these problems. In DANet, we propose two novel blocks: Attentive DEformable-convolution Block (ADEB) and Attentive Recurrent Offset Block (AROB). In ADEB, deformable convolution is collaborated with various attention modules to generate more adaptive receptive fields. AROB transfers more attentive texture information among different scales during the encoding/decoding process for ADEB. To further refine DANet, we propose a knowledge distillation scheme to train a light-weighted DANet (DANet-S) with limited performance loss. Extensive experiments on several image benchmark datasets demonstrate that our method achieves state-of-the-art (SOTA) results for various IR tasks, including image denoising, JPEG artifacts removal, and real-world super resolution.