SR Network Research Articles

Enhanced local multi-windows attention network for lightweight image super-resolution

Since the global self-attention mechanism can capture long-distance dependencies well, Transformer-based methods have achieved remarkable performance in many vision tasks, including single-image super-resolution (SISR). However, there are strong local self-similarities in images, if the global self-attention mechanism is still used for image processing, it may lead to excessive use of computing resources on parts of the image with weak correlation. Especially in the high-resolution large-size image, the global self-attention will lead to a large number of redundant calculations. To solve this problem, we propose the Enhanced Local Multi-windows Attention Network (ELMA), which contains two main designs. First, different from the traditional self-attention based on square window partition, we propose a Multi-windows Self-Attention (M-WSA) which uses a new window partitioning mechanism to obtain different types of local long-distance dependencies. Compared with original self-attention mechanisms commonly used in other SR networks, M-WSA reduces computational complexity and achieves superior performance through analysis and experiments. Secondly, we propose a Spatial Gated Network (SGN) as a feed-forward network, which can effectively overcome the problem of intermediate channel redundancy in traditional MLP, thereby improving the parameter utilization and computational efficiency of the network. Meanwhile, SGN introduces spatial information into the feed-forward network that traditional MLP cannot obtain. It can better understand and use the spatial structure information in the image, and enhances the network performance and generalization ability. Extensive experiments show that ELMA achieves competitive performance compared to state-of-the-art methods while maintaining fewer parameters and computational costs.

Expression level of cardiac ryanodine receptors dictates properties of Ca2+-induced Ca2+ release

The type 2 ryanodine receptor (RyR2) is the major Ca2+ release channel required for Ca2+-induced Ca2+ release (CICR) and cardiac excitation-contraction coupling. The cluster organization of RyR2 at the dyad is critical for efficient CICR. Despite its central role in cardiac Ca2+ signaling, the mechanisms that control CICR are not fully understood. As a single RyR2 Ca2+ flux dictates local CICR that underlies Ca2+ sparks, RyR2 density in a cluster, and therefore the distance between RyR2s, should have a profound impact on local CICR. Here, we studied the effect of the RyR2 expression level ([RyR2]) on CICR activation, termination, and amplitude. The endoplasmic reticulum (ER)-targeted Ca2+ sensor RCEPIA-1er was used to directly measure the ER [Ca2+] (Ca2+]ER) in the T-Rex-293 the sarco/endoplasmic reticulum Ca2+-ATPase (SERCA2a) stable cell line expressing human RyR2. Cells coexpressing RyR2 and SERCA2a produced periodic [Ca2+]ER depletions in the form of spontaneous Ca2+ waves due to propagating CICR. For each studied cell, the [Ca2+]ER at which Ca2+ waves are activated and terminated was analyzed as a function of [RyR2]. CICR parameters, such as [Ca2+]ER activation, termination, and amplitude, were inversely proportional to [RyR2] at low-intermediate levels. Increasing the sensitivity of RyR2 to cytosolic Ca2+ lowered the [Ca2+]ER at which CICR is activated and terminated. Decreasing the sensitivity of RyR2 to cytosolic Ca2+ had the opposite effect on CICR. These results suggest that RyR2 density in the release cluster should have a significant impact on local CICR activation and termination. Since SR Ca2+ load is evenly distributed throughout the SR network, clusters with higher RyR2 density would have a higher probability of initiating spontaneous CICR.

GlobalSR: Global context network for single image super-resolution via deformable convolution attention and fast Fourier convolution

Vision Transformer have achieved impressive performance in image super-resolution. However, they suffer from low inference speed mainly because of the quadratic complexity of multi-head self-attention (MHSA), which is the key to learning long-range dependencies. On the contrary, most CNN-based methods neglect the important effect of global contextual information, resulting in inaccurate and blurring details. If one can make the best of both Transformers and CNNs, it will achieve a better trade-off between image quality and inference speed. Based on this observation, firstly assume that the main factor affecting the performance in the Transformer-based SR models is the general architecture design, not the specific MHSA component. To verify this, some ablation studies are made by replacing MHSA with large kernel convolutions, alongside other essential module replacements. Surprisingly, the derived models achieve competitive performance. Therefore, a general architecture design GlobalSR is extracted by not specifying the core modules including blocks and domain embeddings of Transformer-based SR models. It also contains three practical guidelines for designing a lightweight SR network utilizing image-level global contextual information to reconstruct SR images. Following the guidelines, the blocks and domain embeddings of GlobalSR are instantiated via Deformable Convolution Attention Block (DCAB) and Fast Fourier Convolution Domain Embedding (FCDE), respectively. The instantiation of general architecture, termed GlobalSR-DF, proposes a DCA to extract the global contextual feature by utilizing Deformable Convolution and a Hadamard product as the attention map at the block level. Meanwhile, the FCDE utilizes the Fast Fourier to transform the input spatial feature into frequency space and then extract image-level global information from it by convolutions. Extensive experiments demonstrate that GlobalSR is the key part in achieving a superior trade-off between SR quality and efficiency. Specifically, our proposed GlobalSR-DF outperforms state-of-the-art CNN-based and ViT-based SISR models regarding accuracy-speed trade-offs with sharp and natural details.

REMA: A Rich Elastic Mixed Attention Module for Single Image Super-Resolution.

Detail preservation is a major challenge for single image super-resolution (SISR). Many deep learning-based SISR methods focus on lightweight network design, but these may fall short in real-world scenarios where performance is prioritized over network size. To address these problems, we propose a novel plug-and-play attention module, rich elastic mixed attention (REMA), for SISR. REMA comprises the rich spatial attention module (RSAM) and the rich channel attention module (RCAM), both built on Rich Structure. Based on the results of our research on the module's structure, size, performance, and compatibility, Rich Structure is proposed to enhance REMA's adaptability to varying input complexities and task requirements. RSAM learns the mutual dependencies of multiple LR-HR pairs and multi-scale features, while RCAM accentuates key features through interactive learning, effectively addressing detail loss. Extensive experiments demonstrate that REMA significantly improves performance and compatibility in SR networks compared to other attention modules. The REMA-based SR network (REMA-SRNet) outperforms comparative algorithms in both visual effects and objective evaluation quality. Additionally, we find that module compatibility correlates with cardinality and in-branch feature bandwidth, and that networks with high effective parameter counts exhibit enhanced robustness across various datasets and scale factors in SISR.

Efficient blind super-resolution imaging via adaptive degradation-aware estimation

Single image super-resolution (SISR) has achieved prominent success based on deep learning. However, most SISR methods based on the specific degradation pattern, e.g., bicubic interpolation with a bulky network structure, are unsuitable for real-world images reconstructed on edge devices. To mitigate the above limitations, unlike the majority of blind super-resolution methods, which consist of a separate complex degradation kernel projection network and a SR reconstruction network, we propose an end-to-end lightweight multi-degradation oriented network that not only can estimate degradation kernel but also reconstructs HR image fast and accurately. The core components of the designed network adopt a lightweight SR model with region non-local feature similarity learning, i.e., Region Non-Local Feature Block (RNLFB), to establish region-wise global feature correlation. Concretely, more and fewer RNLFBs are used to learn more sophisticated feature representation and relatively simple degradation representation. Then, a degradation kernel projector is equipped to fulfill adaptive degradation-aware estimation from the degradation representation. With the help of the encoded degradation kernel, the reconstruction model can learn to restore a high-resolution image from the feature representation. By doing so, the degradation kernel estimation task and the HR image reconstruction task can be accomplished without designing two complex networks separately, which guides more stable and effective network training under low computational resource costs. Extensive experiments on synthetic and real-world datasets demonstrate that our method can fulfill precise degradation kernel prediction and HR image reconstruction compared with other state-of-the-art SR methods with lower model complexity.

Enhanced Video Super-Resolution Network towards Compressed Data

Video super-resolution (VSR) algorithms aim at recovering a temporally consistent high-resolution (HR) video from its corresponding low-resolution (LR) video sequence. Due to the limited bandwidth during video transmission, most available videos on the internet are compressed. Nevertheless, few existing algorithms consider the compression factor in practical applications. In this paper, we propose an enhanced VSR model towards compressed videos, termed as ECVSR, to simultaneously achieve compression artifacts reduction and SR reconstruction end-to-end. ECVSR contains a motion-excited temporal adaption network (METAN) and a multi-frame SR network (SRNet). The METAN takes decoded LR video frames as input and models inter-frame correlations via bidirectional deformable alignment and motion-excited temporal adaption, where temporal differences are calculated as motion prior to excite the motion-sensitive regions of temporal features. In SRNet, cascaded recurrent multi-scale blocks (RMSB) are employed to learn deep spatio-temporal representations from adapted multi-frame features. Then, we build a reconstruction module for spatio-temporal information integration and HR frame reconstruction, which is followed by a detail refinement module for texture and visual quality enhancement. Extensive experimental results on compressed videos demonstrate the superiority of our method for compressed VSR. Code will be available at https://github.com/lifengcs/ECVSR .

SuperVidConform: Conformation detail-preserving network (CDPN) for video super-resolution

Video Super Resolution (VSR) applications extensively utilize deep learning-based methods. Several VSR methods primarily focus on improving the fine-patterns within reconstructed video frames. It frequently overlooks the crucial aspect of keeping conformation details, particularly sharpness. Therefore, reconstructed video frames often fail to meet expectations. In this paper, we propose a Conformation Detail-Preserving Network (CDPN) named as SuperVidConform. It focuses on restoring local region features and maintaining the sharper details of video frames. The primary focus of this work is to generate the high-resolution (HR) frame from its corresponding low-resolution (LR). It consists of two parts: (i) The proposed model decomposes confirmation details from the ground-truth HR frames to provide additional information for the super-resolution process, and (ii) These video frames pass to the temporal modelling SR network to learn local region features by residual learning that connects the network intra-frame redundancies within video sequences. The proposed approach is designed and validated using VID4, SPMC, and UDM10 datasets. The experimental results show the proposed model presents an improvement of 0.43 dB (VID4), 0.78 dB (SPMC), and 0.84 dB (UDM10) in terms of PSNR. Further, the CDPN model set new standards for the performance of self-generated surveillance datasets.

FDSR: An Interpretable Frequency Division Stepwise Process Based Single-Image Super-Resolution Network.

Deep learning has excelled in single-image super-resolution (SISR) applications, yet the lack of interpretability in most deep learning-based SR networks hinders their applicability, especially in fields like medical imaging that require transparent computation. To address these problems, we present an interpretable frequency division SR network that operates in the image frequency domain. It comprises a frequency division module and a step-wise reconstruction method, which divides the image into different frequencies and performs reconstruction accordingly. We develop a frequency division loss function to ensure that each reconstruction module (ReM) operates solely at one image frequency. These methods establish an interpretable framework for SR networks, visualizing the image reconstruction process and reducing the black box nature of SR networks. Additionally, we revisited the subpixel layer upsampling process by deriving its inverse process and designing a displacement generation module. This interpretable upsampling process incorporates subpixel information and is similar to pre-upsampling frameworks. Furthermore, we develop a new ReM based on interpretable Hessian attention to enhance network performance. Extensive experiments demonstrate that our network, without the frequency division loss, outperforms state-of-the-art methods qualitatively and quantitatively. The inclusion of the frequency division loss enhances the network's interpretability and robustness, and only slightly decreases the PSNR and SSIM metrics by an average of 0.48 dB and 0.0049, respectively.

Multi-scale feature fusion kernel estimation with masked interpolation loss for real-world remote sensing images super-resolution

ABSTRACT In recent years, the application of deep learning to remote sensing image super-resolution has achieved promising results. However, most deep learning-based methods are often trained on remote sensing datasets constructed by bicubic downsampling, and their recovery effects on real-world remote sensing images are often unsatisfactory. This is because the process of generating low-resolution (LR) images from high-resolution (HR) images to construct training data pairs (LR-HR) using simple bicubic downsampling cannot reflect the degradation process of real-world remote sensing dataset images. In this paper, we propose a multi-scale feature fusion kernel estimation with masked interpolation loss for real-world remote sensing images super-resolution (MFFILSR) to address this problem. MFFILSR is divided into two stages: degenerate kernel estimation and SR network training. In the first stage, we propose a multi-scale feature fusion kernel estimation network that can effectively fuse multi-scale information, making the estimated downsampling kernel closer to the degradation patterns of real-world remote sensing images. In the second stage, we introduce a masked interpolation loss during generator training, by masking the interpolation loss, the artefacts of generated images can be effectively reduced. Extensive experiments show that MFFILSR has satisfactory super-resolution reconstruction performance for real-world remote sensing images.

Scale-Aware Frequency Attention network for super-resolution

Scale-arbitrary super-resolution (SASR) has recently gained widespread attention in the community, which aims to super-resolve an image to arbitrary desired resolutions. Existing SASR methods always use a fixed convolution neural network (CNN) as the backbone to share the same image features for different resolutions, leading to limited reconstruction performance due to the lack of distinctive feature extraction for individual resolutions. Meanwhile, these CNN-based methods always tend to overfit low-frequency components while losing high-frequency ones. This frequency gap leads to blurred structures in the reconstructed image. To address these issues, we propose a Scale Aware Frequency Attention network (SAFANet), which consists of Scale-Aware Frequency Attention (SAFA) modules and a Scale-Guided Continuous Reconstruction (SGCR) module. Additionally, frequency-domain attention learning is introduced to optimize the scale-adapted features to explicitly focus on high-frequency components, thereby overcoming the frequency restoration gap. Finally, the SGCR module combines the input scale as guidance with the optimized features to reconstruct images with arbitrary resolutions by using an implicit reconstruction function, thus adaptively achieving continuous image restoration. Our proposed SAFANet can be easily incorporated into existing CNN-based SR networks to achieve interactive SR at arbitrary scales. Extensive experiments demonstrate that SAFANet achieves superior performance, which is competitive with or even better than the state-of-the-art methods across a vast scale range (×1∼×30) via a single model.

Multimodal-Boost: Multimodal Medical Image Super-Resolution Using Multi-Attention Network With Wavelet Transform.

Multimodal medical images are widely used by clinicians and physicians to analyze and retrieve complementary information from high-resolution images in a non-invasive manner. Loss of corresponding image resolution adversely affects the overall performance of medical image interpretation. Deep learning-based single image super resolution (SISR) algorithms have revolutionized the overall diagnosis framework by continually improving the architectural components and training strategies associated with convolutional neural networks (CNN) on low-resolution images. However, existing work lacks in two ways: i) the SR output produced exhibits poor texture details, and often produce blurred edges, ii) most of the models have been developed for a single modality, hence, require modification to adapt to a new one. This work addresses (i) by proposing generative adversarial network (GAN) with deep multi-attention modules to learn high-frequency information from low-frequency data. Existing approaches based on the GAN have yielded good SR results; however, the texture details of their SR output have been experimentally confirmed to be deficient for medical images particularly. The integration of wavelet transform (WT) and GANs in our proposed SR model addresses the aforementioned limitation concerning textons. While the WT divides the LR image into multiple frequency bands, the transferred GAN uses multi-attention and upsample blocks to predict high-frequency components. Additionally, we present a learning method for training domain-specific classifiers as perceptual loss functions. Using a combination of multi-attention GAN loss and a perceptual loss function results in an efficient and reliable performance. Applying the same model for medical images from diverse modalities is challenging, our work addresses (ii) by training and performing on several modalities via transfer learning. Using two medical datasets, we validate our proposed SR network against existing state-of-the-art approaches and achieve promising results in terms of structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR).

Lightweight Implicit Blur Kernel Estimation Network for Blind Image Super-Resolution

Blind image super-resolution (Blind-SR) is the process of leveraging a low-resolution (LR) image, with unknown degradation, to generate its high-resolution (HR) version. Most of the existing blind SR techniques use a degradation estimator network to explicitly estimate the blur kernel to guide the SR network with the supervision of ground truth (GT) kernels. To solve this issue, it is necessary to design an implicit estimator network that can extract discriminative blur kernel representation without relying on the supervision of ground-truth blur kernels. We design a lightweight approach for blind super-resolution (Blind-SR) that estimates the blur kernel and restores the HR image based on a deep convolutional neural network (CNN) and a deep super-resolution residual convolutional generative adversarial network. Since the blur kernel for blind image SR is unknown, following the image formation model of blind super-resolution problem, we firstly introduce a neural network-based model to estimate the blur kernel. This is achieved by (i) a Super Resolver that, from a low-resolution input, generates the corresponding SR image; and (ii) an Estimator Network generating the blur kernel from the input datum. The output of both models is used in a novel loss formulation. The proposed network is end-to-end trainable. The methodology proposed is substantiated by both quantitative and qualitative experiments. Results on benchmarks demonstrate that our computationally efficient approach (12x fewer parameters than the state-of-the-art models) performs favorably with respect to existing approaches and can be used on devices with limited computational capabilities.

Lightweight Image Super-Resolution with ConvNeXt Residual Network

Single image super-resolution based on convolutional neural networks has been very successful in recent years. However, as the computational cost is too high, making it difficult to apply to resource-constrained devices, a big challenge for existing approaches is to find a balance between the complexity of the CNN model and the quality of the resulting SR. To solve this problem, various lightweight SR networks have been proposed. In this paper, we propose lightweight and efficient residual networks (IRN), which differ from previous lightweight SR networks that aggregate more powerful features by improving feature utilization through complex layer-connection strategies. The main idea is to simplify feature aggregation by using simple and efficient residual modules for feature learning, thus achieving a good trade-off between the computational cost of the model and the quality of the resulting SR. In addition, we revisit the impact of the activation function in the model and observe that different activation functions have an impact on the performance of the model. The experiment results show that IRN outperforms previous state-of-the-art methods in benchmark tests while maintaining a relatively low computational cost. The code will be available at https://github.com/kptx666/IRN .

Iterative dual regression network for blind image super-resolution

Previous single-image super-resolution methods assume that the blur kernel is known (e.g., bicubic) when degrading from high-resolution (HR) images to low-resolution (LR) images. They use a single degradation to train a model to restore HR images. However, the actual degradation in the real world is often unknown. It is difficult to deal with LR images caused by different degradations. To cope with the above situation, previous methods attempt to restore SR images using a blur kernel estimation structure that combines with a non-blind SR network. There are two problems that should be earnestly considered: (1) For accurate blur kernel estimation, insufficient correlation of consecutive kernels lead to an unsatisfied reconstruction result. (2) For ill-posed issue of image reconstruction, a more efficient constraint condition is worth trying. To solve the two problems, we propose an iterative dual regression network for an adaptive and precision blur kernel estimation, which improves the speed of kernel estimation by learning a dual mapping. Specifically, we design a Predictor-Generator structure: the Predictor, through several iterations, searching for accurate kernels through intermediate kernels and generated SR images; the Generator, generating final SR images with the help of the predicted kernels. More importantly, the elaborately designed dual learning strategy can not only provide additional constraints for accurate kernel estimation but also reduce the domain gap between SR images and HR images. Experiments on synthetic degraded images and real-world images show that our network is competitive in performance and superior in visual results.

Lightweight Remote-Sensing Image Super-Resolution via Re-Parameterized Feature Distillation Network

Recent deep learning based-works have made remarkable progress in Remote-Sensing Image Super-Resolution (RSISR). However, the complicated network architecture as well as a huge amount of parameters increase computational cost, hindering their practical deployment. To alleviate this problem, we propose a novel Re-parameterized Feature Distillation Network (ReFDN) for lightweight and efficient RSISR tasks. Feature distillation, refinement, condensation, and enhancement are efficiently integrated into the re-parameterized feature distillation block named ReFDB for lighter and stronger feature extraction. With the help of elaborate re-parameterized convolution (ReConv) design, we further boost the feature refinement capability without extra inference costs. Additionally, we design an efficient channel and spatial attention module (ECSA) to enhance the important objects and regions of the intermediate features adaptively. Conducted on both commonly used datasets and additional Google Earth data, the experimental results demonstrate our method can achieve a good trade-off between SR performance and network complexity. Our code will be publicly available at https://github.com/DaxingZ/ReFDN.

Meta-Learning based Degradation Representation for Blind Super-Resolution.

Blind image super-resolution (blind SR) aims to generate high-resolution (HR) images from low-resolution (LR) input images with unknown degradations. To enhance the performance of SR, the majority of blind SR methods introduce an explicit degradation estimator, which helps the SR model adjust to unknown degradation scenarios. Unfortunately, it is impractical to provide concrete labels for the multiple combinations of degradations (e.g., blurring, noise, or JPEG compression) to guide the training of the degradation estimator. Moreover, the special designs for certain degradations hinder the models from being generalized for dealing with other degradations. Thus, it is imperative to devise an implicit degradation estimator that can extract discriminative degradation representations for all types of degradations without requiring the supervision of degradation ground-truth. To this end, we propose a Meta-Learning based Region Degradation Aware SR Network (MRDA), including Meta-Learning Network (MLN), Degradation Extraction Network (DEN), and Region Degradation Aware SR Network (RDAN). To handle the lack of ground-truth degradation, we use the MLN to rapidly adapt to the specific complex degradation after several iterations and extract implicit degradation information. Subsequently, a teacher network MRDAT is designed to further utilize the degradation information extracted by MLN for SR. However, MLN requires iterating on paired LR and HR images, which is unavailable in the inference phase. Therefore, we adopt knowledge distillation (KD) to make the student network learn to directly extract the same implicit degradation representation (IDR) as the teacher from LR images. Furthermore, we introduce an RDAN module that is capable of discerning regional degradations, allowing IDR to adaptively influence various texture patterns. Extensive experiments under classic and real-world degradation settings show that MRDA achieves SOTA performance and can generalize to various degradation processes.

Epistemic-Uncertainty-Based Divide-and-Conquer Network for Single-Image Super-Resolution

The introduction of convolutional neural networks (CNNs) into single-image super-resolution (SISR) has resulted in remarkable performance in the last decade. There is a contradiction in SISR between indiscriminate processing and the different processing difficulties in different regions, leading to the need for locally differentiated processing of SR networks. In this paper, we propose an epistemic-uncertainty-based divide-and-conquer network (EU-DC) in order to address this problem. Firstly, we build an image-gradient-based divide-and-conquer network (IG-DC) that utilizes gradient-based division to separate degraded images into easy and hard processing regions. Secondly, we model the IG-DC’s epistemic uncertainty map (EUM) by using Monte Carlo dropout and, thus, measure the output confidence of the IG-DC. The lower the output confidence is, the more difficult the IG-DC is to process. The EUM-based division is generated by quantizing the EUM into two levels. Finally, the IG-DC is transformed into an EU-DC by substituting the gradient-based division with EUM-based division. Our extensive experiments demonstrate that the proposed EU-DC achieves better reconstruction performance than that of multiple state-of-the-art SISR methods in terms of both quantitative and visual quality.

Structure-Preserving Image Super-Resolution.

Structures matter in single image super-resolution (SISR). Benefiting from generative adversarial networks (GANs), recent studies have promoted the development of SISR by recovering photo-realistic images. However, there are still undesired structural distortions in the recovered images. In this paper, we propose a structure-preserving super-resolution (SPSR) method to alleviate the above issue while maintaining the merits of GAN-based methods to generate perceptual-pleasant details. First, we propose SPSR with gradient guidance (SPSR-G) by exploiting gradient maps of images to guide the recovery in two aspects. On the one hand, we restore high-resolution gradient maps by a gradient branch to provide additional structure priors for the SR process. On the other hand, we propose a gradient loss to impose a second-order restriction on the super-resolved images, which helps generative networks concentrate more on geometric structures. Second, since the gradient maps are handcrafted and may only be able to capture limited aspects of structural information, we further extend SPSR-G by introducing a learnable neural structure extractor (NSE) to unearth richer local structures and provide stronger supervision for SR. We propose two self-supervised structure learning methods, contrastive prediction and solving jigsaw puzzles, to train the NSEs. Our methods are model-agnostic, which can be potentially used for off-the-shelf SR networks. Experimental results on five benchmark datasets show that the proposed methods outperform state-of-the-art perceptual-driven SR methods under LPIPS, PSNR, and SSIM metrics. Visual results demonstrate the superiority of our methods in restoring structures while generating natural SR images. Code is available at https://github.com/Maclory/SPSR.

Pedestrian detection using a translation-invariant wavelet residual dense super-resolution.

Pedestrian detection is an important research area and technology for car driving, gait recognition, and other applications. Although a lot of pedestrian detection techniques have been introduced, low-resolution imaging devices still exist in real life, so detection in low-resolution images remains a challenging problem. To address this issue, we propose a novel end-to-end Translation-invariant Wavelet Residual Dense Super-Resolution (TiWRD-SR) method to upscale LR images to SR images and then use Yolov4 for detection to address the low detection problem performance on low-resolution images. To make the enlarged SR image not only effectively distinguish the foreground and background of images but also highlight the characteristic structure of pedestrians, we decompose the image into low-frequency and high-frequency parts by stationary wavelet transform (SWT). The high- and low-frequency sub-images are trained through different network structures so that the network can reconstruct the high-frequency image edge information and the low-frequency image structure in a more detailed manner. In addition, a high-to-low branch information transmission (H2LBIT) is proposed to import high-frequency image edge information into the low-frequency network to make the reconstructed low-frequency structure more detailed. In addition, we also propose a novel loss function, which enables the SR network to focus on the reconstruction of image structure in the network by the characteristics of wavelet decomposition, thereby improving its detection performance. The experimental results indicate that the proposed TiWRD-SR can effectively improve detection performance.

Unsupervised Remote Sensing Image Super-Resolution Guided by Visible Images

Remote sensing images are widely used in many applications. However, due to being limited by the sensors, it is difficult to obtain high-resolution (HR) images from remote sensing images. In this paper, we propose a novel unsupervised cross-domain super-resolution method devoted to reconstructing a low-resolution (LR) remote sensing image guided by an unpaired HR visible natural image. Therefore, an unsupervised visible image-guided remote sensing image super-resolution network (UVRSR) is built. The network is divided into two learnable branches: a visible image-guided branch (VIG) and a remote sensing image-guided branch (RIG). As HR visible images can provide rich textures and sufficient high-frequency information, the purpose of VIG is to treat them as targets and make full use of their advantages in reconstruction. Specially, we first use a CycleGAN to drag the LR visible natural images to the remote sensing domain; then, we apply an SR network to upscale these simulated remote sensing domain LR images. However, the domain gap between SR remote sensing images and HR visible targets is massive. To enforce domain consistency, we propose a novel domain-ruled discriminator in the reconstruction. Furthermore, inspired by the zero-shot super-resolution network (ZSSR) to explore the internal information of remote sensing images, we add a remote sensing domain inner study to train the SR network in RIG. Sufficient experimental works show UVRSR can achieve superior results with state-of-the-art unpaired and remote sensing SR methods on several challenging remote sensing image datasets.