Super-resolution Framework Research Articles

Learning face super-resolution through identity features and distilling facial prior knowledge

Deep learning techniques in electronic surveillance have shown impressive performance for super-resolution (SR) of captured low-quality face images. Most of these techniques adopt facial priors to improve the feature details in the resultant super-resolved images. However, the estimation of facial priors from the captured low-quality images is often inaccurate in real-life situations because of their tiny, noisy, and blurry nature. Thus, the fusion of such priors badly affects the performance of these models. Therefore, this work presents a teacher–student-based face SR framework that efficiently preserves the personal facial structure information in the super-resolved faces. In the proposed framework, the teacher network exploits the facial heatmap-based ground-truth-prior to learn the facial structure that is utilized by the student network. The student network is trained with the identity feature loss for maintaining the identity and facial structure information in reconstructed high-resolution (HR) face images. The performance of the proposed framework is evaluated by conducting the experimental study on standard datasets namely CelebA-HQ and LFW face. The experimental results reveal that the proposed technique conquers the existing methods for the face SR task.

FuBay: An Integrated Fusion Framework for Hyperspectral Super-Resolution Based on Bayesian Tensor Ring.

Fusion with corresponding finer-resolution images has been a promising way to enhance hyperspectral images (HSIs) spatially. Recently, low-rank tensor-based methods have shown advantages compared with other kind of ones. However, these current methods either relent to blind manual selection of latent tensor rank, whereas the prior knowledge about tensor rank is surprisingly limited, or resort to regularization to make the role of low rankness without exploration on the underlying low-dimensional factors, both of which are leaving the computational burden of parameter tuning. To address that, a novel Bayesian sparse learning-based tensor ring (TR) fusion model is proposed, named as FuBay. Through specifying hierarchical sprasity-inducing prior distribution, the proposed method becomes the first fully Bayesian probabilistic tensor framework for hyperspectral fusion. With the relationship between component sparseness and the corresponding hyperprior parameter being well studied, a component pruning part is established to asymptotically approaching true latent rank. Furthermore, a variational inference (VI)-based algorithm is derived to learn the posterior of TR factors, circumventing nonconvex optimization that bothers the most tensor decomposition-based fusion methods. As a Bayesian learning methods, our model is characterized to be parameter tuning-free. Finally, extensive experiments demonstrate its superior performance when compared with state-of-the-art methods.

Reference-Based OCT Angiogram Super-Resolution With Learnable Texture Generation.

Optical coherence tomography angiography (OCTA) can visualize retinal microvasculature and is important to qualitatively and quantitatively identify potential biomarkers for different retinal diseases. However, the resolution of optical coherence tomography (OCT) angiograms inevitably decreases when increasing the field-of-view (FOV) given a fixed acquisition time. To address this issue, we propose a novel reference-based super-resolution (RefSR) framework to preserve the resolution of the OCT angiograms while increasing the scanning area. Specifically, textures from the normal RefSR pipeline are used to train a learnable texture generator (LTG), which is designed to generate textures according to the input. The key difference between the proposed method and traditional RefSR models is that the textures used during inference are generated by the LTG instead of being searched from a single reference (Ref) image. Since the LTG is optimized throughout the whole training process, the available texture space is significantly enlarged and no longer limited to a single Ref image, but extends to all textures contained in the training samples. Moreover, our proposed LTGNet does not require an Ref image at the inference phase, thereby becoming invulnerable to the selection of the Ref image. Both experimental and visual results show that LTGNet has competitive performance and robustness over state-of-the-art methods, indicating good reliability and promise in real-life deployment. The source code is available at https://github.com/RYY0722/LTGNet.

ConvMambaSR: Leveraging State-Space Models and CNNs in a Dual-Branch Architecture for Remote Sensing Imagery Super-Resolution

Deep learning-based super-resolution (SR) techniques play a crucial role in enhancing the spatial resolution of images. However, remote sensing images present substantial challenges due to their diverse features, complex structures, and significant size variations in ground objects. Moreover, recovering lost details from low-resolution remote sensing images with complex and unknown degradations, such as downsampling, noise, and compression, remains a critical issue. To address these challenges, we propose ConvMambaSR, a novel super-resolution framework that integrates state-space models (SSMs) and Convolutional Neural Networks (CNNs). This framework is specifically designed to handle heterogeneous and complex ground features, as well as unknown degradations in remote sensing imagery. ConvMambaSR leverages SSMs to model global dependencies, activating more pixels in the super-resolution task. Concurrently, it employs CNNs to extract local detail features, enhancing the model’s ability to capture image textures and edges. Furthermore, we have developed a global–detail reconstruction module (GDRM) to integrate diverse levels of global and local information efficiently. We rigorously validated the proposed method on two distinct datasets, RSSCN7 and RSSRD-KQ, and benchmarked its performance against state-of-the-art SR models. Experiments show that our method achieves SOTA PSNR values of 26.06 and 24.29 on these datasets, respectively, and is visually superior, effectively addressing a variety of scenarios and significantly outperforming existing methods.

High-flexibility reconstruction of small-scale motions in wall turbulence using a generalized zero-shot learning

This study proposes a novel super-resolution (or SR) framework for generating high-resolution turbulent boundary layer (TBL) flow from low-resolution inputs. The framework combines a super-resolution generative adversarial neural network (SRGAN) with down-sampling modules (DMs), integrating the residual of the continuity equation into the loss function. The DMs selectively filter out components with excessive energy dissipation in low-resolution fields prior to the super-resolution process. The framework iteratively applies the SRGAN and DM procedure to fully capture the energy cascade of multi-scale flow structures, collectively termed the SRGAN-based energy cascade reconstruction framework (EC-SRGAN). Despite being trained solely on turbulent channel flow data (via ‘zero-shot transfer’), EC-SRGAN exhibits remarkable generalization in predicting TBL small-scale velocity fields, accurately reproducing wavenumber spectra compared to direct numerical simulation (DNS) results. Furthermore, a super-resolution core is trained at a specific super-resolution ratio. By leveraging this pretrained super-resolution core, EC-SRGAN efficiently reconstructs TBL fields at multiple super-resolution ratios from various levels of low-resolution inputs, showcasing strong flexibility. By learning turbulent scale invariance, EC-SRGAN demonstrates robustness across different TBL datasets. These results underscore the potential of EC-SRGAN for generating and predicting wall turbulence with high flexibility, offering promising applications in addressing diverse TBL-related challenges.

Unsupervised Test-Time Adaptation Learning for Effective Hyperspectral Image Super-Resolution With Unknown Degeneration.

Fusing a low-resolution hyperspectral image (HSI) with a high-resolution (HR) multi-spectral image has provided an effective way for HSI super-resolution (SR). The key lies on inferring the posteriori of the latent (i.e., HR) HSI using an appropriate image prior and the likelihood determined by the degeneration between the latent HSI and the observed images. However, in scenarios with complex imaging environments and various imaging scenes, the prior of HSIs can be prohibitively complicated and the degeneration is often unknown, which causes it difficult to accurately infer the posteriori of each latent HSI. To tackle this problem, we present an unsupervised test-time adaptation learning (UTAL) framework for HSI SR under unknown degeneration. Instead of directly modeling the complicated image prior, it first implicitly learns a content-agnostic prior shared across different images through supervisedly pre-training a mutual-guiding fusion module on extensive synthetic data. Then, it adapts the shared prior to those private characteristics in the latent HSI for posteriori inference through unsupervisedly learning a self-guiding adaptation module and a degeneration estimation network on two observed images in the test phase. Such a two-stage learning scheme models the complicated image prior in a divide-and-conquer manner, which eases the modeling difficulty and improves the prior accuracy. Moreover, the unknown degeneration can be estimated properly. Both of these two advantages empower us to accurately infer the posteriori of the latent HSI, thereby increasing the generalization performance in real applications. Additionally, in order to further mitigate the over-fitting in coping with more challenging cases (e.g., degenerations in both spectral and spatial domains are unknown) and speed up, we propose to meta-train UTAL on extensive synthetic SR tasks and solve it using an alternative optimization strategy such that UTAL learns to produce good generalization performance in real challenging cases with a small number of gradient descent steps. To verify the efficacy of UTAL, we evaluate it on HSI SR tasks with different unknown degenerations as well as some other HSI restoration tasks (e.g., compressive sensing), and report strong results superior to that of existing competitors.

Progressive reconstruction-decoupled face super-resolution framework with controllable knowledge guidance

Large-factor face super-resolution is a severely ill-posed problem that requires additional information to assist in image reconstruction. Existing methods improve the super-resolution performance by introducing estimated face or pre-trained generative adversarial network (GAN) priors. However, they ignore the interference between contour and detail reconstruction as well as the errors in pretrained GAN and estimated face priors. To address these problems, we propose a progressive reconstruction-decoupled face super-resolution (PDFSR) framework with controllable knowledge guidance. First, a three-level index (pose, semantics, geometry) is proposed to utilize the high-resolution face texture to construct a lightweight knowledge base robust to bias. Then, the face contours are reconstructed by a basic super-resolution network. Finally, a constraint module is proposed to constrain the output of the knowledge base to provide accurate and controllable explicit knowledge for the reconstruction of facial details. The progressive framework decouples contour and detail reconstruction during training for the reconstructed images to consider both fidelity and perceptual quality. The knowledge base provides accurate and controllable low-dimensional explicit knowledge without estimation or pre-training, which reduces prior errors. Experiments demonstrate that our PDFSR outperforms state-of-the-art face super-resolution methods. Moreover, during the testing phase, faces can be edited by adjusting the explicit knowledge output by the knowledge base to obtain diverse reconstruction results.

DEM super-resolution framework based on deep learning: decomposing terrain trends and residuals

DEM super-resolution framework based on deep learning: decomposing terrain trends and residuals

An Intelligent Co-Scheduling Framework for Efficient Super-Resolution on Edge Platforms With Heterogeneous Processors

An Intelligent Co-Scheduling Framework for Efficient Super-Resolution on Edge Platforms With Heterogeneous Processors

Super-resolution with perceptual quality for improved live streaming delivery on edge computing

According to Cisco projections, mobile connection throughput is anticipated to more than triple by the year 2023, with the expected typical 5G connection throughput reaching 575 Mbps. Additionally, smartphones and TV sets equipped with computational capabilities for decoding 4K videos have significantly redefined the landscape of Internet video traffic, accounting for 82% of the total Internet traffic. This surge in connection throughput, coupled with the escalating demand for video-centric content, presents an unprecedented challenge in effectively managing traffic across the backhaul infrastructure of 5G networks. In this article, we introduce and evaluate the On-edge enhanced live streaming with super-resolution (ELiveSR) framework for live video delivery, which takes advantage of throughput improvements, in the access networks, and makes use of real-time super-resolution procedure to reduce transmission rate demands in the backhaul segment of the network. This SR procedure is carried by a deep neural network based model and is capable of coping with video content of different subjects. Experimental assessments conducted revealed that the proposed framework can reduce video-related traffic in the backhaul segment by up to 88.37%. Additionally, it simultaneously enhances the quality of experience metrics observed during live video streaming sessions.

Transformer-based self-supervised image super-resolution method for Rotating Synthetic Aperture system via multi-temporal fusion

Rotating Synthetic Aperture (RSA) technology is one of the distinctly advantageous Earth geostationary orbit optical remote sensing technologies. However, the continuous rotation of the RSA system’s rectangular primary mirror results in a discernible drop in resolution along the shorter side of the mirror. Additionally, the captured images exhibit periodic and time-varying characteristics. To improve the image quality to meet interpretation needs, we first delineate the imaging process of the rotating primary mirror and analyze the characteristics of image degradation based on the system’s imaging mechanism. Then, we propose a dual super-resolution (SR) framework based on Swin Transformer and introduce a self-supervised learning method for jointly training the unified SR network using wavelet fusion. The self-supervised learning method effectively utilizes the spatiotemporal correlation of the information contained in images captured at different rotation directions of the rectangular pupil. Moreover, the attention mechanism in Transformer can adopt a global perspective and utilize content-based interactions between image content and attention weights to model strong long-range dependencies in remote sensing images. This approach significantly enhances image quality along the pupil’s shorter side, consequently yielding superior results. Extensive digital and semi-physical imaging experiments, involving six aspect ratios of the primary mirror, demonstrate that our SR method surpasses state-of-the-art methods. The work in this paper can serve as a valuable reference for future space applications of the RSA technology.

Low-Light Face Super-resolution via Illumination, Structure, and Texture Associated Representation

Human face captured at night or in dimly lit environments has become a common practice, accompanied by complex low-light and low-resolution degradations. However, the existing face super-resolution (FSR) technologies and derived cascaded schemes are inadequate to recover credible textures. In this paper, we propose a novel approach that decomposes the restoration task into face structural fidelity maintaining and texture consistency learning. The former aims to enhance the quality of face images while improving the structural fidelity, while the latter focuses on eliminating perturbations and artifacts caused by low-light degradation and reconstruction. Based on this, we develop a novel low-light low-resolution face super-resolution framework. Our method consists of two steps: an illumination correction face super-resolution network (IC-FSRNet) for lighting the face and recovering the structural information, and a detail enhancement model (DENet) for improving facial details, thus making them more visually appealing and easier to analyze. As the relighted regions could provide complementary information to boost face super-resolution and vice versa, we introduce the mutual learning to harness the informative components from relighted regions and reconstruction, and achieve the iterative refinement. In addition, DENet equipped with diffusion probabilistic model is built to further improve face image visual quality. Experiments demonstrate that the proposed joint optimization framework achieves significant improvements in reconstruction quality and perceptual quality over existing two-stage sequential solutions. Code is available at https://github.com/wcy-cs/IC-FSRDENet.

A multi-view references image super-resolution framework for generating the large-FOV and high-resolution image

This paper investigates the generation of large field-of-view (FOV) and high-resolution (HR) panorama images for smartphones. Existing techniques like image stitching struggle to produce satisfactory results due to geometry misalignment and inconsistent appearance. To circumvent the inherent challenges of image stitching methods and generate high-quality panoramas, we treat the image stitching problem as a multiple-reference-based super-resolution problem. Specifically, one large-FOV low-resolution (LR) image and several overlapped small-FOV HR images are taken as inputs, where the LR image acts as a base and multi-view references provide rich HR information. Building on this foundation, a novel multi-view references image super-resolution framework (MVRefSR) is proposed. Within this framework, to address the residual geometric misalignment between the LR-Ref image pairs after coarse alignment, a flow-based RefSR network (FlowSRNet) is proposed, which super-resolves LR patches with corresponding HR references. To facilitate adaptive feature fusion and minimize distortion in structured regions, a fusion weight estimation module and a gradient branch are introduced in FlowSRNet. Finally, the large-FOV HR image is generated by combining these SR patches together. Furthermore, the lack of real-world RefSR datasets for smartphones is addressed by designing an innovative dataset construction pipeline. Extensive experiments demonstrate the superior performance of FlowSRNet and MVRefSR over the compared SR methods and image stitching software, which proves the effectiveness of generating panoramas from a new perspective.

Coupled image and kernel prior learning for high-generalized super-resolution

Deep convolution neural networks (DCNNs) have demonstrated great success on single image super resolution, where most existing methods aim to construct a deep model in an fully-supervised way using large amount of synthetic image pairs under ideal degradation assumption. The super-resolution performance would be significantly degraded for the low-resolution images captured under uncontrolled imaging conditions with complicated degradation procedures. To handle the above limitations, this work proposes a high-generalized image super-resolution framework by synergistically learning the latent image and the degradation process to achieve the specific priors for an under-studying observation in an unsupervised manner. Specifically, we incorporate dual branches of networks to configure our framework, where one is structured with both convolution and transformer blocks to learn local-to-global prior to generate high-quality image while the other aims to learn the kernel prior with a simple convolution-based architecture. The kernel prior subnet is pre-trained to enhance the stability of the joint optimization for the overall unsupervised learning procedure. With the estimated degradation kernel and target image, we produce the approximated low-resolution version with a convolution-based degradation block to formulate the loss function for the specific learning. Moreover, to pursue visually plausible image generation, we further incorporate a perceptual loss by leveraging a pre-trained discriminator in Gan-based SR model. Extensive experiments on several benchmark datasets have demonstrated the effectiveness of our proposed high-generalized super-resolution model, manifesting superiority over both supervised and unsupervised state-of-the-art SR models.

NAWQ-SR: A Hybrid-Precision NPU Engine for Efficient On-Device Super-Resolution

In recent years, image and video delivery systems have begun integrating deep learning super-resolution (SR) approaches, leveraging their unprecedented visual enhancement capabilities while reducing reliance on networking conditions. Nevertheless, deploying these solutions on mobile devices still remains an active challenge as SR models are excessively demanding with respect to workload and memory footprint. Despite recent progress on on-device SR frameworks, existing systems either penalize visual quality, lead to excessive energy consumption or make inefficient use of the available resources. This work presents NAWQ-SR, a novel framework for the efficient on-device execution of SR models. Through a novel hybrid-precision quantization technique and a runtime neural image codec, NAWQ-SR exploits the multi-precision capabilities of modern mobile NPUs in order to minimize latency, while meeting user-specified quality constraints. Moreover, NAWQ-SR selectively adapts the arithmetic precision at run time to equip the SR DNN's layers with wider representational power, improving visual quality beyond what was previously possible on NPUs. Altogether, NAWQ-SR achieves an average speedup of 7.9×, 3× and 1.91× over the state-of-the-art on-device SR systems that use heterogeneous processors (MobiSR), CPU (SplitSR) and NPU (XLSR), respectively. Furthermore, NAWQ-SR delivers an average of 3.2× speedup and 0.39 dB higher PSNR over status-quo INT8 NPU designs, but most importantly mitigates the negative effects of quantization on visual quality, setting a new state-of-the-art in the attainable quality of NPU-based SR.

Quantifying trade-offs in satellite hardware configurations using a super-resolution framework with realistic image degradation

ABSTRACT When designing and operating Earth observation satellites, trade-offs must often be made between different hardware components, operational considerations, etc… For example, considerations include how much of the mass budget should be spent on imaging lenses?, or how long should image exposure times be?. Resulting limitations in image resolution and quality can be partially compensated for using super-resolution (SR) techniques. However, deep SR networks recently applied to satellite imagery are often trained and tested on data that lacks the typical image degrading noise present in real satellite images. In this work, we combine a method for generating realistically degraded satellite images with a deep SR network, in the context of different satellite hardware configurations and geographical types. We use this framework to assess deep SR performance given realistic remote sensing payloads across different terrain types, by evaluating payload- and terrain-dependent SR performance in reconstructing realistically degraded images. The framework allows us to model the effect of alternative satellite hardware configurations on resulting SR image quality, providing insight into optimal satellite operations and payload design in the context of SR-based image quality enhancements.

Dimension reduction, exact recovery, and error estimates for sparse reconstruction in phase space

An important theme in modern inverse problems is the reconstruction of time-dependent data from only finitely many measurements. To obtain satisfactory reconstruction results in this setting it is essential to strongly exploit temporal consistency between the different measurement times. The strongest consistency can be achieved by reconstructing data directly in phase space, the space of positions and velocities. However, this space is usually too high-dimensional for feasible computations. We introduce a novel dimension reduction technique, based on projections of phase space onto lower-dimensional subspaces, which provably circumvents this curse of dimensionality: Indeed, in the exemplary framework of superresolution we prove that known exact reconstruction results stay true after dimension reduction, and we additionally prove new error estimates of reconstructions from noisy data in optimal transport metrics which are of the same quality as one would obtain in the non-dimension-reduced case.

A Practical Contrastive Learning Framework for Single-Image Super-Resolution.

Contrastive learning has achieved remarkable success on various high-level tasks, but there are fewer contrastive learning-based methods proposed for low-level tasks. It is challenging to adopt vanilla contrastive learning technologies proposed for high-level visual tasks to low-level image restoration problems straightly. Because the acquired high-level global visual representations are insufficient for low-level tasks requiring rich texture and context information. In this article, we investigate the contrastive learning-based single-image super-resolution (SISR) from two perspectives: positive and negative sample construction and feature embedding. The existing methods take naive sample construction approaches (e.g., considering the low-quality input as a negative sample and the ground truth as a positive sample) and adopt a prior model (e.g., pretrained very deep convolutional networks proposed by visual geometry group (VGG) model) to obtain the feature embedding. To this end, we propose a practical contrastive learning framework for SISR (PCL-SR). We involve the generation of many informative positive and hard negative samples in frequency space. Instead of utilizing an additional pretrained network, we design a simple but effective embedding network inherited from the discriminator network, which is more task-friendly. Compared with the existing benchmark methods, we retrain them by our proposed PCL-SR framework and achieve superior performance. Extensive experiments have been conducted to show the effectiveness and technical contributions of our proposed PCL-SR thorough ablation studies. The code and resulting models will be released via https://github.com/Aitical/PCL-SISR.

A Generic Multicorrespondence Matching Framework for Reference-Based Image Super-Resolution

A Generic Multicorrespondence Matching Framework for Reference-Based Image Super-Resolution

A Unified Super-Resolution Framework of Remote-Sensing Satellite Images Classification Based on Information Fusion of Novel Deep Convolutional Neural Network Architectures

A Unified Super-Resolution Framework of Remote-Sensing Satellite Images Classification Based on Information Fusion of Novel Deep Convolutional Neural Network Architectures