Super-resolution Framework Research Articles

A super-resolution framework for emotion recognition based on EEG signals

A super-resolution framework for emotion recognition based on EEG signals

R2D2-GAN: Robust Dual Discriminator Generative Adversarial Network for Microscopy Hyperspectral Image Super-Resolution.

High-resolution microscopy hyperspectral (HS) images can provide highly detailed spatial and spectral information, enabling the identification and analysis of biological tissues at a microscale level. Recently, significant efforts have been devoted to enhancing the resolution of HS images by leveraging high spatial resolution multispectral (MS) images. However, the inherent hardware constraints lead to a significant distribution gap between HS and MS images, posing challenges for image super-resolution within biomedical domains. This discrepancy may arise from various factors, including variations in camera imaging principles (e.g., snapshot and push-broom imaging), shooting positions, and the presence of noise interference. To address these challenges, we introduced a unique unsupervised super-resolution framework named R2D2-GAN. This framework utilizes a generative adversarial network (GAN) to efficiently merge the two data modalities and improve the resolution of microscopy HS images. Traditionally, supervised approaches have relied on intuitive and sensitive loss functions, such as mean squared error (MSE). Our method, trained in a real-world unsupervised setting, benefits from exploiting consistent information across the two modalities. It employs a game-theoretic strategy and dynamic adversarial loss, rather than relying solely on fixed training strategies for reconstruction loss. Furthermore, we have augmented our proposed model with a central consistency regularization (CCR) module, aiming to further enhance the robustness of the R2D2-GAN. Our experimental results show that the proposed method is accurate and robust for super-resolution images. We specifically tested our proposed method on both a real and a synthetic dataset, obtaining promising results in comparison to other state-of-the-art methods. Our code and datasets are accessible through Multimedia Content.

General Hyperspectral Image Super-Resolution via Meta-Transfer Learning.

Recent advances in deep learning-based methods have led to significant progress in the hyperspectral super-resolution (SR). However, the scarcity and the high dimension of data have hindered further development since deep models require sufficient data to learn stable patterns. Moreover, the huge domain differences between hyperspectral image (HSI) datasets pose a significant challenge in generalizability. To address these problems, we present a general hyperspectral SR framework via meta-transfer learning (MTL). We randomly sample various spectral ranges for SR tasks during MTL, allowing the model to accumulate diverse task experiences. Additionally, we implement a task schedule to gradually expand the number of bands, bridging the significant domain differences between datasets. By leveraging multiple datasets, we are able to achieve better performance and greater generalizability, making it applicable under various circumstances. Meanwhile, as a general framework, our scheme can be applied to existing methods to obtain performance improvements. In addition, we design an advanced network architecture based on the multifusion features to further improve the performance. Experiments demonstrate that our method not only achieves superior performance in both qualitative and quantitative terms but also can adapt robustly to a new and difficult sample, where few epochs can yield quite considerable results.

A General Deep Learning Point–Surface Fusion Framework for RGB Image Super-Resolution

Hyperspectral images are usually acquired in a scanning-based way, which can cause inconvenience in some situations. In these cases, RGB image spectral super-resolution technology emerges as an alternative. However, current mainstream spectral super-resolution methods aim to generate continuous spectral information at a very narrow range, limited to the visible light range. Some researchers introduce hyperspectral images as auxiliary data. But it is usually required that the auxiliary hyperspectral images have the same spatial range as RGB images. To address this issue, a general point–surface data fusion method is designed to achieve the RGB image spectral super-resolution goal in this paper, named GRSS-Net. The proposed method utilizes hyperspectral point data as auxiliary data to provide spectral reference information. Thus, the spectral super-resolution can extend the spectral reconstruction range according to spectral data. The proposed method utilizes compressed sensing theory as a fundamental physical mechanism and then unfolds the traditional hyperspectral image reconstruction optimization problem into a deep network. Finally, a high-spatial-resolution hyperspectral image can be obtained. Thus, the proposed method combines the non-linear feature extraction ability of deep learning and the interpretability of traditional physical models simultaneously. A series of experiments demonstrates that the proposed method can effectively reconstruct spectral information in RGB images. Meanwhile, the proposed method provides a framework of spectral super-resolution for different applications.

Reconstruction with robustness: A semantic prior guided face super-resolution framework for multiple degradations

Despite the rapid advancements made with the support of deep learning, Face Super-Resolution (FSR) methods still suffer from challenges under multiple degradations. These challenges significantly impede the practical applications of FSR methods in real-world scenarios. Incorporating facial priors could potentially relieve this issue. However, ground truth priors are not feasible in real-world applications, meanwhile the accuracy of predicted priors is difficult to guarantee, especially for low-resolution faces under multiple degradations. Hence, it is worth exploring how to effectively leverage facial priors for improving the robustness of FSR under multiple degradations. To tackle these problems, we propose RSemFace, a robust semantic prior guided FSR framework to reconstruct multiple degraded faces. In RSemFace, we design the Degradation Stage to synthesize multiple degraded low-resolution faces with a variety of interpolations, noise levels, blurring kernels, and even the real-world interference. The Generation Stage generates Coarse-SR faces, and extracts semantic features from the Coarse-SR as priors, which are used to the reconstruction of Fine-SR faces with the support of Semantic Feature Attention Blocks (SFABs) and Semantic Loss. Both quantitative and qualitative results show the better robustness of our RSemFace for content recovery and perceptual quality in simultaneously handling multiple degraded faces compared with other state-of-the-art methods. Lastly, faces reconstructed by RSemFace are proven to improve the high-level vision task due to better recovered identities.

Joint Image Reconstruction and Super-Resolution for Accelerated Magnetic Resonance Imaging.

Magnetic resonance (MR) image reconstruction and super-resolution are two prominent techniques to restore high-quality images from undersampled or low-resolution k-space data to accelerate MR imaging. Combining undersampled and low-resolution acquisition can further improve the acceleration factor. Existing methods often treat the techniques of image reconstruction and super-resolution separately or combine them sequentially for image recovery, which can result in error propagation and suboptimal results. In this work, we propose a novel framework for joint image reconstruction and super-resolution, aiming to efficiently image recovery and enable fast imaging. Specifically, we designed a framework with a reconstruction module and a super-resolution module to formulate multi-task learning. The reconstruction module utilizes a model-based optimization approach, ensuring data fidelity with the acquired k-space data. Moreover, a deep spatial feature transform is employed to enhance the information transition between the two modules, facilitating better integration of image reconstruction and super-resolution. Experimental evaluations on two datasets demonstrate that our proposed method can provide superior performance both quantitatively and qualitatively.

Frequency-aware optical coherence tomography image super-resolution via conditional generative adversarial neural network.

Optical coherence tomography (OCT) has stimulated a wide range of medical image-based diagnosis and treatment in fields such as cardiology and ophthalmology. Such applications can be further facilitated by deep learning-based super-resolution technology, which improves the capability of resolving morphological structures. However, existing deep learning-based method only focuses on spatial distribution and disregards frequency fidelity in image reconstruction, leading to a frequency bias. To overcome this limitation, we propose a frequency-aware super-resolution framework that integrates three critical frequency-based modules (i.e., frequency transformation, frequency skip connection, and frequency alignment) and frequency-based loss function into a conditional generative adversarial network (cGAN). We conducted a large-scale quantitative study from an existing coronary OCT dataset to demonstrate the superiority of our proposed framework over existing deep learning frameworks. In addition, we confirmed the generalizability of our framework by applying it to fish corneal images and rat retinal images, demonstrating its capability to super-resolve morphological details in eye imaging.

PhySR: Physics-informed deep super-resolution for spatiotemporal data

High-fidelity simulation of complex physical systems is exorbitantly expensive and inaccessible across spatiotemporal scales. Recently, there has been an increasing interest in leveraging deep learning to augment scientific data based on the coarse-grained simulations, which is of cheap computational expense and retains satisfactory solution accuracy. However, the major existing work focuses on data-driven approaches which rely on rich training datasets and lack sufficient physical constraints. To this end, we propose a novel and efficient spatiotemporal super-resolution framework via physics-informed learning (i.e., PhySR), inspired by the independence between temporal and spatial derivatives in partial differential equations (PDEs). The general principle is to leverage the temporal interpolation for flow estimation, and then introduce convolutional-recurrent neural networks for learning temporal refinement. Furthermore, we employ the stacked residual blocks with wide activation and sub-pixel layers with pixelshuffle for spatial reconstruction, where feature extraction is conducted in a low-resolution latent space. Moreover, we consider hard imposition of boundary conditions in the network to improve reconstruction accuracy. Results demonstrate the superior effectiveness and efficiency of the proposed method, which outperforms existing baseline algorithms, based on extensive numerical experiments.

Multiscale and multidirection depth map super resolution with semantic inference

AbstractDepth map super resolution has been paid much attention in 3D applications due to the limitation of depth sensors. Few textures in objects with clear contours along them is the most important characteristic of depth map. An efficient image representation should be directional, multiscale and anisotropic. From this we propose a novel multiscale and multidirection depth map super resolution framework with semantic inference to improve the quality of depth maps. In this framework, a multiscale and multidirection depth map contour fusion scheme captures and assembles intrinsic geometrical structures through a multiview non‐subsampled contourlet transform manner. This scheme not only isolates the discontinuities of contours but retains the smoothness along the contours. The semantic inference is also utilized to segment and label the depth map into objects/backgrounds‐level which are coplanar. Furthermore, a semantic‐aware label refinement strategy is introduced to correct the rarely inaccurate labels of the label map for upscaling the target pixel with pixels in the same object or background. Experimental results on benchmark depth map dataset demonstrate that the proposed multiscale and multidirection depth map super resolution framework with semantic inference has a significant improvement than the state‐of‐the‐art algorithms both visually and quantitatively.

Domain migration representation learning for blind magnetic resonance image super-resolution

Recently, magnetic resonance (MR) imaging has been widely used in clinical treatment to assist doctors in diagnosis because of its high resolution of soft tissue and no radiation. However, limited by imaging equipment and imaging time, the resolution of MR imaging is low. The previous super-resolution algorithms used for MR images use paired datasets to generate down-sampled images by artificially assuming the image degradation process, and forming a sample pair to monitor and train the super-resolution network. However, the super-resolution network obtained by this way will cause serious degradation of the network performance in practical application. In this paper, we proposed DMN, a blind super-resolution framework based on domain migration, which decouples the blind super-resolution network with the idea of domain migration, and avoids the false detail generation problem of the blind super-resolution network in the MR images super-resolution task through independent training. At the same time, we propose a multi-level parallel transformer feature enhancement module (MPTB), which takes noise as the original input and embeds noise features step by step to enhance the fitting ability of the network while reducing the complexity of the network structure design. Besides, we also use the k-space loss function to form a constraint function, so that the domain migration network can fully consider the k-space characteristics of MR images. The experimental results on the MR images of the knee joint and brain in the FastMRI dataset show that the super-resolution performance of our proposed DMN network is better than the previous super-resolution networks.

MAML-SR: Self-adaptive super-resolution networks via multi-scale optimized attention-aware meta-learning

The deep-learning-based super-resolution (SR) methods require an avalanche of training images. However, they do not adapt model parameters in test time to cope with the novel blur kernel scenarios. Even though the recent meta-learning-based SR techniques adapt trained model parameters leveraging a test image’s internal patch recurrence, they need heavy pre-training on an external dataset to initialize. Also, the shallow internal information exploration and failure to amplify the salient edges impacts blurry SR image generation. Besides, model inferencing gets delayed due to a threshold-dependent adaptation phase. In this paper, we present Multi-scale Optimized Attention-aware Meta-Learning framework for SR (MAML-SR) to explore the multi-scale hierarchical self-similarity of recurring patches in a test image. Precisely, without any pre-training, we directly meta-train our model with a second-order optimization having the first-order adapted parameters from the intermediate scales, which are again directly supervised with the ground-truth HR images. At each scale, non-local self-similarity is maximized along with the amplification of salient edges using a novel cross-scale spectro-spatial attention learning unit. Also, we drastically reduce the inference delay by putting a metric-dependent constraint on the gradient updates for a test image. We demonstrate our method’s superior super-resolving capability over four benchmark SR datasets.

SRECG: ECG Signal Super-Resolution Framework for Portable/Wearable Devices in Cardiac Arrhythmias Classification

A combination of cloud-based deep learning (DL) al-gorithms with portable/wearable (P/W) devices has been devel-oped as a smart heath care system to support automatic cardiac arrhythmias (CAs) classification using electrocardiography (ECG). However, long-term and continuous ECG monitoring is challenging because of limitations of batteries and transmission bandwidth of P/W devices while incorporated with consumer elec-tronics (CE). A feasible approach to address this challenge is to decrease sampling rates. However, low sampling rates lead to low-resolution signals that hinder the CAs classification performance. In this study, we propose a DL-based ECG signal super-resolution framework (called SRECG) to enhance low-resolution ECG sig-nals by jointly considering the accuracies when applied to the DL-based high-resolution multiclass classifier (HMC) of CAs. In our experiments, we downsampled the ECG signals from the CPSC2018 dataset and evaluated their HMC accuracies with and without the SRECG. Experimental results show that SRECG can well improve the HMC accuracies as compared to traditional in-terpolation methods. Moreover, approximately half of the CAs classification accuracies of HMC were maintained within the en-hanced ECG signals by SRECG. The promising results confirm that SRECG can be suitably used to enhance low-resolution ECG signals from P/W devices with CE to improve their cloud-based HMC performances.

CNN framework for optical image super-resolution and fusion

CNN framework for optical image super-resolution and fusion

Thangka Hyperspectral Image Super-Resolution Based on a Spatial–Spectral Integration Network

Thangka refers to a form of Tibetan Buddhist painting on a fabric, scroll, or Thangka, often depicting deities, scenes, or mandalas. Deep-learning-based super-resolution techniques have been applied to improve the spatial resolution of hyperspectral images (HSIs), especially for the preservation and analysis of Thangka cultural heritage. However, existing CNN-based methods encounter difficulties in effectively preserving spatial information, due to challenges such as registration errors and spectral variability. To overcome these limitations, we present a novel cross-sensor super-resolution (SR) framework that utilizes high-resolution RGBs (HR-RGBs) to enhance the spectral features in low-resolution hyperspectral images (LR-HSIs). Our approach utilizes spatial–spectral integration (SSI) blocks and spatial–spectral restoration (SSR) blocks to effectively integrate and reconstruct spatial and spectral features. Furthermore, we introduce a frequency multi-head self-attention (F-MSA) mechanism that treats high-, medium-, and low-frequency features as tokens, enabling self-attention computations across the frequency dimension. We evaluate our method on a custom dataset of ancient Thangka paintings and demonstrate its effectiveness in enhancing the spectral resolution in high-resolution hyperspectral images (HR-HSIs), while preserving the spatial characteristics of Thangka artwork with minimal information loss.

Deep EEG Superresolution via Correlating Brain Structural and Functional Connectivities.

Electroencephalogram (EEG) excels in portraying rapid neural dynamics at the level of milliseconds, but its spatial resolution has often been lagging behind the increasing demands in neuroscience research or subject to limitations imposed by emerging neuroengineering scenarios, especially those centering on consumer EEG devices. Current superresolution (SR) methods generally do not suffice in the reconstruction of high-resolution (HR) EEG as it remains a grand challenge to properly handle the connection relationship amongst EEG electrodes (channels) and the intensive individuality of subjects. This study proposes a deep EEG SR framework correlating brain structural and functional connectivities (Deep-EEGSR), which consists of a compact convolutional network and an auxiliary fully connected network for filter generation (FGN). Deep-EEGSR applies graph convolution adapting to the structural connectivity amongst EEG channels when coding SR EEG. Sample-specific dynamic convolution is designed with filter parameters adjusted by FGN conforming to functional connectivity of intensive subject individuality. Overall, Deep-EEGSR operates on low-resolution (LR) EEG and reconstructs the corresponding HR acquisitions through an end-to-end SR course. The experimental results on three EEG datasets (autism spectrum disorder, emotion, and motor imagery) indicate that: 1) Deep-EEGSR significantly outperforms the state-of-the-art counterparts with normalized mean squared error (NMSE) decreased by 1% - 6% and the improvement of signal-to-noise ratio (SNR) up to 1.2 dB and 2) the SR EEG manifests superiority to the LR alternative in ASD discrimination and spatial localization of typical ASD EEG characteristics, and this superiority even increases with the scale of SR.

Development of image super-resolution framework

There are some scenarios where the images taken are of low resolution and it is hard to judge the features from them, resulting in the need for enhancement. Super-resolution is a technique to produce a high-resolution image from a lower-resolution image. The intention here is to develop a system that enhances images of faces and satellite images by integrating these models and providing an interface to access this model. There have been various ways of achieving super-resolution using different techniques. Throughout the years, techniques involving deep learning methods, interpolation techniques, and recursive networks have been explored. We find it promising to use generative adversarial networks (GANs). The system has been deployed through Google Collaborate, Python libraries, and the TensorFlow framework. To assess the developed system, which consists of images, three metrics have been calculated. namely, peak signal-to-noise ratio, mean squared error, and structural similarity index. The model successfully demonstrated the capability of GANs by efficiently generating a high-resolution image from a low-resolution image for the given cases. The model would then be run on a standalone server for free Internet access for users to use super-resolution facial images and satellite images.

Deep learning architecture for 3D image super-resolution of late gadolinium enhanced cardiac MRI.

High-resolution late gadolinium enhanced (LGE) cardiac magnetic resonance imaging (MRI) volumes are difficult to acquire due to the limitations of the maximal breath-hold time achievable by the patient. This results in anisotropic 3D volumes of the heart with high in-plane resolution, but low-through-plane resolution. Thus, we propose a 3D convolutional neural network (CNN) approach to improve the through-plane resolution of the cardiac LGE-MRI volumes. We present a 3D CNN-based framework with two branches: a super-resolution branch to learn the mapping between low-resolution and high-resolution LGE-MRI volumes, and a gradient branch that learns the mapping between the gradient map of low-resolution LGE-MRI volumes and the gradient map of high-resolution LGE-MRI volumes. The gradient branch provides structural guidance to the CNN-based super-resolution framework. To assess the performance of the proposed CNN-based framework, we train two CNN models with and without gradient guidance, namely, dense deep back-projection network (DBPN) and enhanced deep super-resolution network. We train and evaluate our method on the 2018 atrial segmentation challenge dataset. Additionally, we also evaluate these trained models on the left atrial and scar quantification and segmentation challenge 2022 dataset to assess their generalization ability. Finally, we investigate the effect of the proposed CNN-based super-resolution framework on the 3D segmentation of the left atrium (LA) from these cardiac LGE-MRI image volumes. Experimental results demonstrate that our proposed CNN method with gradient guidance consistently outperforms bicubic interpolation and the CNN models without gradient guidance. Furthermore, the segmentation results, evaluated using Dice score, obtained using the super-resolved images generated by our proposed method are superior to the segmentation results obtained using the images generated by bicubic interpolation () and the CNN models without gradient guidance (). The presented CNN-based super-resolution method with gradient guidance improves the through-plane resolution of the LGE-MRI volumes and the structure guidance provided by the gradient branch can be useful to aid the 3D segmentation of cardiac chambers, such as LA, from the 3D LGE-MRI images.

Multiscale Attention Fusion for Depth Map Super-Resolution Generative Adversarial Networks.

Color images have long been used as an important supplementary information to guide the super-resolution of depth maps. However, how to quantitatively measure the guiding effect of color images on depth maps has always been a neglected issue. To solve this problem, inspired by the recent excellent results achieved in color image super-resolution by generative adversarial networks, we propose a depth map super-resolution framework with generative adversarial networks using multiscale attention fusion. Fusion of the color features and depth features at the same scale under the hierarchical fusion attention module effectively measure the guiding effect of the color image on the depth map. The fusion of joint color-depth features at different scales balances the impact of different scale features on the super-resolution of the depth map. The loss function of a generator composed of content loss, adversarial loss, and edge loss helps restore clearer edges of the depth map. Experimental results on different types of benchmark depth map datasets show that the proposed multiscale attention fusion based depth map super-resolution framework has significant subjective and objective improvements over the latest algorithms, verifying the validity and generalization ability of the model.

Multi-image hybrid super-resolution reconstruction via interpolation and multi-scale residual networks

High spatial resolution is necessary for several applications such as visual inspection, and can be achieved using high-resolution (HR) image sensors or through image super-resolution (SR) algorithms. Currently, SR algorithms are applied to either single low-resolution (LR) images or multiple LR image sequences. In this paper, we propose a hybrid super-resolution (HYSR) framework to generate HR images by combining multi-image super-resolution (MISR) and single-image super-resolution (SISR) to obtain high spatial resolution images. This method comprehensively utilizes sub-pixel-level high-frequency detail information between multiple images and co-occurrence prior of a single image to reconstruct SR images with a larger scale factor than the existing methods. Generally, the HYSR reconstruction results have more satisfactory details and visual quality than the SISR or MISR reconstruction results. A large number of qualitative and quantitative evaluation results demonstrate the effectiveness and superiority of the HYSR method over traditional MISR and SISR methods.

Cross-sensor remote sensing imagery super-resolution via an edge-guided attention-based network

The deep learning based super-resolution (SR) methods have recently achieved remarkable progress in the reconstruction of ideally simulated high-quality remote sensing image datasets. However, due to the large variation in image quality caused by the complex degradation factors, their performance decreases dramatically on real-world images acquired by different satellite sensors. To this end, we propose a cross-sensor SR framework that consists of a cross-sensor degradation modeling strategy for bridging the gap between the images obtained by the source and target sensors, and an edge-guided attention-based SR (EGASR) network to promote the learning of high-frequency feature representation. Specifically, we build a degradation pool on the low-resolution (LR) target sensor to produce a degraded training dataset simulated from the high-resolution (HR) images obtained by the source sensor. Furthermore, the EGASR network, which employs the edge-guided residual attention block (EGRAB) to introduce implicit edge prior to enhance edge-related information, is embedded in the cross-sensor SR framework for reconstructing HR results with sharp details. The proposed method is applied on images from the Chinese Gaofen (GF) satellite sensors and compared to several representative SR methods. An ideally simulated GF-2 LR/HR image set with only downsampling considered is first used to evaluate the effectiveness of the proposed EGASR network. Moreover, GF-2/GF-1 and GF-2/GF-6 cross-sensor SR datasets are constructed by synthesizing GF-2 degraded image pairs with the degradation pools estimated from the GF-1 and GF-6 images, respectively. The results show that: 1) the proposed EGASR model shows superiority in reconstructing textural details and edge features, and achieves the best results among the state-of-art SR methods involved in comparison; 2) the cross-sensor SR framework significantly promotes the model’s ability to super-resolve real-world LR images acquired by the target satellite sensors, e.g., the NIQE values are improved by at least 30% and 34% on average with respect to other comparative methods for GF-2/GF-1 and GF-2/GF-6 datasets in the real experiments, respectively. Code is available at https://github.com/zhonghangqiu/EGASR.