Super-resolution Algorithm Research Articles

Improving the performance of web-streaming by super-resolution upscaling techniques

In recent years, we have seen significant progress in advanced image upscaling techniques, sometimes called super-resolution, ML-based, or AI-based upscaling. Such algorithms are now available not only in form of specialized software but also in drivers and SDKs supplied with modern graphics cards. Upscaling functions in NVIDIA Maxine SDK is one of the recent examples. However, to take advantage of this functionality in video streaming applications, one needs to (a) quantify the impacts of super-resolution techniques on the perceived visual quality, (b) implement video rendering incorporating super-resolution upscaling techniques, and (c) implement new bitrate+resolution adaptation algorithms in streaming players, enabling such players to deliver better quality of experience or better efficiency (e.g. reduce bandwidth usage) or both. Towards this end, in this paper, we propose several techniques that may be helpful to the implementation community. First, we offer a model quantifying the impacts of super resolution upscaling on the perceived quality. Our model is based on the Westerink-Roufs model connecting the true resolution of images/videos to perceived quality, with several additional parameters added, allowing its tuning to specific implementations of super-resolution techniques. We verify this model by using several recent datasets including MOS scores measured for several conventional up-scaling and super-resolution algorithms. Then, we propose an improved adaptation logic for video streaming players, considering video bitrates, encoded video resolutions, player size, and the upscaling method. This improved logic relies on our modified Westerink-Roufs model to predict perceived quality and suggests choices of renditions that would deliver the best quality for given display and upscaling method characteristics. Finally, we study the impacts of the proposed techniques and show that they can deliver practically appreciable results in terms of the expected QoE improvements and bandwidth savings.

High-Resolution WiFi Imaging With Reconfigurable Intelligent Surfaces

WiFi-based imaging enables pervasive sensing in a privacy-preserving and cost-effective way. However, most of existing methods either require specialized hardware modification or suffer from the poor imaging performance due to the fundamental limit of off-the-shelf commodity WiFi devices in spatial resolution. We observe that the recently developed reconfigurable intelligent surface (RIS) could be a promising solution to overcome these challenges. Thus, in this article, we propose an RIS-aided WiFi imaging framework to achieve high-resolution imaging with the off-the-shelf WiFi devices. Specifically, we first design a beamforming method to achieve the first-stage imaging by separating the signals from different spatial locations with the aid of the RIS. Then, we propose an optimization-based super-resolution imaging algorithm by leveraging the low-rank nature of the reconstructed object. During the optimization, we also explicitly take into account the effect of finite phase quantization in RIS to avoid the resolution degradation due to quantization errors. Simulation results demonstrate that our framework achieves median root-mean-square error (RMSE) of 0.03 and median structural similarity (SSIM) of 0.52. The visual results show that high-resolution imaging results are achieved with simulation signals at 5 GHz that are matched with commercial WiFi 802.11n/ac protocols.

Angular Super-Resolution of Multi-Channel APAR in Interference Environments

Aiming to resolve azimuth-dense targets in interference environments, the radar needs to have the ability of single snap echo angular super-resolution with anti-interference. To solve the problem, the angular super-resolution algorithm based on single snap echo while anti-interference with blocking matrix method is studied for active phased array radar (APAR) in this paper. Since the super-resolution ability of the conventional MUSIC algorithm and iterative adaptive algorithm (IAA) algorithm are limited in single snap echo, the iterative re-weighted least squares (IRLS) algorithm under p-norm constraint is proposed. Further, the near main lobe interference suppression ability is enhanced by the adaptive diagonal loading method. The performance differences of IAA and IRLS algorithms for single-target detection and double-target angular super-resolution are analyzed in detail by numerical simulation on three scenes of no interference, side-lobe interference, and near-main-lobe interference. The simulation results show that the proposed algorithm can effectively solve the problem of target angle estimation and super-resolution based on single sample echo in an interference environment, including near main-lobe interference.

An improved generative adversarial networks for remote sensing image super-resolution reconstruction via multi-scale residual block

Existing image super-resolution algorithms still suffer from the problems of not extracting rich image features and losing realistic high-frequency details. In order to solve these problems, this paper proposes an improved generative adversarial network algorithm for super-resolution reconstruction of remote sensing images by multi-scale residual blocks. The original generative adversarial network (GAN) structure is improved and multi-scale residual blocks are introduced in the generator to fuse features at different scales. After extracting the parallel information of multi-scale features, information is exchanged between multi-resolution information streams to obtain contextual information through spatial and channel attention mechanisms, and multi-scale features are fused according to the attention mechanism. In the discriminator, the concept of relative average GAN (RaGAN) is introduced, and the loss function of the network is redesigned so that the discriminator can predict relative probabilities instead of absolute probabilities thus enabling clear learning of edge and texture details. Experimental results show that the proposed method in this paper significantly outperforms state-of-the-art (SOTA) methods in terms of both subjective and objective metrics.In three test datasets, compared with SOTA methods, the Peak Signal to Noise Ratio(PSNR) is improved by a maximum of 1.18 dB, 0.84 dB and 1.29 dB respectively, and the Structural Similarity Index (SSIM) is improved by 0.0264, 0.0077 and 0.0109 respectively in scale of 2, 3 and 4 times images super-resolution.The model proposed in this paper effectively improved the super-resolution re-construction results of remote sensing images.

Meta-Learning Multi-Scale Radiology Medical Image Super-Resolution

High-resolution medical images have important medical value, but are difficult to obtain directly. Limited by hardware equipment and patient’s physical condition, the resolution of directly acquired medical images is often not high. Therefore, many researchers have thought of using super-resolution algorithms for secondary processing to obtain high-resolution medical images. However, current super-resolution algorithms only work on a single scale, and multiple networks need to be trained when super-resolution images of different scales are needed. This definitely raises the cost of acquiring high-resolution medical images. Thus, we propose a multi-scale super-resolution algorithm using meta-learning. The algorithm combines a meta-learning approach with an enhanced depth of residual super-resolution network to design a meta-upscale module. The meta-upscale module utilizes the weight prediction property of meta-learning and is able to perform the super-resolution task of medical images at any scale. Meanwhile, we design a non-integer mapping relation for super-resolution, which allows the network to be trained under non-integer magnification requirements. Compared to the state-of-the-art single-image super-resolution algorithm on computed tomography images of the pelvic region. The meta-learning multiscale super-resolution algorithm obtained a surpassing of about 2% at a smaller model volume. Testing on different parts proves the high generalizability of our algorithm. Multi-scale super-resolution algorithms using meta-learning can compensate for hardware device defects and reduce secondary harm to patients while obtaining high-resolution medical images. It can be of great use in imaging related fields.

TnTViT-G: Transformer in Transformer Network for Guidance Super Resolution

Image Super Resolution is a potential approach that can improve the image quality of low-resolution optical sensors, leading to improved performance in various industrial applications. It is important to emphasize that most state-of-the-art super resolution algorithms often use a single channel of input data for training and inference. However, this practice ignores the fact that the cost of acquiring high-resolution images in various spectral domains can differ a lot from one another. In this paper, we attempt to exploit complementary information from a low-cost channel (visible image) to increase the image quality of an expensive channel (infrared image). We propose a dual stream Transformer-based super resolution approach that uses the visible image as a guide to super-resolve another spectral band image. To this end, we introduce Transformer in Transformer network for Guidance super resolution, named TnTViT-G, an efficient and effective method that extracts the features of input images via different streams and fuses them together at various stages. In addition, unlike other guidance super resolution approaches, TnTViT-G is not limited to a fixed upsample size and it can generate super-resolved images of any size. Extensive experiments on various datasets show that the proposed model outperforms other state-of-the-art super resolution approaches. TnTViT-G surpasses state-of-the-art methods by up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.19 \sim 2.3dB $ </tex-math></inline-formula> , while it is memory efficient.

A Review of Spatial Enhancement of Hyperspectral Remote Sensing Imaging Techniques

Remote sensing technology has undeniable importance in various industrial applications, such as mineral exploration, plant detection, defect detection in aerospace and shipbuilding, and optical gas imaging, to name a few. Remote sensing technology has been continuously evolving, offering a range of image modalities that can facilitate the aforementioned applications. One such modality is Hyperspectral Imaging (HSI). Unlike Multispectral Images (MSI) and natural images, HSI consist of hundreds of bands. Despite their high spectral resolution, HSI suffer from low spatial resolution in comparison to their MSI counterpart, which hinders the utilization of their full potential. Therefore, spatial enhancement, or Super Resolution (SR), of HSI is a classical problem that has been gaining rapid attention over the past two decades. The literature is rich with various SR algorithms that enhance the spatial resolution of HSI while preserving their spectral fidelity. This paper reviews and discusses the most important algorithms relevant to this area of research between 2002-2022, along with the most frequently used datasets, HSI sensors, and quality metrics. Meta-analysis are drawn based on the aforementioned information, which is used as a foundation that summarizes the state of the field in a way that bridges the past and the present, identifies the current gap in it, and recommends possible future directions.

MSAGAN: A New Super-Resolution Algorithm for Multispectral Remote Sensing Image Based on a Multiscale Attention GAN Network

In the absence of high-resolution sensors, super-resolution( SR) algorithms for remote sensing imagery improve the spatial resolution of the images. Currently, most of the SR algorithms are based on deep learning methods e.g., convolutional neural networks(CNN). Particularly, the generative adversarial networks(GANs) have demonstrated accepted performances in image super-resolution owing to their powerful generative capabilities. However, remote sensing images have complex feature types, which largely limits the performances of GAN-based SR methods for real satellite images. To address this issue, an attention mechanism and a multi-scale structure are introduced into the generator of the GAN network, and a multi-scale attention GAN(MSAGAN) is constructed in this study. We sequentially arrange the channel attention module and the spatial attention module after the multi-scale structure to emphasize important information, suppress unimportant information details, improve the model’s performance. Furthermore, we add residual connections and dense blocks to further enhance the performance of the generative network by increasing its depth. We compared to other existing deep learning-based SR methods, our proposed MSAGAN algorithm performed better in generating high spatial satellite images.

Super-resolution reconstruction of binocular image based on multi-level fusion attention network

Super-resolution reconstruction of binocular image based on multi-level fusion attention network

Super-Resolution Based on Curvelet Transform and Sparse Representation

Super-resolution techniques are used to reconstruct an image with a high resolution from one or more low-resolution image(s). In this paper, we proposed a single image super-resolution algorithm. It uses the nonlocal mean filter as a prior step to produce a denoised image. The proposed algorithm is based on curvelet transform. It converts the denoised image into low and high frequencies (sub-bands). Then we applied a multi-dimensional interpolation called Lancozos interpolation over both sub-bands. In parallel, we applied sparse representation with over complete dictionary for the denoised image. The proposed algorithm then combines the dictionary learning in the sparse representation and the interpolated sub-bands using inverse curvelet transform to have an image with a higher resolution. The experimental results of the proposed super-resolution algorithm show superior performance and obviously better-recovering images with enhanced edges. The comparison study shows that the proposed super-resolution algorithm outperforms the state-of-the-art. The mean absolute error is 0.021 ± 0.008 and the structural similarity index measure is 0.89 ± 0.08.

Multi-Frame Compressed Video Quality Enhancement by Spatio-Temporal Information Balance

In recent years, the performance of multi-frame quality enhancement algorithms for compressed videos has been greatly improved compared with single-frame based algorithms. However, the existing methods mainly focus on mining the temporal information of multiple frames. The large number of reference frames reduces the exploration of spatial information, although the existing single-frame based algorithms for enhancement, denoising, and super-resolution demonstrate the significance of the spatial information. To address this problem, we propose a plug-and-play module called Spatio-temporal Information Balance (STIB) to adaptively balance the spatial and temporal information. In our method, we use a feature extractor to exploit richer spatial information, and use a refinement module to refine the aligned temporal information, to be more conducive to the fusion of spatio-temporal information. Finally, we use the deformable convolution based re-alignment module to do alignment and fusion in feature space for balancing the spatio-temporal information. Experiments show that our module can significantly improve the performance of the existing multi-frame based enhancement algorithms.

Learning Dynamic Scale Awareness and Global Implicit Functions for Continuous-Scale Super-Resolution of Remote Sensing Images

The mainstream remote sensing image (RSI) super-resolution (SR) algorithms treat tasks with different scale factors independently, and a single model can only process a fixed integer scale factor. However, in practical applications, it is important to continuously super-resolve RSIs to multiple resolutions, as different resolutions present various levels of detail. Retraining the model for each scale factor consumes huge computational resources and storage space. Existing continuous-scale SR models employ static convolutions, and most are designed for natural scenes, ignoring dynamic feature extraction needs for different scale factors and the inherent properties of RSIs. In addition, efficiently obtaining the continuous representation of RSIs and avoiding the artifacts of RSI SR results is still a challenging problem. To address the above problems, we propose a scale-aware dynamic network (SADN) for RSI continuous-scale SR. First, we devise a scale-aware dynamic convolutional (SAD-Conv) layer to handle the strong randomness of the RSI textural distribution and achieve dynamic feature extraction according to scale factors. Second, we devise a continuous-scale upsampling module (CSUM) with the multi-bilinear global implicit function (MBGIF) for any-scale upsampling. The CSUM constructs multiple feature spaces with asymptotic resolutions to approximate the continuous representation of an image, and then, the MBGIF makes full use of multiresolution features to map arbitrary coordinates to spectral values. We evaluate our SADN using various benchmarks, and the experimental results show that the CSUM can efficiently achieve continuous-scale upsampling while maintaining excellent objective and visual performance. Moreover, our SADN uses fewer parameters and even outperforms the state-of-the-art fixed-scale SR methods. The source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/hanlinwu/SADN</uri> .

Технология сейсмической локации. III. Построение глубинно-скоростной сейсмической модели геологической среды

An original technology for construction of seismic images of high-resolution andaccuracy is proposed. In given article the third part of technology is stated. A new approachto the construction of seismic images is presented here, using stable and unambiguous solutions to several direct kinematic problems of seismic exploration instead of solving oneinverse problem. It is based on the location of the seismic reflection impulse source, which iscarried out by a seismic digital antenna array using a super-resolution location algorithm anda virtual location grid. The accuracy of constructing reflective boundaries is determined bythe specified parameters of the selection and location algorithms used, as well as the nodes ofthe location grid. The seismic location technology has a number of theoretical and practicaladvantages over known seismic imaging technologies.

Super-Resolution Reconstruction Algorithm of Underwater Image Based on Information Distillation Mechanism

Super-Resolution Reconstruction Algorithm of Underwater Image Based on Information Distillation Mechanism

Terahertz Super-Resolution Nondestructive Detection Algorithm Based on Edge Feature Convolution Network

Much research has been conducted to improve the defect-detection rate and detection accuracy of the imaging technology used in terahertz nondestructive testing. Due to the power limit of light sources and noise interference in terahertz equipment, images have low resolution and fuzzy defect edges. Hence, improving the resolution is crucial for detecting defects. We designed an edge detection network structure based on a traditional deep neural network. Besides, we devised a node-fusing strategy to train the network. It demonstrates significant improvement of the resolution of the terahertz defect contour. A quartz fiber composites with embedded defects was tested with our network. The results showed that the proposed super-resolution reconstruction algorithm improves resolution, particularly on the edges of defect contours.

Joint Super-Resolution and Head Pose Estimation for Extreme Low-Resolution Faces

State-of-the-art deep learning-based Head Pose Estimation (HPE) techniques have reached spectacular performance on High-Resolution (HR) face images. However, they still fail to achieve expected performance on low-resolution images at large scales. This work presents an end-to-end HPE framework assisted by a Face Super-Resolution (FSR) algorithm. The proposed FSR model is specifically guided to enhance the HPE performance rather than considering FSR as an independent task. To this end, we utilized a Multi-Stage Generative Adversarial Network (MSGAN) which benefit from a pose-aware adversarial loss and head pose estimation feedback to generate super-resolved images that are properly aligned for HPE. Also, we propose a degradation strategy rather than simple down-sampling approach to mimic the diverse properties of real-world Low-Resolution (LR) images. We evaluate the performance of our proposed method on both synthetic and real-world LR datasets and show the superiority of our approach in both visual and HPE metrics on the AFLW2000, BIWI, and WiderFace Datasets.

Improved Upsampling Based Depth Image Super-Resolution Reconstruction

Constrained by current sensing technology, depth camera only acquires a low-resolution depth image that does not meet actual requirements. To solve this problem, this paper take a divide-and-conquer strategy to synthesize a high-resolution depth image from a low-resolution range image under the guidance of a registered high-resolution color image. Initially, the depth image is divided into planar areas and edge regions. For different zones, we exploit different methods to interpolate the missing depths. At planar area, the linear interpolation method is employed to perform upsampling. At edge region, a segmentation-separation upsampling method is used to interpolate the missing values. Then the upsampling result are refined on the Depth CNN that is built in this paper. We conduct extensive experiments on the benchmark database and real world data with various upsampling rates to illustrate the upsampling ability of our method. The comparison with classical super-resolution algorithms demonstrates that our upsampling algorithm achieves the best quality with fewer artifacts and our depth CNN outperforms the most state-of-the-art methods in terms of qualitative and quantitative evaluations.

Unmixing Guided Unsupervised Network for RGB Spectral Super-Resolution.

Spectral super-resolution has attracted research attention recently, which aims to generate hyperspectral images from RGB images. However, most of the existing spectral super-resolution algorithms work in a supervised manner, requiring pairwise data for training, which is difficult to obtain. In this paper, we propose an Unmixing Guided Unsupervised Network (UnGUN), which does not require pairwise imagery to achieve unsupervised spectral super-resolution. In addition, UnGUN utilizes arbitrary other hyperspectral imagery as the guidance image to guide the reconstruction of spectral information. The UnGUN mainly includes three branches: two unmixing branches and a reconstruction branch. Hyperspectral unmixing branch and RGB unmixing branch decompose the guidance and RGB images into corresponding endmembers and abundances respectively, from which the spectral and spatial priors are extracted. Meanwhile, the reconstruction branch integrates the above spectral-spatial priors to generate a coarse hyperspectral image and then refined it. Besides, we design a discriminator to ensure that the distribution of generated image is close to the guidance hyperspectral imagery, so that the reconstructed image follows the characteristics of a real hyperspectral image. The major contribution is that we develop an unsupervised framework based on spectral unmixing, which realizes spectral super-resolution without paired hyperspectral-RGB images. Experiments demonstrate the superiority of UnGUN when compared with some SOTA methods.

Image super-resolution algorithm based on V-transform combined with neural network

Image super-resolution algorithm based on V-transform combined with neural network

Sparse Target Batch-Processing Framework for Scanning Radar Superresolution Imaging

Sparse superresolution algorithms have been applied in scanning radar imaging to improve its azimuth resolution. However, the inverse matrix in each iteration is usually diagonal loading by the updating result, which leads to huge computational complexity for two-dimensional echo data. In this letter, a batch-processing superresolution framework is proposed to process the echo data in parallel. On the one hand, the optimization problem for sparse target recovery is modified as matrix form, which presents batch-processing potential for two-dimensional echo data. On the other hand, the optimization problem is solved by the proposed alternating direction method of multipliers (ADMM)-based batch-processing framework, which can avoid high-dimensional matrix inversion along different range bins. Compared with traditional sparse superresolution methods, the proposed batch-processing framework is much suitable for two-dimensional echo data superresolution.