Image Super-resolution Method Research Articles

Feedback Refined Local-Global Network for Super-Resolution of Hyperspectral Imagery

Powered by advanced deep-learning technology, multi-spectral image super-resolution methods based on convolutional neural networks have recently achieved great progress. However, the single hyperspectral image super-resolution remains a challenging problem due to the high-dimensional and complex spectral characteristics of hyperspectral data, which make it difficult for general 2D convolutional neural networks to simultaneously capture spatial and spectral prior information. To deal with this issue, we propose a novel Feedback Refined Local-Global Network (FRLGN) for the super-resolution of hyperspectral image. To be specific, we develop a new Feedback Structure and a Local-Global Spectral block to alleviate the difficulty in spatial and spectral feature extraction. The Feedback Structure can transfer the high-level information to guide the generation process of low-level features, which is achieved by a recurrent structure with finite unfoldings. Furthermore, in order to effectively use the high-level information passed back, a Local-Global Spectral block is constructed to handle the feedback connections. The Local-Global Spectral block utilizes the feedback high-level information to correct the low-level feature from local spectral bands and generates powerful high-level representations among global spectral bands. By incorporating the Feedback Structure and Local-Global Spectral block, the FRLGN can fully exploit spatial-spectral correlations among spectral bands and gradually reconstruct high-resolution hyperspectral images. Experimental results indicate that FRLGN presents advantages on three public hyperspectral datasets.

Structure-Preserving and Color-Restoring Up-Sampling for Single Low-Light Image

Single image super-resolution (SR), as a basic computer vision task, has been widely studied. However, existing single image SR methods, applied in images with normal light, perform poorly on low-light images. To address this limitation, a method of structure-preserving and color-restoring up-sampling for single low-light image is proposed. Theoretical and experimental analysis indicates that existing methods suffer negative effects due to the suppressed color and the weakened texture in low-light image. Therefore, combined with the Retinex theory, the single low-light image up-sampling model is established for the first time, which avoids the conflict between color and texture by distinguishing reflectance and illumination. Further, we develop a structure-preserving and color-restoring up-sampling network for single low-light image SR. In the network, the reflectance and illumination components are obtained by decomposing the observed image, and then up-sampling of reflectance and enhancement of illumination are performed to complete the primary SR. In addition, the gradient information is reconstructed and fused into the up-sampling process to further enrich the SR texture. Experiments demonstrate that our method obtains competitive qualitative and quantitative evaluation on the produced dataset and real-world images.

A Lightweight Remote Sensing Image Super-Resolution Method and Its Application in Smart Cities

With the growth of urban population, a series of urban problems have emerged, and how to speed up smart city construction has received extensive attention. Remote sensing images have the advantages of wide spatial coverage and rich information, and it is suitable for use as research data for smart cities. However, due to limitations in the imaging sensor conditions and complex weather, remote sensing images face the problems of insufficient resolution and cloud occlusion, which cannot meet the resolution requirements of smart city tasks. The remote sensing image super-resolution (SR) technique can improve the details and texture information without upgrading the imaging sensor system, which becomes a feasible solution for the above problems. In this paper, we propose a novel remote sensing image super-resolution method which leverages the texture features from internal and external references to help with SR reconstruction. We introduce the transformer attention mechanism to select and extract parts of texture features with high reference values to ensure that the network is lightweight, effective, and easier to deploy on edge computing devices. In addition, our network can automatically learn and adjust the alignment angles and scales of texture features for better SR results. Extensive comparison experiments show that our proposed method achieves superior performance compared with several state-of-the-art SR methods. In addition, we also evaluate the application value of our proposed SR method in urban region function recognition in smart cities. The dataset used in this task is low-quality. The comparative experiment between the original dataset and the SR dataset generated by our proposed SR method indicates that our method can effectively improve the recognition accuracy.

Super-resolution acoustic imaging

This work reports a super-resolution acoustic imaging method that inverses source distribution, strength, and structure in three-dimensional space. The nonlinear coupling process between a low-frequency sound field and a high-frequency plane wave is endowed to break the resolution limit. The reconstructed results for different source strengths and frequencies demonstrate the capability of the proposed method. The results show that the proposed method is very suitable for low-frequency source imaging, which could strengthen the super-resolution analysis capability of acoustic imaging tests and, consequently, lead to deepened physical insights into various propulsion systems in underwater and aerospace systems.

Gradient-Guided and Multi-Scale Feature Network for Image Super-Resolution

Recently, deep-learning-based image super-resolution methods have made remarkable progress. However, most of these methods do not fully exploit the structural feature of the input image, as well as the intermediate features from the intermediate layers, which hinders the ability of detail recovery. To deal with this issue, we propose a gradient-guided and multi-scale feature network for image super-resolution (GFSR). Specifically, a dual-branch structure network is proposed, including the trunk branch and the gradient one, where the latter is used to extract the gradient feature map as structural prior to guide the image reconstruction process. Then, to absorb features from different layers, two effective multi-scale feature extraction modules, namely residual of residual inception block (RRIB) and residual of residual receptive field block (RRRFB), are proposed and embedded in different network layers. In our RRIB and RRRFB structures, an adaptive weighted residual feature fusion block (RFFB) is investigated to fuse the intermediate features to generate more beneficial representations, and an adaptive channel attention block (ACAB) is introduced to effectively explore the dependencies between channel features to further boost the feature representation capacity. Experimental results on several benchmark datasets demonstrate that our method achieves superior performance against state-of-the-art methods in terms of both subjective visual quality and objective quantitative metrics.

Video Super Resolution Using a Selective Edge Aggregation Network

An edge map is a feature map representing the contours of the object in the image. There was a Single Image Super Resolution (SISR) method using the edge map, which achieved a notable SSIM performance improvement. Unlike SISR, Video Super Resolution (VSR) uses video, which consists of consecutive images with temporal features. Therefore, some VSR models adopted motion estimation and motion compensation to apply spatio-temporal feature maps. Unlike the models above, we tried a different method by adding edge structure information and its related post-processing to the existing model. Our model “Video Super Resolution Using a Selective Edge Aggregation Network (SEAN)” consists of a total of two stages. First, the model selectively generates an edge map using the target frame and also the neighboring frame. At this stage, we adopt the magnitude loss function so that the output of SEAN more clearly learns the contours of each object. Second, the final output is generated using the refinement (post-processing) module. SEAN shows more distinct object contours and better color correction compared to other existing models.

Single infrared image super‐resolution based on lightweight multi‐path feature fusion network

Single infrared (IR) image super-resolution methods can help to reduce the cost and difficulty in manufacturing IR sensors for the imaging system. However, the deep learning-based image SR methods need to build a complex network and thus consume a lot of computational power, which limits the application of SR technology on devices with low computing resources in practice. To solve this problem, the authors present a lightweight multi-path feature fusion network (MFFN) for the single infrared (IR) image SR. A multi-path feature fusion block (MFFB) is developed to extract and fuse multiple and discriminative features in a recursive feedback way. Specifically, the multiple features are refined via the linear feature extraction branch, shared-source residual feature extraction branch, and channel attention branch in MFFB. Finally, the authors reconstruct the high-resolution IR images from the low-resolution counterpart based on the refined multiple features. The experimental results demonstrate that MFFN achieves high-quality single infrared image SR and shows superiority over previous methods for several scale factors (e.g. ×2, ×3, and ×4). MFFN has potential applications in the mobile infrared imaging system.

Using Physics-Informed Generative Adversarial Networks to Perform Super-Resolution for Multiphase Fluid Simulations

AbstractComputational fluid dynamics (CFD) simulations are useful in the field of engineering design as they provide deep insights on product or system performance without the need to construct and test physical prototypes. However, they can be very computationally intensive to run. Machine learning methods have been shown to reconstruct high-resolution single-phase turbulent fluid flow simulations from low-resolution inputs. This offers a potential avenue towards alleviating computational cost in iterative engineering design applications. However, little work thus far has explored the application of machine learning image super-resolution methods to multiphase fluid flow (which is important for emerging fields such as marine hydrokinetic energy conversion). In this work, we apply a modified version of the super-resolution generative adversarial network (SRGAN) model to a multiphase turbulent fluid flow problem, specifically to reconstruct fluid phase fraction at a higher resolution. Two models were created in this work, one which incorporates a physics-informed term in the loss function and one which does not, and the results are discussed and compared. We found that both models significantly outperform non-machine learning upsampling methods and can preserve a substantial amount of detail, showing the versatility of the SRGAN model for upsampling multiphase fluid simulations. However, the difference in accuracy between the two models is minimal indicating that, in the context studied here, the additional complexity of a physics-informed approach may not be justified.

Super‐resolution forward‐looking imaging method for manoeuvering platform with optimised dictionary and extended sparsity adaptive matching pursuit

In forward-looking imaging, left-right Doppler is ambiguous and cross-range resolution is limited by real aperture, which result in traditional synthetic aperture radar imaging methods no longer applicable. Besides, the impact of three-dimensional acceleration on an over-complete dictionary matrix and sparse reconstruction kernel is not considered in conventional super-resolution forward-looking imaging methods. To solve these problems, a super-resolution forward-looking imaging method for manoeuvering platforms is proposed. First, the range walk caused by forward-looking geometry and higher-order phase introduced by three-dimensional acceleration are eliminated by compensating a pre-processing function. Subsequently, according to beam steering-based geometry and linear regression model, the over-complete dictionary matrix is optimised to more accurately describe the complicated forward-looking geometry with curved trajectory. Finally, sparsity adaptive matching pursuit kernel with an improved halting condition is adopted to reconstruct the forward-looking scene. The proposed method is supported by point target simulation and surface target simulation.

An efficient and lightweight image super-resolution with feature supplement network

Despite the great success of deep learning approaches in image super-resolution tasks in recent years, a large proportion of the approaches emphasize designing complex network structures to continuously deepen the network to achieve superior reconstruction performance. However, some shortcomings need to be addressed, such as the large number of parameters required for training the complex network structure, which makes the model difficult to optimize and train. To address the above problems, we propose an accurate and lightweight neural network which is called the feature supplement network (FSN) to achieve image reconstruction. Specifically, we design a novel residual feature supplement structure(RFSS), which is composed of some residual feature supplement groups (RFSGs) and layer feature attention module (LFAM). Each RFSG contains several heterogeneous residual channel attention blocks (HRCAB) with different levels of skip connections. In RFSS, the stacked RFSG is able to extract features efficiently with limited parameters. Meanwhile, LFAM further improves the feature representation capability of the network because it is capable of capturing more discriminative features from the original image by considering the correlation between the extracted features of different depths. Experiments of various models conducted on benchmark datasets show that our proposed method not only can achieve comparable results compared to the state-of-the-art methods, but also effectively recover high-quality images, even when using limited parameters than the existing method for image super-resolution. The codes of our proposed structure-FSN are accessible on https://github.com/ReckonerInheritor/RFSN.

High-Refractive-Index Chip with Periodically Fine-Tuning Gratings for Tunable Virtual-Wavevector Spatial Frequency Shift Universal Super-Resolution Imaging.

Continued research in fields such as materials science and biomedicine requires the development of a super‐resolution imaging technique with a large field of view (FOV) and deep subwavelength resolution that is compatible with both fluorescent and nonfluorescent samples. Existing on‐chip super‐resolution methods exclusively focus on either fluorescent or nonfluorescent imaging, and, as such, there is an urgent requirement for a more general technique that is capable of both modes of imaging. In this study, to realize labeled and label‐free super‐resolution imaging on a single scalable photonic chip, a universal super‐resolution imaging method based on the tunable virtual‐wavevector spatial frequency shift (TVSFS) principle is introduced. Using this principle, imaging resolution can be improved more than threefold over the diffraction limit of a linear optical system. Here, diffractive units are fabricated on the chip's surface to provide wavevector‐variable evanescent wave illumination, enabling tunable spatial frequency shifts in the Fourier space. A large FOV and resolutions of λ/4.7 and λ/7.1 were achieved for label‐free and fluorescently labeled samples using a gallium phosphide (GaP) chip. With its large FOV, compatibility with different imaging modes, and monolithic integration, the proposed TVSFS chip may advance fields such as cell engineering, precision industry inspection, and chemical research.

Comparison of DEM Super-Resolution Methods Based on Interpolation and Neural Networks.

High-resolution digital elevation models (DEMs) play a critical role in geospatial databases, which can be applied to many terrain-related studies such as facility siting, hydrological analysis, and urban design. However, due to the limitation of precision of equipment, there are big gaps to collect high-resolution DEM data. A practical idea is to recover high-resolution DEMs from easily obtained low-resolution DEMs, and this process is termed DEM super-resolution (SR). However, traditional DEM SR methods (e.g., bicubic interpolation) tend to over-smooth high-frequency regions on account of the operation of averaging local variations. With the recent development of machine learning, image SR methods have made great progress. Nevertheless, due to the complexity of terrain characters (e.g., peak and valley) and the huge difference between elevation field and image RGB (Red, Green, and Blue) value field, there are few works that apply image SR methods to the task of DEM SR. Therefore, this paper investigates the question of whether the state-of-the-art image SR methods are appropriate for DEM SR. More specifically, the traditional interpolation method and three excellent SR methods based on neural networks are chosen for comparison. Experimental results suggest that SRGAN (Super-Resolution with Generative Adversarial Network) presents the best performance on accuracy evaluation over a series of DEM SR experiments.

Fast, super‐resolution sparse inverse synthetic aperture radar imaging via continuous compressive sensing

Conventional inverse synthetic aperture radar (ISAR) imaging with sparse aperture usually suffers from high side lobes and wide main lobes, which limit the applications of radar super-resolution imaging, multi-target resolution, and cognitive reconfiguration. This paper proposes a fast, super-resolution imaging method employing continuous compressive sensing for sparse-aperture ISAR. First, the received echo in each range bin is characterised as a linear combination of multiple frequencies shown in a continuous atomic set, established into an atomic norm minimisation (ANM) mode. Second, to improve the resolution and reduce the computational burden significantly, a locally convergent iterative algorithm based on the alternating direction method of multipliers, which iteratively performs ANM with a sound reweighting strategy, is implemented. Then, the low-rank Toeplitz covariance matrix, which contains the information of the target, is obtained. Subsequently, the Vandermonde decomposition of the Toeplitz covariance matrix is performed to acquire the locations and intensities of the scattering points. Finally, the super-resolution result is generated by depicting the estimated scatterers in the image. Extensive numerical experiments demonstrate that the proposal is highly effective in recovering the super-resolution image and shows better performance than state-of-the-art methods.

Hyperspectral Image Super-Resolution via Recurrent Feedback Embedding and Spatial–Spectral Consistency Regularization

Hyperspectral images with tens to hundreds of spectral bands usually suffer from low spatial resolution due to the limitation of the amount of incident energy. Without auxiliary images, the single hyperspectral image super-resolution (SR) method is still a challenging problem because of the high-dimensionality characteristic and special spectral patterns of hyperspectral images. Failing to thoroughly explore the coherence among hyperspectral bands and preserve the spatial–spectral structure of the scene, the performance of existing methods is still limited. In this article, we propose a novel single hyperspectral image SR method termed RFSR, which models the spectrum correlations from a sequence perspective. Specifically, we introduce a recurrent feedback network to fully exploit the complementary and consecutive information among the spectra of the hyperspectral data. With the group strategy, each grouping band is first super-resolved by exploring the consecutive information among groups via feedback embedding. For better preservation of the spatial–spectral structure among hyperspectral data, a regularization network is subsequently appended to enforce spatial–spectral correlations over the intermediate estimation. Experimental results on both natural and remote sensing hyperspectral images demonstrate the advantage of our approach over the state-of-the-art methods.

Advanced Scalability for Light Field Image Coding.

Light field imaging, which captures both spatial and angular information, improves user immersion by enabling post-capture actions, such as refocusing and changing view perspective. However, light fields represent very large volumes of data with a lot of redundancy that coding methods try to remove. State-of-the-art coding methods indeed usually focus on improving compression efficiency and overlook other important features in light field compression such as scalability. In this paper, we propose a novel light field image compression method that enables (i) viewport scalability, (ii) quality scalability, (iii) spatial scalability, (iv) random access, and (v) uniform quality distribution among viewports, while keeping compression efficiency high. To this end, light fields in each spatial resolution are divided into sequential viewport layers, and viewports in each layer are encoded using the previously encoded viewports. In each viewport layer, the available viewports are used to synthesize intermediate viewports using a video interpolation deep learning network. The synthesized views are used as virtual reference images to enhance the quality of intermediate views. An image super-resolution method is applied to improve the quality of the lower spatial resolution layer. The super-resolved images are also used as virtual reference images to improve the quality of the higher spatial resolution layer. The proposed structure also improves the flexibility of light field streaming, provides random access to the viewports, and increases error resiliency. The experimental results demonstrate that the proposed method achieves a high compression efficiency and it can adapt to the display type, transmission channel, network condition, processing power, and user needs.

A Group-Based Embedding Learning and Integration Network for Hyperspectral Image Super-Resolution

Although natural image super-resolution methods have achieved impressive performance, single hyperspectral image super-resolution still remains a challenge due to the high dimensionality. In recent years, many single hyperspectral image super-resolution methods adopted the group-convolution strategy to design the network for reducing the computational burden. However, these methods still process all spectral bands at once during the deep feature extraction and reconstruction, which increases the difficulty of fully exploring the inherent data characteristic of hyperspectral images. Moreover, the advanced group-based methods make insufficient exploitation of complementary information contained in different bands, resulting in limited reconstruction performance. In this paper, we propose a novel group-based single hyperspectral image super-resolution method termed GELIN to reconstruct high-resolution images in a group-by-group manner, which alleviates the difficulty of feature extraction and reconstruction for hyperspectral images. Specifically, a spatial-spectral embedding learning module is designed to extract rewarding spatial details and explore the correlations among spectra simultaneously. Considering the high similarity among different bands, a neighboring group integration module is proposed to fully exploit the complementary information contained in neighboring image groups to recover missing details in the target image group. Experimental results on both natural and remote sensing hyperspectral datasets demonstrate that the proposed method is superior to other state-of-the-art methods both visually and metrically.

Light Field Image Super-Resolution With Transformers

Light field (LF) image super-resolution (SR) aims at reconstructing high-resolution LF images from their low-resolution counterparts. Although CNN-based methods have achieved remarkable performance in LF image SR, these methods cannot fully model the non-local properties of the 4D LF data. In this paper, we propose a simple but effective Transformer-based method for LF image SR. In our method, an angular Transformer is designed to incorporate complementary information among different views, and a spatial Transformer is developed to capture both local and long-range dependencies within each sub-aperture image. With the proposed angular and spatial Transformers, the beneficial information in an LF can be fully exploited and the SR performance is boosted. We validate the effectiveness of our angular and spatial Transformers through extensive ablation studies, and compare our method to recent state-of-the-art methods on five public LF datasets. Our method achieves superior SR performance with a small model size and low computational cost. Code is available at https://github.com/ZhengyuLiang24/LFT.

High-Order Coupled Fully Connected Tensor Network Decomposition for Hyperspectral Image Super-Resolution

Hyperspectral image super-resolution addresses the problem of fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) to produce a high-resolution hyperspectral image (HR-HSI). Tensor analysis has been proven to be an efficient method for hyperspectral image processing. However, the existing tensor-based methods of hyperspectral image super-resolution like the tensor train and tensor ring decomposition only establish an operation between adjacent two factors and are highly sensitive to the permutation of tensor modes, leading to an inadequate and inflexible representation. In this paper, we propose a novel method for hyperspectral image super-resolution by utilizing the specific properties of high-order tensors in fully-connected tensor network decomposition. The proposed method first tensorizes the target HR-HSI into a high-order tensor that has multiscale spatial structures. Then, a coupled fully-connected tensor network decomposition model is proposed to fuse the corresponding high-order tensors of LR-HSI and HR-MSI. Moreover, a weighted-graph regularization is imposed on the spectral core tensors to preserve spectral information. In the proposed model, the superiorities of the fully-connected tensor network decomposition lie in the outstanding capability for characterizing adequately the intrinsic correlations between any two modes of tensors and the essential invariance for transposition. Experimental results on three data sets show the effectiveness of the proposed approach as compared to other hyperspectral image super-resolution methods.

Image Super-Resolution Reconstruction Based on Dual Residual Network

Research shows that deep learning algorithms can effectively improve a single image's super-resolution quality. However, if the algorithm is solely focused on increasing network depth and the desired result is not achieved, difficulties in the training process are more likely to arise. Simultaneously, the function space that can be transferred from a low-resolution image to a high-resolution image is enormous, making finding a satisfactory solution difficult. In this paper, we propose a deep learning method for single image super-resolution. The MDRN network framework uses multi-scale residual blocks and dual learning to fully acquire features in low-resolution images. Finally, these features will be sent to the image reconstruction module to restore high-quality images. The function space is constrained by the closed-loop formed by dual learning, which provides additional supervision for the super-resolution reconstruction of the image. The up-sampling process includes residual blocks with short-hop connections, so that the network focuses on learning high-frequency information, and strives to reconstruct images with richer feature details. The experimental results of ×4 and ×8 super-resolution reconstruction of the image show that the quality of the reconstructed image with this method is better than some existing experimental results of image super-resolution reconstruction in subjective visual effects and objective evaluation indicators.

Development of a fluorescent labeling method for the live-cell superresolution imaging of synaptic molecules.

Development of a fluorescent labeling method for the live-cell superresolution imaging of synaptic molecules.