Upscaling Factors Research Articles

A Survey on Image Super-Resolution with Generative Adversarial Networks

Super-resolution is a process to increase image dimensions with a specific upscaling factor while trying to preserve details that matche with the original high-resolution form. Super-resolution can be done with many techniques. But the most effective technique is the one that takes advantage of several neural network designs. Some network designs are more appropriate than others on the specific subject. This study focuses on super resolution studies using Generative Adversarial Network. Many studies use this neural network type to look at various topics such as artificial data production and making the data more meaningful. The key point of this neural network type is having two different sub-networks that try to defeat each other in order to make more realistic results. Performance metrics that measure the quality of a generated image, loss functions used in a neural network and research papers on super-resolution with Generative Adversarial Network are the main domains of this study. 

Breast Cancer Histopathology Image Super-Resolution Using Wide-Attention GAN With Improved Wasserstein Gradient Penalty and Perceptual Loss

In the realm of image processing, enhancing the quality of the images is known as a super-resolution problem (SR). Among SR methods, a super-resolution generative adversarial network, or SRGAN, has been introduced to generate SR images from low-resolution images. As it is of the utmost importance to keep the size and the shape of the images, while enlarging the medical images, we propose a novel super-resolution model with a generative adversarial network to generate SR images with finer details and higher quality to encourage less blurring. By widening residual blocks and using a self-attention layer, our model becomes robust and generalizable as it is able to extract the most important part of the images before up-sampling. We named our proposed model as wide-attention SRGAN (WA-SRGAN). Moreover, we have applied improved Wasserstein with a Gradient penalty to stabilize the model while training. To train our model, we have applied images from Camylon 16 database and enlarged them by $2\times $ , $4\times $ , $8\times $ , and $16\times $ upscale factors with the ground truth of the size of $256\times 256\times 3$ . Furthermore, two normalization methods, including batch normalization, and weight normalization have been applied and we observed that weight normalization is an enabling factor to improve metric performance in terms of SSIM. Moreover, several evaluation metrics, such as PSNR, MSE, SSIM, MS-SSIM, and QILV have been applied for having a comprehensive objective comparison with other methods, including SRGAN, A-SRGAN, and bicubial. Also, we performed the job of classification by using a deep learning model called ResNeXt-101 ( $32\times 8\text{d}$ ) for super-resolution, high-resolution, and low-resolution images and compared the outcomes in terms of accuracy score. Finally, the results on breast cancer histopathology images show the superiority of our model by using weight normalization and a batch size of one in terms of restoration of the color and the texture details.

A Novel SRTSR Model for Cross-Resolution Person Re-Identification

Compared to the general person re-identification (re-id), pedestrian images in cross-resolution person re-id tasks always have variant resolutions, thus the resolution mismatching problem between low-resolution (LR) images and high-resolution (HR) images significantly limits the performance of neural networks. The general solution is to introduce interpolation methods or super-resolution (SR) module to normalize all the images to the same resolution. Unfortunately, current SR methods can only upscale the entire image to a certain scale, but fail to unify the resolution of images with different width-to-height ratio. To solve this problem, we propose a Specific-Resolution-Targeted Super-Resolution (SRTSR) embedded person re-id method. Instead of using deep CNN-based SR methods, our SRTSR network consists of a fast and light-weight feature learning module with a self-adaptive Swish (SA-Swish) as its activation function to improve its performance, and a specific-resolution-targeted upscale module to upscale the images to any specific resolution. Compared to the traditional upscale module which can only accept the upscale factor for the entire image, our upscale module can either accept two independent and arbitrary upscale factors for either height and width, e.g., $2.4\times $ for height and $3.1\times $ for width, or any specific resolution, e.g., $256\times 128$ , helping to unify the resolution of images with different width-to-height ratio. Extensive experiments are conducted and the result shows that the performance of our method is over the SR-based state-of-the-art person re-id methods on common re-id benchmarks.

DeepInterpolation: fusion of multiple interpolations and CNN to obtain super‐resolution

The authors propose architectures that learn end-to-end mapping functions to improve the spatial resolution of the input natural images. The models are unique in forming non-linear combinations of three image interpolation techniques using the convolutional neural network. Another proposed architecture uses a skip connection with nearest-neighbour interpolation, achieving almost similar results. The architectures have been carefully designed to ensure that the reconstructed images lie precisely in the manifold of high-resolution images, thereby preserving the high-frequency components with fine details. They have compared with the state-of-the-art and recent deep learning-based natural image super-resolution techniques and found that their methods can preserve the sharp details in the image, while also obtaining comparable or better peak-signal-to-noise ratio values than them. Since their methods use image interpolations and a shallow convolutional neural network (CNN) with a fewer number of smaller filters, the computational cost is kept low. They have reported the results of the best two proposed architectures on five standard data sets for an upscale factor of 2. Their methods generalise well in most cases, which is evident from the better results obtained with increasingly complex data sets. For four times upscaling, they have designed similar architectures for comparing with other methods.

A Real-Time Learning-Based Super-Resolution System on FPGA

This paper proposes a real-time super-resolution (SR) system. The proposed system performs a fast SR algorithm that generates a high-resolution image from a low-resolution image using direct regression functions with an up-scaling factor of 2. This algorithm contained two processes: feature learning and SR image prediction. The feature learning stage is performed offline, in which several regression functions were trained. The SR image prediction stage is implemented on the proposed system to generate high-resolution image patches. The system implemented on a Xilinx Virtex 7 field-programmable gate array achieves output resolution of [Formula: see text] (UHD) at 85 fps and 700Mpixels/s throughput. Structure similarity (SSIM) is measured for image quality. Experimental results show that the proposed system provides high image quality for real-time applications. And the proposed system possesses high scalability for resolution.

N2NSR-OCT: Simultaneous denoising and super-resolution in optical coherence tomography images using semisupervised deep learning.

Optical coherence tomography (OCT) imaging shows a significant potential in clinical routines due to its noninvasive property. However, the quality of OCT images is generally limited by inherent speckle noise of OCT imaging and low sampling rate. To obtain high signal-to-noise ratio (SNR) and high-resolution (HR) OCT images within a short scanning time, we presented a learning-based method to recover high-quality OCT images from noisy and low-resolution OCT images. We proposed a semisupervised learning approach named N2NSR-OCT, to generate denoised and super-resolved OCT images simultaneously using up- and down-sampling networks (U-Net (Semi) and DBPN (Semi)). Additionally, two different super-resolution and denoising models with different upscale factors (2× and 4×) were trained to recover the high-quality OCT image of the corresponding down-sampling rates. The new semisupervised learning approach is able to achieve results comparable with those of supervised learning using up- and down-sampling networks, and can produce better performance than other related state-of-the-art methods in the aspects of maintaining subtle fine retinal structures.

CT super-resolution using multiple dense residual block based GAN

High-resolution computed tomography (CT) can provide accurate diagnostic information for clinical applications. However, using CT scanning equipment to obtain high-resolution CT directly may cause significant radiation damage to human body. Low-dose CT super-resolution using generative adversarial network (GAN) can improve the visual quality of CT while maintaining a low radiation dose to human body. The existing GAN networks for super-resolution still suffer from the issues such as weak feature expression and network redundancy. This work proposes a novel lightweight multiple dense residual block structure-based GAN network for CT super-resolution. The new structure reduces the number of residual units and establishes a dense link among all residual blocks, which can reduce network redundancy and ensure maximum information transmission. In addition, in order to avoid the gradient vanishing phenomena, the Wasserstein distance is introduced into the loss function. Experimental results show that the presented method achieved a more desirable visual quality with more high-frequency details for different upscaling factors than other popular methods did.

Progressive perception-oriented network for single image super-resolution

Recently, it has been demonstrated that deep neural networks can significantly improve the performance of single image super-resolution (SISR). Numerous studies have concentrated on raising the quantitative quality of super-resolved (SR) images. However, these methods that target PSNR maximization usually produce blurred images at large upscaling factor. The introduction of generative adversarial networks (GANs) can mitigate this issue and show impressive results with synthetic high-frequency textures. Nevertheless, these GAN-based approaches always have a tendency to add fake textures and even artifacts to make the SR image of visually higher-resolution. In this paper, we propose a novel perceptual image super-resolution method that progressively generates visually high-quality results by constructing a stage-wise network. Specifically, the first phase concentrates on minimizing pixel-wise error, and the second stage utilizes the features extracted by the previous stage to pursue results with better structural retention. The final stage employs fine structure features distilled by the second phase to produce more realistic results. In this way, we can maintain the pixel, and structural level information in the perceptual image as much as possible. It is useful to note that the proposed method can build three types of images in a feed-forward process. Also, we explore a new generator that adopts multi-scale hierarchical features fusion. Extensive experiments on benchmark datasets show that our approach is superior to the state-of-the-art methods. Code is available at https://github.com/Zheng222/PPON.

ISeeBetter: Spatio-temporal video super-resolution using recurrent generative back-projection networks

Recently, learning-based models have enhanced the performance of single-image super-resolution (SISR). However, applying SISR successively to each video frame leads to a lack of temporal coherency. Convolutional neural networks (CNNs) outperform traditional approaches in terms of image quality metrics such as peak signal to noise ratio (PSNR) and structural similarity (SSIM). On the other hand, generative adversarial networks (GANs) offer a competitive advantage by being able to mitigate the issue of a lack of finer texture details, usually seen with CNNs when super-resolving at large upscaling factors. We present iSeeBetter, a novel GAN-based spatio-temporal approach to video super-resolution (VSR) that renders temporally consistent super-resolution videos. iSeeBetter extracts spatial and temporal information from the current and neighboring frames using the concept of recurrent back-projection networks as its generator. Furthermore, to improve the “naturality” of the super-resolved output while eliminating artifacts seen with traditional algorithms, we utilize the discriminator from super-resolution generative adversarial network. Although mean squared error (MSE) as a primary loss-minimization objective improves PSNR/SSIM, these metrics may not capture fine details in the image resulting in misrepresentation of perceptual quality. To address this, we use a four-fold (MSE, perceptual, adversarial, and total-variation loss function. Our results demonstrate that iSeeBetter offers superior VSR fidelity and surpasses state-of-the-art performance.

Optimization-Driven Kernel and Deep Convolutional Neural Network for Multi-View Face Video Super Resolution

One of the major challenges faced by video surveillance is recognition from low-resolution videos or person identification. Image enhancement methods play a significant role in enhancing the resolution of the video. This article introduces a technique for face super resolution based on a deep convolutional neural network (Deep CNN). At first, the video frames are extracted from the input video and the face detection is performed using the Viola-Jones algorithm. The detected face image and the scaling factors are fed into the Fractional-Grey Wolf Optimizer (FGWO)-based kernel weighted regression model and the proposed Deep CNN separately. Finally, the results obtained from both the techniques are integrated using a fuzzy logic system, offering a face image with enhanced resolution. Experimentation is carried out using the UCSD face video dataset, and the effectiveness of the proposed Deep CNN is checked depending on the block size and the upscaling factor values and is evaluated to be the best when compared to other existing techniques with an improved SDME value of 80.888.

Data-driven three-dimensional super-resolution imaging of a turbulent jet flame using a generative adversarial network.

Three-dimensional (3D) computed tomography (CT) is becoming a well-established tool for turbulent combustion diagnostics. However, the 3D CT technique suffers from contradictory demands of spatial resolution and domain size. This work therefore reports a data-driven 3D super-resolution approach to enhance the spatial resolution by two times along each spatial direction. The approach, named 3D super-resolution generative adversarial network (3D-SR-GAN), builds a generator and a discriminator network to learn the topographic information and infer high-resolution 3D turbulent flame structure with a given low-resolution counterpart. This work uses numerically simulated 3D turbulent jet flame structures as training data to update model parameters of the GAN network. Extensive performance evaluations are then conducted to show the superiority of the proposed 3D-SR-GAN network, compared with other direct interpolation methods. The results show that a convincing super-resolution (SR) operation with the overall error of ∼4% and the peak signal-to-noise ratio of 37 dB can be reached with an upscaling factor of 2, representing an eight times enhancement of the total voxel number. Moreover, the trained network can predict the SR structure of the jet flame with a different Reynolds number without retraining the network parameters.

Hyperspectral Image Super-Resolution by Band Attention Through Adversarial Learning

Hyperspectral image (HSI) super-resolution (SR) is a challenging task due to the problems of texture blur and spectral distortion when the upscaling factor is large. To meet these two challenges, band attention through the adversarial learning method is proposed in this article. First, we put the SR process in a generative adversarial network (GAN) framework, so that the resulted high-resolution HSI can keep more texture details. Second, different from the other band-by-band SR method, the input of our method is of full bands. In order to explore the correlation of spectral bands and avoid the spectral distortion, a band attention mechanism is proposed in our generative network. A series of spatial–spectral constraints or loss functions is imposed to guide the training of our generative network so as to further alleviate spectral distortion and texture blur. The experiments on the Pavia and Cave data sets demonstrate that the proposed GAN-based SR method can yield very high-quality results, even under large upscaling factor (e.g., $8\times $ ). More importantly, it can outperform the other state-of-the-art methods by a margin which demonstrates its superiority and effectiveness.

Mass and Metallurgical Balance Forecast for a Zinc Processing Plant Using Artificial Neural Networks

The forecasting of ore concentrate and tailings mass and metallurgical recovery at a processing plant is not a simple task. It starts with data collection, which is expensive and laborious, and progresses to multivariate data analysis, which is used to identify the independent variables that should be used to build a prediction model. This is followed by the choice of a statistical technique that is able to deal with data particularity. After building a model, the differences between the flotation batch test and the true plant circuit need to be considered because it is difficult to build a laboratory test that exactly mimics the plant configuration. When the model and its up-scaling factor have been defined, the last step is to check the efficiency of the model in terms of forecasting the geometallurgical variables under study. Bearing in mind that such geometallurgical predictions help in mine planning, economic forecasting and environmental studies (tailings mass and metallurgical recoveries), this paper proposes a methodology that is able to predict six plant outputs simultaneously. These are metallurgical recovery of Zn from Zn concentrate, metallurgical recovery of Zn from Pb concentrate and metallurgical recoveries of Zn from tailings, Zn concentrate mass, Pb concentrate mass and tailings mass. A neural networks technique was used, and the predictions of the model with an up-scaling factor were reconciled with the plant responses, which showed consistent results.

Facial Image Synthesis and Super-Resolution With Stacked Generative Adversarial Network

Image synthesis and super-resolution (SR) have always been a hot spot for computer vision and image processing research. Since the development of Deep Learning, especially after the Deep Convolutional Generative Adversarial Network (DC-GAN) methods, facial image synthesis and SR problem had been solved in many circumstances. But most of the existing works were focused on natural-looking of the synthesized result rather than keeping facial information of the original image. Our paper presented an end-to-end method of getting high-resolution photo-realistic facial images from low-resolution (LR) in-the-wild images without losing the facial identity details. The pipeline used a flexible stacked GAN structure for the SR process with different target image resolutions on different upscaling factors. To avoid getting blur or nonsensical image output and realize the flexibility, “U-Net” architecture and upsampling layers with residual learning blocks were stacked. The stacked network structure makes applying different loss functions in different parts of the network possible, which helps to solve the two problems of keeping identical facial details of the LR input image and generating high-quality output images simultaneously. By using 3 different loss functions in different positions of the stacked network separately, through experimental comparison, we found the best stacked residual block parameters which could get the best output image quality. Experimental results also explicated that the network had a good SR ability compare to state of the art methods in different resolution and upscaling factor.

Depth upsampling based on deep edge-aware learning

Depth map upsampling will unavoidably smoothen the edges leading to blurry results on the depth boundaries, especially at large upscaling factors. Given that edges represent the most important cue in addressing the task of depth upsampling, we propose a novel depth upsampling framework based on deep edge-aware learning. Unlike existing CNN-based approaches that directly predict depth values from low resolution (LR) depth input, our framework firstly learns edge information of depth boundaries from the known LR depth map and its corresponding high resolution (HR) color image as reconstruction cues. Then, two depth restoration modules, i.e., a fast depth filling strategy and a cascaded restoration network, are proposed to recover HR depth map by leveraging the predicted edge map and the HR color image. Extensive comparisons on both edge inference and depth upsampling under noisy and noiseless cases demonstrate the superiority of the proposed approaches.

Upscaling factor estimation on double JPEG compressed images

Resampling detection is one of the most important topics in image forensics. Some methods can even estimate the resampling factors. However, it remains a challenge on the post-compressed images because the JPEG compression destroys the statistical characteristics of the resampled images which greatly affect the detection ability. In this paper, we propose a method to estimate the upscaling factors of double JPEG compressed images in the presence of image upscaling between the two compressions. Both of the pre-compression and the post-compression have a great impact on the estimation. We first analyze the pre-compressed image upscaling and find that the peaks produced by the pre-compression will be suppressed by the post-compression. Therefore, the post-compression has a greater impact than the pre-compression. Through analyzing the influence of post-compression and the causes of the periodic JPEG peaks, we find that the image block estimation can effectively reduce the influence of JPEG block artifacts. To further improve the detection ability, a Gaussian filtering model is developed to deal with JPEG block artifacts and a detector which can more effective extract resampling characteristic is proposed to estimate the upscaling factor. The experimental results demonstrate that the proposed detection method outperforms some state-of-the-art methods.

Image Super-Resolution Reconstruction Based on a Generative Adversarial Network

In the field of computer vision, super-resolution reconstruction techniques based on deep learning have undergone considerable advancement; however, certain limitations remain, such as insufficient feature extraction and blurred image generation. To address these problems, we propose an image super-resolution reconstruction model based on a generative adversarial network. First, we employ a dual network structure in the generator network to solve the problem of insufficient feature extraction. The dual network structure is divided into an upsample subnetwork and a refinement subnetwork, which upsample and optimize a low-resolution image, respectively. In a scene with large upscaling factors, this structure can reduce the negative effect of noise and enhance the utilization of high-frequency details, thereby generating high-quality reconstruction results. Second, to generate sharper super-resolution images, we use the perceptual loss, which exhibits a fast convergence and excellent visual effect, to guide the generator network training. We apply the ResNeXt-50- $32\times 4\text{d}$ network, which has few parameters and a large depth, to calculate the loss to obtain a reconstructed super-resolution image that is highly realistic. Finally, we introduce the Wasserstein distance into the discriminator network to enhance the discrimination ability and stability of the model. Specifically, this distance is employed to eliminate the activation function in the last layer of the network and avoid the use of the logarithm in calculating the loss function. Extensive experiments on the DIV2K, Set5, Set14, and BSD100 datasets demonstrate the effectiveness of the proposed model.

Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

Computed Tomography (CT) scanners that are commonly-used in hospitals and medical centers nowadays produce low-resolution images, e.g. one voxel in the image corresponds to at most one-cubic millimeter of tissue. In order to accurately segment tumors and make treatment plans, radiologists and oncologists need CT scans of higher resolution. The same problem appears in Magnetic Resonance Imaging (MRI). In this paper, we propose an approach for the single-image super-resolution of 3D CT or MRI scans. Our method is based on deep convolutional neural networks (CNNs) composed of 10 convolutional layers and an intermediate upscaling layer that is placed after the first 6 convolutional layers. Our first CNN, which increases the resolution on two axes (width and height), is followed by a second CNN, which increases the resolution on the third axis (depth). Different from other methods, we compute the loss with respect to the ground-truth high-resolution image right after the upscaling layer, in addition to computing the loss after the last convolutional layer. The intermediate loss forces our network to produce a better output, closer to the ground-truth. A widely-used approach to obtain sharp results is to add Gaussian blur using a fixed standard deviation. In order to avoid overfitting to a fixed standard deviation, we apply Gaussian smoothing with various standard deviations, unlike other approaches. We evaluate the proposed method in the context of 2D and 3D super-resolution of CT and MRI scans from two databases, comparing it to related works from the literature and baselines based on various interpolation schemes, using 2× and 4× scaling factors. The empirical study shows that our approach attains superior results to all other methods. Moreover, our subjective image quality assessment by human observers reveals that both doctors and regular annotators chose our method in favor of Lanczos interpolation in 97.55% cases for an upscaling factor of 2× and in 96.69% cases for an upscaling factor of 4×. In order to allow others to reproduce our state-of-the-art results, we provide our code as open source at https://github.com/lilygeorgescu/3d-super-res-cnn.

A Light Weight Convolutional Neural Network for Single Image Super-Resolution

Abstract Recently, many convolutional neural network based models obtain remarkable performance in single-image super-resolution task by stacking more number of convolution layers. However, those models require a huge amount of network parameters which increases the computational complexity of their single image super-resolution models. Due to this, they are no longer appropriate for many real-world applications. Hence, to design a network which can obtain better super-resolution performance with less number of network parameters is always an active area of research in the computer vision community. In this paper, we propose a light weight convolutional neural network based SR model called LWSRNet for the upscaling factor x4. In LWSRNet, we introduce a novel basic block which helps to extract complex features of the given low-resolution observation. Additionally, we use a weighted L2 loss function in order to train the network which is more effective than L1 and L2 loss functions. Various experiments have been carried out to validate the proposed method and observe that the super-resolution results obtained using the proposed LWSRNet method are better than that of the other existing single image super-resolution methods. Also, the proposed LWSRNet outperforms to the recently proposed state-of-the-art methods with approximately 20% to 60% less number of training parameters.

Joint-SRVDNet: Joint Super Resolution and Vehicle Detection Network

In many domestic and military applications, aerial vehicle detection and super-resolution algorithms are frequently developed and applied independently. However, aerial vehicle detection on super-resolved images remains a challenging task due to the lack of discriminative information in the super-resolved images. To address this problem, we propose a Joint Super-Resolution and Vehicle Detection Network (Joint-SRVDNet) that tries to generate discriminative, high-resolution images of vehicles from low-resolution aerial images. First, aerial images are up-scaled by a factor of 4x using a Multi-scale Generative Adversarial Network (MsGAN), which has multiple intermediate outputs with increasing resolutions. Second, a detector is trained on super-resolved images that are upscaled by factor 4x using MsGAN architecture and finally, the detection loss is minimized jointly with the super-resolution loss to encourage the target detector to be sensitive to the subsequent super-resolution training. The network jointly learns hierarchical and discriminative features of targets and produces optimal super-resolution results. We perform both quantitative and qualitative evaluation of our proposed network on VEDAI, xView and DOTA datasets. The experimental results show that our proposed framework achieves better visual quality than the state-of-the-art methods for aerial super-resolution with 4x up-scaling factor and improves the accuracy of aerial vehicle detection.