Single-frame Super-resolution Research Articles

A Single-Frame and Multi-Frame Cascaded Image Super-Resolution Method.

The objective of image super-resolution is to reconstruct a high-resolution (HR) image with the prior knowledge from one or several low-resolution (LR) images. However, in the real world, due to the limited complementary information, the performance of both single-frame and multi-frame super-resolution reconstruction degrades rapidly as the magnification increases. In this paper, we propose a novel two-step image super resolution method concatenating multi-frame super-resolution (MFSR) with single-frame super-resolution (SFSR), to progressively upsample images to the desired resolution. The proposed method consisting of an L0-norm constrained reconstruction scheme and an enhanced residual back-projection network, integrating the flexibility of the variational model-based method and the feature learning capacity of the deep learning-based method. To verify the effectiveness of the proposed algorithm, extensive experiments with both simulated and real world sequences were implemented. The experimental results show that the proposed method yields superior performance in both objective and perceptual quality measurements. The average PSNRs of the cascade model in set5 and set14 are 33.413 dB and 29.658 dB respectively, which are 0.76 dB and 0.621 dB more than the baseline method. In addition, the experiment indicates that this cascade model can be robustly applied to different SFSR and MFSR methods.

PRINCIPAL COMPONENTS VERSUS AUTOENCODERS FOR DIMENSIONALITY REDUCTION: A CASE OF SUPER-RESOLVED OUTPUTS FROM PRISMA HYPERSPECTRAL MISSION DATA

Abstract. This study attempts to solve these issues associated with hyperspectral (HS) data, i.e., coarse spatial resolution and high volume, by understanding the effect of deep learning and traditional dimensionality reduction on super-resolved products generated from the recently launched PRecursore IperSpettrale della Missione Applicativa (PRISMA) HS mission. Four single-frame super-resolution (SR) algorithms have been used to super-resolve a 30 m PRISMA scene of Ahmedabad, India and generate 15 m spatial resolution images with both spatial and spectral fidelity. Iterative back projection (IBP) and sparse representation (SIS) are the best and worst-performing SR algorithms following a comparative assessment and validation protocol. Next, denoising autoencoders and PCT computed using singular and eigenvalue decompositions have been executed on the original PRISMA, IBP and SIS-based super-resolved datasets. The resulting low-dimensional representations have been assessed to preserve the original dataset's topology using label-independent Lee and Verleysen's co-ranking matrix and loss of quality measure. Findings suggest that autoencoders are computationally expensive and require a higher neighbourhood size than PCT and its variants to produce a high-quality encoding. These insights remain significant for urban information extraction as there are few direct comparative assessments between machine learning-based linear and non-linear data compression methods in earlier studies.

BLUR KERNEL’S EFFECT ON PERFORMANCE OF SINGLE-FRAME SUPER-RESOLUTION ALGORITHMS FOR SPATIALLY ENHANCING HYPERION AND PRISMA DATA

Abstract. Single-frame super-resolution (SFSR) achieves the goal of generating a high-resolution image from a single low-resolution input in a three-step process, namely, noise removal, up-sampling and deblurring. Scale factor and blur kernel are essential parameters of the up-sampling and deblurring steps. Few studies document the impact of these parameters on the performance of SFSR algorithms for improving the spatial resolution of real-world remotely-sensed datasets. Here, the effect of changing blur kernel has been studied on the behaviour of two classic SFSR algorithms: iterative back projection (IBP) and gaussian process regression (GPR), which are applied to two spaceborne hyperspectral datasets for scale factors 2, 3 and 4. Eight full-reference image quality metrics and algorithm processing time are deployed for this purpose. A literature-based re-interpretation of Wald’s reduced resolution protocol has also been used in this work for choosing the reference image. Intensive intra-algorithm comparisons of various simulation scenarios reveal each algorithm’s best performing Gaussian blur kernel parameters. Inter-algorithm comparison shows the better performing algorithm out of the two, thereby paving the way for further research in SFSR of remotely-sensed images.

Exploring single-frame super-resolution on real-world Hyperion and PRISMA datasets of an urban area in a developing nation

ABSTRACT Few studies on single-frame super-resolution (SFSR) exist for real-world hyperspectral images due to fewer training samples and paucity of freely available high-resolution ground truth for validation. Consequently, a user relies on the simulated results from test datasets. This gap is filled by super-resolving real-time Hyperion and PRISMA data of Ahmedabad in India and introducing a structure of comparing and validating super-resolved outputs. Proposing open-source panchromatic and multispectral data as visual examination references, the first-ever SFSR results on PRISMA are presented. Findings suggest reconstruction-based SFSR algorithm outperforms at scale factors (SFs) 2 and 4 for Hyperion and SFs 2, 3 and 4 for PRISMA. The hybrid SFSR algorithm stands out at SF 3 for Hyperion. External dictionaries utilized by learning-based SFSR algorithms experience degradation with increasing spatial resolution. Although image quality declines slightly at higher SFs, out of the chosen algorithms, reconstruction and hybrid-based algorithms outperform learning-based algorithms in producing enhanced spatial and spectral fidelity outputs in the present case.

Deep-Learning-Based Super-Resolution of Video Satellite Imagery by the Coupling of Multiframe and Single-Frame Models

Image super-resolution (SR) is an effective sol-ution to the limitation of the spatial resolution of video satellite images, which is caused by the degradation and compression in the imaging phase. For the processing of satellite videos, the commonly employed deep learning based single-frame SR (SFSR) framework have limited performance without using complementary information between the video frames. On the other side, the multi-frame SR (MFSR) can utilize temporal sub-pixel information to super-resolve the high-resolution (HR) imagery. However, although deeper and wider deep learning network provide powerful feature representations for SR methods, it has always been a challenge to accurately recon-struct the boundaries of ground objects in video satellite images. In this paper, to address these issues, we propose an edge-guided video super-resolution (EGVSR) framework for video satellite image SR, which couple MFSR model and edge-SFSR (E-SFSR) model in a unified network. The EGVSR framework is composed of an MFSR branch and an edge branch. The MFSR branch is used to extract the complementary features from the consecutive video frames. Concurrently, the edge branch acts as an SFSR model to translate the edge maps from the low-resolution modality to the HR one. At the final SR stage, the DBFM is built to focus on the promising inner repre-sentations of the features of the two branches and fuse them. Extensive experiments on video satellite imagery show that the proposed EGVSR method can achieve a superior performance when compared to the representative deep learning based SR methods.

Single-frame super-resolution for high resolution optical remote-sensing data products

ABSTRACT The resolution of remote-sensing images is directly related to the application value of data products. Due to differences in imaging characteristics between digital cameras and remote-sensing cameras, the existing network models cannot get the best super resolution (SR) results of satellite images. In response to the requirements of remote-sensing image production, we propose a single image super-resolution (SISR) reconstruction method for specific type of remote-sensing satellite. First, we measure and model the imaging degradation phenomenon of remote-sensing satellite, including the image blur and noise model. Then, high-quality aerial images are down-sampled and degraded to construct paired training image datasets. We chose the most popular Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN) as the basic structure and optimized the number of Residual-in-Residual Dense Block (RRDB) modules to further improve the processing efficiency. Finally, we perform a series of quantitative measurements of the SR image results, including image interpretation capability, reconstruction accuracy, ground resolution distance, and data processing efficiency, using higher resolution remote-sensing images as the benchmark. The experimental results demonstrate that our proposed method has higher interpretation capability and reconstruction accuracy for the SR processing of specific type remote-sensing satellite. Our proposed method is evaluated within a real satellite image product, that demonstrated it has the capability of pipeline super-resolution processing of remote sensing data products.

Single-frame super-resolution for remote sensing images based on improved deep recursive residual network

Single-frame image super-resolution (SISR) technology in remote sensing is improving fast from a performance point of view. Deep learning methods have been widely used in SISR to improve the details of rebuilt images and speed up network training. However, these supervised techniques usually tend to overfit quickly due to the models’ complexity and the lack of training data. In this paper, an Improved Deep Recursive Residual Network (IDRRN) super-resolution model is proposed to decrease the difficulty of network training. The deep recursive structure is configured to control the model parameter number while increasing the network depth. At the same time, the short-path recursive connections are used to alleviate the gradient disappearance and enhance the feature propagation. Comprehensive experiments show that IDRRN has a better improvement in both quantitation and visual perception.

Video super resolution using convolutional neural network and image fusion techniques

Recently, huge progress has been achieved in the field of single image super resolution which augments the resolution of images. The idea behind super resolution is to convert low-resolution images into high-resolution images. SRCNN (Single Resolution Convolutional Neural Network) was a huge improvement over the existing methods of single-image super resolution. However, video super-resolution, despite being an active field of research, is yet to benefit from deep learning. Using still images and videos downloaded from various sources, we explore the possibility of using SRCNN along with image fusion techniques (minima, maxima, average, PCA, DWT) to improve over existing video super resolution methods. Video Super-Resolution has inherent difficulties such as unexpected motion, blur and noise. We propose Video Super Resolution – Image Fusion (VSR-IF) architecture which utilizes information from multiple frames to produce a single high- resolution frame for a video. We use SRCNN as a reference model to obtain high resolution adjacent frames and use a concatenation layer to group those frames into a single frame. Since, our method is data-driven and requires only minimal initial training, it is faster than other video super resolution methods. After testing our program, we find that our technique shows a significant improvement over SCRNN and other single image and frame super resolution techniques.

Gated Fusion Network for Degraded Image Super Resolution

Single image super resolution aims to enhance image quality with respect to spatial content, which is a fundamental task in computer vision. In this work, we address the task of single frame super resolution with the presence of image degradation, e.g., blur, haze, or rain streaks. Due to the limitations of frame capturing and formation processes, image degradation is inevitable, and the artifacts would be exacerbated by super resolution methods. To address this problem, we propose a dual-branch convolutional neural network to extract base features and recovered features separately. The base features contain local and global information of the input image. On the other hand, the recovered features focus on the degraded regions and are used to remove the degradation. Those features are then fused through a recursive gate module to obtain sharp features for super resolution. By decomposing the feature extraction step into two task-independent streams, the dual-branch model can facilitate the training process by avoiding learning the mixed degradation all-in-one and thus enhance the final high-resolution prediction results. We evaluate the proposed method in three degradation scenarios. Experiments on these scenarios demonstrate that the proposed method performs more efficiently and favorably against the state-of-the-art approaches on benchmark datasets.

Characterization and Retrieval of Snow and Urban Land Cover Parameters using Hyperspectral Imaging

Author(s): Singh, SK; Jain, G; Siddiqui, A; Naik, S; Rathore, BP; Garg, V; Snehmani; Kumar, V; Bahuguna, IM; Sharma, SA; Shekhar, C; Thakur, PK; Mishra, K; Kumar, P; Painter, TH; Dozier, J | Abstract: Snow and urban land cover are important due to their role in hydrological management and utility, climate response, social aspects and economic viability, along with influencing the Earth's environment at local, regional and global scale. Hyperspectral data enable identification, characterization and retrieval of these land-cover features based on physical and chemical properties of compositional materials. AVIRISNG hyperspectral airborne data, with synchronous ground observations using field spectroradiometer and collateral instruments, were collected over two widely varied land-cover types, viz. a relatively homogenous area covered by snow in the extreme cold environment of the Himalaya (Bhaga sub-basin, Himachal Pradesh), and a completely heterogeneous urban area of a metropolitan city (Ahmedabad, Gujarat). AVIRIS-NG airborne data were analysed to understand the effect of terrain parameters such as slope and aspect on snow reflectance. Snow grain index using visible and near-infrared (VNIR) bands and absorption peak in the near-infrared (NIR) were used to retrieve grain size in parts of the Himalayan region. A radiative transfer model was used to understand the grain size variability and its effect on absorption peak in NIR. Continuum removal was performed for snow spectral observations obtained from airborne, modelled and field platforms to estimate band depth at 1030 nm. Grain size was observed to vary with altitude from 100 to 500 μm using AVIRIS-NG image. In the urban area, the data also separated pervious and impervious surface cover using spectral unmixing technique, identified several urban features over multispectral data such as buildings with red tiled roofs, metallic surfaces and tarpaulin sheets using the material spectral profiles. Two single-frame superresolution methods namely sparse regression and natural prior (SRP), and gradient profile prior (GPP) were applied on AVIRIS-NG data for the mixed environment around Kankaria Lake in the city of Ahmedabad, which revealed that SRP method was better than GPP, and affirmed by eight indices. Preliminary analysis of AVIRIS-NG imaging over snow-covered areas and densely populated cities indicated utility of future spaceborne hyperspectral missions, particularly for hydrological and climatological applications in such diverse environments.

An Extra-Rate Spatial Enhancement Constructed by MSRR using Regularized Technique and SSRR using High-Frequency Pre-Forecasting

Under many circumstances, a high spatial resolution image (HR) is greatly needed for modern applications nevertheless the HR image captured device is usually overpriced cost. Hence, Super Resolution Reconstruction (SRR) technique, which can reconstruct a HR image from a single LR image or many LR images by using algebraic formulation, is one of the modern research fields in digital image processing (DIP) and Computer Vision (CV). In this paper, an extra-rate spatial enhancement constructed by MSRR (Multi-frame Super Resolution Reconstruction) using regularized technique and SSRR (Single-frame Super Resolution Reconstruction) using high-frequency pre-forecasting is presented in order to enlarge up to 16x ratio rate. Initially, a group of captured images with low spatial resolution are mathematically fused by MSRR using regularized technique established on a recursive Maximum Likelihood and Tukey's Biweight norm in order to enlarge up to 4x ratio rate. Next, this 4x enhanced image is enlarged to be 16x spatial resolution image by SSRR established on the high-frequency pre-forecasting. In the verification experimentation section, the verification outcome demonstrates that the proposed spatial enhancement is successful for enlarging HR image with 16x ratio rate with finer quality.

Single-frame super resolution of remote-sensing images by convolutional neural networks

ABSTRACTSuper resolution (SR) refers to generation of a high-resolution (HR) image from a decimated, blurred, low-resolution (LR) image set, which can be either a single-frame or multi-frame that contains a collection of images acquired from slightly different views of the same observation area. In this study, two convolutional neural network (CNN)-based deep learning techniques are adapted in single-frame SR to increase the resolution of remote sensing (RS) images by a factor of 2, 3, and 4. In order to both preserve the colour information and speed up the algorithm, first an intensity hue saturation (IHS) transform is utilized and the SR techniques are only applied to the intensity channel of the images. Colour information is then restored with an inverse IHS transformation. We demonstrate the results of the proposed method on RS images acquired from Satellites Pour l’Observation de la Terre (SPOT) or Earth-observing satellites and Pleiades satellites with different spatial resolution. First synthetic LR images are created by downsampling, then structural similarity (SSIM) Index, peak signal-to-noise ratio (PSNR), Spectral Angle Mapper (SAM) and Erreur Relative Globale Adimensionnelle de Synthese (ERGAS) values are calculated for a quantitative evaluation of the methods. Finally, the method, with better performance results, is tested within a real scenario, that is, with original LR images as the input. The obtained HR images demonstrated visible qualitative enhancements.

Super-resolution imaging applied to moving object tracking

Moving object tracking in a video is a method used to detect and analyze changes that occur in an object that being observed. Visual quality and the precision of the tracked target are highly wished in modern tracking system. The fact that the tracked object does not always seem clear causes the tracking result less precise. The reasons are low quality video, system noise, small object, and other factors. In order to improve the precision of the tracked object especially for small object, we propose a two step solution that integrates a super-resolution technique into tracking approach. First step is super-resolution imaging applied into frame sequences. This step was done by cropping the frame in several frame or all of frame. Second step is tracking the result of super-resolution images. Super-resolution image is a technique to obtain high-resolution images from low-resolution images. In this research single frame super-resolution technique is proposed for tracking approach. Single frame super-resolution was a kind of super-resolution that it has the advantage of fast computation time. The method used for tracking is Camshift. The advantages of Camshift was simple calculation based on HSV color that use its histogram for some condition and color of the object varies. The computational complexity and large memory requirements required for the implementation of super-resolution and tracking were reduced and the precision of the tracked target was good. Experiment showed that integrate a super-resolution imaging into tracking technique can track the object precisely with various background, shape changes of the object, and in a good light conditions.

Sparse Representation-Based Multiple Frame Video Super-Resolution.

In this paper, we propose two multiple-frame super-resolution (SR) algorithms based on dictionary learning (DL) and motion estimation. First, we adopt the use of video bilevel DL, which has been used for single-frame SR. It is extended to multiple frames by using motion estimation with sub-pixel accuracy. We propose a batch and a temporally recursive multi-frame SR algorithm, which improves over single-frame SR. Finally, we propose a novel DL algorithm utilizing consecutive video frames, rather than still images or individual video frames, which further improves the performance of the video SR algorithms. Extensive experimental comparisons with the state-of-the-art SR algorithms verify the effectiveness of our proposed multiple-frame video SR approach.

Machine Learning Based Single-Frame Super-Resolution Processing for Lensless Blood Cell Counting.

A lensless blood cell counting system integrating microfluidic channel and a complementary metal oxide semiconductor (CMOS) image sensor is a promising technique to miniaturize the conventional optical lens based imaging system for point-of-care testing (POCT). However, such a system has limited resolution, making it imperative to improve resolution from the system-level using super-resolution (SR) processing. Yet, how to improve resolution towards better cell detection and recognition with low cost of processing resources and without degrading system throughput is still a challenge. In this article, two machine learning based single-frame SR processing types are proposed and compared for lensless blood cell counting, namely the Extreme Learning Machine based SR (ELMSR) and Convolutional Neural Network based SR (CNNSR). Moreover, lensless blood cell counting prototypes using commercial CMOS image sensors and custom designed backside-illuminated CMOS image sensors are demonstrated with ELMSR and CNNSR. When one captured low-resolution lensless cell image is input, an improved high-resolution cell image will be output. The experimental results show that the cell resolution is improved by 4×, and CNNSR has 9.5% improvement over the ELMSR on resolution enhancing performance. The cell counting results also match well with a commercial flow cytometer. Such ELMSR and CNNSR therefore have the potential for efficient resolution improvement in lensless blood cell counting systems towards POCT applications.

An image enhancement tool: Pattern Recognition Image Augmented Resolution

PRIAR (Pattern Recognition Image Augmented Resolution) is an innovative approach to singleframe super-resolution that combines common single-frame super-resolution with pattern-recognition algorithms. PRIAR uses the information gained through pattern-recognition to enhance resolution for low quality images, and to allow the end user to explore, recognize and super-resolve low-resolution images. In this paper, we present the basic functionality of the PRIAR algorithm that we have implemented. The program is modular and each module is easily combined. In addition, such modularity permits us to work on images where single modules can be changed in order to resolve different classes of problems. In this paper, we firstly present the features of the PRIAR program processing images reproducing animal cells recorded with a scanning probe microscope.

Super-Resolution of Dynamic Scenes Using Sampling Rate Diversity.

In earlier work, we proposed a super-resolution (SR) method that required the availability of two low resolution (LR) sequences corresponding to two different sampling rates, where images from one sequence were used as a basis to represent the polyphase components (PPCs) of the high resolution (HR) image, while the other LR sequences provided the reference LR image (to be super-resolved). The (simple) algorithm implemented by Salem and Yagle is only applicable when the scene is static. In this paper, we recast our approach to SR as a two-stage example-based algorithm to process dynamic scenes. We employ feature selection to create, from the LR frames, local LR dictionaries to represent PPCs of HR patches. To enforce sparsity, we implement Gaussian generative models as an efficient alternative to L1-norm minimization. Estimation errors are further reduced using what we refer to as the anchors, which are based on the relationship between PPCs corresponding to different sampling rates. In the second stage, we revert to simple single frame SR (applied to each frame), using HR dictionaries extracted from the super-resolved sequence of the previous stage. The second stage is thus a reiteration of the sparsity coding scheme, using only one LR sequence, and without involving PPCs. The ability of the modified algorithm to super-resolve challenging LR sequences reintroduces sampling rate diversity as a prerequisite of robust multiframe SR.

Introduction to special issue on ‘Recent Computing Paradigms, Network Protocols, and Applications’

This special issue of Innovations in Systems and Software Engineering: A NASA Journal is devoted to selected contributions from the 3rd International Conference on Advanced Computing, Networking and Informatics (ICACNI-2015), organized by School of Computer Engineering, KIIT University, Odisha, India, during 23–25 June, 2015. The conference commenced with a keynote by Prof. Nikhil R. Pal (Fellow IEEE, Vice President for Publications IEEE Computational Intelligence Society (2015–2016), Indian Statistical Institute, Kolkata, India) on ‘A Fuzzy Rule-Based Approach to Single Frame Super Resolution’. Apart from three regular tracks on advanced computing, networking, and informatics, the conference hosted three invited special sessions. While a total of 558 articles across different tracks of the conference were received, 132 articles are finally selected for presentation and publication by Smart Innovation, Systems and Technologies series of Springer as Volume 43 and 44. The conference showcased a technical talk by Prof. Nabendu Chaki (Senior Member IEEE, Calcutta University, India) on ‘Evolution from Web-based Applications to Cloud Services: A Case Study with Remote Healthcare’. The conference identified some wonderful works and has given away eight awards in different categories.

Adaptive single-frame superresolution for detecting closely spaced IR targets in clutter

This paper addresses the problem of individually resolving very closely spaced infrared (IR) point targets in clutter, an important requirement in a variety of applications including air defense, where groups of targets may be operating in close formations, and astronomy, where stars can be closely spaced in angle as seen from the sensor's perspective. Novel linear and nonlinear (order statistic) adaptive noncoherent superresolution spatial-processing methods are introduced and employed to solve this problem using a single frame of IR data.

A fuzzy-rule-based approach for single frame super resolution.

In this paper, a novel fuzzy rule-based prediction framework is developed for high-quality image zooming. In classical interpolation-based image zooming, resolution is increased by inserting pixels using certain interpolation techniques. Here, we propose a patch-based image zooming technique, where each low-resolution (LR) image patch is replaced by an estimated high-resolution (HR) patch. Since an LR patch can be generated from any of the many possible HR patches, it would be natural to develop rules to find different possible HR patches and then to combine them according to rule strength to get the estimated HR patch. Here, we generate a large number of LR–HR patch pairs from a collection of natural images, group them into different clusters, and then generate a fuzzy rule for each of these clusters. The rule parameters are also learned from these LR-HR patch pairs. As a result, an efficient mapping from LR patch space to HR patch space can be formulated. The performance of the proposed method is tested on different images, and is also compared with other representative as well as state-of-the-art image zooming techniques. Experimental results show that the proposed method is better than the competing methods and is capable of reconstructing thin lines, edges, fine details, and textures in the image efficiently.