Multiple Low-resolution Frames Research Articles

Efficient Video Super-Resolution via Hierarchical Temporal Residual Networks

Super-Resolving (SR) video is more challenging compared with image super-resolution because of the demanding computation time. To enlarge a low-resolution video, the temporal relationship among frames must be fully exploited. We can model video SR as a multi-frame SR problem and use deep learning methods to estimate the spatial and temporal information. This paper proposes a lighter residual network, based on a multi-stage back projection for multi-frame SR. We improve the back projection based residual block by adding weights for adaptive feature tuning, and add global & local connections to explore deeper feature representation. We jointly learn spatial-temporal feature maps by using the proposed Spatial Convolution Packing scheme as an attention mechanism to extract more information from both spatial and temporal domains. Different from others, our proposed network can input multiple low-resolution frames to obtain multiple super-resolved frames simultaneously. We can then further improve the video SR quality by self-ensemble enhancement to meet videos with different motions and distortions. Results of much experimental work show that our proposed approaches give large improvement over other state-of-the-art video SR methods. Compared to recent CNN based video SR works, our approaches can save, up to 60% computation time and achieve 0.6 dB PSNR improvement.

FAST AND ACCURATE MULTI-FRAME SUPER-RESOLUTION OF SATELLITE IMAGES

Abstract. Recent constellations of small satellites, such as Planet’s SkySats, offer new acquisition modes where very short videos or bursts of images are acquired instead of a single still image. Compared to sequences of multi-date images, these sequences of consecutive video frames yield a large redundancy of information within the range of seconds. This redundancy enables to increase the spatial resolution using multi-frame super-resolution algorithms. In this paper, we propose a novel super-resolution method based on a high-order spline interpolation model that combines multiple low-resolution frames to produce a high-resolution image. Moreover this method can be implemented efficiently on GPU to process entire images from real satellite acquisitions. Synthetic and real experiments show that the proposed method is able to recover fine details, and measurements of the resulting resolution indicate a gain of 10 cm / pixel with respect to Planet’s SkySat standard imagery products.

Frame and Feature-Context Video Super-Resolution

For video super-resolution, current state-of-the-art approaches either process multiple low-resolution (LR) frames to produce each output high-resolution (HR) frame separately in a sliding window fashion or recurrently exploit the previously estimated HR frames to super-resolve the following frame. The main weaknesses of these approaches are: 1) separately generating each output frame may obtain high-quality HR estimates while resulting in unsatisfactory flickering artifacts, and 2) combining previously generated HR frames can produce temporally consistent results in the case of short information flow, but it will cause significant jitter and jagged artifacts because the previous super-resolving errors are constantly accumulated to the subsequent frames.In this paper, we propose a fully end-to-end trainable frame and feature-context video super-resolution (FFCVSR) network that consists of two key sub-networks: local network and context network, where the first one explicitly utilizes a sequence of consecutive LR frames to generate local feature and local SR frame, and the other combines the outputs of local network and the previously estimated HR frames and features to super-resolve the subsequent frame. Our approach takes full advantage of the inter-frame information from multiple LR frames and the context information from previously predicted HR frames, producing temporally consistent highquality results while maintaining real-time speed by directly reusing previous features and frames. Extensive evaluations and comparisons demonstrate that our approach produces state-of-the-art results on a standard benchmark dataset, with advantages in terms of accuracy, efficiency, and visual quality over the existing approaches.

Robust Multiframe Super-Resolution Employing Iteratively Re-Weighted Minimization

Multiframe super-resolution algorithms reconstruct high-resolution images by exploiting complementary information in multiple low-resolution frames. However, despite their success under ideal conditions, most existing methods rely on simplistic approximations to the physics of image acquisition and show limited robustness in real-world applications. This paper proposes spatially adaptive Bayesian modeling and an iterative algorithm for robust super-resolution imaging. In particular, we introduce a weighted Gaussian observation model to consider space variant noise and weighted bilateral total variation to exploit sparsity of natural images. Based on this model, we develop a majorization–minimization algorithm implemented as iteratively re-weighted minimization. The proposed method simultaneously estimates model parameters and the super-resolved image in an iterative coarse-to-fine scheme. Compared to prior work, our approach combines the benefits of achieving robust and edge preserving image reconstruction with small amount of parameter tuning, while being flexible in terms of motion models, computationally efficient and easy to implement. Our experimental evaluation confirms that our approach outperforms state-of-the-art algorithms under various practical conditions, e.g., inaccurate geometric and photometric registration or invalid measurements.

Multi-sensor super-resolution for hybrid range imaging with application to 3-D endoscopy and open surgery

In this paper, we propose a multi-sensor super-resolution framework for hybrid imaging to super-resolve data from one modality by taking advantage of additional guidance images of a complementary modality. This concept is applied to hybrid 3-D range imaging in image-guided surgery, where high-quality photometric data is exploited to enhance range images of low spatial resolution. We formulate super-resolution based on the maximum a-posteriori (MAP) principle and reconstruct high-resolution range data from multiple low-resolution frames and complementary photometric information. Robust motion estimation as required for super-resolution is performed on photometric data to derive displacement fields of subpixel accuracy for the associated range images. For improved reconstruction of depth discontinuities, a novel adaptive regularizer exploiting correlations between both modalities is embedded to MAP estimation. We evaluated our method on synthetic data as well as ex-vivo images in open surgery and endoscopy. The proposed multi-sensor framework improves the peak signal-to-noise ratio by 2 dB and structural similarity by 0.03 on average compared to conventional single-sensor approaches. In ex-vivo experiments on porcine organs, our method achieves substantial improvements in terms of depth discontinuity reconstruction.

Exploring super-resolution implementations across multiple platforms

The performance of many applications, such as video streaming, webcam conferencing, and aerial surveillance, all greatly depend on video quality. A major issue with higher quality video is that either more data bandwidth or storage resources must be dedicated for transferring or storing the video. However, if the low-resolution video is transferred or stored in order to conserve data bandwidth and storage space, super-resolution is a viable solution that can be applied afterwards on the receiving end to rectify the poor quality of the low-resolution video. Super-resolution is an imaging technique that leverages motion blur and multiple low-resolution frames to construct a high-resolution frame. In our paper, we implement and analyze a super-resolution algorithm across multiple platforms ranging from purely hardware to purely software and even a mix of both hardware and software. More specifically, we examine the performance for a field-programmable gate array (FPGA) implementation on two different FPGAs, a software/hardware solution on a FPGA with a soft core processor, a general purpose graphics processing unit (GPGPU) implementation, and a MATLAB implementation. Overall, we found that the GPGPU provides the best overall performance with up to 29 FPS with 35 iterations of the super-resolution algorithm. A high-performance FPGA can have comparable performance and rival the GPGPU in some cases. One of the interesting results was that the hardware/software FPGA combination performed worse than the pure software implementation.

Robust super resolution of compressed video

This paper presents a robust algorithm to recover high-frequency information from compressed low-resolution (LR) video sequences. Previous super-resolution (SR) approaches have succeeded in resolution enhancement when the motion in the LR sequence is simple. However, when the motion is complex, new artifacts will be introduced in the SR processing. To solve this problem, we develop a robust Bayesian SR algorithm with two steps. We first isolate the frames individually to get their corresponding initial SR solutions within the Bayesian framework. Secondly, with a robust cost function to reject outliers and noise, final SR images are achieved with multiple LR frames. In the mean time, we impose the constraint that the distribution of high-resolution (HR) image gradient should be equal to one of the corresponding decompressed LR images to sharpen the edges of the results. As a result of these steps, we are able to produce high-quality deblurred results, which show a suppressing of high-frequency artifacts and less ringing artifacts, with a higher peak signal-to-noise ratio (PSNR).

Improving image resolution by adaptive back-projection correction techniques

The paper describes a back-projection based algorithm, improved by adaptive techniques, able to reconstruct a high-resolution image from multiple low-resolution frames. The proposed approach is mainly based on a specific metric, the uncertainty degree, used in pixel reconstruction and on a local auto-iterative approach for error correction. Using uncertainty, the amount of computation is sensibly reduced. The auto-iterative technique reduces drastically the number of iterations needed to decrease approximation errors. This method is suitable to be applied when real-time processing is required. Experimental results are presented both in terms of perceived and measured quality.