Video Super-resolution Research Articles

Super-resolution network with dynamic cleanup and temporal–spatial attention for compressed videos

Video super-resolution methods focus on restoring high-resolution video frames from low-resolution videos with pre-defined degradations, and few consider compression. However, the videos on the Internet are compressed to reduce the massive size of video data. Thus, they also contain visual noises and artifacts due to the lossy compression. In this paper, we propose a video super-resolution model specific for compressed videos to suppress artifacts and restore high-resolution contents. We propose a two-stage super-resolution network for compressed videos that contains three new modules: cleanup and local alignment in the first stage, 3D-convolution temporal–spatial attention in the second stage. Specifically, the cleanup module is designed in stage-1 to suppress the artifacts and noises of low-quality inputs. The subsequent local alignment is designed to aggregate adjacent frames to maintain high-frequency details. The 3D-convolution attention is used in stage-2 to refine the weights of the feature maps. In addition, we propose the new CrfVideos dataset to facilitate fair comparisons, containing HEVC/H.265 compressed videos of different compression levels. Extensive experiments substantiate that our method outperforms state-of-the-art methods on the HEVC/H.265 compressed benchmark with medium to high compression rates by 0.3–0.7 dB in terms of PSNR and exceeds state-of-the-art methods on benchmark Vid4 by 0.2–0.8 dB. The code will be released at https://github.com/cvygkhv/decompression.

Temporal Consistency Learning of Inter-Frames for Video Super-Resolution

Video super-resolution (VSR) is a task that aims to reconstruct high-resolution (HR) frames from the low-resolution (LR) reference frame and multiple neighboring frames. The vital operation is to utilize the relative misaligned frames for the current frame reconstruction and preserve the consistency of the results. Existing methods generally explore information propagation and frame alignment to improve the performance of VSR. However, few studies focus on the temporal consistency of inter-frames. In this paper, we propose a Temporal Consistency learning Network (TCNet) for VSR in an end-to-end manner, to enhance the consistency of the reconstructed videos. A spatio-temporal stability module is designed to learn the self-alignment from inter-frames. Especially, the correlative matching is employed to exploit the spatial dependency from each frame to maintain structural stability. Moreover, a self-attention mechanism is utilized to learn the temporal correspondence to implement an adaptive warping operation for temporal consistency among multi-frames. Besides, a hybrid recurrent architecture is designed to leverage short-term and long-term information. We further present a progressive fusion module to perform a multistage fusion of spatio-temporal features. And the final reconstructed frames are refined by these fused features. Objective and subjective results of various experiments demonstrate that TCNet has superior performance on different benchmark datasets, compared to several state-of-the-art methods.

Knowledge Distillation for Optical Flow-Based Video Super-resolution

Recently, deep learning-based super-resolution (SR) models have been used to improve SR performance by equipping preprocessing networks with baseline SR networks. In particular, in video SR, which creates a high-resolution (HR) image with multiple frames, optical flow extraction is accompanied by a preprocessing process. These preprocessing networks work effectively in terms of quality, but at the cost of increased network parameters, which increase the computational complexity and memory consumption for SR tasks with restricted resources. One well-known approach is the knowledge distillation (KD) method, which can transfer the original model’s knowledge to a lightweight model with less performance degradation. Moreover, KD may improve SR quality with reduced model parameters. In this study, we propose an effective KD method that can effectively reduce the original SR model parameters and even improve network performance. The experimental results demonstrated that our method achieved a better PSNR than the original state-of-the-art SR network despite having fewer parameters.

Video super-resolution via nonlocal deformable alignment and frame recursive progressive fusion network

Video super-resolution (VSR) is a process of high-resolution reconstruction of low-resolution video. To address the problems of the previous VSR methods with poor temporal consistency and unsatisfactory SR results, we proposed a nonlocal deformable alignment and frame recursive progressive fusion (RPF) network combining sliding window and recursive methods, which uses nonlocal operations to align sequential frame features and later applies recursion to temporally model the hidden information and alignment features of the previous moment, thus improving temporal consistency. The RPF unit is used to fully fuse the hidden information with the currently aligned features, acquiring more supporting information to be obtained from adjacent frames, resulting in better SR results. The results were evaluated on the three public VSR datasets of Vid4, udm10, and Vimeo-90K, and the experimental results show that the proposed method can achieve state-of-the-art performance on VSR task.

Revisiting video super-resolution: you only look outstanding frames

Video super-resolution (VSR) has been improved with various deep learning architectures and datasets that mostly contain clean images. However, most real-world videos are compressed and here arises a critical issue: each frame in a video usually varies in quality. We discover an important but simple coding prior that affects the performance of the existing VSR models because the prior can tell us which frame is outstanding than others in terms of quality, namely outstanding-frames. Exploiting the prior, we propose a method that allows you only look outstanding frames (YOLOF) to enhance the existing VSR models as a universal approach, which feeds VSR models the best quality of frames near the reference frame with given distance. Extensive evaluations with various VSR models show that our YOLOF method enhances existing VSR models substantially without harming original architectures.

Advanced Reference-Constrained Image Restoration Algorithm

Image restoration is always a long-term problem in low-level computer vision, which has many practical applications. In this paper, we devote to reconstructing a single image from compressive measurements with the assistance of one or multiple reference images, by using the similarity prior information of them. More specifically, we propose to bring in the concept of Regularization by Denoising, which leverages existing denoising engine to regularize inverse problems, and illustrate how the reference images and this powerful algorithm can be merged into a new highly effective recovery approach, which is named Reference Image Constrained Regularization by Denoising for solving various image restoration tasks. Since the similarity between the reference and the ground truth is not guaranteed, or the degree of the similarity is different, we extend the proposed algorithm by adding adaptive weighting matrix to the difference between the target image and the reference images to further improve the performance. We evaluate experimentally the proposed algorithm in the image deblurring and single image super-resolution problems. Furthermore, we also carry out our method on video super-resolution task, in the case of multiple reference images available. We compare the advanced reference-constrained image restoration algorithm with previous methods about reconstruction accuracy and computation cost. The experimental results show that the proposed method outperforms state-of-the-art algorithms, including Regularization by Denoising, in terms of both the Peak Signal to Noise Ratio and the visual performance.

SATVSR: Scenario Adaptive Transformer for Cross Scenarios Video Super-Resolution

Video Super-Resolution (VSR) aims to recover sequences of high-resolution (HR) frames from low-resolution (LR) frames. Previous methods mainly utilize temporally adjacent frames to assist the reconstruction of target frames. However, in the real world, there is a lot of irrelevant information in adjacent frames of videos with fast scene switching, these VSR methods cannot adaptively distinguish and select useful information. In contrast, with a transformer structure suitable for temporal tasks, we devise a novel adaptive scenario video super-resolution method. Specifically, we use optical flow to label the patches in each video frame, only calculate the attention of patches with the same label. Then select the most relevant label among them to supplement the spatial-temporal information of the target frame. This design can directly make the supplementary information come from the same scene as much as possible. We further propose a cross-scale feature aggregation module to better handle the scale variation problem. Compared with other video super-resolution methods, our method not only achieves significant performance gains on single-scene videos but also has better robustness on cross-scene datasets.

Joint Video Super-Resolution and Frame Interpolation via Permutation Invariance

We propose a joint super resolution (SR) and frame interpolation framework that can perform both spatial and temporal super resolution. We identify performance variation according to permutation of inputs in video super-resolution and video frame interpolation. We postulate that favorable features extracted from multiple frames should be consistent regardless of input order if the features are optimally complementary for respective frames. With this motivation, we propose a permutation invariant deep architecture that makes use of the multi-frame SR principles by virtue of our order (permutation) invariant network. Specifically, given two adjacent frames, our model employs a permutation invariant convolutional neural network module to extract "complementary" feature representations facilitating both the SR and temporal interpolation tasks. We demonstrate the effectiveness of our end-to-end joint method against various combinations of the competing SR and frame interpolation methods on challenging video datasets, and thereby we verify our hypothesis.

Video super-resolution for single-photon LIDAR.

3D time-of-flight (ToF) image sensors are used widely in applications such as self-driving cars, augmented reality (AR), and robotics. When implemented with single-photon avalanche diodes (SPADs), compact, array format sensors can be made that offer accurate depth maps over long distances, without the need for mechanical scanning. However, array sizes tend to be small, leading to low lateral resolution, which combined with low signal-to-background ratio (SBR) levels under high ambient illumination, may lead to difficulties in scene interpretation. In this paper, we use synthetic depth sequences to train a 3D convolutional neural network (CNN) for denoising and upscaling (×4) depth data. Experimental results, based on synthetic as well as real ToF data, are used to demonstrate the effectiveness of the scheme. With GPU acceleration, frames are processed at >30 frames per second, making the approach suitable for low-latency imaging, as required for obstacle avoidance.

Lightweight Video Super-Resolution for Compressed Video

Video compression technology for Ultra-High Definition (UHD) and 8K UHD video has been established and is being widely adopted by major broadcasting companies and video content providers, allowing them to produce high-quality videos that meet the demands of today’s consumers. However, high-resolution video content broadcasting is not an easy problem to be resolved in the near future due to limited resources in network bandwidth and data storage. An alternative solution to overcome the challenges of broadcasting high-resolution video content is to downsample UHD or 8K video at the transmission side using existing infrastructure, and then utilizing Video Super-Resolution (VSR) technology at the receiving end to recover the original quality of the video content. Current deep learning-based methods for Video Super-Resolution (VSR) fail to consider the fact that the delivered video to viewers goes through a compression and decompression process, which can introduce additional distortion and loss of information. Therefore, it is crucial to develop VSR methods that are specifically designed to work with the compression–decompression pipeline. In general, various information in the compressed video is not utilized enough to realize the VSR model. This research proposes a highly efficient VSR network making use of data from decompressed video such as frame type, Group of Pictures (GOP), macroblock type and motion vector. The proposed Convolutional Neural Network (CNN)-based lightweight VSR model is suitable for real-time video services. The performance of the model is extensively evaluated through a series of experiments, demonstrating its effectiveness and applicability in practical scenarios.

Light-weight recurrent network for real-time video super-resolution

Light-weight recurrent network for real-time video super-resolution

CycMuNet+: Cycle-Projected Mutual Learning for Spatial-Temporal Video Super-Resolution.

Spatial-Temporal Video Super-Resolution (ST-VSR) aims to generate high-quality videos with higher resolution (HR) and higher frame rate (HFR). Quite intuitively, pioneering two-stage based methods complete ST-VSR by directly combining two sub-tasks: Spatial Video Super-Resolution (S-VSR) and Temporal Video Super-Resolution (T-VSR) but ignore the reciprocal relations among them. 1) T-VSR to S-VSR: temporal correlations help accurate spatial detail representation; 2) S-VSR to T-VSR: abundant spatial information contributes to the refinement of temporal prediction. To this end, we propose a one-stage based Cycle-projected Mutual learning network (CycMuNet) for ST-VSR, which makes full use of spatial-temporal correlations via the mutual learning between S-VSR and T-VSR. Specifically, we propose to exploit the mutual information among them via iterative up- and down projections, where spatial and temporal features are fully fused and distilled, helping high-quality video reconstruction. In addition, we also show interesting extensions for efficient network design (CycMuNet+), such as parameter sharing and dense connection on projection units and feedback mechanism in CycMuNet. Besides extensive experiments on benchmark datasets, we also compare our proposed CycMuNet (+) with S-VSR and T-VSR tasks, demonstrating that our method significantly outperforms the state-of-the-art methods.

A Comparative Analysis of End-To-End Trainable Motion Compensation based Video Super-Resolution Techniques

Video Super-Resolution (VSR) is gaining massive attention in the digital age across various application domains such as video surveillance and ultra-high-definition displays. Deep learning (DL) has recently made significant progress in research and industry domains. Existing researchers used numerous VSR methods based on deep learning to improve the resolution. VSR is based on inter-frame information utilization. Video frames provide the leverage information of data that modalities record sequentially, i.e.; the object's temporal changes cause inconsistent measurements. Consequently, suitable models and algorithms must be developed to provide artifact-free images. This paper goes through two reconstruction methods incorporating various motion data sources. Initially, it describes an iterative approach that implicitly treats dynamic behavior as uncertainty in the forward model. Further direct reconstruction techniques can be derived for an explicit motion map. This paper compares both explicit and implicit motion compensation-based methods. Convolutional neural networks (CNN) and recurrent, residual network (RRN) based methods from a deep learning environment are analyzed to reconstruct the High-Resolution (HR) frames. This paper shows numerical findings for both methods using simulated data. Observations show that RRN can boost the super-resolution (SR) performance and achieve considerable visual quality after reconstruction; however, performance is limited to small datasets due to shallow network parameters. Extensive experiments show that RRN significantly impacts SR results due to its substantial computational efficiency.

FFFN: Frame-By-Frame Feedback Fusion Network for Video Super-Resolution

Video super-resolution (VSR) is a fundamental and challenging task in computer vision. Many of the existing VSR works focus on how to effectively align neighboring frames to better incorporate temporal information, while little work is devoted to the important subsequent step of inter-frame information fusion, and the existing methods on frame fusion have shortcomings such as not being able to make full use of spatio-temporal information. In this work, we propose a Frame-by-frame Feedback Fusion Network (FFFN) for VSR tasks. By applying the feedback learning mechanism commonly existing in the human cognitive system to the frame fusion stage, FFFN can refine low-level representation of the fused frames with high-level information in a coarse-to-fine manner. Specifically, after the neighboring frames are aligned, we first rearrange them from near to far according to the distance from the reference frame in the temporal space, and then feed them one-by-one into a proposed recurrent structure called Feedback Fusion Module (FFM), which is then able to iteratively generate high-level representation of the fused frames with several Feature Refinement Groups (FRGs) and feedback connections. Finally, we design a Dual-path Residual Reconstruction Module (DRRM) to reconstruct the final high-resolution image. The proposed FFFN comes with a strong frame fusion and reconstruction ability, and extensive experiments on several benchmark data sets show that it achieves favorable performance against state-of-the-art methods.

Learning a Local-Global Alignment Network for Satellite Video Super-Resolution

Satellite video is a novel data source for earth observation, which can be applied in multiple fields for dynamic monitoring. It is always equipped with high temporal resolution at the cost of low spatial resolution of tiny moving objects. Video super-resolution (VSR) is utilized to improve the spatial resolution of satellite video and obtain high spatial-temporal resolution data. However, most existing VSR methods mainly focus on the local inter-frame information during feature alignment, which lack the ability to model long-distance correspondence. In this article, a novel two-branch alignment network with an efficient fusion module is proposed for satellite VSR. Both deformable convolution and transformer-like attention are employed to fully explore the local and global information between frames. Further, a fusion module is proposed to model the residuals between fusion features and compensate them for better fusion. Experiments on Jilin-1 satellite videos demonstrate that the proposed network can achieve comparable results to current state-of-the-art VSR methods with tiny parameters.

AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling.

Pooling layers are essential building blocks of convolutional neural networks (CNNs), to reduce computational overhead and increase the receptive fields of proceeding convolutional operations. Their goal is to produce downsampled volumes that closely resemble the input volume while, ideally, also being computationally and memory efficient. Meeting both these requirements remains a challenge. To this end, we propose an adaptive and exponentially weighted pooling method: adaPool. Our method learns a regional-specific fusion of two sets of pooling kernels that are based on the exponent of the Dice-Sørensen coefficient and the exponential maximum, respectively. AdaPool improves the preservation of detail on a range of tasks including image and video classification and object detection. A key property of adaPool is its bidirectional nature. In contrast to common pooling methods, the learned weights can also be used to upsample activation maps. We term this method adaUnPool. We evaluate adaUnPool on image and video super-resolution and frame interpolation. For benchmarking, we introduce Inter4K, a novel high-quality, high frame-rate video dataset. Our experiments demonstrate that adaPool systematically achieves better results across tasks and backbones, while introducing a minor additional computational and memory overhead.

A Multiscale Aligned Video Super-Resolution Network for Improving Vibration Signal Measurement Accuracy

The quality of sequence frames is the premise of obtaining accurate vibration curves by visual vibration measurement method. Existing methods for video super-resolution reconstruction (VSR) have large models and slow inference speeds, which are not suitable for practical industrial applications. This paper proposes a video super-resolution network framework based on the pyramid and deformable convolution to measure the three key indicators of algorithm accuracy, speed, and model size. Specifically, we designed a pyramid alignment module (PAM) that includes a deformable self-attention module (DSA) to focus on the motion estimation of the region of interest while suppressing unrealistic motion estimation. In addition, the two-way propagation method combined with the PAM can be used to ensure that all frames in the input sequence can be balanced to enjoy the information of the full sequence of frames when aligned. The experimental results show that the proposed method improves the PSNR by 0.58dB compared with the existing reconstruction algorithm on the high-speed rotor image data set produced in this paper, and reduces the reasoning time by 41.5% and the model size by 9.1%. Finally, the rotor vibration displacement analysis experiment shows that the vibration signal obtained from the rotor data enhanced by this algorithm has better periodicity and stability. Our code will be available at https://github.com/candygogogogo/Rotor-Sequence-Image-Enhancement.

Online Video Super-Resolution With Convolutional Kernel Bypass Grafts

Deep learning-based models have achieved remarkable performance in video super-resolution (VSR) in recent years, but most of these models are less applicable to online video applications. These methods solely consider the distortion quality and ignore crucial requirements for online applications, e.g., low latency and low model complexity. In this paper, we focus on online video transmission in which VSR algorithms are required to generate high-resolution video sequences frame by frame in real time. To address such challenges, we propose an extremely low-latency VSR algorithm based on a novel kernel knowledge transfer method, named the convolutional kernel bypass graft (CKBG). First, we design a lightweight network structure that does not require future frames as inputs and saves extra time for caching these frames. Then, our proposed CKBG method enhances this lightweight base model by bypassing the original network with “kernel grafts”, which are extra convolutional kernels containing the prior knowledge of the external pretrained image SR models. During the testing phase, we further accelerate the grafted multibranch network by converting it into a simple single-path structure. The experimental results show that our proposed method can process online video sequences up to 110 FPS with very low model complexity and competitive SR performance.

Blind Superresolution of Satellite Videos by Ghost Module-Based Convolutional Networks

Deep learning (DL)-based video satellite superresolution (SR) methods have recently yielded superior performance over traditional model-based methods by using an end-to-end manner. Existing DL-based methods usually assume that the blur kernels are known and, thus, do not model the blur kernels during restoration. However, this assumption is rarely held for real satellite videos and leads to oversmoothed results. In this article, we propose a Ghost module-based convolution network model for blind SR of satellite videos. The proposed Ghost module-based video SR (GVSR) method, which assumes that the blur kernel is unknown, consists of two main modules, i.e., the preliminary image generation module and the SR results’ reconstruction module. First, the motion information from adjacent video frames and the wrapped images are explored by an optical flow estimation network, the blur kernel is flexibly obtained by a blur kernel estimation network, and the preliminary high-resolution image is generated by feeding both blur kernel and wrapped images. Second, a reconstruction network consisting of three paths with attention-based Ghost (AG) bottlenecks is designed to remove artifacts in the preliminary image and obtain the final high-quality SR results. Experiments conducted on Jilin-1 and OVS-1 satellite videos demonstrate that the qualitative and quantitative performance of our proposed method is superior to current state-of-the-art methods.

CLSA: A Contrastive Learning Framework with Selective Aggregation for Video Rescaling.

Video rescaling has recently drawn extensive attention for its practical applications such as video compression. Compared to video super-resolution, which focuses on upscaling bicubic-downscaled videos, video rescaling methods jointly optimize a downscaler and a upscaler. However, the inevitable loss of information during downscaling makes the upscaling procedure still ill-posed. Furthermore, the network architecture of previous methods mostly relies on convolution to aggregate information within local regions, which cannot effectively capture the relationship between distant locations. To address the above two issues, we propose a unified video rescaling framework by introducing the following designs. First, we propose to regularize the information of the downscaled videos via a contrastive learning framework, where, particularly, hard negative samples for learning are synthesized online. With this auxiliary contrastive learning objective, the downscaler tends to retain more information that benefits the upscaler. Second, we present a selective global aggregation module (SGAM) to efficiently capture long-range redundancy in high-resolution videos, where only a few representative locations are adaptively selected to participate in the computationally-heavy self-attention (SA) operations. SGAM enjoys the efficiency of the sparse modeling scheme while preserving the global modeling capability of SA. We refer to the proposed framework as Contrastive Learning framework with Selective Aggregation (CLSA) for video rescaling. Comprehensive experimental results show that CLSA outperforms video rescaling and rescaling-based video compression methods on five datasets, achieving state-of-the-art performance.