Video Super-resolution Research Articles

TempDiff: Enhancing Temporal‐awareness in Latent Diffusion for Real‐World Video Super‐Resolution

AbstractLatent diffusion models (LDMs) have demonstrated remarkable success in generative modeling. It is promising to leverage the potential of diffusion priors to enhance performance in image and video tasks. However, applying LDMs to video super‐resolution (VSR) presents significant challenges due to the high demands for realistic details and temporal consistency in generated videos, exacerbated by the inherent stochasticity in the diffusion process. In this work, we propose a novel diffusion‐based framework, Temporal‐awareness Latent Diffusion Model (TempDiff), specifically designed for real‐world video super‐resolution, where degradations are diverse and complex. TempDiff harnesses the powerful generative prior of a pre‐trained diffusion model and enhances temporal awareness through the following mechanisms: 1) Incorporating temporal layers into the denoising U‐Net and VAE‐Decoder, and fine‐tuning these added modules to maintain temporal coherency; 2) Estimating optical flow guidance using a pre‐trained flow net for latent optimization and propagation across video sequences, ensuring overall stability in the generated high‐quality video. Extensive experiments demonstrate that TempDiff achieves compelling results, outperforming state‐of‐the‐art methods on both synthetic and real‐world VSR benchmark datasets. Code will be available at https://github.com/jiangqin567/TempDiff

A lightweight distillation recurrent convolution network on FPGA for real-time video super-resolution

A lightweight distillation recurrent convolution network on FPGA for real-time video super-resolution

Continuous Space-Time Video Super-Resolution with Multi-Stage Motion Information Reorganization

Space-time video super-resolution (ST-VSR) aims to simultaneously expand a given source video to a higher frame rate and resolution. However, most existing schemes either consider fixed intermediate time and scale or fail to exploit long-range temporal information due to model design or inefficient motion estimation and compensation. To address these problems, we propose a continuous ST-VSR method to convert the given video to any frame rate and spatial resolution with Multi- s tage M otion information r eorganization (MsMr). To achieve time-arbitrary interpolation, we propose a forward warping guided frame synthesis module and an optical flow-guided context consistency loss to better approximate extreme motion and preserve similar structures among input and prediction frames. To realize continuous spatial upsampling, we design a memory-friendly cascading depth-to-space module. Meanwhile, with the sophisticated reorganization of optical flow, MsMr realizes more efficient motion estimation and motion compensation, making it possible to propagate information from long-range neighboring frames and achieve better reconstruction quality. Extensive experiments show that the proposed algorithm is flexible and performs better on various datasets than the state-of-the-art methods. The code will be available at https://github.com/hahazh/LD-STVSR .

Vibration displacement measurement of bridge structural models using image super-resolution reconstruction and visual object detection network

Abstract Visual vibration measurement has emerged in the field of structural health monitoring in recent years, but it still has some shortcomings in terms of resolution, recognition rate and real-time performance. Considering the three aspects of recovering high-frequency image details, improving the compactness of the target bounding box, and reducing the computational time, we use the constructed image super-resolution reconstruction model and target detection model to measure the vibration displacement of the bridge structural model. First, we integrate the Transformer module into the Unet network with a simple structure. The Swin and Global Transformer Unet (SGTU) module constructed in this form can reduce the computational cost while reconstructing the large-resolution feature map target, and it can sharply edge information of the vibration target. We use the framework of the YOLOv5 algorithm as the backbone, and use the GhostBottleneck (GB) module to reduce the time for convolution operations to generate similar features. In addition, the proposed DWCBottleneck (DWCB) fusion module is also able to achieve high-level semantic fusion and network depth expansion with minimal computational cost. Finally, the center point offset of the bounding box predicted by the model can be used to obtain the displacement offset of the object in the image sequence. The position information of the target in the first frame image is used as the reference frame for calculating the offset, and the vibration displacement of the flexible structure in the image coordinate system is obtained by calculating the deviation of the displacement between the remaining frames and the first frame. We perform qualitative and quantitative comparisons in three aspects: video super-resolution reconstruction, visual detection robustness, and sensor vibration measurement displacement using a homemade vibration image dataset. The time-frequency domain displacement curves regressed by the visual vibration measurement algorithm are compared with the curves acquired after accelerometer acquisition, indicating the necessity of super-resolution reconstruction in visual vibration measurement.

FDDCC-VSR: a lightweight video super-resolution network based on deformable 3D convolution and cheap convolution

FDDCC-VSR: a lightweight video super-resolution network based on deformable 3D convolution and cheap convolution

SR4KVQA: Video quality assessment database and metric for 4K super-resolution

The quality assessment for 4K super-resolution (SR) videos can be conducive to the optimization of video SR algorithms. To improve the subjective and objective consistency of the SR quality assessment, a 4K video database and a blind metric are proposed in this paper. In the database SR4KVQA, there are 30 4K pristine videos, from which 600 SR 4K distorted videos with mean opinion score (MOS) labels are generated by three classic interpolation methods, six SR algorithms based on the deep neural network (DNN), and two SR algorithms based on the generative adversarial network (GAN). The benchmark experiment of the proposed database indicates that video quality assessment (VQA) of the 4K SR videos is challenging for the existing metrics. Among those metrics, the Video-Swin-Transformer backbone demonstrates tremendous potential in the VQA task. Accordingly, a blind VQA metric based on the Video-Swin-Transformer backbone is established, where the normalized loss function and optimized spatio-temporal sampling strategy are applied. The experiment result manifests that the Pearson linear correlation coefficient (PLCC) and Spearman rank-order correlation coefficient (SROCC) of the proposed metric reach 0.8011 and 0.8275 respectively on the SR4KVQA database, which outperforms or competes with the state-of-the-art VQA metrics. The database and the code proposed in this paper are available in the GitHub repository, https://github.com/AlexReadyNico/SR4KVQA.

Real-world video superresolution enhancement method based on the adaptive down-sampling model

With the 5G and the popularity of high-definition and ultrahigh-definition equipment, people have increasingly higher requirements for the resolution of images or videos. However, the transmission pressure on servers is also gradually increasing. Therefore, superresolution technology has attracted much attention in recent years. Simultaneously, with the further development of deep learning techniques, superresolution research is shifting from the calculation of traditional algorithms to the deep learning method, which exhibits a greatly superior final display. First, the traditional block-matching-3D (BM3D) algorithm is formed as the postprocessing module, which can avoid the uneven edge of GAN network recovery, make the picture appear more authentic, and improve the viewer’s subjective feelings. Next, the adaptive-downsampling model (ADM) is utilized to train models for specific camera styles. The high-resolution (HR) data sequence is subsequently downsampled to a low-resolution (LR) data sequence, enabling the superresolution algorithm to utilize this training set. This method can obtain better results and improve overall performance by 0.1~0.3 dB.

Combining optical flow and Swin Transformer for Space-Time video super-resolution

Space–time video super-resolution is a task that aims to interpolate low frame rate, low resolution videos to high frame rate, high resolution ones. While existing Transformer-based methods have achieved results comparable to convolutional neural networks-based methods, the computational cost of Transformer limits its performance with constrained computational resources. Moreover, Swin Transformer may fail to fully exploit the spatio-temporal information of video frames due to the limitation of window size, impeding its effectiveness in handling large motions. To address these limitations, we propose an end-to-end space–time video super-resolution architecture based on optical flow alignment and Swin Transformer. The alignment module is introduced to extract spatio-temporal information from adjacent frames without significantly increasing the computational burden. Additionally, we design a residual convolution layer to enhance the translational invariance of the features extracted by Swin Transformer and introduces additional nonlinear transformations. Experimental results demonstrate that our proposed method achieves superior performance on various benchmark datasets compared to state-of-the-art methods. In terms of Peak Signal-to-Noise Ratio, our method outperforms the state-of-the-art methods by at least 0.15 dB on Vimeo-Medium dataset.

Recurrent feature supplementation network for video super-resolution

ABSTRACT Efficient aggregation of temporal information is the basis for achieving video super-resolution. Most researchers have employed alignment or propagation to exploit the temporal information of consecutive frames. However, they frequently overlook the centrality of the reference frame in the model reconstruction when using temporal features. Thus, in this paper, we design a novel recurrent feature supplementation network. We divide the temporal information into three parts: surrounding, back propagation and forward propagation, and extract and fuse them separately. A new grouping approach is proposed for extracting features from the reference frame and its surroundings. The backward temporal fusion module and the forward temporal fusion module are designed to aggregate the backward and forward temporal information at a distance. The temporal fusion module is designed to aggregate temporal information from different parts. Moreover, we propose a feature supplementation mechanism to improve the stability of the model. The feature supplement module is devised to improve the utilization of input features and the stability of the model. Experiments demonstrate that our model achieves the state-of-the-art performance.

Space-time video super-resolution via multi-scale feature interpolation and temporal feature fusion

Space-time video super-resolution via multi-scale feature interpolation and temporal feature fusion

STDAN: Deformable Attention Network for Space-Time Video Super-Resolution.

The target of space-time video super-resolution (STVSR) is to increase the spatial-temporal resolution of low-resolution (LR) and low-frame-rate (LFR) videos. Recent approaches based on deep learning have made significant improvements, but most of them only use two adjacent frames, that is, short-term features, to synthesize the missing frame embedding, which cannot fully explore the information flow of consecutive input LR frames. In addition, existing STVSR models hardly exploit the temporal contexts explicitly to assist high-resolution (HR) frame reconstruction. To address these issues, in this article, we propose a deformable attention network called STDAN for STVSR. First, we devise a long short-term feature interpolation (LSTFI) module that is capable of excavating abundant content from more neighboring input frames for the interpolation process through a bidirectional recurrent neural network (RNN) structure. Second, we put forward a spatial-temporal deformable feature aggregation (STDFA) module, in which spatial and temporal contexts in dynamic video frames are adaptively captured and aggregated to enhance SR reconstruction. Experimental results on several datasets demonstrate that our approach outperforms state-of-the-art STVSR methods. The code is available at https://github.com/littlewhitesea/STDAN.

A ‘deep’ review of video super-resolution

Video super-resolution (VSR) is an ill-posed inverse problem where the goal is to obtain high-resolution video content from a low-resolution counterpart. In this survey, we trace the history of video super-resolution techniques beginning with traditional methods, showing the evolution towards techniques that use shallow networks and finally, the recent trends where deep learning algorithms result in state-of-the-art performance. Specifically, we consider 60 neural network-based VSR techniques in addition to 8 traditional VSR techniques. We extensively cover the deep learning-based techniques including the latest models and introduce a novel taxonomy depending on their architecture. We discuss the pros and cons of each category of techniques. We consider the various components of the problem including the choice of loss functions, evaluation criteria and the benchmark datasets used for evaluation. We present a comparison of the existing techniques using common datasets, providing insights into the relative rankings of these methods. We compare the network architectures based on their computation speed and the network complexity. We also discuss the limitations of existing loss functions and the evaluation criteria that are currently used and propose alternate suggestions. Finally, we identify some of the current challenges and provide future research directions towards video super-resolution, thus providing a comprehensive understanding of the problem.

Semantic guidance incremental network for efficiency video super-resolution

AbstractIn video streaming, bandwidth constraints significantly affect client-side video quality. Addressing this, deep neural networks offer a promising avenue for implementing video super-resolution (VSR) at the user end, leveraging advancements in modern hardware, including mobile devices. The principal challenge in VSR is the computational intensity involved in processing temporal/spatial video data. Conventional methods, uniformly processing entire scenes, often result in inefficient resource allocation. This is evident in the over-processing of simpler regions and insufficient attention to complex regions, leading to edge artifacts in merged regions. Our innovative approach employs semantic segmentation and spatial frequency-based categorization to divide each video frame into regions of varying complexity: simple, medium, and complex. These are then processed through an efficient incremental model, optimizing computational resources. A key innovation is the sparse temporal/spatial feature transformation layer, which mitigates edge artifacts and ensures seamless integration of regional features, enhancing the naturalness of the super-resolution outcome. Experimental results demonstrate that our method significantly boosts VSR efficiency while maintaining effectiveness. This marks a notable advancement in streaming video technology, optimizing video quality with reduced computational demands. This approach, featuring semantic segmentation, spatial frequency analysis, and an incremental network structure, represents a substantial improvement over traditional VSR methodologies, addressing the core challenges of efficiency and quality in high-resolution video streaming.

Self-Supervised Deep Blind Video Super-Resolution.

Existing deep learning-based video super-resolution (SR) methods usually depend on the supervised learning approach, where the training data is usually generated by the blurring operation with known or predefined kernels (e.g., Bicubic kernel) followed by a decimation operation. However, this does not hold for real applications as the degradation process is complex and cannot be approximated by these idea cases well. Moreover, obtaining high-resolution (HR) videos and the corresponding low-resolution (LR) ones in real-world scenarios is difficult. To overcome these problems, we propose a self-supervised learning method to solve the blind video SR problem, which simultaneously estimates blur kernels and HR videos from the LR videos. As directly using LR videos as supervision usually leads to trivial solutions, we develop a simple and effective method to generate auxiliary paired data from original LR videos according to the image formation of video SR, so that the networks can be better constrained by the generated paired data for both blur kernel estimation and latent HR video restoration. In addition, we introduce an optical flow estimation module to exploit the information from adjacent frames for HR video restoration. Experiments show that our method performs favorably against state-of-the-art ones on benchmarks and real-world videos.

Bidirectional scale-aware upsampling network for arbitrary-scale video super-resolution

The performance of video super-resolution (VSR) has significantly improved. However, the current methods only focus on a single scale factor, treating the VSR of different scale factors independently and disregarding video super-resolution of arbitrary-scale factors. To address this issue, we propose a model, the Bidirectional Scale-Aware Upsampling Network for Arbitrary-Scale Video Super-Resolution, which eliminates the need for multiple models for various scale factors. We design a Bidirectional Scale-Aware Upsampling module in the proposed model, consisting of a Bidirectional Scale-Aware Module (BSAM) and a Spatial Pyramid Upsampling section. The BSAM extracts feature for various scale factors and allows feature information of different scales to interact bidirectionally. Additionally, we propose a Spatial Pyramid Loss that optimizes the network based on upsampling and maps the results of different scales to a unified spatial set to find the arbitrary-scale factor's loss. Along with this, we introduce an Explicit Feature Pyramid module, which uses Spatial Pyramid Upsampling to learn arbitrary-scale factor details explicitly. Finally, we demonstrate the extensibility of the model through a VSR algorithm integration with the Bidirectional Scale-Aware Upsampling, ensuring high-resolution results of arbitrary-scale factors without affecting the performance. Our comprehensive experiments on public benchmarks show promising results for video super-resolution of arbitrary-scale factors.

MSTG: Multi-Scale Transformer with Gradient for joint spatio-temporal enhancement

Despite the widespread availability of HDR (High Dynamic Range) display devices, the majority of video sources are still stored in SDR (Standard Dynamic Range) format, leading to an urgent need for UHD (Ultra High Definition) video reconstruction. A simple solution is to divide the task into two sub-tasks: video super-resolution (SR) and video Bit-Depth Enhancement (BDE). While some joint enhancement tasks led by SR have been explored in multiple dimensions, the quantitative dimension has often been overlooked. In this paper, we address the first joint task of BDE and SR and propose a succinct network called MSTG. We conduct a systematic analysis to explain how this effect can be achieved and overcome the core challenge arising from distortions exacerbated by the differing optimization directions of BDE and SR. This challenge is attributed to indistinguishable contours and detailed textures. To tackle this, we employ a dual-branch module that leverages both gradient and image information to discern between BDE and SR. Furthermore, we propose a novel model named Cross-scale Transformer (CTF) embedded with Cross-scale Attention (CSA). This model facilitates information aggregation while adaptively utilizing structural similarity features. It jointly extracts, aligns, and fuses frame features at multiple scales, forming the MST module. Experiments demonstrate that the proposed algorithm outperforms the cascaded video enhancement methods by large margins on both synthetic and realistic datasets.

Deep Compressed Video Super-Resolution With Guidance of Coding Priors

Deep Compressed Video Super-Resolution With Guidance of Coding Priors

A Database and Model for the Visual Quality Assessment of Super-Resolution Videos

A Database and Model for the Visual Quality Assessment of Super-Resolution Videos

Looking beyond input frames: Self-supervised adaptation for video super-resolution

Recent test-time adaptive video super-resolution (VSR) methods have elevated the performance by exploiting the self-similar patches within the low-resolution (LR) frames to adapt to given input frames. However, the LR frames contain a limited amount of such patches, limiting the performance of such adaptation methods, especially for a challenging scale factor (e.g., ×4). In this work, we propose to explore beyond the input LR frames. In particular, we observe that a greater amount of self-similar patches across various scales can be found from estimated high-resolution (HR) frames (i.e., initially restored frames) produced by a pre-trained VSR network. Upon the observation, we propose a new self-supervision test-time adaptation approach via self-distillation to exploit such rich amount of self-similar patches from initially restored frames. Specifically, we perform self-distillation by exploiting multi-scale relationship: distilling knowledge from larger patches to smaller ones with similar semantics. Our framework is flexible and effective as the knowledge can be distilled either from the network itself or the larger one. Furthermore, our framework demonstrates the robustness, being able to recover from undesirable artifacts present in initially restored frames. Extensive evaluation with various VSR networks on numerous datasets reveals that our algorithm consistently improves the restoration quality by a large margin without ground-truth HR video frames. Code is available at: https://github.com/jinsuyoo/bissa.

Enhanced Video Super-Resolution Network towards Compressed Data

Video super-resolution (VSR) algorithms aim at recovering a temporally consistent high-resolution (HR) video from its corresponding low-resolution (LR) video sequence. Due to the limited bandwidth during video transmission, most available videos on the internet are compressed. Nevertheless, few existing algorithms consider the compression factor in practical applications. In this paper, we propose an enhanced VSR model towards compressed videos, termed as ECVSR, to simultaneously achieve compression artifacts reduction and SR reconstruction end-to-end. ECVSR contains a motion-excited temporal adaption network (METAN) and a multi-frame SR network (SRNet). The METAN takes decoded LR video frames as input and models inter-frame correlations via bidirectional deformable alignment and motion-excited temporal adaption, where temporal differences are calculated as motion prior to excite the motion-sensitive regions of temporal features. In SRNet, cascaded recurrent multi-scale blocks (RMSB) are employed to learn deep spatio-temporal representations from adapted multi-frame features. Then, we build a reconstruction module for spatio-temporal information integration and HR frame reconstruction, which is followed by a detail refinement module for texture and visual quality enhancement. Extensive experimental results on compressed videos demonstrate the superiority of our method for compressed VSR. Code will be available at https://github.com/lifengcs/ECVSR .