Video Super-resolution Reconstruction Research Articles

Vibration displacement measurement of bridge structural models using image super-resolution reconstruction and visual object detection network

Abstract Visual vibration measurement has emerged in the field of structural health monitoring in recent years, but it still has some shortcomings in terms of resolution, recognition rate and real-time performance. Considering the three aspects of recovering high-frequency image details, improving the compactness of the target bounding box, and reducing the computational time, we use the constructed image super-resolution reconstruction model and target detection model to measure the vibration displacement of the bridge structural model. First, we integrate the Transformer module into the Unet network with a simple structure. The Swin and Global Transformer Unet (SGTU) module constructed in this form can reduce the computational cost while reconstructing the large-resolution feature map target, and it can sharply edge information of the vibration target. We use the framework of the YOLOv5 algorithm as the backbone, and use the GhostBottleneck (GB) module to reduce the time for convolution operations to generate similar features. In addition, the proposed DWCBottleneck (DWCB) fusion module is also able to achieve high-level semantic fusion and network depth expansion with minimal computational cost. Finally, the center point offset of the bounding box predicted by the model can be used to obtain the displacement offset of the object in the image sequence. The position information of the target in the first frame image is used as the reference frame for calculating the offset, and the vibration displacement of the flexible structure in the image coordinate system is obtained by calculating the deviation of the displacement between the remaining frames and the first frame. We perform qualitative and quantitative comparisons in three aspects: video super-resolution reconstruction, visual detection robustness, and sensor vibration measurement displacement using a homemade vibration image dataset. The time-frequency domain displacement curves regressed by the visual vibration measurement algorithm are compared with the curves acquired after accelerometer acquisition, indicating the necessity of super-resolution reconstruction in visual vibration measurement.

A proposed video super-resolution reconstruction strategy using wavelet multi-scale convolutional neural networks

A proposed video super-resolution reconstruction strategy using wavelet multi-scale convolutional neural networks

Video Super-Resolution Network with Gated High-Low Resolution Frames

In scenes with large inter-frame motion variations, distant targets, and blurred targets, the lack of inter-frame alignment can greatly affect the effectiveness of subsequent video super-resolution reconstruction. How to perform inter-frame alignment in such scenes is the key to super-resolution reconstruction. In this paper, a new motion compensation method is proposed to design an alignment network based on gated high-low resolution frames. The core idea is to introduce a gating mechanism while using the information of high-low resolution neighboring frames to perform motion compensation adaptively. Meanwhile, within this alignment framework, we further introduce a pre-initial hidden state network and a local scale hierarchical salient feature fusion network. The pre-initial hidden state network is mainly used to reduce the impact of unbalanced quality effects between frames that occur in one-way cyclical networks; the local scale hierarchical salient feature fusion network is used to fuse the features of aligned video frames to extract contextual information and locally salient features to improve the reconstruction quality of the video. Compared with existing video super-resolution methods, this method achieves good performance and clearer edge and texture details.

Dual feature enhanced video super-resolution network based on low-light scenarios

Video super-resolution (VSR) reconstruction technique aims to improve the spatiotemporal resolution of consecutive low resolution (LR) frames. Most of previous methods use the correlation of inter-frame to restore pixels. It is still a challenge to exploit intra-frame and inter-frame correlations to recover high-resolution (HR) frames, especially reconstruction of video in low-light conditions for sharp imaging results has been a bottleneck in current industrial environments. Therefore, this paper proposes a novel VSR network. Specifically, we first generate the hidden information of the current frame by using the appropriate combination of the front and rear frames. Then, a re-fusion block (RFB) is designed, which utilizes the hidden information to re-fuse with corresponding LR frame. After that, we integrate an improved dual attention mechanism (DAM) into network to extract more accurate feature of intra-frame without increasing the number of parameters. The technology has important practical significance in low-light and dim scenes, so we collect some industrial video sequences and make datasets to complete VSR task. Experimental results show that our model significantly outperforms state-of-the-art methods in performance.

A Multiscale Aligned Video Super-Resolution Network for Improving Vibration Signal Measurement Accuracy

The quality of sequence frames is the premise of obtaining accurate vibration curves by visual vibration measurement method. Existing methods for video super-resolution reconstruction (VSR) have large models and slow inference speeds, which are not suitable for practical industrial applications. This paper proposes a video super-resolution network framework based on the pyramid and deformable convolution to measure the three key indicators of algorithm accuracy, speed, and model size. Specifically, we designed a pyramid alignment module (PAM) that includes a deformable self-attention module (DSA) to focus on the motion estimation of the region of interest while suppressing unrealistic motion estimation. In addition, the two-way propagation method combined with the PAM can be used to ensure that all frames in the input sequence can be balanced to enjoy the information of the full sequence of frames when aligned. The experimental results show that the proposed method improves the PSNR by 0.58dB compared with the existing reconstruction algorithm on the high-speed rotor image data set produced in this paper, and reduces the reasoning time by 41.5% and the model size by 9.1%. Finally, the rotor vibration displacement analysis experiment shows that the vibration signal obtained from the rotor data enhanced by this algorithm has better periodicity and stability. Our code will be available at https://github.com/candygogogogo/Rotor-Sequence-Image-Enhancement.

Video Super-Resolution with Regional Focus for Recurrent Network

Video super-resolution reconstruction is the process of reconstructing low-resolution video frames into high-resolution video frames. Most of the current methods use motion estimation and motion compensation to extract temporal series information, but the inaccuracy of motion estimation will lead to the degradation of the quality of video super-resolution results. Additionally, when using convolution network to extract feature information, the number of feature information is limited by the number of feature channels, resulting in poor reconstruction results. In this paper, we propose a recurrent structure of regional focus network for video super-resolution, which can avoid the influence of inaccurate motion compensation on super-resolution results. Meanwhile, regional focus blocks in the network can focus on different areas of video frames, extract different features from shallow to deep layers, and skip-connect to the last layer of the network through feature aggregation to improve the richness of features participating in the reconstruction. The experimental results show that our method has higher computational efficiency and better video super-resolution results than other temporal modeling methods.

The influence of digital twins on the methods of film and television creation

All walks of life in the world have entered the "big data era", and China's film and television industry is also developing rapidly. Digital twins can also have a huge impact on how film and television are created. This paper aims to study the theoretical knowledge of digital twins and film and television creation methods, and propose an image and video super-resolution reconstruction algorithm based on video computing and reconstruction. Both algorithms make perfect sense in digital twins. This paper proposes how to study the impact of digital twins on the way of film and television creation. It can be seen from the data in Table 1 of the experiment in this paper that the director's attention to the way of film and television creation has increased from 54% in 2013 to 70% in 2019. It can be seen that the decoding quality of the two reconstruction algorithms is higher than that of the traditional reconstruction algorithm. It can be seen that it is very meaningful to explore the impact of digital twins on the way of film and television creation.Digital twin is a multi-disciplinary, multi-physical, multi-scale, multi-probability simulation process that makes full use of physical models, sensor updates, and operating history to complete the mapping in virtual space, thereby reflecting the full life cycle of the corresponding physical equipment. process. A digital twin is a concept beyond reality that can be viewed as a digital mapping system of one or more important, interdependent equipment systems.

Three-Dimensional Video Super-Resolution Reconstruction Scheme Based on Histogram Matching and Recursive Bayesian Algorithms

Multimedia Super-Resolution (SR) reconstruction is an essential and mandatory process for different visualization functions. Recently, several schemes have been suggested for single- and multi-image SR reconstruction. This work presents an effective SR reconstruction scheme for visual quality and resolution enhancement of 3D Video (3DV) sequences. The idea behind the proposed 3DV SR reconstruction scheme is the utilization of a recursive Bayesian algorithm for improving the resolution of the degraded 3DV sequences with down-sampling, blurring, and noise effects. In addition, a significant stage of histogram matching based on a visual image with a better-distributed histogram is employed. The main aim of employing the histogram matching stage for enhancing the 3DV sequence is to introduce a dynamic range modification of each 3DV frame. Hence, it presents a 3DV sequence with an enhanced distribution of intensities. This improves the whole performance efficiency of the suggested scheme. The performance of the proposed SR reconstruction scheme is compared with that of the conventional bicubic interpolation scheme. Comparisons with recent and related SR reconstruction schemes are also introduced. Simulation results reveal that the proposed scheme achieves superior outcomes in terms of Structural Similarity (SSIM) index, local contrast, average gradient, Mean Square Error (MSE), edge intensity, entropy, and Peak Signal-to-Noise Ratio (PSNR) of the resulting 3DV frames.

Real-time video super resolution network using recurrent multi-branch dilated convolutions

Recent developments of video super-resolution reconstruction often exploit spatial and temporal contexts from input frame sequence by making use of explicit motion estimation, e.g., optical flow, which may introduce accumulated errors and requires huge computations to obtain an accurate estimation. In this paper, we propose a novel multi-branch dilated convolution module for real-time frame alignment without explicit motion estimation, which is incorporated with the depthwise separable up-sampling module to formulate a sophisticated real-time video super-resolution network. Specifically, the proposed video super-resolution framework can efficiently acquire a larger receptive field and learn spatial–temporal features of multiple scales with minimal computational operations and memory requirements. Extensive experiments show that the proposed super-resolution network outperforms current state-of-the-art real-time video super-resolution networks, e.g., VESPCN and 3DVSRnet, in terms of PSNR values (0.49 dB and 0.17 dB) on average in various datasets, but requires less multiplication operations.

Video super-resolution reconstruction method based on deep Back projection and motion feature fusion

How to effectively utilize inter-frame redundancies is the key to improve the accuracy and speed of video super-resolution reconstruction methods. Previous methods usually process every frame in the whole video in the same way, and do not make full use of redundant information between frames, resulting in low accuracy or long reconstruction time. In this paper, we propose the idea of reconstructing key frames and non-key frames respectively, and give a video super-resolution reconstruction method based on deep back projection and motion feature fusion. Key-frame reconstruction subnet can obtain key frame features and reconstruction results with high accuracy. For non-key frames, key frame features can be reused by fusing them and motion features, so as to obtain accurate non-key frame features and reconstruction results quickly. Experiments on several public datasets show that the proposed method performs better than the state-of-the-art methods, and has good robustness.

Super Resolution Reconstruction Algorithm of UAV Image Based on Residual Neural Network

With the development of convolutional neural network, video super-resolution algorithm has achieved remarkable success. Because the dependence between frames is complex, traditional methods lack the ability to model the complex dependence, and it is difficult to estimate and compensate the motion accurately in the process of video super-resolution reconstruction. Therefore, a reconstruction network based on optical flow residuals is proposed. In low resolution space, the dense residual network is used to obtain the complementary information of adjacent video frames, and then the optical flow of high-resolution video frames is predicted through the pyramid structure, and then the low resolution video frames are transformed into high-resolution video frames through the sub-pixel convolution layer, The high-resolution video frame is compensated with the predicted high-resolution optical flow. Finally, it is input into the super-resolution fusion network to get better effect. A new loss function training network is proposed to better constrain the network. Experimental results on public data sets show that the reconstruction effect is improved in PSNR, structural similarity and subjective visual effect.

Video Super-Resolution Reconstruction Based on Deep Learning and Spatio-Temporal Feature Self-Similarity

To address the problems in the existing video super-resolution methods, such as noise, over smooth and visual artifacts, which are caused by the reliance on limited external training or mismatch of internal similarity patch instances, this study proposes a novel video super-resolution reconstruction algorithm based on deep learning and spatio-temporal feature similarity (DLSS-VSR). The video super-resolution reconstruction mechanism with the joint internal and external constraints is established utilizing the complementary advantages of both external deep correlation mapping learning and internal spatio-temporal nonlocal self-similarity prior constraint. A deep learning model based on deep convolutional neural network is constructed to learn the nonlinear correlation mapping between low-resolution and high-resolution video frame patches. A novel spatio-temporal feature similarity calculation method is proposed, which considers both internal video spatio-temporal self-similarity and external clean nonlocal similarity. For the internal spatio-temporal feature self-similarity, we improve the accuracy and robustness of similarity matching by proposing a similarity measure strategy based on spatio-temporal moment feature similarity and structural similarity. The external nonlocal similarity prior constraint is learned by the patch group-based Gaussian mixture model. The time efficiency for spatio-temporal similarity matching is further improved based on saliency detection and region correlation judgment strategy, which achieves a better tradeoff between super-resolution accuracy and speed. Experimental results demonstrate that the DLSS-VSR algorithm achieves competitive super-resolution quality compared to other state-of-the-art algorithms in both subjective and objective evaluations.

Robust online video super-resolution using an efficient alternating projections scheme

Video super-resolution reconstruction (SRR) algorithms attempt to reconstruct high-resolution (HR) video sequences from low-resolution observations. Although recent progress in video SRR has significantly improved the quality of the reconstructed HR sequences, it remains challenging to design SRR algorithms that achieve good quality and robustness at a small computational complexity, being thus suitable for online applications. In this paper, we propose a new adaptive video SRR algorithm that achieves state-of-the-art performance at a very small computational cost. Using a nonlinear cost function constructed considering characteristics of typical innovation outliers in natural image sequences and an edge-preserving regularization strategy, we achieve state-of-the-art reconstructed image quality and robustness. This cost function is optimized using a specific alternating projections strategy over non-convex sets that is able to converge in a very few iterations. An accurate and very efficient approximation for the projection operations is also obtained using tools from multidimensional multirate signal processing. This solves the slow convergence issue of stochastic gradient-based methods while keeping a small computational complexity. Simulation results with both synthetic and real image sequences show that the performance of the proposed algorithm is similar or better than state-of-the-art SRR algorithms, while requiring only a small fraction of their computational cost.

SAR Analysis for Real Video Super- Resolution Improvement

This paper presents a simultaneous autoregressive (SAR) analysis method to describe the unknown signal-to-noise ratio (SNR) and the texture feature of low-quality real video frames when the ground-truth images are not available. The real video images degraded by the factors, such as electronic noise, oversaturated pixels, motion blur, and compression artifacts, often result in poor motion registration estimation, which makes the performance of the existing video super-resolution (VSR) algorithms lower than expected. It is hard to estimate the SNR of the low-quality real frames without any prior knowledge. To solve this problem, we made a connection between SAR hyperparameters and the SNR of real images. The relationship expression of them was given in this paper. Using the proposed method, well-registered low-quality real video frames can be selected to decrease the root mean squared error (RMSE) of motion estimation of video frames for VSR reconstruction improvement. The anomalous low-quality frame images whose SAR hyperparameters values are inconsistent with others will be considered for removal. Synthetic experiments were designed to illustrate how the SAR hyperparameters values vary with the variation of synthetic parameters. In order to better illustrate the effectiveness of the proposed method, real low-quality videos captured under different conditions were tested under VSR reconstruction experiments. The VSR reconstruction results show that the results obtained using SAR prior analysis have sharper edges and fewer ringing artifacts than the original results. It indicates that the proposed method is helpful to obtain better results of motion registration estimation for low-quality real video images.

A New Adaptive Video Super-Resolution Algorithm With Improved Robustness to Innovations.

In this paper, a new video super-resolution reconstruction (SRR) method with improved robustness to outliers is proposed. Although the regularized least mean squares (R-LMSs) are one of the SRR algorithms with the best reconstruction quality for its computational cost, and is naturally robust to registration inaccuracies, its performance is known to degrade severely in the presence of innovation outliers. By studying the proximal point cost function representation of the R-LMS iterative equation, a better understanding of its performance under different situations is attained. Using statistical properties of typical innovation outliers, a new cost function is then proposed and two new algorithms are derived, which present improved robustness to outliers while maintaining computational costs comparable with that of R-LMS. The Monte-Carlo simulation results illustrate that the proposed method outperforms the traditional and regularized versions of LMS, and is competitive with state-of-the-art SRR methods at a much smaller computational cost.

Global motion based video super-resolution reconstruction using discrete wavelet transform

Different from the existing super-resolution (SR) reconstruction approaches working under either the frequency-domain or the spatial- domain, this paper proposes an improved video SR approach based on both frequency and spatial-domains to improve the spatial resolution and recover the noiseless high-frequency components of the observed noisy low-resolution video sequences with global motion. An iterative planar motion estimation algorithm followed by a structure-adaptive normalised convolution reconstruction method are applied to produce the estimated low-frequency sub-band. The discrete wavelet transform process is employed to decompose the input low-resolution reference frame into four sub-bands, and then the new edge-directed interpolation method is used to interpolate each of the high-frequency sub-bands. The novelty of this algorithm is the introduction and integration of a nonlinear soft thresholding process to filter the estimated high-frequency sub-bands in order to better preserve the edges and remove potential noise. Another novelty of this algorithm is to provide flexibility with various motion levels, noise levels, wavelet functions, and the number of used low-resolution frames. The performance of the proposed method has been tested on three well-known videos. Both visual and quantitative results demonstrate the high performance and improved flexibility of the proposed technique over the conventional interpolation and the state-of-the-art video SR techniques in the wavelet- domain.

Video super-resolution reconstruction based on deep convolutional neural network and spatio-temporal similarity

Existing learning-based super-resolution (SR) reconstruction algorithms are mainly designed for single image, which ignore the spatio-temporal relationship between video frames. Aiming at applying the advantages of learning-based algorithms to video SR field, a novel video SR reconstruction algorithm based on deep convolutional neural network (CNN) and spatio-temporal similarity (STCNN-SR) was proposed in this paper. It is a deep learning method for video SR reconstruction, which considers not only the mapping relationship among associated low-resolution (LR) and high-resolution (HR) image blocks, but also the spatio-temporal non-local complementary and redundant information between adjacent low-resolution video frames. The reconstruction speed can be improved obviously with the pre-trained end-to-end reconstructed coefficients. Moreover, the performance of video SR will be further improved by the optimization process with spatio-temporal similarity. Experimental results demonstrated that the proposed algorithm achieves a competitive SR quality on both subjective and objective evaluations, when compared to other state-of-the-art algorithms.

Video super-resolution reconstruction based on correlation learning and spatio-temporal nonlocal similarity

A novel video super-resolution reconstruction algorithm based on correlation learning and spatio-temporal nonlocal similarity is proposed in this paper. Objective high-resolution (HR) estimates of low-resolution (LR) video frames can be obtained by learning LR-HR correlation mapping and fusing the spatio-temporal nonlocal similarity information between video frames. First, the LR-HR correlation mapping between LR and HR patches is established based on semi-coupled dictionary learning. With the aim of improving algorithm efficiency while guaranteeing super-resolution quality, LR-HR correlation mapping is performed only for the salient object region, and then an improved visual saliency-based nonlocal fuzzy registration scheme using the pseudo-Zernike moment feature and structural similarity is proposed for spatio-temporal similarity matching and fusion. Visual saliency and self-adaptive regional correlation evaluation strategies are used in spatio-temporal similarity matching to improve algorithm efficiency further. Experimental results demonstrate that the proposed algorithm achieves competitive super-resolution quality compared to other state-of-the-art algorithms in terms of both subjective and objective evaluations.

Learning-Based Video Superresolution Reconstruction Using Spatiotemporal Nonlocal Similarity

Aiming at improving the video visual resolution quality and details clarity, a novel learning-based video superresolution reconstruction algorithm using spatiotemporal nonlocal similarity is proposed in this paper. Objective high-resolution (HR) estimations of low-resolution (LR) video frames can be obtained by learning LR-HR correlation mapping and fusing spatiotemporal nonlocal similarities between video frames. With the objective of improving algorithm efficiency while guaranteeing superresolution quality, a novel visual saliency-based LR-HR correlation mapping strategy between LR and HR patches is proposed based on semicoupled dictionary learning. Moreover, aiming at improving performance and efficiency of spatiotemporal similarity matching and fusion, an improved spatiotemporal nonlocal fuzzy registration scheme is established using the similarity weighting strategy based on pseudo-Zernike moment feature similarity and structural similarity, and the self-adaptive regional correlation evaluation strategy. The proposed spatiotemporal fuzzy registration scheme does not rely on accurate estimation of subpixel motion, and therefore it can be adapted to complex motion patterns and is robust to noise and rotation. Experimental results demonstrate that the proposed algorithm achieves competitive superresolution quality compared to other state-of-the-art algorithms in terms of both subjective and objective evaluations.

Superresolution Reconstruction of Video Sequence Using a Coarse-to-Fine Registration and Optimal Interpolation Strategy

Abstract In this paper, a novel video superresolution (SR) reconstruction approach using a coarse-to-fine registration and optimal interpolation strategy is proposed. Firstly, initial correspondence between two consecutive frames is identified by scale invariant feature transform (SIFT) features under the homography transformation model. Then, a least square matching (LSM) method is employed to achieve a high-precision estimation of homography. Finally, optimal interpolation using the iterative back projection-based deblurring method was designed to obtain a high-resolution grid interpolation and to improve the visual quality of the final SR results. The experiment results demonstrate that the proposed method provides an available SR approach in engineering applications.