Low-resolution Frames Research Articles

Improved automated parallel implementation of GMM background subtraction on a multicore digital signal processor

<p><span>Scene segmentation is an essential step in a wide range of video processing applications, for instance, object recognition and tracking. The Gaussian mixture model (GMM) for background subtraction (BS) has gained widespread usage in scene segmentation, despite its known computational intensity. To tackle this challenge, we propose a practical solution to accelerate processing through a parallel implementation on an embedded multicore platform. In this paper, we present an improved automated parallel implementation of the GMM algorithm using the Orphan directive provided by open multiprocessing (OpenMP). Experimental assessments conducted on the eight cores of the C6678 digital signal processor (DSP) demonstrate significant advancements in parallel efficiency, particularly when handling high-resolution frames, including high-definition (HD) and full-HD resolutions. The achieved parallel efficiency surpasses the results obtained with classical OpenMP scheduling modes, encompassing dynamic, static, and guided approaches. Specifically, the parallel efficiency reaches approximately 82% for full-HD resolution frames and, 99.3% for low-resolution frames, respectively.</span></p>

STDAN: Deformable Attention Network for Space-Time Video Super-Resolution.

The target of space-time video super-resolution (STVSR) is to increase the spatial-temporal resolution of low-resolution (LR) and low-frame-rate (LFR) videos. Recent approaches based on deep learning have made significant improvements, but most of them only use two adjacent frames, that is, short-term features, to synthesize the missing frame embedding, which cannot fully explore the information flow of consecutive input LR frames. In addition, existing STVSR models hardly exploit the temporal contexts explicitly to assist high-resolution (HR) frame reconstruction. To address these issues, in this article, we propose a deformable attention network called STDAN for STVSR. First, we devise a long short-term feature interpolation (LSTFI) module that is capable of excavating abundant content from more neighboring input frames for the interpolation process through a bidirectional recurrent neural network (RNN) structure. Second, we put forward a spatial-temporal deformable feature aggregation (STDFA) module, in which spatial and temporal contexts in dynamic video frames are adaptively captured and aggregated to enhance SR reconstruction. Experimental results on several datasets demonstrate that our approach outperforms state-of-the-art STVSR methods. The code is available at https://github.com/littlewhitesea/STDAN.

Looking beyond input frames: Self-supervised adaptation for video super-resolution

Recent test-time adaptive video super-resolution (VSR) methods have elevated the performance by exploiting the self-similar patches within the low-resolution (LR) frames to adapt to given input frames. However, the LR frames contain a limited amount of such patches, limiting the performance of such adaptation methods, especially for a challenging scale factor (e.g., ×4). In this work, we propose to explore beyond the input LR frames. In particular, we observe that a greater amount of self-similar patches across various scales can be found from estimated high-resolution (HR) frames (i.e., initially restored frames) produced by a pre-trained VSR network. Upon the observation, we propose a new self-supervision test-time adaptation approach via self-distillation to exploit such rich amount of self-similar patches from initially restored frames. Specifically, we perform self-distillation by exploiting multi-scale relationship: distilling knowledge from larger patches to smaller ones with similar semantics. Our framework is flexible and effective as the knowledge can be distilled either from the network itself or the larger one. Furthermore, our framework demonstrates the robustness, being able to recover from undesirable artifacts present in initially restored frames. Extensive evaluation with various VSR networks on numerous datasets reveals that our algorithm consistently improves the restoration quality by a large margin without ground-truth HR video frames. Code is available at: https://github.com/jinsuyoo/bissa.

Omniscient Video Super-Resolution with Explicit-Implicit Alignment

When considering the temporal relationships, most previous video super-resolution (VSR) methods follow the iterative or recurrent framework. The iterative framework adopts neighboring low-resolution (LR) frames from a sliding window, while the recurrent framework utilizes the output generated in the previous SR procedure. The hybrid framework combines them but still cannot fully leverage the temporal relationships. Meanwhile, the existing methods are limited in the receptive field of the optical flow or lack semantic constrains on motion information. In this work, we propose an omniscient framework to fully explore the temporal relationships in the video, which encompasses both LR frames and SR outputs from the past, present, and future. The omniscient framework is more generic because the iterative, recurrent, and hybrid frameworks can be regarded as its special cases. Besides, when addressing the motion information, most previous VSR methods adopt the explicit motion estimation and compensation, while many recent methods turn to implicit alignment. In implicit alignment methods, because basic non-local means suffers from heavy computational costs, we improve it by capturing the non-local correlations in a relatively local manner to reduce the complexity. Moreover, we integrate the explicit and implicit methods into an explicit-implicit alignment module to better utilize motion information. We have conducted extensive experiments on public datasets, which show that our method is superior over the state-of-the-art methods in objective metrics, subjective visual quality, and complexity. In particular, on datasets of Vid4 and UDM10, our method improves PSNR by 0.19 dB, 0.49 dB against the most advanced method BasicVSR++, respectively.

Channel Attention-Based Approach with Autoencoder Network for Human Action Recognition in Low-Resolution Frames

Action recognition (AR) has many applications, including surveillance, health/disabilities care, man-machine interactions, video-content-based monitoring, and activity recognition. Because human action videos contain a large number of frames, implemented models must minimize computation by reducing the number, size, and resolution of frames. We propose an improved method for detecting human actions in low-size and low-resolution videos by employing convolutional neural networks (CNNs) with channel attention mechanisms (CAMs) and autoencoders (AEs). By enhancing blocks with more representative features, convolutional layers extract discriminating features from various networks. Additionally, we use random sampling of frames before main processing to improve accuracy while employing less data. The goal is to increase performance while overcoming challenges such as overfitting, computational complexity, and uncertainty by utilizing CNN-CAM and AE. Identifying patterns and features associated with selective high-level performance is the next step. To validate the method, low-resolution and low-size video frames were used in the UCF50, UCF101, and HMDB51 datasets. Additionally, the algorithm has relatively minimal computational complexity. Consequently, the proposed method performs satisfactorily compared to other similar methods. It has accuracy estimates of 77.29, 98.87, and 97.16%, respectively, for HMDB51, UCF50, and UCF101 datasets. These results indicate that the method can effectively classify human actions. Furthermore, the proposed method can be used as a processing model for low-resolution and low-size video frames.

An assisted multi-frame approach for super-resolution in hyperspectral images of rock samples

Imaging spectroscopy is decisive for accurately characterizing rock samples for geological and mineral applications, since slight differences in mineral composition can be easily recognized in spectral signatures. Although, instrument limitations prevent high spatial resolution pixels scanned along hundreds of spectral channels, which considerably reduces the data capabilities. Multi frame Super-Resolution (SR) techniques can overcome this issue by retrieving high-quality images using purely computational methods. However, these approaches still have important drawbacks because of inherent uncertainties when estimating the spatial variation between frames, which strongly penalize the spectral information, essential for lithology. In this paper we propose an assisted framework to provide fully controlled motion parameters of the imaging process, skipping the intricate registration task. Essentially, we use a stepping device to successively change the position of rock samples for each new frame acquisition. With accurate knowledge about motion parameters, we were able to overcome the uncertainties regarding spatial registration between frames. Therefore, a reasonable number of equations connecting the low-resolution frames and the super resolved image could be used to solve the optimization problem. Extensive experiments proved that the proposed assisted method achieved best performances (both qualitatively and quantitatively) compared with similar approaches. Moreover, detailed spectral analysis showed almost absolute consistency between the original low-resolution image and the super-resolved result, which is crucial to guarantee reliable mineral characterization.

Dual feature enhanced video super-resolution network based on low-light scenarios

Video super-resolution (VSR) reconstruction technique aims to improve the spatiotemporal resolution of consecutive low resolution (LR) frames. Most of previous methods use the correlation of inter-frame to restore pixels. It is still a challenge to exploit intra-frame and inter-frame correlations to recover high-resolution (HR) frames, especially reconstruction of video in low-light conditions for sharp imaging results has been a bottleneck in current industrial environments. Therefore, this paper proposes a novel VSR network. Specifically, we first generate the hidden information of the current frame by using the appropriate combination of the front and rear frames. Then, a re-fusion block (RFB) is designed, which utilizes the hidden information to re-fuse with corresponding LR frame. After that, we integrate an improved dual attention mechanism (DAM) into network to extract more accurate feature of intra-frame without increasing the number of parameters. The technology has important practical significance in low-light and dim scenes, so we collect some industrial video sequences and make datasets to complete VSR task. Experimental results show that our model significantly outperforms state-of-the-art methods in performance.

A four-aperture super-resolution camera based on adaptive regularization parameter tuning

Multi-aperture imaging can improve spatial resolution by fusing complementary information from multiple low-resolution (LR) frames using super-resolution (SR) algorithms. The existing SR algorithms are offline used due to the lack of fast regularization parameter update solution. The classical general cross validation (GCV) and the L-curve algorithms search for a suitable solution on each updated logarithmic parameter set for a long time. In this work, we proposed an adaptive regularization parameter tuning (ARPT) method. The regularization parameter is updated by a function of the fidelity term, the standard deviation of the noise, the weighted bilateral total variation and its standard deviation. Simulations demonstrate that the PSNR and SSIM of the ARPT algorithm, are 0.27 dB and 0.002 larger than the GCV algorithm on average, and the computation time is reduced by more than four times. A four-aperture SR camera is built by randomly shifting the sensors on a sub-pixel scale. An SR image was obtained by the ARPT method using 20 LR images. The average consuming time is reduced by 6.17 times compared with the GCV algorithm. Experiments indicate that the proposed four-aperture SR camera can improve the spatial resolution, suppress the noise, and realize a high-quality and more efficient reconstruction.

SATVSR: Scenario Adaptive Transformer for Cross Scenarios Video Super-Resolution

Video Super-Resolution (VSR) aims to recover sequences of high-resolution (HR) frames from low-resolution (LR) frames. Previous methods mainly utilize temporally adjacent frames to assist the reconstruction of target frames. However, in the real world, there is a lot of irrelevant information in adjacent frames of videos with fast scene switching, these VSR methods cannot adaptively distinguish and select useful information. In contrast, with a transformer structure suitable for temporal tasks, we devise a novel adaptive scenario video super-resolution method. Specifically, we use optical flow to label the patches in each video frame, only calculate the attention of patches with the same label. Then select the most relevant label among them to supplement the spatial-temporal information of the target frame. This design can directly make the supplementary information come from the same scene as much as possible. We further propose a cross-scale feature aggregation module to better handle the scale variation problem. Compared with other video super-resolution methods, our method not only achieves significant performance gains on single-scene videos but also has better robustness on cross-scene datasets.

A New Model for Person Reidentification Using Deep CNN and Autoencoders

Person re-identification (re-id) is one of the most critical and challenging topics in image processing and artificial intelligence. In general, person re-identification means that a person seen in the field of view of one camera can be found and tracked by other non-overlapped cameras. Low-resolution frames, high occlusion in crowded scene, and few samples for training supervised models make re-id challenging. This paper proposes a new model for person re-identification to overcome the noisy frames and extract robust features from each frame. To this end, a noise-aware system is implemented by training an auto-encoder on artificially damaged frames to overcome noise and occlusion. A model for person re-identification is implemented based on deep convolutional neural networks. Experimental results on two actual databases, CUHK01 and CUHK03, demonstrate that the proposed method performs better than state-of-the-art methods.

Attention-guided video super-resolution with recurrent multi-scale spatial–temporal transformer

Video super-resolution (VSR) aims to recover the high-resolution (HR) contents from the low-resolution (LR) observations relying on compositing the spatial–temporal information in the LR frames. It is crucial to propagate and aggregate spatial–temporal information. Recently, while transformers show impressive performance on high-level vision tasks, few attempts have been made on image restoration, especially on VSR. In addition, previous transformers simultaneously process spatial–temporal information, easily synthesizing confused textures and high computational cost limit its development. Towards this end, we construct a novel bidirectional recurrent VSR architecture. Our model disentangles the task of learning spatial–temporal information into two easier sub-tasks, each sub-task focuses on propagating and aggregating specific information with a multi-scale transformer-based design, which alleviates the difficulty of learning. Additionally, an attention-guided motion compensation module is applied to get rid of the influence of misalignment between frames. Experiments on three widely used benchmark datasets show that, relying on superior feature correlation learning, the proposed network can outperform previous state-of-the-art methods, especially for recovering the fine details.

Group-based bi-directional recurrent wavelet neural network for efficient video super-resolution (VSR)

Video super-resolution (VSR) is an important technology for enhancing the quality of video frames. The recurrent neural network (RNN)-based approach is suitable for sequential data because it can use accumulated temporal information. However, since existing methods only tend to capture slow and symmetrical motion with low frame rate, there are still limitations to restore the missing details for more dynamic motion. Most of the previous methods using spatial information treat different types of the spatial features identically. It leads to lack of obtaining meaningful information and enhancing the fine details. We propose a group-based bi-directional recurrent wavelet neural network (GBR-WNN) to exploit spatio-temporal information effectively. The proposed group-based bi-directional RNN (GBR) framework is built on the well-structured process with the group of pictures (GOP). In a GOP, we resolves the low-resolution (LR) frames from border frames to center target frame. Because super-resolved features in a GOP are cumulative, neighboring features are improved progressively and asymmetrical motion can be dealt with. Also, we propose a temporal wavelet attention (TWA) adopting attention module for both spatial and temporal features simultaneously based on discrete wavelet transform. Experiments show that the proposed scheme achieves superior performance compared with state-of-the-art methods.

HEVC Video Quality Enhancement Using Deep Learning with Super Interpolation and Laplacian Filter

Enhancing video quality is a vital area of research that has piqued the interest of academia and industry alike. The High-Efficiency Video Coding (HEVC) compressed videos suffer from significant quality loss, especially at low bit rates. As a result, the decoder must improve the image quality of HEVC videos. Deep Learning with Super Interpolation based Super Resolution (DLSI-SR) and Laplacian Filter (LF) method is proposed to eliminate artifacts. A Multi-Layered Deep Convolutional Neural Network (ML-DCNN) with LF has been introduced to reduce the artifacts quickly in compressed video and increase the subjective video quality. The quality of Low-Resolution (LR) frames can be effectively enhanced using Super Interpolation and Laplacian filter. The Edge orientation analysis is performed on various external training video frames during the training stage. The LR video frame is up-sampled and interpolated using the Super Interpolation algorithm during the upscaling phase. The interpolated frame is then submitted to edge detection using a Canny Edge detection algorithm for frame smoothing. Using local Laplacian filter, the reconstructed high quality video frame is sharpened with an edge preservation approach. The simulation results of the proposed DLSI-SR method shows higher performance in terms of PSNR and SSIM with less processing time in comparison with the existing approaches. The experimental results show that the proposed approach achieves a maximum PSNR of 42.5 dB (enhanced by 2.56 dB) and produced an SSIM improvement of 0.948 (enhanced by +0.022) as compared to the existing approaches.

A multi-stage spatio-temporal adaptive network for video super-resolution

Video super-resolution aims at restoring the spatial resolution of the reference frame based on consecutive input low-resolution (LR) frames. Existing implicit alignment-based video super-resolution methods commonly utilize convolutional LSTM (ConvLSTM) to handle sequential input frames. However, vanilla ConvLSTM processes input features and hidden states independently in operations and has limited ability to handle the inter-frame temporal redundancy in low-resolution fields. In this paper, we propose a multi-stage spatio-temporal adaptive network (MS-STAN). A spatio-temporal adaptive ConvLSTM (STAC) module is proposed to handle input features in low-resolution fields. The proposed STAC module utilizes the correlation between input features and hidden states in the ConvLSTM unit and modulates the hidden states adaptively conditioned on fused spatio-temporal features. A residual stacked bidirectional (RSB) architecture is further proposed to fully exploit the processing ability of the STAC unit. The proposed STAC and RSB architecture promote the vanilla ConvLSTM’s ability to exploit the inter-frame correlations, thus improving the reconstruction quality. Furthermore, different from existing methods that only aggregate features from the temporal branch once at a specified stage of the network, the proposed network is organized in a multi-stage manner. The corresponding temporal correlation in features at different stages can be fully exploited. Experimental results on Vimeo-90K-T and UMD10 datasets show that the proposed method has comparable performance with current video super-resolution methods. The code is available at https://github.com/yhjoker/MS-STAN.

Video Satellite Imagery Super-Resolution via Model-Based Deep Neural Networks

Video satellite imagery has become a hot research topic in Earth observation due to its ability to capture dynamic information. However, its high temporal resolution comes at the expense of spatial resolution. In recent years, deep learning (DL) based super-resolution (SR) methods have played an essential role to improve the spatial resolution of video satellite images. Instead of fully considering the degradation process, most existing DL-based methods attempt to learn the relationship between low-resolution (LR) satellite video frames and their corresponding high-resolution (HR) ones. In this paper, we propose model-based deep neural networks for video satellite imagery SR (VSSR). The VSSR is composed of three main modules: degradation estimation module, intermediate image generation module, and multi-frame feature fusion module. First, the blur kernel and noise level of LR video frames are flexibly estimated by the degradation estimation module. Second, an intermediate image generation module is proposed to iteratively solve two optimal subproblems and the outputs of this module are intermediate SR frames. Third, a three-dimensional (3D) feature fusion subnetwork is leveraged to fuse the features from multiple video frames. Different from previous video satellite SR methods, the proposed VSSR is a multi-frame-based method that can merge the advantages of both learning-based and model-based methods. Experiments on real-world Jilin-1 and OVS-1 video satellite images have been conducted and the SR results demonstrate that the proposed VSSR achieves superior visual effects and quantitative performance compared with the state-of-the-art methods.

Video super-resolution via mixed spatial-temporal convolution and selective fusion

Video super-resolution aims to recover the high-resolution (HR) contents from the low-resolution (LR) observations relying on compositing the spatial-temporal information in the LR frames. It is crucial to model the spatial-temporal information jointly since the video sequences are three-dimensional spatial-temporal signals. Compared with explicitly estimating motions between the 2D frames, 3D convolutional neural networks (CNNs) have been shown its efficiency and effectiveness for video super-resolution (SR), as a natural way of spatial-temporal data modelling. Though promising, the performance of 3D CNNs is still far from satisfactory. The high computational and memory requirements limit the development of more advanced designs to extract and fuse the information from a larger spatial and temporal scale. We thus propose a Mixed Spatial-Temporal Convolution (MSTC) block that simultaneously extracts the spatial information and the supplemented temporal dependency among frames by jointly applying 2D and 3D convolution. To further fuse the learned features corresponding to different frames, we propose a novel similarity-based selective features strategy, unlike precious methods directly stacking the learned features. Additionally, an attention-based motion compensation module is applied to alleviate the influence of misalignment between frames. Experiments on three widely used benchmark datasets and real-world dataset show that, relying on superior feature extraction and fusion ability, the proposed network can outperform previous state-of-the-art methods, especially for recovering the confusing details.

Audio Matters in Video Super-Resolution by Implicit Semantic Guidance

Video super-resolution (VSR) aims to use multiple consecutive low-resolution frames to recover the corresponding high-resolution frames. However, existing VSR methods only consider videos as image sequences, ignoring another essential timing informationaudio, while in fact, there is a semantic link between audio and vision, and extensive studies have shown that audio can provide supervisory information in visual networks. Meanwhile, the addition of semantic priors has been proven to be effective in super-resolution (SR) tasks, but a pretrained segmentation network is required to obtain semantic segmentation maps. By contrast, audio as the information contained in the video itself can be directly used. Therefore, in this study, we propose a novel and pluggable multiscale audiovisual fusion (MS-AVF) module to enhance VSR performance by exploiting the relevant audio information, which can be regarded as implicit semantic guidance compared with the kind of explicit segmentation priors. Specifically, we first fuse audiovisual features on the semantic feature maps of different granularities of the target frames, and then through a top-down multiscale fusion approach, feedback high-level semantics to the underlying global visual features layer by layer, thereby providing effective audio implicit semantic guidance for VSR. Experimental results show that audio can further improve the VSR effect. Moreover, by visualizing the learned attention mask, the proposed end-to-end model can automatically learn potential audiovisual semantic links, especially improving the accuracy and effectiveness of the SR of sound sources and their surrounding regions.

Video super-resolution using a hierarchical recurrent multireceptive-field integration network

Video super-resolution methods always suffer from being computationally expensive and time-consuming because of sequentially performing the temporal frame alignment and spatial feature fusion. To address the issue, a novel video super-resolution method based on a hierarchical recurrent multireceptive-field integration network is proposed. Specifically, a hierarchical recurrent structure, which can fully model the temporal correlation of sequential low-resolution frames, is proposed. Then, a residual multireceptive-field integration module is introduced to obtain the hidden state and rich context features. Finally, an adaptive fusion strategy is employed to further boost video-reconstruction quality using fewer parameters. Moreover, the proposed method is capable of improving the performance and speed of video super-resolution. Extensive experiments on benchmark datasets demonstrate that the proposed method achieves a better balance between the reconstruction performance and speed than existing video super-resolution methods.

Attention deep and shallow network union HR optical flow compensation for video super-resolution

Video super-resolution (VSR) aims to generate high-resolution (HR) video by exploiting temporal consistency and contextual similarity of low-resolution (LR) video sequences. The key to improving the quality of VSR lies in accurate frame alignment and the feature fusion of adjacent frames. We propose a dual channel attention deep and shallow super-resolution network, which combines with HR optical flow compensation to construct an end-to-end VSR framework HOFADS-VSR (attention deep and shallow VSR network union HR optical flow compensation). HR optical flow calculated by spatiotemporal dependency of consecutive LR frames is used to compensate adjacent frames to implement accurate frame alignment. Deep and shallow channels with attention residual block restore small-scale detail features and large-scale contour features, respectively, and strengthen the rich features of global and local regions through weight adjustment. Extensive experiments have been performed to demonstrate the effectiveness and robustness of HOFADS-VSR. Comparative results on the Vid4, SPMC-12, and Harmonic-8 datasets show that our network not only achieves good performance on peak signal-to-noise ratio and structural similarity index but also the restored structure and texture have excellent fidelity.

Deep Gradient Prior Regularized Robust Video Super-Resolution

This paper proposes a robust multi-frame video super-resolution (SR) scheme to obtain high SR performance under large upscaling factors. Although the reference low-resolution frames can provide complementary information for the high-resolution frame, an effective regularizer is required to rectify the unreliable information from the reference frames. As the high-frequency information is mostly contained in the image gradient field, we propose to learn the gradient-mapping function between the high-resolution (HR) and the low-resolution (LR) image to regularize the fusion of multiple frames. In contrast to the existing spatial-domain networks, we train a deep gradient-mapping network to learn the horizontal and vertical gradients. We found that adding the low-frequency information (mainly from the LR image) to the gradient-learning network can boost the performance of the network. A forward and backward motion field prior is used to regularize the estimation of the motion flow between frames. For robust SR reconstruction, a weighting scheme is proposed to exclude the outlier data. Visual and quantitative evaluations on benchmark datasets demonstrate that our method is superior to many state-of-the-art methods and can recover better details with less artifacts.