Video Quality Enhancement Research Articles

AI-Powered Video and Image Processing for Video Meetings

The increasing dependence, on meetings has led to a need for advanced technologies that improve communication and teamwork dynamics swiftly evolving video and image processing technologies backed by AI are reshaping the realm of video conferencing by enabling instant enhancements and enhancing interaction quality and usability This research paper delves into how artificial intelligence is being incorporated into video meetings with a focus on practical uses, like real time background blurring and replacement automatic noise reduction facial expression recognition and participant monitoringIn addition, to that we will explore how AI can enhance video quality, compress images and optimize bandwidth. This showcases its ability to decrease delays and enhance video clarity in areas with bandwidth. We will delve into progress in AI algorithms like using learning for recognizing faces utilizing natural language processing, for instant transcription and detecting emotions.

A compressed video quality enhancement algorithm based on CNN and transformer hybrid network

A compressed video quality enhancement algorithm based on CNN and transformer hybrid network

HVASR: Enhancing 360-degree video delivery with viewport-aware super resolution

In recent years, 360-degree videos have gained significant traction due to their capacity to provide immersive experiences. However, the adoption of 360-degree videos substantially escalates bandwidth demands, necessitating approximately four to ten times more bandwidth than traditional video formats do. This presents a considerable challenge in maintaining high-quality videos in environments characterized by limited bandwidth or unstable networks. A trend has emerged where client-side computational power and deep neural networks are employed to enhance video quality while mitigating bandwidth requirements within contemporary video delivery systems. These approaches segment a video into discrete chunks and apply super resolution (SR) models to each segment, streaming low-resolution (LR) chunks alongside their corresponding SR models to the client. Although these methods enhance both video quality and transmission efficiency for conventional videos, they impose greater computational resource demands when applied to 360-degree content, thereby constraining widespread implementation. This paper introduces an innovative method called HVASR for 360-degree videos that leverages viewport information for more precise segmentation and minimizes model training costs as well as bandwidth requirements. Additionally, HVASR incorporates a viewport-aware training strategy that is aimed at further enhancing performance while reducing computational expenses. The experimental results demonstrate that HVASR achieves an average utility increase ranging from 12.46% to 40.89% across various scenes.

Spatio-temporal feature learning for enhancing video quality based on screen content characteristics

With the rising demands for remote desktops and online meetings, screen content videos have drawn significant attention. Different from natural videos, screen content videos often exhibit scene switches where the content abruptly changes from one frame to the next. These scene switches result in obvious distortions in compressed videos. Besides, frame freezing, where the content remains unchanged for a certain duration, is also very common in screen content videos. Existing alignment-based models struggle to effectively enhance scene switch frames and lack efficiency when dealing with frame freezing situations. Therefore, we propose a novel alignment-free method that effectively handles both scene switches and frame freezing. In our approach, we develop a spatial and temporal feature extraction module that compresses and extracts spatio-temporal information from three groups of frame inputs. This enables efficient handling of scene switches. In addition, an edge aware block is proposed for extracting edge information, which guides the model to focus on restoring the high-frequency components in frame freezing situations. The fusion module is then designed to adaptively fuse the features from three groups, considering different positions of video frames, to enhance frames during scene switch and frame freezing scenarios. Experimental results demonstrate the significant advancements achieved by the proposed edge aware with spatio-temporal information fusion network (EAST) in enhancing the quality of compressed videos, surpassing the current state-of-the-art methods.

Fusing steering kernel guided filtering with U-NET ConvLSTM for elevated video quality enhancement

Fusing steering kernel guided filtering with U-NET ConvLSTM for elevated video quality enhancement

Reconstruction flow recurrent network for compressed video quality enhancement

We present a reconstruction flow for the task of compressed video quality enhancement (VQE). Compressed videos often suffer from various coding artifacts, such as blocking and blurring, especially under low bit-rate. VQE aims to suppress these artifacts to improve the visual quality. Frame similarity can be utilized to enhance low-quality frames given their neighboring high-quality frames, for which motion estimation becomes important. Previous approaches often calculate optical flow for the motion compensation. On the other hand, video coding contains a rich set of block motion vectors, forming a coding flow, which may or may not correspond to the scene motion, but to places that deliver the minimum compression error. In contrast, such a valuable coding flow has always been ignored in VQE previously. In this work, we combine these two motion sources into a new flow, namely reconstruction flow, for the purpose of high-quality VQE. Specifically, we estimate optical flows from RGB frames and extract coding flows from coding streams, which are then merged by a fusion module to generate reconstruction flow. Besides, our network is built upon a recurrent network to utilize global temporal information. The deep features are warped according to the reconstruction flow and fed into the subsequent reconstruction module with spatial-variant kernel attention. Our method is evaluated on the leading MFQE2.0 dataset, which demonstrates superior performances when compared to the existing state-of-the-art methods.

Video Processing Based Tracking and Vehicle Identification

Video surveillance and analysis have become integral components of various domains such as security, traffic management, and urban planning. However, effective tracking and identification of vehicles in video streams remain challenging due to environmental factors, occlusions, and complex motion patterns. This research proposes a novel approach leveraging YOLOv5, a state-of-the-art object detection algorithm, for real-time vehicle tracking and identification. By integrating YOLOv5 with advanced video processing techniques, including preprocessing for enhancing video quality and Kalman filtering for object tracking, the proposed system achieves improved accuracy and robustness in diverse scenarios. Experimental results demonstrate the effectiveness of the approach, showcasing high accuracy in vehicle tracking and reliable identification performance. The findings suggest significant potential for practical applications in enhancing video surveillance systems for better security and traffic management. Additionally, avenues for future research are discussed to further enhance the capabilities of video-based vehicle tracking and identification systems

Scalable Video Fidelity Enhancement: Leveraging the state-of-the-art AI Models

Improving visual quality is crucial as we navigate through the vast world of data. State-of-the-art (SOTA) artificial intelligence (AI) models provide highly effective solutions. Driven by the ever-growing demand for high-fidelity multimedia content, this research explores the groundbreaking capabilities of SOTA AI models to revolutionize video quality enhancement. Existing video capture methods often struggle with limitations in hardware, bandwidth, and compression, leading to subpar visual experiences. To address this challenge, we propose a novel Video Quality Enhancement Solution (VQES) that synergistically combines Google FILM for frame interpolation and Real-ESRGAN for image super-resolution. By applying these models to each video frame and integrating scalable post-processing techniques, a comprehensive VQES has been devised. Extensive experiments demonstrate that our VQES outperforms existing methods in terms of peak signal-to-noise ratio (PSNR) improvement and user-perceived visual quality. By advancing video fidelity, this research paves the way for consistently immersive, informative, and enjoyable visual experiences.

STDF: Spatio-Temporal Deformable Fusion for Video Quality Enhancement on Embedded Platforms

With the development of embedded systems and deep learning, it is feasible to combine them for offering various and convenient human-centered services, which is based on high-quality (HQ) videos. However, due to the limit of video traffic load and unavoidable noise, the visual quality of an image from an edge camera may degrade significantly, influencing the overall video and service quality. To maintain video stability, video quality enhancement (QE), aiming at recovering HQ videos from their distorted low-quality (LQ) sources, has aroused increasing attention in recent years. The key challenge for video QE lies in how to effectively aggregate complementary information from multiple frames (i.e., temporal fusion). To handle diverse motion in videos, existing methods commonly apply motion compensation before the temporal fusion. However, the motion field estimated from the distorted LQ video tends to be inaccurate and unreliable, thereby resulting in ineffective fusion and restoration. In addition, motion estimation for consecutive frames is generally conducted in a pairwise manner, which leads to expensive and inefficient computation. In this article, we propose a fast yet effective temporal fusion scheme for video QE by incorporating a novel Spatio-Temporal Deformable Convolution (STDC) to simultaneously compensate motion and aggregate temporal information. Specifically, the proposed temporal fusion scheme takes a target frame along with its adjacent reference frames as input to jointly estimate an offset field to deform the spatio-temporal sampling positions of convolution. As a result, complementary information from multiple frames can be fused within the STDC operation in one forward pass. Extensive experimental results on three benchmark datasets show that our method performs favorably to the state of the art in terms of accuracy and efficiency.

PixRevive: Latent Feature Diffusion Model for Compressed Video Quality Enhancement.

In recent years, the rapid prevalence of high-definition video in Internet of Things (IoT) systems has been directly facilitated by advances in imaging sensor technology. To adapt to limited uplink bandwidth, most media platforms opt to compress videos to bitrate streams for transmission. However, this compression often leads to significant texture loss and artifacts, which severely degrade the Quality of Experience (QoE). We propose a latent feature diffusion model (LFDM) for compressed video quality enhancement, which comprises a compact edge latent feature prior network (ELPN) and a conditional noise prediction network (CNPN). Specifically, we first pre-train ELPNet to construct a latent feature space that captures rich detail information for representing sharpness latent variables. Second, we incorporate these latent variables into the prediction network to iteratively guide the generation direction, thus resolving the problem that the direct application of diffusion models to temporal prediction disrupts inter-frame dependencies, thereby completing the modeling of temporal correlations. Lastly, we innovatively develop a Grouped Domain Fusion module that effectively addresses the challenges of diffusion distortion caused by naive cross-domain information fusion. Comparative experiments on the MFQEv2 benchmark validate our algorithm's superior performance in terms of both objective and subjective metrics. By integrating with codecs and image sensors, our method can provide higher video quality.

End-to-End Transformer for Compressed Video Quality Enhancement

End-to-End Transformer for Compressed Video Quality Enhancement

Video stabilization based on low‐rank constraint and trajectory optimization

AbstractVideo stabilization plays a pivotal role in enhancing video quality by eliminating unwanted jitter in shaky videos. This paper introduces a novel video stabilization algorithm that leverages low‐rank constraint and trajectory optimization to effectively eliminate undesirable motion and generate stabilized videos. In the proposed algorithm, a low‐rank constraint regularization term is incorporated to enhance the smoothness of motion trajectories. Additionally, a predictive path smoothness term is integrated to ensure the consistency of motion across neighbouring frames. To address the problem of excessive cropping resulting from aggressive smoothing, a flexible local window strategy that emphasizes local motion relationships within the trajectories is introduced. The experimental results show that, compared to other excellent video stabilization algorithms, the proposed algorithm improves the stability metric by approximately 2.3%. Furthermore, in the stabilized videos generated by the algorithm, an approximate improvement of 2.18 dB in average image temporal fidelity and a 5.7% increase in average structural similarity between adjacent frames are achieved. The code that implements the proposed method is publicly accessible at https://github.com/CZS0319/VS_Low‐Rank_Constraint_Trajectory_Optimization.

MDEformer: Mixed Difference Equation Inspired Transformer for Compressed Video Quality Enhancement.

Deep learning methods have achieved impressive performance in compressed video quality enhancement tasks. However, these methods rely excessively on practical experience by manually designing the network structure and do not fully exploit the potential of the feature information contained in the video sequences, i.e., not taking full advantage of the multiscale similarity of the compressed artifact information and not seriously considering the impact of the partition boundaries in the compressed video on the overall video quality. In this article, we propose a novel Mixed Difference Equation inspired Transformer (MDEformer) for compressed video quality enhancement, which provides a relatively reliable principle to guide the network design and yields a new insight into the interpretable transformer. Specifically, drawing on the graphical concept of the mixed difference equation (MDE), we utilize multiple cross-layer cross-attention aggregation (CCA) modules to establish long-range dependencies between encoders and decoders of the transformer, where partition boundary smoothing (PBS) modules are inserted as feedforward networks. The CCA module can make full use of the multiscale similarity of compression artifacts to effectively remove compression artifacts, and recover the texture and detail information of the frame. The PBS module leverages the sensitivity of smoothing convolution to partition boundaries to eliminate the impact of partition boundaries on the quality of compressed video and improve its overall quality, while not having too much impacts on non-boundary pixels. Extensive experiments on the MFQE 2.0 dataset demonstrate that the proposed MDEformer can eliminate compression artifacts for improving the quality of the compressed video, and surpasses the state-of-the-arts (SOTAs) in terms of both objective metrics and visual quality.

Sports Video Classification Method Based on Improved Deep Learning

Classifying sports videos is complex due to their dynamic nature. Traditional methods, like optical flow and the Histogram of Oriented Gradient (HOG), are limited by their need for expertise and lack of universality. Deep learning, particularly Convolutional Neural Networks (CNNs), offers more effective feature recognition in sports videos, but standard CNNs struggle with fast-paced or low-resolution sports videos. Our novel neural network model addresses these challenges. It begins by selecting important frames from sports footage and applying a fuzzy noise reduction algorithm to enhance video quality. The model then uses a bifurcated neural network to extract detailed features, leading to a densely connected neural network with a specific activation function for categorizing videos. We tested our model on a High-Definition Sports Video Dataset covering over 20 sports and a low-resolution dataset. Our model outperformed established classifiers like DenseNet, VggNet, Inception v3, and ResNet-50. It achieved high precision (0.9718), accuracy (0.9804), F-score (0.9761), and recall (0.9723) on the high-resolution dataset, and significantly better precision (0.8725) on the low-resolution dataset. Correspondingly, the highest values on the matrix of four traditional models are: precision (0.9690), accuracy (0.9781), F-score (0.9670), recall (0.9681) on the high-resolution dataset, and precision (0.8627) on the low-resolution dataset. This demonstrates our model’s superior performance in sports video classification under various conditions, including rapid motion and low resolution. It marks a significant step forward in sports data analytics and content categorization.

Multi-Swin Transformer based Spatio-Temporal Information Exploration for Compressed Video Quality Enhancement

Multi-Swin Transformer based Spatio-Temporal Information Exploration for Compressed Video Quality Enhancement

Multi-View Video Quality Enhancement Method Based on Multi-Scale Fusion Convolutional Neural Network and Visual Saliency

Multi-View Video Quality Enhancement Method Based on Multi-Scale Fusion Convolutional Neural Network and Visual Saliency

Compressed Video Quality Enhancement With Temporal Group Alignment and Fusion

Compressed Video Quality Enhancement With Temporal Group Alignment and Fusion

Cloud-Based Solutions for Video Streaming and Big Data Testing

In the era of digital transformation, cloud-based solutions have revolutionized the video streaming industry and big data testing methodologies. As consumer demand for high-quality, on-demand video content surges, traditional on-premises infrastructure often falls short in meeting scalability, performance, and cost-efficiency requirements. Cloud-based platforms, with their scalable resources and flexibility, have become essential in addressing these challenges. This paper explores the benefits and implementation strategies of cloud-based solutions for video streaming and big data testing, focusing on how these technologies enhance performance, scalability, and cost-effectiveness Video streaming services require a robust infrastructure capable of handling large volumes of data and delivering seamless user experiences across diverse devices and network conditions. Cloud-based video streaming solutions offer several advantages, including the ability to dynamically scale resources based on user demand, optimize content delivery through content delivery networks (CDNs), and provide a global reach without the need for extensive physical infrastructure. By leveraging cloud services, streaming platforms can reduce latency, enhance video quality, and improve overall user satisfaction. Traditional testing methods often struggle to cope with the complexity and volume of big data. Cloud-based testing platforms provide scalable environments where testers can simulate various data scenarios, execute performance tests, and analyze results with greater efficiency. The cloud's elasticity allows for on-demand provisioning of testing resources, making it easier to handle the dynamic nature of big data and conduct comprehensive testing without investing in expensive hardware.

Methods of video quality-improving

Video content has become integral to our daily lives, but poor video quality can significantly reduce viewers' experience and engagement. Various super-resolution methods are used to correct this, thereby reconstructing high-resolution videos from low-resolution ones. Two main categories of super-resolution methods exist traditional image processing and deep learning-based techniques. Deep learning-based techniques, such as Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), and Recurrent Neural Networks (RNNs), have shown great promise in enhancing video quality. The article discusses multiple adaptations of contemporary deep learning models to enhance video resolution. It also briefly explains the framework's design and implementation aspects. Lastly, the paper presents an overview and comparative analysis of the VSR techniques' efficiency on various benchmark datasets. At the same time, the paper describes potential challenges when choosing training sets; performance metrics, which can be used to compare different algorithms quantitatively. This work does not describe absolutely all existing VSR methods, but it is expected to contribute to the development of recent research in this field and potentially deepen our understanding of deep learning-based VSR methods, as well as stimulate further research in this area. In this work, new solutions for improving the performance of the methods are proposed, in particular, new quality metrics and datasets for model training. Overall, AI-based methods for VSR are becoming increasingly crucial with the rising demand for high-quality video content

Deep Learning-Based Cross-Layer Power Allocation for Downlink Cell-Free Massive Multiple-Input–Multiple-Output Video Communication Systems

We propose a deep learning-based cross-layer power allocation method for asymmetric cell-free massive MIMO video communication systems. The proposed cross-layer approach considers physical layer channel state information (CSI) and the application layer rate distortion (RD) function, and it aims to enhance video quality in terms of peak signal-to-noise ratio (PSNR). Our study develops a decentralized deep neural network (DNN) model to capture intricate system patterns, enabling accurate and efficient power allocation decisions. The proposed cross-layer approach includes unsupervised and hybrid (supervised/unsupervised) learning models. The numerical results show that the hybrid method achieves convergence with just 50% of the iterations required by the unsupervised learning model and that it achieves a 1 dB gain in PSNR over the baseline physical layer scheme.