Video Quality Degradation Research Articles

Versatile Video Coding-Post Processing Feature Fusion: A Post-Processing Convolutional Neural Network with Progressive Feature Fusion for Efficient Video Enhancement

Advanced video codecs such as High Efficiency Video Coding/H.265 (HEVC) and Versatile Video Coding/H.266 (VVC) are vital for streaming high-quality online video content, as they compress and transmit data efficiently. However, these codecs can occasionally degrade video quality by adding undesirable artifacts such as blockiness, blurriness, and ringing, which can detract from the viewer’s experience. To ensure a seamless and engaging video experience, it is essential to remove these artifacts, which improves viewer comfort and engagement. In this paper, we propose a deep feature fusion based convolutional neural network (CNN) architecture (VVC-PPFF) for post-processing approach to further enhance the performance of VVC. The proposed network, VVC-PPFF, harnesses the power of CNNs to enhance decoded frames, significantly improving the coding efficiency of the state-of-the-art VVC video coding standard. By combining deep features from early and later convolution layers, the network learns to extract both low-level and high-level features, resulting in more generalized outputs that adapt to different quantization parameter (QP) values. The proposed VVC-PPFF network achieves outstanding performance, with Bjøntegaard Delta Rate (BD-Rate) improvements of 5.81% and 6.98% for luma components in random access (RA) and low-delay (LD) configurations, respectively, while also boosting peak signal-to-noise ratio (PSNR).

Vision-based Human Activity Recognition Using Local Phase Quantization

Human activity recognition (HAR) has been the most active and interesting area of research in recent years due to its wide range of applications in the field, such as healthcare, security and surveillance, robotics, gaming, entertainment, etc. However, recognizing vision-based human activity is still a challenging as input sequences may have cluttered background, illumination conditions, occlusions, degradation of video quality, blurring, etc. In the literature, several state-of-the-art methods have been trained and tested on different datasets but have yet to perform adequately to a certain extent. Moreover, extracting potential features and combining appropriate methods is one of the most challenging tasks in realistic video. This paper proposes an efficient frequency-based blur-invariance local phase quantization feature extractor and multiclass SVM classifier that overcomes these challenges. The feature is invariant towards camera motion, misfocused optics, movements in the scene, and environmental conditions. The proposed feature vector is then fed to the classifier to recognize human activities. The experiment has conducted on two publicly available datasets, UCF101 and HMDB51, and has achieved 99.79% and 98.67% accuracies, respectively. The approach has also outperformed the existing state-of-the-art approaches in terms of computational cost without compromising the accuracy of HAR.

MagView++: Data Exfiltration via CPU Magnetic Signals Under Video Decoding

Air-gapped networks achieve security by using physical isolation to keep the computers and network from the Internet. However, magnetic covert channels based on CPU utilization have been proposed to help secret data to exfiltrate from the Faraday-cage and the air gap. Despite the success of such covert channels, they suffer from the high risk of being detected by the transmitter computer and the challenge of installing malware into such a computer. In this paper, we propose <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MagView++</monospace> , where sensitive information is embedded in other data such as video and can be transmitted over the internal network. When any computer uses the data such as playing the video, the sensitive information will leak through the magnetic signals. The “separation” of information embedding and leaking, combined with the fact that the data can be exfiltrated from any computer in a distributed manner, overcomes these limitations. We demonstrate that CPU utilization for video decoding can be effectively controlled by changing the video frame type, reducing the quantization parameter, and changing the timestamp of the frame, without video quality degradation. We prototype <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MagView++</monospace> and achieve 8.9 bps throughput with 0.0057 BER when using a smartphone as the receiver, and 59 bps throughput with 0.0025 BER when using a dedicated devices with high sampling rate as the receiver. Experiments under various environments are conducted to show the robustness of <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MagView++</monospace> . Limitations and possible countermeasures are also discussed.

Optimized in-loop filtering in versatile video coding using improved fast guided filter

&lt;p&gt;Devices with varying display capabilities from a common source may face degradation in video quality because of the limitation in transmission bandwidth and storage. The solution to overcome this challenge is to enrich the video quality. For the mentioned purpose, this paper introduces an improved fast guided filter (IFGF) for the contemporary video coding standard H.266/VVC (versatile video coding), a continuation of H.265/HEVC (high efficiency video coding). VVC includes several types of coding techniques to enhance video coding efficiency over existing video coding standards. Despite that, blocking artifacts are still present in the images. Hence, the proposed method focuses on denoising the image and the increase of video quality, which is measured in terms of peak signal-to-noise (PSNR). The objective is achieved by using an IFGF for in-loop filtering in VVC to denoise the reconstructed images. VTM (VVC test model)-17.2 is used to simulate the various video sequences with the proposed filter. This method achieves a 0.67% Bjontegaard delta (BD)-rate reduction in low-delay configuration accompanied by an encoder run time increase of 4%.&lt;/p&gt;

QoE-Based Performance Comparison of AVC, HEVC, and VP9 on Mobile Devices with Additional Influencing Factors

While current video quality assessment research predominantly revolves around resolutions of 4 K and beyond, targeted at ultra high-definition (UHD) displays, effective video quality for mobile video streaming remains primarily within the range of 480 p to 1080 p. In this study, we conducted a comparative analysis of the quality of experience (QoE) for widely implemented video codecs on mobile devices, specifically Advanced Video Coding (AVC), its successor High-Efficiency Video Coding (HEVC), and Google’s VP9. Our choice of 720 p video sequences from a newly developed database, all with identical bitrates, aimed to maintain a manageable subjective assessment duration, capped at 35–40 min. To mimic real-time network conditions, we generated stimuli by streaming original video clips over a controlled emulated setup, subjecting them to eight different packet-loss scenarios. We evaluated the quality and structural similarity of the distorted video clips using objective metrics, including the Video Quality Metric (VQM), Peak Signal-to-Noise Ratio (PSNR), Video Multi-Method Assessment Fusion (VMAF), and Multi-Scale Structural Similarity Index (MS-SSIM). Subsequently, we collected subjective ratings through a custom mobile application developed for Android devices. Our findings revealed that VMAF accurately represented the degradation in video quality compared to other metrics. Moreover, in most cases, HEVC exhibited an advantage over both AVC and VP9 under low packet-loss scenarios. However, it is noteworthy that in our test cases, AVC outperformed HEVC and VP9 in scenarios with high packet loss, based on both subjective and objective assessments. Our observations further indicate that user preferences for the presented content contributed to video quality ratings, emphasizing the importance of additional factors that influence the perceived video quality of end users.

Multi-Modality Deep Restoration of Extremely Compressed Face Videos.

Arguably the most common and salient object in daily video communications is the talking head, as encountered in social media, virtual classrooms, teleconferences, news broadcasting, talk shows, etc. When communication bandwidth is limited by network congestions or cost effectiveness, compression artifacts in talking head videos are inevitable. The resulting video quality degradation is highly visible and objectionable due to high acuity of human visual system to faces. To solve this problem, we develop a multi-modality deep convolutional neural network method for restoring face videos that are aggressively compressed. The main innovation is a new DCNN architecture that incorporates known priors of multiple modalities: the video-synchronized speech signal and semantic elements of the compression code stream, including motion vectors, code partition map and quantization parameters. These priors strongly correlate with the latent video and hence they are able to enhance the capability of deep learning to remove compression artifacts. Ample empirical evidences are presented to validate the superior performance of the proposed DCNN method on face videos over the existing state-of-the-art methods.

Enhancing Media Streaming in Wireless Networks using IFW-CFH Algorithm

One of the major concerns for service providers and application developer are Quality of experience (QoE), where high traffic congestion on the Internet leads to the degradation of video quality. However, the effectiveness of video transmission is minimized due to the network based on packet loss, bandwidth, and delay. Because of bandwidth limitations, the videos transmitted are obtained in low quality. Meanwhile, various outcomes such as reduction in throughput, re-buffering, or mosaic are determined in packet loss which validated the video streaming obtained in reliable or unreliable mode. Therefore this paper proposes an Improved Fuzzy Weighted queueing based Crossover Fire Hawk (IFW-CFH) algorithm for effective real-time video transmission. The objective of the IFW-CFH approach is to reduce the delay, packet loss, and bandwidth to enhance the video quality via two key mechanisms namely congestion control mechanism as well as packet scheduling mechanism. During the generation of encoded video frames, the packaged packets to the local buffer are transmitted by the scheduler using our proposed IFW-CFH algorithm. Finally, the experimentation is conducted and the results show that the proposed method minimized transmission delay, packet loss, and bandwidth by 13.8% for effective real-time video transmission compared to the existing methods.

PRAM: Penalized Resource Allocation Method for Video Services

Abstract The human visual system response to picture quality degradation due to packet loss is very different from the responses of objective quality measures. While video quality due to packet loss may be impaired by at most for one Group of Pictures (GOP), its subjective quality degradation may last for several GOPs. This has a great impact on resource allocation strategies, which normally make decisions on instantaneous conditions of multiplexing buffer. This is because, when the perceptual impact of degraded video quality is much longer than its objective degradation period, any assigned resources to the degraded flow is wasted. This paper, through both simulations and analysis shows that, during resource allocation, if the quality of a video stream is significantly degraded, it is better to penalize this degraded flow from getting its full bandwidth share and instead assign the remaining share to other flows preventing them from undergoing quality degradation.

Reference picture selection with decreased temporal dependency for HEVC error resilience

The high level of compression efficiency achieved by the High-Efficiency Video Coding standard (HEVC) decreases the robustness of the encoded bitstreams. This increased susceptibility to network errors leads to end video quality degradation. Moreover, due to the high computational complexity of HEVC, high-resolution video transmission with time constraints over hostile channels such as wireless networks becomes more challenging. This paper proposes a reference picture selection-based error-resilient method to reduce the temporal error propagation due to high-trip delay and frame-copy concealment error. First, the encoder selects the reference pictures based on the error status received from the feedback channel, taking into consideration the Rate-Distortion-Optimization (RDO). Second, the temporal information mismatch prediction resulting from the error concealment is reduced by decreasing the temporal dependency between adjacent frames based on new motion-estimation tools. Results show a PSNR gain of about 6.13 dB, 5.20 dB and 4.72 dB for 1080p, 720p and 480p resolutions respectively.

Spatio-Temporal Context Based Adaptive Camcorder Recording Watermarking

Video watermarking technology has attracted increasing attention in the past few years, and a great deal of traditional and deep learning-based methods have been proposed. However, these existing methods usually suffer from the following two challenges: First, most algorithms cannot resist camcorder recording attack, which limits their practical application. Second, watermark embedding may cause substantial degradation of video quality. Through analyzing the unique distortions presented in the camcorder recording process, including geometric distortion, temporal sampling distortion, sensor distortion and processing distortion, this paper proposes a novel spatio-temporal context based adaptive camcorder recording watermarking scheme STACR. In STACR, considering the geometric distortion and video visual quality, we embed the watermark by constructing a spatio-temporal histogram and incorporate a content features based adaptive locating algorithm to select embedding blocks and embedding strengths. As for the temporal sampling attack, we put forward a watermark correlation-based synchronization algorithm and combine it with cross-validation. Moreover, to resist the sensor distortion, we design a local matching-based algorithm to improve the extraction accuracy. In addition, grouped and repeated embedding strategies are combined to cope with the processing distortion. Experimental results compared with the state-of-the-art show that the proposed scheme achieves high video quality and is robust to geometric attacks, compression, scaling, transcoding, recoding, frame rate changes and especially for camcorder recording.

VQProtect: Lightweight Visual Quality Protection for Error-Prone Selectively Encrypted Video Streaming.

Mobile multimedia communication requires considerable resources such as bandwidth and efficiency to support Quality-of-Service (QoS) and user Quality-of-Experience (QoE). To increase the available bandwidth, 5G network designers have incorporated Cognitive Radio (CR), which can adjust communication parameters according to the needs of an application. The transmission errors occur in wireless networks, which, without remedial action, will result in degraded video quality. Secure transmission is also a challenge for such channels. Therefore, this paper’s innovative scheme “VQProtect” focuses on the visual quality protection of compressed videos by detecting and correcting channel errors while at the same time maintaining video end-to-end confidentiality so that the content remains unwatchable. For the purpose, a two-round secure process is implemented on selected syntax elements of the compressed H.264/AVC bitstreams. To uphold the visual quality of data affected by channel errors, a computationally efficient Forward Error Correction (FEC) method using Random Linear Block coding (with complexity of ) is implemented to correct the erroneous data bits, effectively eliminating the need for retransmission. Errors affecting an average of 7–10% of the video data bits were simulated with the Gilbert–Elliot model when experimental results demonstrated that 90% of the resulting channel errors were observed to be recoverable by correctly inferring the values of erroneous bits. The proposed solution’s effectiveness over selectively encrypted and error-prone video has been validated through a range of Video Quality Assessment (VQA) metrics.

Spatiotemporal Neural Network for Video-Based Pose Estimation

The application of 2D pose estimation methods often suffers from performance degeneration because of the severe video quality degradation. To mitigate the problem, a novel model is proposed, namely spatiotemporal net (STNet). STNet utilizes convolution modules to extract the 2D joint heatmaps of each frame and exploits recurrent convolution modules to encode the time information between the adjacent frames. This decoupling learning of spatiotemporal information improves the temporal coherence and spatial accuracy of the estimated poses and reduces the difficulty of extracting spatiotemporal features. The application of ConvGRU effectively reduces the computational cost while ensuring recognition accuracy. Proposed model is compared with other existing methods on two benchmarks: Penn Action and Sub-JHMDB. The results show that STNet can better trade-off prediction performance and computational complexity and have more practical value.

User Fairness Aware Power Allocation for NOMA-Assisted Video Transmission With Adaptive Quality Adjustment

In this paper, we investigate the user fairness power allocation (PA) for video transmission in a non-orthogonal multiple access (NOMA)-assisted wireless system with a pair of users. Considering that video transmission is usually delay-sensitive, the video quality can be adaptively adjusted based on the network conditions. First, a new performance metric, called the average video quality degradation probability, is proposed to describe the adaptive video quality. Then, based on the principles of NOMA and user queue model, the average video quality degradation probability of both users is derived. Next, a bisection search based power allocation (PA) algorithm is proposed to balance the video quality between the NOMA pair. Simulation results demonstrate that the proposed NOMA scheme is more effective to achieve the user fairness compared with the fixed NOMA scheme, the fractional transmit power allocation (FTPA) based NOMA scheme, the achievable rate fairness (ARF) NOMA scheme, and the optimized time division multiple access (TDMA) scheme.

Flexible Low-Cost Power-Efficient Video Memory With ECC-Adaptation

In this article, a flexible power-efficient video memory is presented that can dynamically adjust the strength of error correction code (ECC), thereby enabling power-quality tradeoff based on application requirements. Specifically, we utilize the bit significance characteristics of video data to develop a low-cost parity storage scheme that supports both hamming code-74 (ECC74) and hamming code-1511 (ECC1511). Based on this, we propose a flexible memory with three dynamic power-quality adaptation schemes (i.e., ECC74, ECC1511, and no ECC) to meet different video application requirements. Our simulation results in 45-nm CMOS technology show that the proposed memory can enable up to 35.37% power savings without a noticeable degradation in video quality, as compared to the conventional design. We also design an integrated ECC encoder/decoder that handles both ECC74 and ECC1511, which reduces area overhead. To evaluate the effectiveness of the proposed technique, we further develop a system-level video storage embedded test platform based on a commercial 65-nm SRAM chip, which shows that the proposed technique results in significant supply voltage reduction without noticeable video quality degradation.

An efficient field-programmable gate array-based hardware oriented block motion estimation algorithm based on diamond adaptive rood pattern search algorithm for multi-standard video codec

Motion estimation and motion compensation techniques are effectively used for removing temporal redundancy between adjacent frames during video compression, but video quality will be degraded with the traditional pixel-based matching algorithms. To overcome this issue, diamond adaptive rood pattern search algorithm-based block-matching motion estimation algorithms are proposed in this manuscript for multi-stranded codec hardware design, which provide a high compression rate with less computational complexity. In this work, a diamond adaptive rood pattern search algorithm is used, which is a combination of a diamond-search algorithm and adaptive rood pattern search. In addition, the proposed algorithm applies multiple search strategies and terminates unnecessary searching while reserving high data regularity. The proposed diamond adaptive rood pattern search algorithm-based block-matching motion estimation algorithm for multi-stranded codec hardware design provides good performance in terms of complex calculations reduction and reaches a high speed in comparison to traditional search algorithms. The proposed architecture is implemented in Xilinx ISE design 14.5 suite and the implemented results are compared to the existing motion estimation techniques on field-programmable gate array devices. Simulation results shows that the diamond adaptive rood pattern search algorithm for multi-stranded codec hardware design provides a very good improvement in terms of computations overhead reduction, with 96.4%, 97.3% latency and 98.7% throughput and achieves a high compression rate without video quality degradation.

Efficient mode decision scheme based on edge detection with Gaussian pulse for Intra-prediction in H.264/AVC

Advanced Video Coding (H.264/AVC) is one of the most important video coding standards which was developed by the ITU-T Video Coding Experts Group (VCEG) together with the ISO/IEC Moving Picture Experts Group (MPEG). Due to the complex procedures needed to find the optimal mode in intra prediction mode stage of H.264/AVC, an efficient mode decision is suggested in this paper. The main objectives of the proposed work are improving the compression efficiency of video coding and minimizing the degradation of video quality. The proposed algorithm is based on the gradient method for edge detection using the Prewitt filter to determine the texture direction of a block to be predicted in the luminance component of a video frame. To decide either implemented the DC mode or the edge-detection based gradient method to predict block under consideration, a predefined threshold value is used for the homogeneity test of each macro-block. To reduce the Bit Rate of the compressed video, Gaussian pulse modulation is also suggested to the Discrete Cosine Transformation (DCT) components of each macro-block during the coding process. The proposed algorithm is carried out on a luminance component of a set of different YUV video sequences with different resolutions and different quantization parameters using MATLAB. The suggested algorithm is evaluated using different metrics which are Peak Signal to Noise Ratio (PSNR) and bit rate measured by Bjontegaard Delta method. The simulation results show the ability of the proposed algorithm in enhancing the BDPSNR averagely by 1.14 dB and reducing the BDBR averagely by 11.34 %for QCIF sequences compared to the existing state-of-the-art fast intra-prediction mode decision algorithms.

Joint model of gradient magnitude and Gabor features via Spatio-Temporal slice

To form a high-performance video quality predictor, we developed a framework for full-reference (FR) video quality assessment that integrates Spatio-temporal slice analysis (STS) to create a high-performance predictor of video quality. However, both gradient and Gabor are spatial–temporal structural capturers used for the simultaneous extraction of both spatial and temporal features. In this paper, we proposed a novel VQA algorithm via a joint model of gradient magnitude and Gabor features (JMG) between the STS images of the reference videos and their distorted counterparts to assess the degradation of video quality effectively. Firstly, gradient magnitude and the Gabor filter were constructed to extract the spatiotemporal features of the video sequence. However, the two-feature model combined to predict the perceptual quality of frames. This new proposed VQA model is known as the horizontal and time STS (HT-JMG) model. To further investigate the influence of spatial dissimilarity, we combined the frame-by-frame spatial T-JMG(S) factor with the HT-JMG and propose another VQA model, called the time, horizontal, and vertical STS (THV-JMG) model. Finally, the results of the experiment showed that the proposed method has a strong correlation with subjective perception and is competitive with state-of-the-art full reference VQA models.

Improving human action recognition by jointly exploiting video and WiFi clues

Recent years have witnessed the increasing attentions on human action recognition(HAR). Traditional methods are prone to explore the optimum spatiotemporal feature representation of human actions in video clips so as to achieve high recognition performance. However, the optical limitations, such as inappropriate view, dim illumination and object occlusion, usually degrade video quality and affect the recognition performance a lot. Considering that wireless signals are robust against optical limitations, we thus incorporate WiFi signals with video streams for HAR. Specifically, we use WiFi Channel State Information as a compensator for video streams. A great challenge is how to effectively fuse the video and WiFi information to achieve better prediction performance. To this end, we employ convolution neural networks and statistic analysis algorithms to extract video and WiFi features respectively, and propose a novel multi-modal learning approach for video and WiFi feature fusion, where the video and WiFi features are projected to a common space by supervised learning. The experimental results indicate that the recognition precision of human actions in videos improved obviously with the aid of WiFi signals and the proposed multi-modal learning approach rivals the state of art methods.

Data Embedding in SHVC Video Using Threshold-Controlled Block Splitting

With the increasing number of video applications, it is essential to resolve issues such as ineffective search of video content, tampered/forged video content, packet loss, to name a few. Data embedding is typically utilized as one of the solutions to address the aforementioned issues. One of the important requirements of data embedding is to maximize embedding capacity with minimal bit rate overhead while ensuring imperceptibility of the inserted data. However, embedding capacity varies depending on the video content and increasing the embedding capacity usually leads to video quality degradation. In this work, a threshold-controlled block splitting technique is proposed for embedding data into SHVC video. Specifically, the embedding capacity can be increased by coding the host video by using more small blocks, which can be achieved by tuning a threshold-controlled parameter in the rate distortion optimization process. Subsequently, the predictive syntax elements in both intra and inter-coded blocks are jointly utilized to embed data, which ensures that data can be embedded regardless of the prediction mode used in coding a block. Results suggest that the proposed method can achieve a trade-off between the increase in embedding capacity and bit rate overhead while maintaining video quality. In the best case scenario, the sequence PartyScene can embed 516.9 kbps with an average bit rate overhead of +7.0% for the Low Delay P configuration, while the same video can embed 1578.6 kbps with an average bit rate overhead of +2.9% for the All Intra configuration.

Optical flow estimation combining with illumination adjustment and edge refinement in livestock UAV videos

The performance of optical flow estimation is affected by many factors. The impact of illumination changes and video quality degradation in unmanned aerial vehicles videos on optical flow estimation cannot be ignored. Inspired by the human retina’s visual adaptation mechanism, we propose a mechanism for illumination adjustment that imitates retinal processing in order to reduce illumination variation. We further introduce an edge refinement mechanism into optical flow estimation that is based on a weighted neighborhood filtering. The experimental results on public benchmarks inlcuding KITTI 2012, KITTI 2015, MPI Sintel and Middlebury show that the proposed approach is robust on illumination and preserves accurate motion details. Further, experiments on outdoor livestock UAV videos show that the proposed approach implements illumination robustness and preserves the accurate detection of motion edges in other types of video. The performance of the proposed method on public benchmarks and livestock UAV videos demonstrates that the proposed approach improves motion edge accuracy of optical flow fields in varying illumination.