Perceived Video Quality Research Articles

Perceptual Quality Assessment of NeRF and Neural View Synthesis Methods for Front‐Facing Views

AbstractNeural view synthesis (NVS) is one of the most successful techniques for synthesizing free viewpoint videos, capable of achieving high fidelity from only a sparse set of captured images. This success has led to many variants of the techniques, each evaluated on a set of test views typically using image quality metrics such as PSNR, SSIM, or LPIPS. There has been a lack of research on how NVS methods perform with respect to perceived video quality. We present the first study on perceptual evaluation of NVS and NeRF variants. For this study, we collected two datasets of scenes captured in a controlled lab environment as well as in‐the‐wild. In contrast to existing datasets, these scenes come with reference video sequences, allowing us to test for temporal artifacts and subtle distortions that are easily overlooked when viewing only static images. We measured the quality of videos synthesized by several NVS methods in a well‐controlled perceptual quality assessment experiment as well as with many existing state‐of‐the‐art image/video quality metrics. We present a detailed analysis of the results and recommendations for dataset and metric selection for NVS evaluation.

QoE-Based Performance Comparison of AVC, HEVC, and VP9 on Mobile Devices with Additional Influencing Factors

While current video quality assessment research predominantly revolves around resolutions of 4 K and beyond, targeted at ultra high-definition (UHD) displays, effective video quality for mobile video streaming remains primarily within the range of 480 p to 1080 p. In this study, we conducted a comparative analysis of the quality of experience (QoE) for widely implemented video codecs on mobile devices, specifically Advanced Video Coding (AVC), its successor High-Efficiency Video Coding (HEVC), and Google’s VP9. Our choice of 720 p video sequences from a newly developed database, all with identical bitrates, aimed to maintain a manageable subjective assessment duration, capped at 35–40 min. To mimic real-time network conditions, we generated stimuli by streaming original video clips over a controlled emulated setup, subjecting them to eight different packet-loss scenarios. We evaluated the quality and structural similarity of the distorted video clips using objective metrics, including the Video Quality Metric (VQM), Peak Signal-to-Noise Ratio (PSNR), Video Multi-Method Assessment Fusion (VMAF), and Multi-Scale Structural Similarity Index (MS-SSIM). Subsequently, we collected subjective ratings through a custom mobile application developed for Android devices. Our findings revealed that VMAF accurately represented the degradation in video quality compared to other metrics. Moreover, in most cases, HEVC exhibited an advantage over both AVC and VP9 under low packet-loss scenarios. However, it is noteworthy that in our test cases, AVC outperformed HEVC and VP9 in scenarios with high packet loss, based on both subjective and objective assessments. Our observations further indicate that user preferences for the presented content contributed to video quality ratings, emphasizing the importance of additional factors that influence the perceived video quality of end users.

Perceptual Video Coding for Machines via Satisfied Machine Ratio Modeling.

Video Coding for Machines (VCM) aims to compress visual signals for machine analysis. However, existing methods only consider a few machines, neglecting the majority. Moreover, the machine's perceptual characteristics are not leveraged effectively, resulting in suboptimal compression efficiency. To overcome these limitations, this paper introduces Satisfied Machine Ratio (SMR), a metric that statistically evaluates the perceptual quality of compressed images and videos for machines by aggregating satisfaction scores from them. Each score is derived from machine perceptual differences between original and compressed images. Targeting image classification and object detection tasks, we build two representative machine libraries for SMR annotation and create a large-scale SMR dataset to facilitate SMR studies. We then propose an SMR prediction model based on the correlation between deep feature differences and SMR. Furthermore, we introduce an auxiliary task to increase the prediction accuracy by predicting the SMR difference between two images in different quality. Extensive experiments demonstrate that SMR models significantly improve compression performance for machines and exhibit robust generalizability on unseen machines, codecs, datasets, and frame types. Code is available at https://github.com/ywwynm/SMR.

Semi-Supervised Learning of Perceptual Video Quality by Generating Consistent Pairwise Pseudo-Ranks

Semi-Supervised Learning of Perceptual Video Quality by Generating Consistent Pairwise Pseudo-Ranks

A contrast sensitivity model of the human visual system in modern conditions for presenting video content.

Digital video incurs many distortions during processing, compression, storage, and transmission, which can reduce perceived video quality. Developing adaptive video transmission methods that provide increased bandwidth and reduced storage space while preserving visual quality requires quality metrics that accurately describe how people perceive distortion. A severe problem for developing new video quality metrics is the limited data on how the early human visual system simultaneously processes spatial and temporal information. The problem is exacerbated by the fact that the few data collected in the middle of the last century do not consider current display equipment and are subject to medical intervention during collection, which does not guarantee a proper description of the conditions under which media content is currently consumed. In this paper, the 27840 thresholds of the visibility of spatio-temporal sinusoidal variations necessary to determine the artefacts that a human perceives were measured by a new method using different spatial sizes and temporal modulation rates. A multidimensional model of human contrast sensitivity in modern conditions of video content presentation is proposed based on new large-scale data obtained during the experiment. We demonstrate that the presented visibility model has a distinct advantage in predicting subjective video quality by testing with video quality metrics and including our and other visibility models against three publicly available video datasets.

A Visual Sensitivity Aware ABR Algorithm for DASH via Deep Reinforcement Learning

In order to cope with the fluctuation of network bandwidth and provide smooth video services, adaptive video streaming technology is proposed. In particular, the adaptive bitrate (ABR) algorithm is widely used in dynamic adaptive streaming over HTTP (DASH) to improve quality of experience (QoE). However, existing ABR algorithms still ignore the inherent visual sensitivity of human visual system (HVS). As the final receiver of video, HVS has different sensitivity to the quality distortion of different video content, and video content with high visual sensitivity needs to allocate more bitrate resources. Therefore, existing ABR algorithms still have limitations in reasonably allocating bitrate and maximizing QoE. To solve this problem, this paper designs an adaptive bitrate strategy from the perspective of user vision, studies the modeling of visual sensitivity, and proposes a visual sensitivity aware ABR algorithm. We extract a set of content features and attribute features from the video, and consider the simulation of HVS to establish a total masking effect model that reflects the visual sensitivity more accurately. Further, the network status, buffer occupancy, and visual sensitivity are comprehensively considered under a deep reinforcement learning framework to select the appropriate bitrate for maximizing QoE. We implement the proposed algorithm over a realistic trace-driven evaluation and compare its performance with several latest algorithms. Experimental results show that our algorithm can align ABR strategy with visual sensitivity to achieve better QoE in high visual sensitivity content, and improves the average perceptual video quality and overall user QoE by 18.3% and 22.8%, respectively. Additionally, we prove the feasibility of our algorithm through subjective evaluation in the real environment.

Deep Reinforcement Learning-Based Approach for Video Streaming: Dynamic Adaptive Video Streaming over HTTP

Dynamic adaptive video streaming over HTTP (DASH) plays a crucial role in delivering video across networks. Traditional adaptive bitrate (ABR) algorithms adjust video segment quality based on network conditions and buffer occupancy. However, these algorithms rely on fixed rules, making it challenging to achieve optimal decisions considering the overall context. In this paper, we propose a novel deep-reinforcement-learning-based approach for DASH streaming, with the primary focus of maintaining consistent perceived video quality throughout the streaming session to enhance user experience. Our approach optimizes quality of experience (QoE) by dynamically controlling the quality distance factor between consecutive video segments. We evaluate our approach through a comprehensive simulation model encompassing diverse wireless network environments and various video sequences. We also conduct a comparative analysis with state-of-the-art methods. The experimental results demonstrate significant improvements in QoE, ensuring users enjoy stable, high-quality video streaming sessions.

Video perception enhancement using effective encoding optimization in future generation wireless network

<span>The cumulative effects of network transmission link imperfections and real-time limitations of video data result in multiple challenges for video transmission in massive millimeter wave (mmwave) network. The challenges, together with the increasing expectations of users for good-quality videos, further extend the complexity particularly in dense areas. To address the issues this work proposes perceptual quality aware video encoding (PQAVE) scheme in two phases: in phase 1 an effective perceptual quality encoding method leveraging low-rank approximation to reduce overall video size is done. In phase 2, a novel bitstream optimization technique is introduced to improve perceptual quality of videos. Experiment are conducted using standard video dataset show the proposed PQAVE model attain better bit error rate (BER), symbol error rate (SER), error vector magnitude (EVM), and coding gain in comparison with existing perceptual video encoding scheme.</span>

Blind video watermarking scheme for medical video authentication

Medical video watermarking is one of the beneficial and efficient tools to prohibit important patients' data from illicit enrollment and redistribution. In this paper, a new blind watermarking scheme has been proposed to improve the confidentiality, integrity, authenticity, and perceptual quality of a medical video with minimum distortion. The proposed scheme is based on 2D-DWT and dual Hessenberg-QR decomposition, where the input medical video is initially processed into frames. Then, the processed frames are transformed into sub-bands using 2D-DWT, followed by applying Hessenberg-QR decomposition on the selected wavelet HL2 sub-band. The watermark is scrambled via Arnold cat map to raise confidentiality and then concealed in the modified selected features. The watermark is extracted in a fully blind mode without referencing the original video, which reduces the extraction time. The proposed scheme maintained a fundamental tradeoff between robustness and visual imperceptibility compared to existing methods against many commonly encountered attacks. The visual imperceptibility has been evaluated using well-known metrics PSNR, SSIM, Q-index, and histogram analysis. The proposed scheme achieves a high PSNR value of (70.6899 dB) with minimal distortion and a high robustness level with an average NC value of (0.9998) and BER value of (0.0023) while conserving a large payload capacity. The obtained results show superior performance over similar video watermarking methods. The limitation of this scheme is the elapsed time during the embedding process since we utilized dual Hessenberg-QR decomposition. One possible solution to reduce time consumption is simple decompositions like bound-constrained SVM or similar decompositions.

A Novel Automatic Content Generation and Optimization Framework

With the rapid growth of IoT multimedia devices, more and more content is being delivered in multimedia forms which generally requires more effort and resources from users to create and could be challenging to streamline. In this paper, we propose Text2Animation (T2A), a framework that helps to generate complex multimedia content, and animation, from the simple textual script input. By leveraging recent advances in computational cinematography and video understanding, the purposed workflow further reduces associated knowledge requirements dramatically for cinematography. By jointly incorporating the fidelity and aesthetic models, T2A jointly considers the comprehensiveness of the visual presentation of the input script and the compliance of generated video with given cinematography specifications. The virtual camera placement in a 3D environment is mapped into an optimization problem that can be resolved by using dynamic programming to achieve the lowest computational complexity. Experimental results show that T2A can reduce the manual animation production process by around 74%, and the new optimization framework can improve the perceptual quality of the output video by up to 35%. More video footage can be found at https://www.youtube.com/watch?v=MMTJbmWL3gs.

Perceptual video quality assessment: the journey continues!

Perceptual Video Quality Assessment (VQA) is one of the most fundamental and challenging problems in the field of Video Engineering. Along with video compression, it has become one of two dominant theoretical and algorithmic technologies in television streaming and social media. Over the last 2 decades, the volume of video traffic over the internet has grown exponentially, powered by rapid advancements in cloud services, faster video compression technologies, and increased access to high-speed, low-latency wireless internet connectivity. This has given rise to issues related to delivering extraordinary volumes of picture and video data to an increasingly sophisticated and demanding global audience. Consequently, developing algorithms to measure the quality of pictures and videos as perceived by humans has become increasingly critical since these algorithms can be used to perceptually optimize trade-offs between quality and bandwidth consumption. VQA models have evolved from algorithms developed for generic 2D videos to specialized algorithms explicitly designed for on-demand video streaming, user-generated content (UGC), virtual and augmented reality (VR and AR), cloud gaming, high dynamic range (HDR), and high frame rate (HFR) scenarios. Along the way, we also describe the advancement in algorithm design, beginning with traditional hand-crafted feature-based methods and finishing with current deep-learning models powering accurate VQA algorithms. We also discuss the evolution of Subjective Video Quality databases containing videos and human-annotated quality scores, which are the necessary tools to create, test, compare, and benchmark VQA algorithms. To finish, we discuss emerging trends in VQA algorithm design and general perspectives on the evolution of Video Quality Assessment in the foreseeable future.

Robust Video Watermarking Algorithm based on DCT-SVD approach and Encryption

Sharing of digital media content over the internet is increasing everyday .Digital watermarking is a technique used to protect the intellectual property rights of multimedia content owners. In this paper, we propose a robust video watermarking scheme that utilizes Discrete Cosine Transform (DCT) and Singular Value Decomposition (SVD) for embedding a watermark into video frames. The proposed method uses encryption to make the watermark more robust against malicious attacks. The encryption key is used to modify the watermark before it is embedded in the video frames. The modified watermark is then embedded in the DCT and SVD coefficients of the video frames. The experimental results show that the proposed method provides better robustness against various attacks such as compression, noise addition, and filtering, while maintaining good perceptual quality of the watermarked video. The proposed method also shows better resistance against geometric attacks such as cropping, rotation, and scaling. Overall, the proposed method provides an effective solution for protecting the intellectual property rights of multimedia content owners in video distribution and transmission scenarios.

No-Reference Image Quality Assessment Using the Statistics of Global and Local Image Features

Methods of image quality assessment are widely used for ranking computer vision algorithms or controlling the perceptual quality of video and streaming applications. The ever-increasing number of digital images has encouraged the research in this field at an accelerated pace in recent decades. After the appearance of convolutional neural networks, many researchers have paid attention to different deep architectures to devise no-reference image quality assessment algorithms. However, many systems still rely on handcrafted features to ensure interpretability and restrict the consumption of resources. In this study, our efforts are focused on creating a quality-aware feature vector containing information about both global and local image features. Specifically, the research results of visual physiology indicate that the human visual system first quickly and automatically creates a global perception before gradually focusing on certain local areas to judge the quality of an image. Specifically, a broad spectrum of statistics extracted from global and local image features is utilized to represent the quality-aware aspects of a digital image from various points of view. The experimental results demonstrate that our method’s predicted quality ratings relate strongly with the subjective quality ratings. In particular, the introduced algorithm was compared with 16 other well-known advanced methods and outperformed them by a large margin on 9 accepted benchmark datasets in the literature: CLIVE, KonIQ-10k, SPAQ, BIQ2021, TID2008, TID2013, MDID, KADID-10k, and GFIQA-20k, which are considered de facto standards and generally accepted in image quality assessment.

Enhancing Perceptual Experience of Video Quality in Drone Communications by Using VPN Bonding

Recently, drones are increasingly being used as innovative IoT sensors. The application contexts of UAV systems where there is often a need to transmit time-critical data between the drone and the Internet are increasingly numerous and varied. Keeping this in mind and assuming that most professional UAVs are equipped with multiinterface routers and use virtual private networks (VPNs) to ensure reliable, robust, and secure transmission to the operation room, this letter presents a technique to enhance real-time and time-critical communications such as real-time video streaming. An embedded system, Raspberry-based gateway prototype, was used to create a testbed using a professional drone that captured and transmitted real-time video during a 30-min flight over a rural area characterized by poor mobile radio coverage. A perceptual video quality assessment tool, video multimethod assessment fusion (VMAF), has been employed to evaluate the improvement in terms of video quality perceived by the operator.

Efficacy of Video-based Education in Improving Understanding of Pediatric Fertility Preservation.

Eighty percent of children diagnosed with cancer in childhood survive into adulthood. Fertility preservation (FP) is an important consideration, and procedures are available to reduce the risk of infertility following gonadotoxic therapies. Discussing FP options eases decision-making and minimizes regret; however, poor comprehension of these topics remains a challenge. This study evaluates if video-based educational tools increase understanding of FP options among pediatric patients and families. Videos were first tested among participants not at risk of infertility to ensure objective utility and optimize quality. In part 1, parents of pediatric surgical patients were randomized to view 2 publicly available educational videos on FP in differing orders. Each group completed pre-surveys and post-surveys assessing the comprehension and perception of video quality. In part 2, the parent and patient participants completed a comprehension assessment before and after viewing our institution-specific educational video, designed based on participant feedback from part 1. Part 1 results demonstrated a significant increase in participant knowledge and perceived understanding after viewing the videos ( P <0.001), regardless of order. In part 2, the post-test comprehension scores were significantly improved for all participants and all subgroups, P <0.01. Results suggest that video-based educational tools may help to reduce barriers to FP in pediatric oncology.

Video coding deep learning-based modeling for long life video streaming over next network generation.

Availability is one of the primary goals of smart networks, especially, if the network is under heavy video streaming traffic. In this paper, we propose a deep learning based methodology to enhance availability of video streaming systems by developing a prediction model for video streaming quality, required power consumption, and required bandwidth based on video codec parameters. The H.264/AVC codec, which is one of the most popular codecs used in video steaming and conferencing communications, is chosen as a case study in this paper. We model the predicted consumed power, the predicted perceived video quality, and the predicted required bandwidth for the video codec based on video resolution and quantization parameters. We train, validate, and test the developed models through extensive experiments using several video contents. Results show that an accurate model can be built for the needed purpose and the video streaming quality, required power consumption, and required bandwidth can be predicted accurately which can be utilized to enhance network availability in a cooperative environment.

CONVIQT: Contrastive Video Quality Estimator.

Perceptual video quality assessment (VQA) is an integral component of many streaming and video sharing platforms. Here we consider the problem of learning perceptually relevant video quality representations in a self-supervised manner. Distortion type identification and degradation level determination is employed as an auxiliary task to train a deep learning model containing a deep Convolutional Neural Network (CNN) that extracts spatial features, as well as a recurrent unit that captures temporal information. The model is trained using a contrastive loss and we therefore refer to this training framework and resulting model as CONtrastive VIdeo Quality EstimaTor (CONVIQT). During testing, the weights of the trained model are frozen, and a linear regressor maps the learned features to quality scores in a no-reference (NR) setting. We conduct comprehensive evaluations of the proposed model against leading algorithms on multiple VQA databases containing wide ranges of spatial and temporal distortions. We analyze the correlations between model predictions and ground-truth quality ratings, and show that CONVIQT achieves competitive performance when compared to state-of-the-art NR-VQA models, even though it is not trained on those databases. Our ablation experiments demonstrate that the learned representations are highly robust and generalize well across synthetic and realistic distortions. Our results indicate that compelling representations with perceptual bearing can be obtained using self-supervised learning.

Quality Feature Learning via Multi-Channel CNN and GRU for No-Reference Video Quality Assessment

Nowadays, video quality assessment (VQA) plays a vital role in video-related industries to predict human perceived video quality to maintain the quality of service. Although many deep neural network-based VQA methods have been proposed, the robustness and performance are limited by small scale of available human-label data. Recently, some transfer learning-based methods and pre-trained models in other domains have been adopted in VQA to compensate for the lack of enormous training samples. However, they result in a domain gap between the source and target domains, which provides sub-optimal feature representation for VQA tasks and deteriorates the accuracy. Therefore, in the paper, we propose quality feature learning via a multi-channel convolutional neural network (CNN) with a gated recurrent unit (GRU), taking into account both the motion-aware information and human visual perception (HVP) characteristics to solve the above issue for no-reference VQA. First, inspired by self-supervised learning (SSL), the multi-channel CNN is pre-trained on the image quality assessment (IQA) domain without using human annotation labels. Then, semi-supervised learning is applied on top of the pre-trained multi-channel CNN to fine-tune the model to transfer the domain from IQA to VQA while considering motion-aware information for better frame-level quality feature representation. After that, several HVP features are extracted with frame-level quality feature representation as the input of the GRU model to obtain the final precise predicted video quality. Finally, the experimental results demonstrate the robustness and validity of our model, which is superior to the state-of-the-art approaches and is closely related to human perception.

Qrator: An Interest-Aware Approach to ABR Streaming Based on User Engagement

Conventional approaches to adaptive bitrate (ABR) streaming fail to further improve the quality of experience (QoE) as they are indifferent to video semantics and user behaviors by focusing solely on the network conditions. These ABR schemes cannot enhance the perceptual video quality as they neglect the users’ varying degree of interest (DoI) over video sections and may experience QoE degradation due to frequent rebuffers caused by user behaviors such as selectively watching the interesting sections. To this end, this article proposes <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Qrator</i> , which is the very first attempt to utilize timestamps and likes as user engagement data to infer users’ genuine interests and further elevate the user-perceived QoE. Based on timestamps and likes in video comments, Qrator improves the overall QoE by considering DoI variations and user behaviors in performing interest-aware rate adaptation and prefetching. Qrator can be widely applied to conventional ABR approaches without significantly modifying their implementations. Evaluation results show that Qrator can heighten the bitrates of user interest sections without degrading the average bitrate. Furthermore, applying Qrator under user behavior patterns can reduce the rebuffering ratio and the number of rebuffers by 32% and 31%, respectively, while maintaining other QoE metrics to a similar extent.

Quality-driven video streaming for ultra-dense OFDMA heterogeneous networks

The explosive growth of mobile video traffic has brought enormous traffic pressure to the existing bandwidth-limited cellular network. Ultra-dense heterogeneous networks (HetNets) are deemed as one of the most promising 5G technologies for increasing the cellular network capacity. In this paper, we study quality-driven multiuser video streaming algorithm for two-tier orthogonal frequency division multiple access (OFDMA) HetNets. Specifically, we formulate a stochastic optimization problem for the quality-driven multiuser video streaming in two-tier OFDMA HetNets to maximize the average perceived video quality of small cell users while meeting the average data rate demand of macrocell users. We then decompose the original stochastic problem into a sequence of per-slot deterministic subproblems of adaptive bitrate selection and radio resource management which have no requirement of prior channel knowledge by employing Lyapunov optimization technique. We accordingly propose an online quality-driven video streaming algorithm. In this algorithm, we derive the optimal bitrate decision for the adaptive bitrate selection subproblem, and design an iterative scheduling algorithm of joint subchannel assignment and power control by utilizing alternating optimization and successive convex approximation techniques to solve the radio management subproblem. Extensive simulation results validate the high efficiency of the proposed quality-driven video streaming algorithm in OFDMA HetNets.