Video Quality Assessment Research Articles

Scaling and Masking: A New Paradigm of Data Sampling for Image and Video Quality Assessment

Quality assessment of images and videos emphasizes both local details and global semantics, whereas general data sampling methods (e.g., resizing, cropping or grid-based fragment) fail to catch them simultaneously. To address the deficiency, current approaches have to adopt multi-branch models and take as input the multi-resolution data, which burdens the model complexity. In this work, instead of stacking up models, a more elegant data sampling method (named as SAMA, scaling and masking) is explored, which compacts both the local and global content in a regular input size. The basic idea is to scale the data into a pyramid first, and reduce the pyramid into a regular data dimension with a masking strategy. Benefiting from the spatial and temporal redundancy in images and videos, the processed data maintains the multi-scale characteristics with a regular input size, thus can be processed by a single-branch model. We verify the sampling method in image and video quality assessment. Experiments show that our sampling method can improve the performance of current single-branch models significantly, and achieves competitive performance to the multi-branch models without extra model complexity. The source code will be available at https://github.com/Sissuire/SAMA.

Knowledge Guided Semi-supervised Learning for Quality Assessment of User Generated Videos

Perceptual quality assessment of user generated content (UGC) videos is challenging due to the requirement of large scale human annotated videos for training. In this work, we address this challenge by first designing a self-supervised Spatio-Temporal Visual Quality Representation Learning (ST-VQRL) framework to generate robust quality aware features for videos. Then, we propose a dual-model based Semi Supervised Learning (SSL) method specifically designed for the Video Quality Assessment (SSL-VQA) task, through a novel knowledge transfer of quality predictions between the two models. Our SSL-VQA method uses the ST-VQRL backbone to produce robust performances across various VQA datasets including cross-database settings, despite being learned with limited human annotated videos. Our model improves the state-of-the-art performance when trained only with limited data by around 10%, and by around 15% when unlabelled data is also used in SSL. Source codes and checkpoints are available at https://github.com/Shankhanil006/SSL-VQA.

360° video quality assessment based on saliency-guided viewport extraction

360° video quality assessment based on saliency-guided viewport extraction

Adversarial attacks on video quality assessment models

Most currently developed video quality assessment (VQA) algorithms have achieved excellent performance by using deep neural network (DNN). However, DNN is vulnerable to adversarial attacks, as an efficient surrogate for validating the model robustness, and there lack adversarial attack methods against VQA models. To this end, we propose a spatiotemporal attack network to generate adversarial examples for evaluating the robustness of VQA models that contains a spatial subnetwork and a temporal subnetwork. The proposed network, dubbed the Space-Time Quality Attack Network (STQA-Net1), first computes the just noticeable difference (JND) maps of a video sequence as the input of the spatial subnetwork. The spatial subnetwork encodes the computed maps as spatial features and feeds the spatial features to the temporal subnetwork. Then, the spatial features are fused with the output of the temporal subnetwork and the fused features are decoded as attack weight maps. A visual constraint is used to control the visibility of perturbations and guide the generation of perturbation maps by multiplying JND maps with attack weight maps. Finally, the generated perturbation maps are added to the original video to form an adversarial example. Further, we also try to design a two-branch network to generate two opposite examples in a targeted attack scenario. The proposed attack methods against six state-of-the-art VQA algorithms are thoroughly tested on three VQA databases. The experimental results show that the proposed attack methods are very effective for testing the robustness of VQA models.

Omnidirectional Video Quality Assessment With Causal Intervention

Omnidirectional Video Quality Assessment With Causal Intervention

Multi-Modal Learning-Based Blind Video Quality Assessment Metric for Synthesized Views

Multi-Modal Learning-Based Blind Video Quality Assessment Metric for Synthesized Views

Validity of Material Related to the Anterior Cruciate Ligament on TikTok.

Video material is one of the most common types of digital information on social media platforms. To assess the validity and informational value of TikTok videos with regard to the anterior cruciate ligament (ACL). Cross-sectional study. Using the terms "anterior cruciate ligament" AND/OR "ACL," 420 TikTok videos were retrieved, of which 100 videos were included in the analysis. The video duration and the number of likes, shares, and views were recorded. Videos were categorized based on the source (private user, physical therapist, surgeon, and researcher), subject type (patient experience, physical therapy, injury mechanism, anatomy, clinical test, and surgical technique), content (patient experience/testimony, education, and rehabilitation), and any background audio (music or voice). Video quality assessments were conducted using the DISCERN instrument, The Journal of the American Medical Association (JAMA) benchmark criteria, and the Global Quality Score (GQS). Associations between quantitative variables were tested using the Spearman rank correlation. One-way analysis of variance or Mann-Whitney test was performed to assess whether video quality differed by video characteristics. A total of 41 videos were published by private users (41%), 34 by physical therapists (34%), 23 by surgeons (23%), and 2 by researchers (2%). Most of the information regarded patient experience (34%), followed by physical therapy (20%), injury mechanism (19%), anatomy (11%), clinical test (9%), and surgical technique (7%). The mean video length was 40.55 ± 41.58 seconds. The mean number of views was 151,084.39 ± 487,150.02, while the mean numbers of comments, likes, and shares were 72.80 ± 249.68, 6781.49 ± 29,163.96, and 98.71 ± 307.76, respectively. The mean DISCERN, JAMA, and GQS scores were 15.73 ± 1.44, 0.19 ± 0.39, and 1.16 ± 0.37, respectively, indicating the poorest quality on all 3 indices. The number of views, likes, shares, comments, and video lengths were all positively correlated with DISCERN (except the number of shares), JAMA, and GQS scores (P < .05). A significant difference was found in DISCERN scores between videos by private users and those by surgeons (15.37 ± 0.98 vs 16.22 ± 1.57; P = .007). Our analysis has demonstrated that the educational value of these videos on the ACL on TikTok is poor. Given the rapid growth of TikTok, further research is needed.

YouTube Video Analytics for Patient Education: An Exploratory Clustering of Obstructive Sleep Apnea Videos.

With the growing popularity of content-sharing platforms, patients are increasingly using the Internet as a critical source of health information. As one of the most popular video-sharing sites, YouTube provides easy access to health information seekers, but it is difficult and time-consuming to identify and retrieve high-quality videos that may serve as engaging patient education materials. This paper reports on an exploratory analysis of 317 YouTube videos on Obstructive Sleep Apnea (OSA) to better understand some key features of the videos and the relationships between them to facilitate subsequent video classification and recommendation. Features intrinsic to a video, such as video duration, and extrinsic, such as the number of views, are analyzed using unsupervised clustering methods and the Sankey diagram to discover the relationship between the clusters and their significance across different clusters, providing promising insights for the assessment of video quality.

3D Convolutional Neural Network-Based Multi-Parameter Video Quality Assessment Model on Cloud Platforms

In light of the rapid advancements in big data and artificial intelligence technologies, the trend of uploading local files to cloud servers to mitigate local storage limitations is growing. However, the surge of duplicate files, especially images and videos, results in significant network bandwidth wastage and complicates server management. To tackle these issues, we have developed a multi-parameter video quality assessment model utilizing a 3D convolutional neural network within a video deduplication framework. Our method, inspired by the analytic hierarchy process, thoroughly evaluates the effects of packet loss rate, codec, frame rate, bit rate, and resolution on video quality. The model employs a two-stream 3D convolutional neural network to integrate spatial and temporal streams for capturing video distortion details, with a coding layer configured to remove redundant distortion information. We validated our approach using the LIVE and CSIQ datasets, comparing its performance against the V-BLIINDS and VIDEO schemes across different packet loss rates. Furthermore, we simulated the client-server interaction using a subset of the dataset and assessed the scheme's time efficiency. Our results indicate that the proposed scheme offers a highly efficient solution for video quality assessment.

QoE-Based Performance Comparison of AVC, HEVC, and VP9 on Mobile Devices with Additional Influencing Factors

While current video quality assessment research predominantly revolves around resolutions of 4 K and beyond, targeted at ultra high-definition (UHD) displays, effective video quality for mobile video streaming remains primarily within the range of 480 p to 1080 p. In this study, we conducted a comparative analysis of the quality of experience (QoE) for widely implemented video codecs on mobile devices, specifically Advanced Video Coding (AVC), its successor High-Efficiency Video Coding (HEVC), and Google’s VP9. Our choice of 720 p video sequences from a newly developed database, all with identical bitrates, aimed to maintain a manageable subjective assessment duration, capped at 35–40 min. To mimic real-time network conditions, we generated stimuli by streaming original video clips over a controlled emulated setup, subjecting them to eight different packet-loss scenarios. We evaluated the quality and structural similarity of the distorted video clips using objective metrics, including the Video Quality Metric (VQM), Peak Signal-to-Noise Ratio (PSNR), Video Multi-Method Assessment Fusion (VMAF), and Multi-Scale Structural Similarity Index (MS-SSIM). Subsequently, we collected subjective ratings through a custom mobile application developed for Android devices. Our findings revealed that VMAF accurately represented the degradation in video quality compared to other metrics. Moreover, in most cases, HEVC exhibited an advantage over both AVC and VP9 under low packet-loss scenarios. However, it is noteworthy that in our test cases, AVC outperformed HEVC and VP9 in scenarios with high packet loss, based on both subjective and objective assessments. Our observations further indicate that user preferences for the presented content contributed to video quality ratings, emphasizing the importance of additional factors that influence the perceived video quality of end users.

Feature Maps Correlation-based Video Quality Assessment

Feature Maps Correlation-based Video Quality Assessment

Blind video quality assessment based on Spatio-Temporal Feature Resolver

As a method to measure video quality, blind video quality assessment (BVQA) plays an important role in video related applications. Feature extraction, as a vital component of BVQA, significantly impacts the performance and speed of the method. In the process of designing BVQA method, we make the feature extraction process independent of the whole quality evaluation method. First, a Spatio-Temporal Feature Resolver (STFR) is obtained by training. Then, the STFR is employed to directly extract the spatio-temporal features of the video sequences. Finally, the extracted features are mapped to quality scores using Support Vector Regression (SVR). STFR only needs to be trained in an undistorted video sequence, and can be directly applied to video sequences of various scenes, which has a universality. To evaluate the effectiveness of the proposed method, the experimental results on four published video quality databases show that the proposed BVQA method not only achieves more accurately than the existing BVQA methods, but also exhibits significant competitiveness in computational speed.

DashReStreamer: Framework for Creation of Impaired Video Clips under Realistic Network Conditions

The continuous rise of multimedia entertainment has led to an increased demand for delivering outstanding user experience of multimedia content. However, modelling user-perceived Quality of Experience (QoE) is a challenging task, resulting in efforts for better understanding and measurement of user-perceived QoE. To evaluate user QoE, subjective quality assessment, where people watch and grade videos, and objective quality assessment in which videos are graded using one or many objective metrics are conducted. While there is a plethora of video databases available for subjective and objective video quality assessment, these videos are artificially infused with various temporal and spatial impairments. Videos being assessed are artificially distorted with startup delay, bitrate changes, and stalls due to rebuffering events. To conduct a more credible quality assessment, a reproduction of original user experiences while watching different types of streams on different types and quality of networks is needed. To aid current efforts in bridging the gap between the mapping of objective video QoE metrics to user experience, we developed DashReStreamer, an open-source framework for re-creating adaptively streamed video in real networks. The framework takes inputs in the form of video logs captured by the client in a non-regulated setting, along with an.mpd file or a YouTube URL. The ultimate result is a video sequence that encompasses all the data extracted from the video log. DashReStreamer also calculates popular video quality metrics like PSNR, SSIM, MS-SSIM and VMAF. Finally, DashReStreamer allows creating impaired video sequences from the popular streaming platform, YouTube. As a demonstration of framework usage we created a database of 332 realistic video clips, based on video logs collected from real mobile and wireless networks. Every video clip is supplemented with bandwidth trace and video logs used in its creation and also with objective metrics calculation reports. In addition to dataset, we performed subjective evaluation of video content, assessing its effect on overall user QoE. We believe that this dataset and framework will allow the research community to better understand the impacts of video QoE dynamics.

FAVER: Blind quality prediction of variable frame rate videos

Video quality assessment (VQA) remains an important and challenging problem that affects many applications at the widest scales. Recent advances in mobile devices and cloud computing techniques have made it possible to capture, process, and share high resolution, high frame rate (HFR) videos across the Internet nearly instantaneously. Being able to monitor and control the quality of these streamed videos can enable the delivery of more enjoyable content and perceptually optimized rate control. Accordingly, there is a pressing need to develop VQA models that can be deployed at enormous scales. While some recent effects have been applied to full-reference (FR) analysis of variable frame rate and HFR video quality, the development of no-reference (NR) VQA algorithms targeting frame rate variations has been little studied. Here, we propose a first-of-a-kind blind VQA model for evaluating HFR videos, which we dub the Framerate-Aware Video Evaluator w/o Reference (FAVER). FAVER uses extended models of spatial natural scene statistics that encompass space–time wavelet-decomposed video signals, and leverages the advantages of the deep neural network to provide motion perception, to conduct efficient frame rate sensitive quality prediction. Our extensive experiments on several HFR video quality datasets show that FAVER outperforms other blind VQA algorithms at a reasonable computational cost. To facilitate reproducible research and public evaluation, an implementation of FAVER is being made freely available online: https://github.com/uniqzheng/HFR-BVQA.

Subjective and Objective Quality Assessment of Rendered Human Avatar Videos in Virtual Reality.

We study the visual quality judgments of human subjects on digital human avatars (sometimes referred to as "holograms" in the parlance of virtual reality [VR] and augmented reality [AR] systems) that have been subjected to distortions. We also study the ability of video quality models to predict human judgments. As streaming human avatar videos in VR or AR become increasingly common, the need for more advanced human avatar video compression protocols will be required to address the tradeoffs between faithfully transmitting high-quality visual representations while adjusting to changeable bandwidth scenarios. During transmission over the internet, the perceived quality of compressed human avatar videos can be severely impaired by visual artifacts. To optimize trade-offs between perceptual quality and data volume in practical workflows, video quality assessment (VQA) models are essential tools. However, there are very few VQA algorithms developed specifically to analyze human body avatar videos, due, at least in part, to the dearth of appropriate and comprehensive datasets of adequate size. Towards filling this gap, we introduce the LIVE-Meta Rendered Human Avatar VQA Database, which contains 720 human avatar videos processed using 20 different combinations of encoding parameters, labeled by corresponding human perceptual quality judgments that were collected in six degrees of freedom VR headsets. To demonstrate the usefulness of this new and unique video resource, we use it to study and compare the performances of a variety of state-of-the-art Full Reference and No Reference video quality prediction models, including a new model called HoloQA. As a service to the research community, we publicly releases the metadata of the new database at https://live.ece.utexas.edu/research/LIVE-Meta-rendered-human-avatar/index.html.

Subjective and Objective Quality Assessment of Colonoscopy Videos.

Captured colonoscopy videos usually suffer from multiple real-world distortions, such as motion blur, low brightness, abnormal exposure, and object occlusion, which impede visual interpretation. However, existing works mainly investigate the impacts of synthesized distortions, which differ from real-world distortions greatly. This research aims to carry out an in-depth study for colonoscopy Video Quality Assessment (VQA). In this study, we advance this topic by establishing both subjective and objective solutions. Firstly, we collect 1,000 colonoscopy videos with typical visual quality degradation conditions in practice and construct a multi-attribute VQA database. The quality of each video is annotated by subjective experiments from five distortion attributes (i.e., temporal-spatial visibility, brightness, specular reflection, stability, and utility), as well as an overall perspective. Secondly, we propose a Distortion Attribute Reasoning Network (DARNet) for automatic VQA. DARNet includes two streams to extract features related to spatial and temporal distortions, respectively. It adaptively aggregates the attribute-related features through a multi-attribute association module to predict the quality score of each distortion attribute. Motivated by the observation that the rating behaviors for all attributes are different, a behavior guided reasoning module is further used to fuse the attribute-aware features, resulting in the overall quality. Experimental results on the constructed database show that our DARNet correlates well with subjective ratings and is superior nine state-of-the-art methods.

A Study of Subjective and Objective Quality Assessment of HDR Videos.

As compared to standard dynamic range (SDR) videos, high dynamic range (HDR) content is able to represent and display much wider and more accurate ranges of brightness and color, leading to more engaging and enjoyable visual experiences. HDR also implies increases in data volume, further challenging existing limits on bandwidth consumption and on the quality of delivered content. Perceptual quality models are used to monitor and control the compression of streamed SDR content. A similar strategy should be useful for HDR content, yet there has been limited work on building HDR video quality assessment (VQA) algorithms. One reason for this is a scarcity of high-quality HDR VQA databases representative of contemporary HDR standards. Towards filling this gap, we created the first publicly available HDR VQA database dedicated to HDR10 videos, called the Laboratory for Image and Video Engineering (LIVE) HDR Database. It comprises 310 videos from 31 distinct source sequences processed by ten different compression and resolution combinations, simulating bitrate ladders used by the streaming industry. We used this data to conduct a subjective quality study, gathering more than 20,000 human quality judgments under two different illumination conditions. To demonstrate the usefulness of this new psychometric data resource, we also designed a new framework for creating HDR quality sensitive features, using a nonlinear transform to emphasize distortions occurring in spatial portions of videos that are enhanced by HDR, e.g., having darker blacks and brighter whites. We apply this new method, which we call HDRMAX, to modify the widely-deployed Video Multimethod Assessment Fusion (VMAF) model. We show that VMAF+HDRMAX provides significantly elevated performance on both HDR and SDR videos, exceeding prior state-of-the-art model performance. The database is now accessible at: https://live.ece.utexas.edu/research/LIVEHDR/LIVEHDR_index.html. The model will be made available at a later date at: https://live.ece.utexas.edu//research/Quality/index_algorithms.htm.

Quality assessment of online osteosarcoma-related videos in Mainland China: A cross-sectional study.

Information about orthopedics diseases on the Internet has not been extensively assessed. Our purpose was to evaluate the quality of online information of osteosarcoma on current video-sharing platforms in mainland China. TikTok and Bilibili were independently queried from June to July 2023 by four independent researchers using the Microsoft Edge web browser. Information about the videos and creators was recorded, and descriptive analyses were conducted. After data extraction, a total of 95 videos were included, in which 43 videos were uploaded by certified doctors (45.3%), with 35 videos (36.8%) uploaded by certified orthopedic surgeons. Of the content of these videos, 78.9% were introduction (n = 75), 64.2% were on professional knowledge (n = 61), 28.4% were on treatment (n = 27), while 5.3% were on surgical techniques (n = 5). The mean DISCERN total score was 43.8 ± 13.4, and the mean JAMA score was 3.8 ± 0.3. Videos about osteosarcoma on current video-sharing platforms were extensive, but were not comprehensive and professional. Although current online videos have the potential to improve public awareness on osteosarcoma, due to their quality and content, were not assessed to be good sources for medical education.

Tucker Decomposition and Log-Gabor Feature-Based Quality Assessment for the Screen Content Videos

Tucker Decomposition and Log-Gabor Feature-Based Quality Assessment for the Screen Content Videos

Subjective Quality Assessment of Compressed Tone-Mapped High Dynamic Range Videos.

High Dynamic Range (HDR) videos are able to represent wider ranges of contrasts and colors than Standard Dynamic Range (SDR) videos, giving more vivid experiences. Due to this, HDR videos are expected to grow into the dominant video modality of the future. However, HDR videos are incompatible with existing SDR displays, which form the majority of affordable consumer displays on the market. Because of this, HDR videos must be processed by tone-mapping them to reduced bit-depths to service a broad swath of SDR-limited video consumers. Here, we analyze the impact of tone-mapping operators on the visual quality of streaming HDR videos. To this end, we built the first large-scale subjectively annotated open-source database of compressed tone-mapped HDR videos, containing 15,000 tone-mapped sequences derived from 40 unique HDR source contents. The videos in the database were labeled with more than 750,000 subjective quality annotations, collected from more than 1,600 unique human observers. We demonstrate the usefulness of the new subjective database by benchmarking objective models of visual quality on it. We envision that the new LIVE Tone-Mapped HDR (LIVE-TMHDR) database will enable significant progress on HDR video tone mapping and quality assessment in the future. To this end, we make the database freely available to the community at https://live.ece.utexas.edu/research/LIVE TMHDR/index.html.