Objective Video Quality Assessment Methods Research Articles

Spatial–Temporal Analysis-Based Video Quality Assessment: A Two-Stream Convolutional Network Approach

In system processing, video inevitably suffers from distortion, which leads to quality degradation and affects the user experience. Therefore, it is of great importance to design an accurate and effective objective video quality assessment (VQA) method. In this paper, by considering the multi-dimensional characteristics for video and visual perceptual mechanism, a two-stream convolutional network for VQA is proposed based on spatial–temporal analysis, named TSCNN-VQA. Specifically, for feature extraction, TSCNN-VQA first extracts spatial and temporal features by two different convolutional neural network branches, respectively. After that, the spatial–temporal joint feature fusion is constructed to obtain the joint spatial–temporal features. Meanwhile, the TSCNN-VQA also integrates an attention module to guarantee that the process conforms to the mechanism that the visual system perceives video information. Finally, the overall quality is obtained by non-linear regression. The experimental results in both the LIVE and CSIQ VQA datasets show that the performance indicators obtained by TSCNN-VQA are higher than those of existing VQA methods, which demonstrates that TSCNN-VQA can accurately evaluate video quality and has better consistency with the human visual system.

Objective Video Quality Assessment Method for Object Recognition Tasks

In the field of video quality assessment for object recognition tasks, accurately predicting the impact of different quality factors on recognition algorithms remains a significant challenge. Our study introduces a novel evaluation framework designed to address this gap by focussing on machine vision rather than human perceptual quality metrics. We used advanced machine learning models and custom Video Quality Indicators to enhance the predictive accuracy of object recognition performance under various conditions. Our results indicate a model performance, achieving a mean square error (MSE) of 672.4 and a correlation coefficient of 0.77, which underscores the effectiveness of our approach in real-world scenarios. These findings highlight not only the robustness of our methodology but also its potential applicability in critical areas such as surveillance and telemedicine.

Inverse-tone-mapped HDR video quality assessment: A new dataset and benchmark

The accurate quality assessment for inverse-tone-mapped videos (ITMVs) has become increasingly important due to the ongoing rapid development of inverse tone mapping (ITM) algorithms. We observed that most existing video quality assessment (VQA) datasets and objective VQA methods are based on two underlying assumptions: the videos are in standard dynamic range (SDR) domain and the distortions are introduced in the video degradation process. However, ITMVs are videos in high dynamic range (HDR) domain, and are accompanied by distortions introduced during the enhancement process, which are distinct from those in degradation process. We then pose the question: Can existing VQA methods objectively assess the quality of ITMVs? To address this question and foster the advancement of VQA methods, we contribute the first publicly available dataset specifically tailored for assessing the quality of ITMVs. The dataset is called “VQA4ITM”, which is an acronym for “Video Quality Assessment for Inverse Tone Mapping”. Specifically, 30 distortion-free, professional 4K Rec. 2020 10 bit 50fps HDR source videos and 12 ITM methods were carefully selected, and a total of 360 ITMVs were generated in the end. A subjective experiment was conducted in a professional HDR viewing environment and 6,098 valid quality scores were finally gathered to calculate the mean opinion score (MOS) of each ITMV. Furthermore, we perform benchmark experiments on the VQA4ITM dataset, revealing that existing VQA methods are unable to fairly assess the quality of ITMVs. The inadequate performance of existing VQA methods on the VQA4ITM dataset highlights the challenging nature of our dataset, as well as the imperative need for the development of more effective VQA algorithms in the context of ITM. To facilitate future research, the VQA4ITM dataset will be made publicly accessible.

Objective Video Quality Assessment Method for Face Recognition Tasks

Nowadays, there are many metrics for overall Quality of Experience (QoE), both those with Full Reference (FR), such as Peak Signal-to-Noise Ratio (PSNR) or Structural Similarity (SSIM), and those with No Reference (NR), such as Video Quality Indicators (VQI), which are successfully used in video processing systems to evaluate videos whose quality is degraded by different processing scenarios. However, they are not suitable for video sequences used for recognition tasks (Target Recognition Videos, TRV). Therefore, correctly estimating the performance of the video processing pipeline in both manual and Computer Vision (CV) recognition tasks is still a major research challenge. There is a need for objective methods to evaluate video quality for recognition tasks. In response to this need, we show in this paper that it is possible to develop the new concept of an objective model for evaluating video quality for face recognition tasks. The model is trained, tested and validated on a representative set of image sequences. The set of degradation scenarios is based on the model of a digital camera and how the luminous flux reflected from the scene eventually becomes a digital image. The resulting degraded images are evaluated using a CV library for face recognition as well as VQI. The measured accuracy of a model, expressed as the value of the F-measure parameter, is 0.87.

Evaluation of objective video quality assessment methods on video sequences with different spatial and temporal activity encoded at different spatial resolutions

With the development of Video on Demand applications due to the availability of high-speed internet access, adaptive streaming algorithms have been developing and improving. The focus is on improving user’s Quality of Experience (QoE) and taking it into account as one of the parameters for the adaptation algorithm. Users often experience changing network conditions, so the goal is to ensure stable video playback with satisfying QoE level. Although subjective Video Quality Assessment (VQA) methods provide more accurate results regarding user’s QoE, objective VQA methods cost less and are less time-consuming. In this article, nine different objective VQA methods are compared on a large set of video sequences with various spatial and temporal activities. VQA methods used in this analysis are: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), MultiScale Structural Similarity Index (MS-SSIM), Video Quality Metric (VQM), Mean Sum of Differences (DELTA), Mean Sum of Absolute Differences (MSAD), Mean Squared Error (MSE), Netflix Video Multimethod Assessment Fusion (Netflix VMAF) and Visual Signal-to-Noise Ratio (VSNR). The video sequences used for testing purposes were encoded according to H.264/AVC with twelve different target coding bitrates, at three different spatial resolutions (resulting in a total of 190 sequences). In addition to objective quality assessment, subjective quality assessment was performed for these sequences. All results acquired by objective VQA methods have been compared with subjective Mean Opinion Score (MOS) results using Pearson Linear Correlation Coefficient (PLCC). Measurement results obtained on a large set of video sequences with different spatial resolutions show that VQA methods like SSIM and VQM correlate better with MOS results compared to PSNR, SSIM, VSNR, DELTA, MSE, VMAF and MSAD. However, the PLCC results for SSIM and VQM are too low (0.7799 and 0.7734, respectively), for the usage of these methods in streaming services instead of subjective testing. These results suggest that more efficient VQA methods should be developed to be used in streaming testing procedures as well as to support the video segmentation process. Furthermore, when comparing results obtained for different spatial resolutions, it can be concluded that the quality of video sequences encoded at lower spatial resolutions in cases of lower target coding bitrate is higher compared to the quality of video sequences encoded at higher spatial resolutions at the same target coding bitrate, particularly when video sequences with higher spatial and temporal information are used.

IJNM Special Issue—International Journal of Network Management: QoE‐centric analysis and management of communication networks

IJNM Special Issue—International Journal of Network Management: QoE‐centric analysis and management of communication networks

Stochastic QoE-aware optimization of multisource multimedia content delivery for mobile cloud

The increasing popularity of mobile video streaming in wireless networks has stimulated growing demands for efficient video streaming services. However, due to the time-varying throughput and user mobility, it is still difficult to provide high quality video services for mobile users. Our proposed optimization method considers key factors such as video quality, bitrate level, and quality variations to enhance quality of experience over wireless networks. The mobile network and device parameters are estimated in order to deliver the best quality video for the mobile user. We develop a rate adaptation algorithm using Lyapunov optimization for multi-source multimedia content delivery to minimize the video rate switches and provide higher video quality. The multi-source manager algorithm is developed to select the best stream based on the path quality for each path. The node joining and cluster head election mechanism update the node information. As the proposed approach selects the optimal path, it also achieves fairness and stability among clients. The quality of experience feature metrics like bitrate level, rebuffering events, and bitrate switch frequency are employed to assess video quality. We also employ objective video quality assessment methods like VQM, MS-SSIM, and SSIMplus for video quality measurement closer to human visual assessment. Numerical results show the effectiveness of the proposed method as compared to the existing state-of-the-art methods in providing quality of experience and bandwidth utilization.

An objective and subjective quality assessment study of passive gaming video streaming

SummaryPassive gaming video‐streaming applications have recently gained much attention as evident with the rising popularity of many Over The Top (OTT) providers such as Twitch.tv and YouTube Gaming. For the continued success of such services, it is imperative that the user Quality of Experience (QoE) remains high, which is usually assessed using subjective and objective video quality assessment methods. Recent years have seen tremendous advancement in the field of objective video quality assessment (VQA) metrics, with the development of models that can predict the quality of the videos streamed over the Internet. A study on the performance of objective VQA on gaming videos, which are artificial and synthetic and have different streaming requirements than traditionally streamed videos, is still missing. Towards this end, we present in this paper an objective and subjective quality assessment study on gaming videos considering passive streaming applications. Subjective ratings are obtained for 90 stimuli generated by encoding six different video games in multiple resolution‐bitrate pairs. Objective quality performance evaluation considering eight widely used VQA metrics is performed using the subjective test results and on a data set of 24 reference videos and 576 compressed sequences obtained by encoding them in 24 resolution‐bitrate pairs. Our results indicate that Video Multimethod Assessment Fusion (VMAF) predicts subjective video quality ratings the best, while Naturalness Image Quality Evaluator (NIQE) turns out to be a promising alternative as a no‐reference metric in some scenarios.

A framework for computationally efficient video quality assessment

Objective video quality assessment (VQA) methods are essentially algorithms that estimate video quality. Recent quality assessment methods aim to provide quality predictions that are well correlated with subjective quality scores. However, most of these methods are computationally costly, which limits their use in real-time applications. A possible solution to this problem is to decrease the video resolution (spatial, temporal or both) in order to reduce the amount of processed data. Although reducing the video resolution is a simple way of decreasing the running time of a VQA method, this approach might impact the prediction accuracy of the VQA method. In this paper, we analyze this impact. More specifically, we analyze the effects of resolution reduction on the performance of the VQA methods. Based on this analysis, we propose a framework that decreases the overall processing time of VQA methods, without decreasing significantly the performance accuracy. We test the framework using six different VQA methods and four different video quality databases. Results show that the proposed framework reduces the average runtime performance of the tested VQA methods, without considerably altering their performance accuracy.

Study of the Subjective Visibility of Packet Loss Artifacts in Decoded Video Sequences

Packet loss is a significant cause of visual impairments in video broadcasting over packet-switched networks. There are several subjective and objective video quality assessment methods focused on the overall perception of video quality. However, less attention has been paid on the visibility of packet loss artifacts appearing in spatially and temporally limited regions of a video sequence. In this paper, we present the results of a subjective study, using a methodology where a video sequence is displayed on a touchscreen and the users tap it in the positions where they observe artifacts. We also analyze the objective features derived from those artifacts, and propose different models for combining those features into an objective metric for assessing the noticeability of the artifacts. The practical results show that the proposed metric predicts visibility of packet loss impairments with a reasonable accuracy. The proposed method can be applied for developing packetization and error recovery schemes to minimize the subjectively experienced distortion in error-prone networked video systems.

Objective Video Quality Assessment Based on Machine Learning for Underwater Scientific Applications.

Video services are meant to be a fundamental tool in the development of oceanic research. The current technology for underwater networks (UWNs) imposes strong constraints in the transmission capacity since only a severely limited bitrate is available. However, previous studies have shown that the quality of experience (QoE) is enough for ocean scientists to consider the service useful, although the perceived quality can change significantly for small ranges of variation of video parameters. In this context, objective video quality assessment (VQA) methods become essential in network planning and real time quality adaptation fields. This paper presents two specialized models for objective VQA, designed to match the special requirements of UWNs. The models are built upon machine learning techniques and trained with actual user data gathered from subjective tests. Our performance analysis shows how both of them can successfully estimate quality as a mean opinion score (MOS) value and, for the second model, even compute a distribution function for user scores.

A Video Quality Assessment Method for VoIP Applications Based on User Experience

An objective video quality assessment method is proposed to evaluate the video quality in voice over internet protocol (VoIP) applications under network distortion. The fluency and the clarity of videos are two main parts of the factors that affect user experience, thus the method evaluates these two parts to assess the distortions of videos in VoIP applications caused by codec and packet loss. The clarity of the video is measured by calculating block artifacts and frame blurring. Video blocking artifacts are measured by splitting the picture into small blocks and calculating the difference of the pixels around each border while video blurring is measured by getting edge information through Sobel operator, and counting the gradient histogram. Then the video clarity can be measured by a weighted sum of block artifacts score and blurring score using linear regression. The scores are also normalized in order to eliminate the impact of different video contents. The video fluency is calculated by counting the wrong frame in the video. Finally, a weighted sum of video clarity score and video fluency score can represent the quality of the video. The experimental results show that the objective quality scores have a strong correlation with the subjective quality scores, and the algorithm concludes two parts of user experience other than just image quality, which is more comprehensive and it can be used in video quality assessment in VoIP applications.

Reduced frame set on wireless distorted video for quality assessment

Objective video quality assessment methods often evaluate all the frames regardless of their importance. For wireless distorted videos, not every frame has the same contribution to the final overall quality due to the channel fading and interference, which may lead to the capacity variation in temporal. Besides, with the content similarity and error propagation pattern in temporal domain, it is possible to evaluate the overall quality with only part of the frames. In this paper, a demonstration is performed to show that the video quality can be evaluated with reduced frames set (RFS), and a state transition model is proposed to extract the RFS. At last, a video quality assessment (VQA) method is carried out based on RFS. Compared with several state-of-the-art methods, our method can achieve a suitable accuracy with less frames to be processed.

A No Reference Video Quality Metric Based on Jerkiness Estimation Focusing on Multiple Frame Freezing in Video Streaming

ABSTRACTIn wireless networks, due to limited bandwidth and packet losses, seamless and ubiquitous delivery of high-quality video streaming services is a big challenge for the operators. In order to improve the process of online video quality monitoring, the presence of no reference (NR) objective video quality assessment (VQA) methods is required. In some networks, the video decoder on the reception side adopts a mechanism in which last correctly received frame is frozen and displayed on video display terminal until the next correct frame is received. This phenomenon, employed as an error concealment technique, can cause a perceptual jerkiness on the video display terminal. In this paper, we have proposed an enhanced model of objective VQA based on the estimation of jerkiness. A study of three contemporary NR methods, used for objective VQA and online monitoring of videos, has been included along with subjective VQA tests. The subjective tests were performed for a set of video sequences with specific spatial and temporal information. The proposed NR method is based on our careful observations from the subjective test results and our main focus is to cater the effect of multiple frame freeze impairments in video steaming. Comparison with other NR methods shows that the proposed method performs better, in terms of estimating the impact of multiple frame freezing impairments, and has more affinity with the subjective test results.

Subjective and Objective Video Quality Assessment of 3D Synthesized Views With Texture/Depth Compression Distortion.

The quality assessment for synthesized video with texture/depth compression distortion is important for the design, optimization, and evaluation of the multi-view video plus depth (MVD)-based 3D video system. In this paper, the subjective and objective studies for synthesized view assessment are both conducted. First, a synthesized video quality database with texture/depth compression distortion is presented with subjective scores given by 56 subjects. The 140 videos are synthesized from ten MVD sequences with different texture/depth quantization combinations. Second, a full reference objective video quality assessment (VQA) method is proposed concerning about the annoying temporal flicker distortion and the change of spatio-temporal activity in the synthesized video. The proposed VQA algorithm has a good performance evaluated on the entire synthesized video quality database, and is particularly prominent on the subsets which have significant temporal flicker distortion induced by depth compression and view synthesis process.

4-13 ベースバンド信号解析に基づくNo Reference型客観画質評価方式の検討(第4部門 メディア工学2)

4-13 ベースバンド信号解析に基づくNo Reference型客観画質評価方式の検討(第4部門 メディア工学2)

Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison

With the increasing demand for video-based applications, the reliable prediction of video quality has increased in importance. Numerous video quality assessment methods and metrics have been proposed over the past years with varying computational complexity and accuracy. In this paper, we introduce a classification scheme for full-reference and reduced-reference media-layer objective video quality assessment methods. Our classification scheme first classifies a method according to whether natural visual characteristics or perceptual (human visual system) characteristics are considered. We further subclassify natural visual characteristics methods into methods based on natural visual statistics or natural visual features. We subclassify perceptual characteristics methods into frequency or pixel-domain methods. According to our classification scheme, we comprehensively review and compare the media-layer objective video quality models for both standard resolution and high definition video. We find that the natural visual statistics based MultiScale-Structural SIMilarity index (MS-SSIM), the natural visual feature based Video Quality Metric (VQM), and the perceptual spatio-temporal frequency-domain based MOtion-based Video Integrity Evaluation (MOVIE) index give the best performance for the LIVE Video Quality Database.

Video Quality Assessment Using Spatio-Velocity Contrast Sensitivity Function

Due to the development and popularization of high-definition televisions, digital video cameras, Blu-ray discs, digital broadcasting, IP television and so on, it plays an important role to identify and quantify video quality degradations. In this paper, we propose SV-CIELAB which is an objective video quality assessment (VQA) method using a spatio-velocity contrast sensitivity function (SV-CSF). In SV-CIELAB, motion information in videos is effectively utilized for filtering unnecessary information in the spatial frequency domain. As the filter to apply videos, we used the SV-CSF. It is a modulation transfer function of the human visual system, and consists of the relationship among contrast sensitivities, spatial frequencies and velocities of perceived stimuli. In the filtering process, the SV-CSF cannot be directly applied in the spatial frequency domain because spatial coordinate information is required when using velocity information. For filtering by the SV-CSF, we obtain video frames separated in spatial frequency domain. By using velocity information, the separated frames with limited spatial frequencies are weighted by contrast sensitivities in the SV-CSF model. In SV-CIELAB, the criteria are obtained by calculating image differences between filtered original and distorted videos. For the validation of SV-CIELAB, subjective evaluation experiments were conducted. The subjective experimental results were compared with SV-CIELAB and the conventional VQA methods such as CIELAB color difference, Spatial-CIELAB, signal to noise ratio and so on. From the experimental results, it was shown that SV-CIELAB is a more efficient VQA method than the conventional methods.

A Study about the Relationship between Frame Quality and Single Video Quality

Digital video encapsulates the time series of a frame (still) images, where overall video quality can be obtained by using the quality of each frame image and the temporal information between the frame image. Coding of video produces degradation of these two types of information. These degradations can be classified as spatial degradation (static degradation) of a frame images and temporal degradation between frame image (dynamic degradation). In the framework of video quality evaluation it is necessary to consider those degradations, because their contents are strongly interdependable and quantification is problematic for these degradations. Therefore, the development of an objective video quality assessment method for single video quality requires to investigate how much static degradation and dynamic degradation affect single video quality. In this research, single video quality was predicted highly accuratly by using frame quality as static degradation and frame rate information as dynamic degradation.

Proposal for an objective video quality assessment method that takes temporal and spatial information into consideration

With the proliferation of high-speed IP connection service, video communication service has been attracting attention. In order to realize high-quality video communication service, it is important to carry out appropriate quality design prior to providing service and to adequately monitor and administer the service. To this end, it is necessary to establish a quality evaluation technique that appropriately expresses the quality enjoyed by the user and evaluates the video quality appropriately and rapidly. In this paper, a video quality assessment method is proposed for coding distortions at a wide range of bit rates, assuming use in a PC vision environment. The proposed method performs evaluation with weighting that takes account of blurring of the edge portion, generation of new edges, and deterioration in the time axis direction. The estimation accuracy of the present objective quality evaluation method is confirmed by adding coding distortions to the input video of 36 standard video scenes. Accuracy similar to the fluctuations of the subjective quality assessment is obtained, indicating that the method is sufficient for objective video quality assessment. © 2006 Wiley Periodicals, Inc. Electron Comm Jpn Pt 1, 89(12): 97– 108, 2006; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecja.20265