Subjective Video Quality Assessment Research Articles

Impact of the Stimulus Presentation Structure on Subjective Video Quality Assessment

A study of video quality perceived by user (Quality of Experience, QoE) was conducted with an examination of whether and how the order and structure of the video sequence presentation affects subjective assessment. For this purpose, the influence of content variability/repeatability, the quality of the preceding sequence, and the sequence order were analyzed. Observations on the correlation of QoE with the micro-structure of sequence presentation are described, which can be the basis for hypotheses of the dependence of QoE assessment on the abovementioned factors. The observed relationships regarding the influence of the number of video repetitions and the impact of the predecessor’s quality on subjective evaluation are consistent with research work on the influence of the order/arrangement and structure of research stimuli on the results of subjective evaluations. Areas for further research are indicated, including relating obtained results in the area of QoE to other cognitive sciences, such as psychological, medical, and economic sciences.

Speeding Up Subjective Video Quality Assessment via Hybrid Active Learning

Subjective video quality assessment (VQA) is the most reliable way to get accurate quality scores, providing first-hand data on the research of quality of experience (QoE). However, traditional subjective VQA has the disadvantage of being cumbersome and time-consuming. In this paper, we propose an efficient subjective VQA framework based on hybrid information and active learning, named HA-SVQA. Built on the principle of active learning for data annotation, HA-SVQA allows iterative assessments on the most valuable or informative videos, which are selected based on hybrid information from the subject’s prior decisions and the objective quality predictions. By eliminating the redundant (or less valuable) videos to be assessed, HA-SVQA can speed up the process of subjective VQA. Concretely, our framework starts with a few quality-known videos to initialize dual-regression models. It then uses a scoring-difference stratified sampling strategy to iteratively select the next group of videos to be assessed, which contain high quality uncertainty. The newly scored videos are used to continually update every part of our framework. By this way, subjective VQA can be stopped early while meeting the acceptable goal of a full-time subjective study. We conducted simulation experiments on three different datasets: LIVE, LIVE VQC, and an underwater video quality dataset. The results show that HA-SVQA can effectively speed up the process of subjective VQA and reduce about 1/3 of the human workload or time cost in the presence of data redundancy. In order to investigate the effectiveness of HA-SVQA in more depth, we conducted a field experiment with deep-sea videos. We found that HA-SVQA is still effective in reducing about 1/3 of the overall human workload, which is consistent with the conclusion of simulation experiments. Finally, we discussed some of the factors that potentially affect the QoE modelling and the subjective VQA.

What do users care about? Research on user behavior of mobile interactive video advertising

Mobile interactive video advertising is a relatively new type of media. A combination of interactivity and narrative is used to tell the video story to the audience. This study studied in three steps on mobile interactive video advertising to determine the design basis that should be paid attention to. Firstly, the research framework is proposed through expert interview and qualitative coding of the interview text. Secondly, research samples are made through surveys and conference discussions. The samples were designed into 30 types containing various interactive narrative and involvement. Finally, the qualitative results are verified in a quantitative way. The results show that in order to improve the audience's purchase intention, more attention should be paid to interactive narrative, subjective video quality assessment, immersion, satisfaction, and product involvement.

Study of Subjective and Objective Quality Assessment of Night-Time Videos

With the widespread usage of video capture devices and social media videos, videos are dominating the multimedia landscape. There is an emerging need for video quality assessment (VQA) that forms the backbone of advanced video systems. Night-time videos play an important role in user capturing, hence being able to accurately assess their quality is critical. However, the characteristics of night-time videos differ from those of general in-capture videos; and VQA algorithms that have been developed for general-purpose videos cannot accurately assess the quality of night-time videos. Research is needed to gain a better understanding of how humans perceive the quality of night-time videos, and use this new understanding to develop reliable VQA algorithms. To this end, we construct a large-scale night-time VQA database, namely Mobile In-capture Night-time Database for Video Quality (MIND-VQ), containing 1181 night-time videos, 435 subjects, and over 130000 opinion scores. We perform thorough analyses to reveal subjective quality assessment behaviors of night-time videos. Furthermore, we propose a new VQA model, namely Visibility-based Night-time Video Quality Assessment Network, VINIA. Spatial and temporal visibility-aware components are characterized to reflect properties of human perception of night-time VQA task. A series of experiments are conducted to compare our VINIA with other existing VQA algorithms using our new MIND-VQ database and other public VQA databases. Experimental results show that our subjective VQA database provides new insights and our new VINIA model achieves superior performance in accessing night-time video quality.

Evaluation of objective video quality assessment methods on video sequences with different spatial and temporal activity encoded at different spatial resolutions

With the development of Video on Demand applications due to the availability of high-speed internet access, adaptive streaming algorithms have been developing and improving. The focus is on improving user’s Quality of Experience (QoE) and taking it into account as one of the parameters for the adaptation algorithm. Users often experience changing network conditions, so the goal is to ensure stable video playback with satisfying QoE level. Although subjective Video Quality Assessment (VQA) methods provide more accurate results regarding user’s QoE, objective VQA methods cost less and are less time-consuming. In this article, nine different objective VQA methods are compared on a large set of video sequences with various spatial and temporal activities. VQA methods used in this analysis are: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), MultiScale Structural Similarity Index (MS-SSIM), Video Quality Metric (VQM), Mean Sum of Differences (DELTA), Mean Sum of Absolute Differences (MSAD), Mean Squared Error (MSE), Netflix Video Multimethod Assessment Fusion (Netflix VMAF) and Visual Signal-to-Noise Ratio (VSNR). The video sequences used for testing purposes were encoded according to H.264/AVC with twelve different target coding bitrates, at three different spatial resolutions (resulting in a total of 190 sequences). In addition to objective quality assessment, subjective quality assessment was performed for these sequences. All results acquired by objective VQA methods have been compared with subjective Mean Opinion Score (MOS) results using Pearson Linear Correlation Coefficient (PLCC). Measurement results obtained on a large set of video sequences with different spatial resolutions show that VQA methods like SSIM and VQM correlate better with MOS results compared to PSNR, SSIM, VSNR, DELTA, MSE, VMAF and MSAD. However, the PLCC results for SSIM and VQM are too low (0.7799 and 0.7734, respectively), for the usage of these methods in streaming services instead of subjective testing. These results suggest that more efficient VQA methods should be developed to be used in streaming testing procedures as well as to support the video segmentation process. Furthermore, when comparing results obtained for different spatial resolutions, it can be concluded that the quality of video sequences encoded at lower spatial resolutions in cases of lower target coding bitrate is higher compared to the quality of video sequences encoded at higher spatial resolutions at the same target coding bitrate, particularly when video sequences with higher spatial and temporal information are used.

Neural network-based assessment of the impact induced in video quality assessment by the semantic labels

Subjective video quality assessment generally comes across with semantically labeled evaluation scales (e.g. Excellent, Good, Fair, Poor and Bad on a single stimulus, 5 level grading scale). While suspicions about an eventual bias these labels induce in the quality evaluation always occur, to the best of our knowledge, very few state-of-the-art studies target an objective assessment of such an impact. Our study presents a neural network solution in this respect. We designed a 5-class classifier, with 2 hidden layers, and a softmax output layer. An ADAM optimizer coupled to a Sparse Categorical Cross Entropy function is subsequently considered. The experimental results are obtained out of processing a database composed of 440 observers scoring about 7 hours of video content of 4 types (high-quality stereoscopic video content, low-quality stereoscopic video content, high-quality 2D video, and low-quality 2D video). The experimental results are discussed and confrontment to the reference given by a probability-based estimation method. They show an overall good convergence between the two types of methods while pointing out to some inner applicative differences that are discussed and explained.

Subjective Video QoE With a Hybrid Cognitive-Motor Task

This paper reports on the subjective video quality analysis with and without a parallel task. It aims to understand how the quality perceived by the human observer brain is affected when the subject is not entirely focused on the judgment task. For this purpose, two subjective video quality tests were performed according to the ITU-T P.910 recommendation - Subjective video quality assessment methods for multimedia applications. They consisted of two parts involving a total of 24 subjects. Unlike the first test, the second test included an additional (parallel) psychomotor task where subjects had to sort different colors of pasta into three bowls. Various analyses were provided, including preliminary analysis, group analysis, and RMSE analysis. Surprisingly, the results show that not all the spectrum of quality is affected equally by parallel task introduction.

Standards column: VQEG

The Video Quality Experts Group (VQEG) was born from a need to bring together experts in subjective video quality assessment and objective quality measurement. The first VQEG meeting, held in Turin in 1997, was attended by a small group of experts drawn from ITU-T and ITU-R Study Groups. VQEG was first grounded in basic subjective methodology and objective tool development/verification for video quality assessment such that the industry could be moved forward with standardization and implementation. At the beginning it was focused around measuring the perceived video quality since the distribution path for video and audio were limited and known. Over the last 20 years from the formation of VQEG the ecosystem has changed dramatically and thus so must the work. Multimedia is now pervasive on all devices and methods of distribution from broadcast to cellular data networks. This shift has the expertise within VQEG to move from the visual (no-audio) quality of video to Quality of Experience (QoE). The march forward of technologies means that VQEG needs to react and be on the leading edge of developing, defining and deploying methods and tools that help address these new technologies and move the industry forward. This also means that we need to embrace both qualitative and quantitative ways of defining these new spaces and terms. Taking a holistic approach to QoE will enable VQEG to drive forward and faster with unprecedented collaboration and execution. VQEG is open to all interested from industry, academia, government organizations and Standard-Developing Organizations (SDOs). There are no fees involved, no membership applications and no invitations are needed to participate in VQEG activities. Subscription to the main VQEG email list (ituvidq@its.bldrdoc.gov) constitutes membership in VQEG. VQEG conducts work via discussions over email reflectors, regularly scheduled conference calls and, in general, two face-to-face meetings per year. There are currently more than 500 people registered across 11 email reflectors, including a main reflector for general announcements relevant to the entire group, and different project reflectors dedicated to technical discussions of specific projects. A LinkedIn group exists as well.

No-reference artifacts measurements based video quality metric

In multimedia delivery, perceived quality of video signal has a significant role in overall users Quality of Experience (QoE). Therefore, multimedia service providers must constantly measure and monitor perceived video quality, which is usually performed using no-reference (NR) video quality metrics. In this paper, a novel NR objective video quality metric named Artifacts Measurements Based Video Quality Metric (AMB-VQM) is proposed. In addition to artifacts measures computed by blocking (BL), packet-loss (PL), and freezing (FZ) artifacts detection algorithms, the metric incorporates artifacts masking based on spatial and temporal video content complexity. Furthermore, a newly created FERIT-RTRK-2 video quality database, which contains 486 Full HD test video sequences impaired by video compression (MPEG-2, H.264/AVC, and H.265), packet-loss, frame freezing, and combinations of these procedures, is presented in this paper. FERIT-RTRK-2 database is publicly available for scientific community at http://www.rt-rk.com/other/VideoDBReadme.html. Additionally, newly created user-friendly subjective video quality assessment tool with graphical user interface, which can be used for conducting of subjective video quality experiments, is presented. In experimental part the performance of the proposed AMB-VQM is compared to this of 10 other objective video quality metrics (PSNR, SSIM, VSNR, PSNRHVS, PSNRHVSM, VIFP, ViS3, ST-MAD, BRISQUE, VIIDEO) using distorted video sequences from four different video quality databases: CSIQ, LIVE, FERIT-RTRK and FERIT-RTRK-2. The results show that AMB-VQM achieves high performance when predicting the video quality for videos distorted in different manners. AMB-VQM results outperform the results of most of the analyzed popular and very often used video quality metrics.

Crowdsourced subjective 3D video quality assessment

This article proposes a new method for subjective 3D video quality assessment based on crowdsourced workers—Crowd3D. The limitations of traditional laboratory-based grade collection procedures are outlined, and their solution through the use of a crowd-based approach is described. Several conceptual and technical requirements of crowd-based 3D video quality assessment methods are identified and the solutions adopted described in detail. The system built takes the form of a web-based platform that supports 3D video monitors, and orchestrates the entire process of observer validation, content presentation and quality, depth, and comfort grade recording in a remote database. The crowdsourced subjective 3D quality assessment system uses as source contents a set of 3D video and grades database assembled earlier in a laboratory setting. To evaluate the validity of the crowd-based approach, the grades gathered using the crowdsourced system were analysed and compared to a set of grades obtained in laboratory settings using the same data set. Results show that it is possible to obtain Pearson’s and Spearman’s correlation up to 0.95 for quality Difference Mean Opinion Score and 0.96 for quality Mean Opinion Score, when comparing with laboratory grades. Apart from the present study, the 3D video quality assessment platform proposed can be used with advantage for further related research activities, reducing the time and cost compared to the traditional laboratory-based quality assessments.

Performance analysis of a video quality ruler methodology for subjective quality assessment

Objective video quality metrics are designed to be as reliable as the subjective quality assessments on which they are calibrated and validated. However, existing standard methodologies for subjective video quality assessment provide low reliable results for some conditions. We investigate whether an extension of the quality ruler experimental methodology, originally defined for images and shown to be more reliable than, e.g., standard single stimulus (SS) methods, can be adapted to reliably assess the quality of videos. The video quality ruler methodology allows subjects to assess video quality using a set of reference anchor images (the ruler), spanning a wide range of quality altogether, but closely spaced in function of quality one from the other. Subjects are asked to compare the quality of the displayed test video with the quality of these anchor images, displayed on a tablet, and indicate which of the reference images matches in quality the test video. As a result, the video quality assessment task is reduced to a set of visual comparisons between video and reference image quality. We describe how to adapt the original quality ruler methodology to video quality assessment, and we compare the proposed methodology with two other, widely used experimental methodologies: the single stimulus (SS) and the double stimulus (DS) method. Our results show that video quality ruler is a reliable method to assess video quality according to a multitude of criteria.

Influence of Bit Depth on Subjective Video Quality Assessment for High Resolutions

This paper deals with the influence of bit depth on the subjective video quality assessment. To achieve this goal, eight video sequences, each representing a different content prototype, were analysed. Subjective evaluation was performed using the ACR method. The analysed video sequences were encoded to 8 and 10-bit bit depth. Two most used compression standards H.264 and H.265 were evaluated with 1, 3, 5, 10 and 15 Mbps bitrate in Full HD and UHD resolution. Finally, the perceived quality of both compression standards using the subjective tests with emphasis on bit-depth was compared. From the results we can state, that the practical application of 10-bit bit depth is not appropriate for Full HD resolution in the range of bitrate from 1 to 15 Mbps, for Ultra HD resolution, it is appropriate only for videos encoded by H.265/HEVC compression standard.

Unsupervised deep learning for real-time assessment of video streaming services

Evaluating quality of experience in video streaming services requires a quality metric that works in real time and for a broad range of video types and network conditions. This means that, subjective video quality assessment studies, or complex objective video quality assessment metrics, which would be best suited from the accuracy perspective, cannot be used for this tasks (due to their high requirements in terms of time and complexity, in addition to their lack of scalability). In this paper we propose a light-weight No Reference (NR) method that, by means of unsupervised machine learning techniques and measurements on the client side is able to assess quality in real-time, accurately and in an adaptable and scalable manner. Our method makes use of the excellent density estimation capabilities of the unsupervised deep learning techniques, the restricted Boltzmann machines, and light-weight video features computed just on the impaired video to provide a delta of quality degradation. We have tested our approach in two network impaired video sets, the LIMP and the ReTRiEVED video quality databases, benchmarking the results of our method against the well-known full reference metric VQM. We have obtained levels of accuracy of at least 85% in both datasets using all possible cases.

Performance of Four Subjective Video Quality Assessment Protocols and Impact of Different Rating Preprocessing and Analysis Methods

Standardization bodies recommend various protocols to conduct subjective quality assessment (QA) of imaging systems. While many studies have compared these QA protocols, few have assessed the impact of different approaches for preprocessing and analyzing quality ratings. Furthermore, the effect of large versus small quality differences on the discrimination performance of protocols has not been extensively studied in video QA. This study examines these issues for four QA protocols. H.264 compressed medical videos and denoised natural scene videos were evaluated by expert and naive subjects. Scores were collected with four QA protocols—forced choice (FC), two ratio-scaled paired comparison methods: preference (Pref) and dissimilarity (Dissim), and single stimulus (SS)—and analyzed using combinations of different rating preprocessing approaches, generating a total of 14 methods. Performance metrics—probability and effect size—quantified the ability of the methods to discriminate between quality levels. The Pref and Dissim methods analyzed with classical multidimensional scaling and the SS method with Z-score transformation consistently outperformed the other methods. The type of preprocessing introduced large differences in the performance of individual protocols. Grouping stimuli pairs by small and large quality differences introduced significant differences in the performance rankings of the methods, with Pref and Dissim being most sensitive to small quality differences. For the future, we suggest further validation of the FC method, due to its simplicity and ease of use, and continued investigation into more robust raw scores transformation and statistical analysis methods for both SS and FC ratings.

Perceived Quality of Full HD Video - Subjective Quality Assessment

In recent years, an interest in multimedia services has become a global trend and this trend is still rising. The video quality is a very significant part from the bundle of multimedia services, which leads to a requirement for quality assessment in the video domain. Video quality of a streamed video across IP networks is generally influenced by two factors “transmission link imperfection and efficiency of compression standards. This paper deals with subjective video quality assessment and the impact of the compression standards H.264, H.265 and VP9 on perceived video quality of these compression standards. The evaluation is done for four full HD sequences, the difference of scenes is in the content“ distinction is based on Spatial (SI) and Temporal (TI) Index of test sequences. Finally, experimental results follow up to 30% bitrate reducing of H.265 and VP9 compared with the reference H.264.

Cardiac ultrasonography over 4G wireless networks using a tele-operated robot.

This Letter proposes an end-to-end mobile tele-echography platform using a portable robot for remote cardiac ultrasonography. Performance evaluation investigates the capacity of long-term evolution (LTE) wireless networks to facilitate responsive robot tele-manipulation and real-time ultrasound video streaming that qualifies for clinical practice. Within this context, a thorough video coding standards comparison for cardiac ultrasound applications is performed, using a data set of ten ultrasound videos. Both objective and subjective (clinical) video quality assessment demonstrate that H.264/AVC and high efficiency video coding standards can achieve diagnostically-lossless video quality at bitrates well within the LTE supported data rates. Most importantly, reduced latencies experienced throughout the live tele-echography sessions allow the medical expert to remotely operate the robot in a responsive manner, using the wirelessly communicated cardiac ultrasound video to reach a diagnosis. Based on preliminary results documented in this Letter, the proposed robotised tele-echography platform can provide for reliable, remote diagnosis, achieving comparable quality of experience levels with in-hospital ultrasound examinations.

Support for reduced presentation durations in subjective video quality assessment

Video content distributors, codec developers and researchers in related fields often rely on subjective assessments to ensure that their video processing procedures result in satisfactory quality. The current 10s recommendation for the length of test sequences in subjective video quality assessment, however, has recently been questioned. Not only do sequences of this length depart from modern cinematic shooting styles, the use of shorter sequences would also enable substantial efficiency improvements to the data collection process. Our previous work, using a double-stimulus methodology, indicated that shortening test sequences had a limited impact upon rating behaviour. Here, using a larger database and additional opinion score measures, we also explore the same effect within the popular single-stimulus approach. Two groups of viewers assessed reference and distorted videos ranging in length from 1.5s to 10s. Analyses confirmed our previous findings using the DSCQS paradigm, and were replicated when using a similar single-stimulus paradigm: while viewers' DMOS for 1.5s videos was significantly lower than for 10s, no significant variation was found between the groups of 10s, 7s and 5s videos. Together with our previous research, these data lead us to recommend the use of 5s, temporally-consistent video clips in quality assessment studies that employ either DSCQS or its single-stimulus variant. The extension of our recommendation to further methodologies is also discussed.

A No Reference Video Quality Metric Based on Jerkiness Estimation Focusing on Multiple Frame Freezing in Video Streaming

ABSTRACTIn wireless networks, due to limited bandwidth and packet losses, seamless and ubiquitous delivery of high-quality video streaming services is a big challenge for the operators. In order to improve the process of online video quality monitoring, the presence of no reference (NR) objective video quality assessment (VQA) methods is required. In some networks, the video decoder on the reception side adopts a mechanism in which last correctly received frame is frozen and displayed on video display terminal until the next correct frame is received. This phenomenon, employed as an error concealment technique, can cause a perceptual jerkiness on the video display terminal. In this paper, we have proposed an enhanced model of objective VQA based on the estimation of jerkiness. A study of three contemporary NR methods, used for objective VQA and online monitoring of videos, has been included along with subjective VQA tests. The subjective tests were performed for a set of video sequences with specific spatial and temporal information. The proposed NR method is based on our careful observations from the subjective test results and our main focus is to cater the effect of multiple frame freeze impairments in video steaming. Comparison with other NR methods shows that the proposed method performs better, in terms of estimating the impact of multiple frame freezing impairments, and has more affinity with the subjective test results.

On the Optimal Presentation Duration for Subjective Video Quality Assessment

Subjective quality assessment is an essential component of modern image and video processing for both the validation of objective metrics and the comparison of coding methods. However, the standard procedures used to collect data can be prohibitively time consuming. One way of increasing the efficiency of data collection is to reduce the duration of test sequences from a 10-s length currently used in most subjective video quality assessment (VQA) experiments. Here, we explore the impact of reducing sequence length upon perceptual accuracy when identifying compression artifacts. A group of four reference sequences, together with five levels of distortion, are used to compare the subjective ratings of viewers watching videos between 1.5 and 10 s long. We identify a smooth function indicating that accuracy increases linearly as the length of the sequences increases from 1.5 to 7 s. The accuracy of observers viewing 1.5-s sequences was significantly inferior to those viewing sequences of 5, 7, and 10 s. We argue that sequences between 5 and 10 s produce satisfactory levels of accuracy but the practical benefits of acquiring more data lead us to recommend the use of 5-s sequences for future VQA studies that use the double stimulus continuous quality scale methodology.

Subjective Video Quality Assessment of H.265 Compression Standard for Full HD Resolution

Recently increasing interest in multimedia services leads to requirements for quality assessment, especially in the video domain. There are many factors that influence the video quality. Compression technology and transmission link imperfection can be considered as the main ones. This paper deals with the assessment of the impact of H.265/HEVC compression standard on the video quality using subjective metrics. The evaluation is done for two types of sequences with Full HD resolution depending on content. The paper is divided as follows. In the first part of the article, a short characteristic of the H.265/HEVC compression standard is written. In the second part, the subjective video quality methods used in our experiments are described. The last part of this article deals with the measurements and experimental results. They showed that quality of sequences coded between 5 and 7 Mbps is for observers sufficient, so there is no need for providers to use higher bitrates in streaming than this threshold. These results are part of a new model that is still being created and will be used for predicting the video quality in networks based on IP.