Real-time Quality Of Experience Research Articles

Crowdsourcing Framework for QoE-Aware SD-WAN

Quality of experience (QoE) is an important measure of users’ satisfaction regarding their network-based services, and it is widely employed today to provide a real assessment of the service quality as perceived by the end users. QoE measures can be used to improve application performance, as well as to optimize network resources and reallocate them as needed when the service quality degrades. While quantitative QoE assessments based on network parameters may provide insights into users’ experience, subjective assessments through direct feedback from the users have also gathered interest recently due to their accuracy and interactive nature. In this paper, we propose a framework that can be used to collect real-time QoE feedback through crowdsourcing and forward it to network controllers to enhance streaming routes. We analyze how QoE can be affected by different network conditions, and how different streaming protocols compare against each other when the network parameters change dynamically. We also compare the real-time user feedback to predefined network changes to measure if participants will be able to identify all degradation events, as well as to examine which combination of degradation events are noticeable to the participants. Our aim is to demonstrate that real-time QoE feedback can enhance cloud-based services and can adjust service quality on the basis of real-time, active participants’ interactions.

A low-complexity psychometric curve-fitting approach for the objective quality assessment of streamed game videos

The increasing popularity of video gaming competitions, the so called eSports, has contributed to the rise of a new type of end-user: the passive game video streaming (GVS) user. This user acts as a passive spectator of the gameplay rather than actively interacting with the content. This content, which is streamed over the Internet, can suffer from disturbing network and encoding impairments. Therefore, assessing the user’s perceived quality, i.e the Quality of Experience (QoE), in real-time becomes fundamental. For the case of natural video content, several approaches already exist that tackle the client-side real-time QoE evaluation. The intrinsically different expectations of the passive GVS user, however, call for new real-time quality models for these streaming services. Therefore, this paper presents a real-time Reduced-Reference (RR) quality assessment framework based on a low-complexity psychometric curve-fitting approach. The proposed solution selects the most relevant, low-complexity objective feature. Afterwards, the relationship between this feature and the ground-truth quality is modelled based on the psychometric perception of the human visual system (HVS). This approach is validated on a publicly available dataset of streamed game videos and is benchmarked against both subjective scores and objective models. As a side contribution, a thorough accuracy analysis of existing Objective Video Quality Metrics (OVQMs) applied to passive GVS is provided. Furthermore, this analysis has led to interesting insights on the accuracy of low-complexity client-based metrics as well as to the creation of a new Full-Reference (FR) objective metric for GVS, i.e. the Game Video Streaming Quality Metric (GVSQM).

Using Session Modeling to Estimate HTTP-Based Video QoE Metrics From Encrypted Network Traffic

Understanding the user-perceived quality of experience (QoE) of HTTP-based video has become critical for content providers, distributors, and network operators. For network operators, monitoring QoE is challenging due to lack of access to video streaming applications, user devices, or servers. Thus, network operators need to rely on the network traffic to infer key metrics that influence video QoE. Furthermore, with content providers increasingly encrypting the network traffic, the task of QoE inference from passive measurements has become even more challenging. In this paper, we present a methodology called eMIMIC that uses passive network measurements to estimate key video QoE metrics for encrypted HTTP-based adaptive streaming (HAS) sessions. eMIMIC uses packet headers from network traffic to model an HAS session and estimate video QoE metrics, such as average bitrate and re-buffering ratio. We evaluate our methodology using network traces from a variety of realistic conditions and ground truth collected using a lab testbed for video sessions from three popular services, two video on demand (VoD) and one Live. eMIMIC estimates re-buffering ratio within 1% point of ground truth for up to 75% sessions in VoD (80% in Live) and average bitrate with error under 100 Kb/s for up to 80% sessions in VoD (70% in Live). We also compare eMIMIC with recently proposed machine learning-based QoE estimation methodology. We show that eMIMIC can predict average bitrate with 2.8%–3.2% higher accuracy and re-buffering ratio with 9.8%–24.8% higher accuracy without requiring any training on ground truth QoE metrics. Finally, we show that eMIMIC can estimate real-time QoE metrics with at least 89.6% accuracy in identifying buffer occupancy state and at least 85.7% accuracy in identifying average bitrate class of recently downloaded chunks.

Peripheral Sensing: Monitoring Quality of Experience for Video Services Based on Mobile Terminals

Quality of experience (QoE), which directly relates to both technical evolvement and profit promotion, is a vital concern for mobile video services. However, the wireless network operators have long been troubled by the problem of lacking effective QoE monitoring approaches since the traditional evaluation methods of communication quality are objective metrics oriented. Considering that mobile terminals are the network elements closest to users, it is promising to realize a real-time QoE estimation for video services by fully utilizing the sensing capabilities of mobile terminals. As the first step, we specifically develop a mobile video testing application. With support from China Unicom, one of the three major wireless network operators in China, over 80 000 data records are collected under the real-world conditions. The collected data consist of four types of subjective scores and 13 objective parameters concerning video attributes, network performance, device capability, playback events, and external factors. After preprocessing the data set through correlation analysis, we establish the two QoE estimation models based on the C4.5 method and the gradient boosting decision tree (GBDT) method, respectively. The experimental results demonstrate that the proposed models can achieve remarkable estimation performances and outperform the baseline models. Specifically, the overall estimation accuracy of the GBDT-based model is approximately 80% for a five-level scale and approaches 90% when a more practical 3-level scale is adopted. Finally, we comprehensively discuss the estimation performances based on characteristics of the data and validate the feasibility of estimating QoE based on mobile terminals-the “peripheral sensors” of the mobile networks.

Two-phase rate adaptation strategy for improving real-time video QoE in mobile networks

With the popularity of smart handheld devices, mobile streaming video has multiplied the global network traffic in recent years. A huge concern of users' quality of experience (QoE) has made rate adaptation methods very attractive. In this paper, we propose a two-phase rate adaptation strategy to improve users' real-time video QoE. First, to measure and assess video QoE, we provide a continuous QoE prediction engine modeled by RNN recurrent neural network. Different from traditional QoE models which consider the QoE-aware factors separately or incompletely, our RNN-QoE model accounts for three descriptive factors (video quality, rebuffering, and rate change) and reflects the impact of cognitive memory and recency. Besides, the video playing is separated into the initial startup phase and the steady playback phase, and we takes different optimization goals for each phase: the former aims at shortening the startup delay while the latter ameliorates the video quality and the rebufferings. Simulation results have shown that RNN-QoE can follow the subjective QoE quite well, and the proposed strategy can effectively reduce the occurrence of rebufferings caused by the mismatch between the requested video rates and the fluctuated throughput and attains standout performance on real-time QoE compared with classical rate adaption methods.

Real-Time QoE Monitoring System for Video Streaming Services with Adaptive Media Playout

Quality of Experience (QoE) of video streaming services has been attracting more and more attention recently. Therefore, in this work we designed and implemented a real-time QoE monitoring system for streaming services with Adaptive Media Playout (AMP), which was implemented into the VideoLAN Client (VLC) media player to dynamically adjust the playout rate of videos according to the buffer fullness of the client buffer. The QoE monitoring system reports the QoE of streaming services in real time so that network/content providers can monitor the qualities of their services and resolve troubles immediately whenever their subscribers encounter them. Several experiments including wired and wireless streaming were conducted to show the effectiveness of the implemented AMP and QoE monitoring system. Experimental results demonstrate that AMP significantly improves the QoE of streaming services according to the Mean Opinion Score (MOS) estimated by our developed program. Additionally, some challenging issues in wireless streaming have been easily identified using the developed QoE monitoring system.

A Method for Evaluating QoE of Live Streaming Services

This paper presents a novel method which can provide accurate real-time evaluation for Quality of Experience (QoE) of live streaming services. First, we use the modified Analytic Hierarchy Process (AHP) to build a QoE model which is mainly based on video quality and network transmission quality in live streaming services. Then, for video quality, we take into account the unique multi-resolution feature of Internet live streaming videos and design a video quality assessment algorithm mainly based on bitrate, temporal perceptual information (TI) and spatial perceptual information (SI). This algorithm removes the interference of multi-resolution and has a good correlation with subjective perception. And for network transmission quality, we design another assessment algorithm based on buffering time and frequency. Finally, by using the QoE model and the assessment algorithms, we can build a real-time QoE monitoring and evaluation system for live streaming services.

A cost-effective and real-time QoE evaluation method for multimedia streaming services

Traditionally, subjective tests are conducted to obtain the Mean Opinion Score (MOS) or Quality of Experience (QoE) of a video streaming service. However, subjective tests are time-consuming and rather costly. Thus, a cost-effective and real-time QoE evaluation method for video streaming services is presented in this paper. The proposed scheme first adopts subjective tests to find the relationships between the QoE and individual QoS parameters. Next, based on subjective test results, the corresponding QoS---QoE mapping functions for individual QoS metrics are derived using the regression approach. Finally, the integrated QoE function, which is a multi-variate function of multiple QoS metrics, for assessing the overall QoE of video streaming services is proposed. Hence, network/service providers only need to measure related objective QoS parameters while the corresponding MOS/QoE can be easily derived in real time based on the integrated QoE function. Since subjective tests only need to be conducted occasionally for updating the parameters in the QoS---QoE mapping functions, the cost of QoE assessment decreases significantly.

Quality of experience prediction model for progressive downloading over mobile broadcast networks

The paper introduces a new quality of experience (QoE) prediction model for progressive downloading over the mobile broadcast networks. The proposed model covers new QoE metrics, a novel QoE aware buffering method, and a novel QoE measurement method that predicts the perceived service quality in real time. Progressive download is a streaming technology using "play while download" approach. As a case study, progressive downloading is used over the multimedia broadcast multicast service (3GPP's MBMS) as the underlying network. The broadcast networks are unidirectional delivery platforms, and hence exposed to many unwanted conditions such as the packet losses, delays, and bandwidth problems. Quality of service (QoS) is a way of classification that manages how these conditions are controlled and mapped to the service quality. However, QoS parameters could not reflect how the end-user experience is. At this point, the QoE describes the achieved QoS and the end-user satisfaction with the service. Conventional QoE metrics for multimedia streaming assumes that packet losses cause artifacts, such as blurring and color distortions, on media presentation. However, with the progressive downloading the packet losses could be tolerated, e.g., using forward error correction. With the protection against loss errors, the overall network errors are projected onto the behaviors of the buffering model. The buffering characteristics should be described by well defined states and expected behaviors in that expected behaviors, from the user expectation point of view, are better than the random ones. In this article, first a buffering method for progressive downloading is proposed. Then, a real time QoE measurement method is proposed to map the buffering characteristics to the achieved performance of the service. Finally, some subjective study using mean opinion score are provided to prove the accuracy of the proposed model.

A real-time QoE methodology for AMR codec voice in mobile network

This paper studies a general strategy to predict voice Quality of Experience (QoE) for various mobile networks. Particularly, based on data-mining for Adaptive Multi-Rate (AMR) codec voice, a novel QoE assessment methodology is proposed. The proposed algorithm consists of two parts. The first part is devoted to assessing speech quality of fixed rate codec mode (CM) of AMR while in the other one a adaptive rate CM is designed. Measuring basic network parameters that have much impact on speech quality, QoE can be monitored in real time for operators. Meanwhile, based on the measurement data sets from real mobile network, the QoE prediction strategy can be implemented and QoE assessment model for AMR codec voice is trained and tested. Finally, the numerical results suggest that the correlation coefficient between predicted values and true values is greater than 90% and root mean squared error is less than 0.5 for fixed and adaptive rate CM.

Full-reference video quality metric assisted the development of no-reference bitstream video quality metrics for real-time network monitoring

High-quality video is being increasingly delivered over Internet Protocol networks, which means that network operators and service providers need methods to measure the quality of experience (QoE) of the video services. In this paper, we propose a method to speed up the development of no-reference bitstream objective metrics for estimating QoE. This method uses full-reference objective metrics, which makes the process significantly faster and more convenient than using subjective tests. In this process, we have evaluated six publicly available full-reference objective metrics in three different databases, the EPFL-PoliMI database, the HDTV database, and the Live Video Wireless database, all containing transmission distortions in H.264 coded video. The objective metrics could be used to speed up the development process of no-reference real-time video QoE monitoring methods that are receiving great interest from the research community. We show statistically that the full-reference metric Video Quality Metric (VQM) performs best considering all the databases. In the EPFL-PoliMI database, SPATIAL MOVIE performed best and TEMPORAL MOVIE performed worst. When transmission distortions are evaluated, using the compressed video as the reference provides greater accuracy than using the uncompressed original video as the reference, at least for the studied metrics. Further, we use VQM to train a lightweight no-reference bitstream model, which uses the packet loss rate and the interval between instantaneous decoder refresh frames, both easily accessible in a video quality monitoring system.

H.264/MVC interleaving for real-time multiview video streaming

The multiview video streaming demands high bandwidth even after coding. Any loss during transmission will have effect on the real-time quality of experience (QoE) of the end user due to the prediction structure used in H.264/MVC encoder. This paper proposes an interleaving scheme by considering the prediction structure used in the encoder. The priority for every picture in the prediction structure is computed using proposed mathematical model in order to satisfy any GOP size and any number of views. The validation and verification of the proposed work is done using H.264/MVC reference software JMVC 8.5. The test results show that the proposed interleaving scheme significantly outperforms the non-interleaved scheme.

Multi-Resolution Multimedia QoE Models for IPTV Applications

Internet television (IPTV) is rapidly gaining popularity and is being widely deployed in content delivery networks on the Internet. In order to proactively deliver optimum user quality of experience (QoE) for IPTV, service providers need to identify network bottlenecks in real time. In this paper, we develop psycho-acoustic-visual models that can predict user QoE of multimedia applications inreal timebased on online network status measurements. Our models are neural network based and cater to multi-resolution IPTV applications that include QCIF, QVGA, SD, and HD resolutions encoded using popular audio and video codec combinations. On the network side, our models account for jitter and loss levels, as well as router queuing disciplines: packet-ordered and time-ordered FIFO. We evaluate the performance of our multi-resolution multimedia QoE models in terms of prediction characteristics, accuracy, speed, and consistency. Our evaluation results demonstrate that the models are pertinent for real-time QoE monitoring and resource adaptation in IPTV content delivery networks.

A “GAP-model” based framework for online VVoIP QoE measurement

Increased access to broadband networks has led to a fast-growing demand for voice and video over IP (VVoIP) applications such as Internet telephony (VoIP), videoconferencing, and IP television (IPTV). For pro-active troubleshooting of VVoIP performance bottlenecks that manifest to end-users as performance impairments such as video frame freezing and voice dropouts, network operators cannot rely on actual end-users to report their subjective quality of experience (QoE). Hence, automated and objective techniques that provide real-time or online VVoIP QoE estimates are vital. Objective techniques developed to-date estimate VVoIP QoE by performing frame-to-frame peak-signal-to-noise ratio (PSNR) comparisons of the original video sequence and the reconstructed video sequence obtained from the sender-side and receiver-side, respectively. Since processing such video sequences is time consuming and computationally intensive, existing objective techniques cannot provide online VVoIP QoE. In this paper, we present a novel framework that can provide online estimates of VVoIP QoE on network paths without end-user involvement and without requiring any video sequences. The framework features the G AP-model, which is an offline model of QoE expressed as a function of measurable network factors such as bandwidth, delay, jitter, and loss. Using the GAP-model, our online framework can produce VVoIP QoE estimates in terms of Good, Acceptable, or Poor (GAP) grades of perceptual quality solely from the on-line measured network conditions.