Quality Of Experience Estimation Research Articles

QoE Estimation of WebRTC-based Audio-visual Conversations from Facial and Speech Features

The utilization of user’s facial- and speech-related features for the estimation of the Quality of Experience (QoE) of multimedia services is still underinvestigated despite its potential. Currently, only the use of either facial or speech features individually has been proposed, and relevant limited experiments have been performed. To advance in this respect, in this study, we focused on WebRTC-based videoconferencing, where it is often possible to capture both the facial expressions and vocal speech characteristics of the users. First, we performed thorough statistical analysis to identify the most significant facial- and speech-related features for QoE estimation, which we extracted from the participants’ audio-video data collected during a subjective assessment. Second, we trained individual QoE estimation machine learning-based models on the separated facial and speech datasets. Finally, we employed data fusion techniques to combine the facial and speech datasets into a single dataset to enhance the QoE estimation performance due to the integrated knowledge provided by the fusion of facial and speech features. The obtained results demonstrate that the data fusion technique based on the Improved Centered Kernel Alignment (ICKA) allows for reaching a mean QoE estimation accuracy of 0.93, whereas the values of 0.78 and 0.86 are reached when using only facial or speech features, respectively.

Industrial User Experience Index vs. Quality of Experience Models

Network operators have increased interest in understanding and quantifying the user perceived quality of experience (QoE). Appropriate QoE models allow mapping of measurable QoS parameters to QoE metrics like mean opinion scores (MOS) or poor-or-worse (PoW) ratios. Unfortunately, there are no QoE models for all applications available yet, and development and standardization is rather time-consuming. For that reason, industry often implements simple but effective approaches based on expert knowledge to get a user experience index able to deliver high value insights. This index is often defined based on thresholds of measurable QoS parameters to classify the resulting user experience. The question arises whether such a threshold-based user experience index can properly estimate the QoE in a system. We therefore answer in this work whether the user experience index is able to quantify the expected QoE in a system and if other QoE metrics like PoW ratios can be derived. Furthermore, we discuss the interpretation and the limits of the index. As a key contribution, we present a detailed systematic analysis of the threshold-based QoE concept and discuss its usability as QoE estimation methodology for industry.

Construction of a Video Transmission Scenario in Software-Defined Networks for QoE Estimation

The services supported by data networks have become widespread, so the architectures of the new data networks are service-oriented. They are endowed with intelligence, flexibility, and programmability. The preceding is with the aim of providing acceptability by users. Thus, this paper presents the construction of a video transmission scenario over a software-defined network (SDN, Software-Defined Networking) using free software and modifying its behavior with background traffic, on which the Quality of Experience (QoE) is estimated. Subjective and objective metrics were used for the QoE estimation. For the first one, the Mean Opinion Score (MOS) was used, while the second one was studied with the Full Reference Image Quality Assessment (FR-IQA). Finally, a correlation between the two types of metrics was proposed.

ML-Based QoE Estimation in 5G Networks Using Different Regression Techniques

Monitoring and providing customers with a satisfying Quality of Experience (QoE) is a crucial business incentive for mobile network operators (MNOs). While the MNO is capable of monitoring a vast amount of network-related key performance indicators (KPIs), it typically does not have access to application-specific performance metrics. Among others, this is due to practical obstacles, such as missing standardized interfaces between the network and the application. Existing QoE models allow to map collected KPIs to the user-perceived quality. However, they are not dynamic, cumbersome to obtain, and often rely on application-level information, such as the stalling duration in the case of video streaming. The 5G networking architecture provides new features which can potentially overcome current limitations of in-network QoE monitoring. More specifically, the Application Function (AF) provides a standardized interface for communicating between 5G systems and third parties, such as application providers. The Network Data Analytics Function (NWDAF) is capable of collecting a vast number of network statistics from other 5G network functions and is dedicated to training and deploying Machine Learning (ML) models. This opens new possibilities, unimaginable for earlier mobile network generations, to dynamically learn the relationship between network KPIs and QoE by utilizing ML. Besides elaborating on how the new capabilities introduced with 5G can support an ML-based QoE estimation, we perform a simulation-based feasibility study which evaluates the estimation accuracy of different state-of-the-art regression techniques. In addition, we discuss them with respect to various qualitative aspects from an MNO’s point of view.

From Whole Video to Frames: Weakly-Supervised Domain Adaptive Continuous-Time QoE Evaluation.

Due to the rapid increase in video traffic and relatively limited delivery infrastructure, end users often experience dynamically varying quality over time when viewing streaming videos. The user quality-of-experience (QoE) must be continuously monitored to deliver an optimized service. However, modern approaches for continuous-time video QoE estimation require densely annotating the continuous-time QoE labels, which is labor-intensive and time-consuming. To cope with such limitations, we propose a novel weakly-supervised domain adaptation approach for continuous-time QoE evaluation, by making use of a small amount of continuously labeled data in the source domain and abundant weakly-labeled data (only containing the retrospective QoE labels) in the target domain. Specifically, given a pair of videos from source and target domains, effective spatiotemporal segment-level feature representation is first learned by a combination of 2D and 3D convolutional networks. Then, a multi-task prediction framework is developed to simultaneously achieve continuous-time and retrospective QoE predictions, where a quality attentive adaptation approach is investigated to effectively alleviate the domain discrepancy without hampering the prediction performance. This approach is enabled by explicitly attending to the video-level discrimination and segment-level transferability in terms of the domain discrepancy. Experiments on benchmark databases demonstrate that the proposed method significantly improves the prediction performance under the cross-domain setting.

QoE estimation for different adaptive streaming techniques in mobile networks

Video services are becoming more and more popular for mobile network users and require greater and greater resources and provisions from telecommunications service providers. But operators suffer from problems of interoperability between the different adaptive transmissions techniques they employ in an attempt to satisfy the quality of experience (QoE) of the service provided to users and improve network performance. This article presents a comparison of four such streaming techniques - DASH (dynamic adaptive streaming over HTTP), HDS (HTTP dynamic streaming), HLS (HTTP2 live streaming) and HSS (HTTP smooth streaming) - used in a live video playback by a user in different test scenarios on an emulated long-term evolution (LTE) network. Comparison of performance was carried out using the mean opinion score (MOS) metric calculated based on ITU-T Recommendation P.1203. The streaming techniques that performed best in each of the different test scenarios are revealed.

QoE‐aware traffic monitoring based on user behavior in video streaming services

SummaryIn recent years, online video content has gained a lot of popularity and the exponential growth of video traffic continues in every area of the connected world. Thus, understanding the quality of experience (QoE) perceived by end‐users of video streaming services is important for both network operators and over the top providers since estimating end user's QoE has become one of the main points to meet user expectations. However, it is not trivial to ensure an adequate QoE since user experience is affected by various influencing factors (e.g., context factors, human factors, and system factors), and it is still challenging to identify the QoE key influencing factors. To address these challenges, we focused on improving QoE estimation and management strategies by exploiting valuable human and context information because the influence of social contextual information and user behavior on the perceptual quality is often neglected. In this article, we first proposed a classification of influence factors into four categories which are: system factors, human factors, context factors, and social‐behavioral factors. We developed a monitoring web application where video content is played to end‐users so that subjective and objective video metrics are collected. We built a new machine learning (ML) based model for QoE prediction. We used well‐known supervised ML algorithms like decision tree, k‐nearest neighbors, and support vector machine. Finally, we proposed a QoE management approach in the context of software defined network/multi‐access edge computing that implements the proposed QoE prediction model to optimize the video delivery transmission chain.

Machine Learning based KPI Monitoring of Video Streaming Traffic for QoE Estimation

Quality of Experience (QoE) monitoring of video streaming traffic is crucial task for service providers. Nowadays it is challenging due to the increased usage of end-to-end encryption. In order to overcome this issue, machine learning (ML) approaches for QoE monitoring have gained popularity in the recent years. This work proposes a framework which includes a machine learning pipeline that can be used for detecting key QoE related events such as buffering events and video resolution changes for ongoing YouTube video streaming sessions in real-time. For this purpose, a ML model has been trained using YouTube streaming traffic collected from Android devices. Later on, the trained ML model is deployed in the framework's pipeline to make online predictions. The ML model uses statistical traffic information observed from the network-layer for learning and predicting the video QoE related events. It reaches 88% overall testing accuracy for predicting the video events. Although our work is yet at an early stage, the application of the ML model for online detection and prediction of video events yields quite promising results.

A Sign of Things to Come: Predicting the Perception of Above-the-Fold Time in Web Browsing

Content is a key influencing factor in Web Quality of Experience (QoE) estimation. A web user’s satisfaction can be influenced by how long it takes to render and visualize the visible parts of the web page in the browser. This is referred to as the Above-the-fold (ATF) time. SpeedIndex (SI) has been widely used to estimate perceived web page loading speed of ATF content and a proxy metric for Web QoE estimation. Web application developers have been actively introducing innovative interactive features, such as animated and multimedia content, aiming to capture the users’ attention and improve the functionality and utility of the web applications. However, the literature shows that, for the websites with animated content, the estimated ATF time using the state-of-the-art metrics may not accurately match completed ATF time as perceived by users. This study introduces a new metric, Plausibly Complete Time (PCT), that estimates ATF time for a user’s perception of websites with and without animations. PCT can be integrated with SI and web QoE models. The accuracy of the proposed metric is evaluated based on two publicly available datasets. The proposed metric holds a high positive Spearman’s correlation (rs=0.89) with the Perceived ATF reported by the users for websites with and without animated content. This study demonstrates that using PCT as a KPI in QoE estimation models can improve the robustness of QoE estimation in comparison to using the state-of-the-art ATF time metric. Furthermore, experimental result showed that the estimation of SI using PCT improves the robustness of SI for websites with animated content. The PCT estimation allows web application designers to identify where poor design has significantly increased ATF time and refactor their implementation before it impacts end-user experience.

SDN Based in-Network Two-Staged Video QoE Estimation With Measurement Error Correction for Edge Network

Network resource management is one of the key technologies needed to ensure that multiple applications in edge networks provide reliable and stable performance. Although throughput has previously been seen as the primary network performance metric, recent applications do not focus on throughput alone. Instead, Quality of Experience (QoE) is attracting significant attention as an indicator of network resource management performance because it allows a wide variety of applications to be compared within a single metric. In this study, we tackle QoE measurements for a video streaming service as a way to evaluate QoE-based network management. However, there are several problems related to measuring QoE. For example, in-network components are difficult to measure because QoE is normally measured at end-points, and several properties that are deeply related to application settings are required for those calculations. Additionally, the measurements set forth in the International Telecommunication Union’s ITU-T G.1071 standard require a certain duration, which is too long for network resource management evaluations. Therefore, this paper proposes a two-staged in-network QoE estimation method for video flows that can resolve these issues. In the first stage, we focus on producing a fast and rough QoE estimate to start forwarding the arriving flow onto an appropriate route as soon as possible. Next, the second stage is designed to produce precise QoE estimations based on careful long-duration measurements. In both stages, the proposed method uses a parameter estimation process that converts in-network information to end-point information for QoE calculations by following ITU-T G. 1071 and corrects measurement errors reducing QoE calculation errors to the greatest extent possible. Through experimental evaluations, we then demonstrate that the QoEs of all flows can be maximized by selecting appropriate routes based on the predicted QoE at the first stage, and that the accuracy of the QoE estimation at the second stage can be improved in real-time even when packet losses occur.

QoE Analysis for Improving Multimedia Services Based on Different Operating Situations on Cellular Networks

Cellular networks are constantly evolving to support the new context and demands of users with increasing internet traffic. The key to a service provider’s success is a direct response to endpoints to retain existing users and attract new users. For this reason, Mobile Network Operators (MNOs) must reach user satisfaction under a user-centered perspective with Quality of Experience (QoE) to achieve a Return on Investment (RoI). In the latest QoE modelling, our previous method performed QoE estimation and synthesis using an Artificial Neural Network (ANN). Nevertheless, such work did not prove the characteristics of the QoE model built on different working situations. The essential features of the area and user diversities need to be considered in detail to summarize the proper model. In this paper, the proposed framework focuses on analyzing QoE model characteristics under different operating situations for comprehensive implementation. The experimental results reveal that the created QoE models from areas of different densities affect the different predictive variables. Inter-Rater Reliability (IRR) analysis exposes the reliability of datasets gathered from users of different groups to distinguish the effects on the response of significant parameters. Additionally, the summary specifies the characteristics of the appropriate model by area and user group as a guideline for improving the multimedia services.

Estimation of Quality Scores From Subjective Tests-Beyond Subjects’ MOS

Subjective tests for the assessment of the quality of experience (QoE) are typically run with a pool of subjects providing their opinion scores using a 5-point scale. The subjects’ mean opinion score (MOS) is generally assumed as the best estimation of the average score in the target population. Indeed, for a large enough sample, we may assume that the mean of the variations across the subjects approaches zero, but this is not the case for the limited number of subjects typically considered in subjective tests. In this paper, we propose an approach based on generalized linear models (GLMs) for estimation of the population average QoE. The motivating dataset is composed of the individual scores assigned by 25 subjects to a set of gaming videos evaluated under different resolutions and compression ratios. The approach recognizes the multinomial nature of the data and allows for correlation between scores of the same subject. The resulting estimated average QoE is shown to follow more credible patterns than the MOS, particularly for higher bitrates, for which the model estimates present more coherent behavior. Similar convincing results are found on a second dataset, showing the validity of the approach.

The CASPER user-centric approach for advanced service provisioning in mobile networks

This paper presents an overview of the project CASPER,11http://gain.di.uoa.gr/casper/. a 4-year Marie Curie Research and Innovation Staff Exchange (RISE) project running between 2016 and 2020, describing its objectives, approach, architecture, tools and key achievements. CASPER combines academic and industrial forces towards leveraging the expected benefits of Quality of Experience (QoE) exploitation in future networks. In order to achieve that, a QoE orchestrator has been proposed which implements the basic functionalities of QoE monitoring, estimation and management. With means of simulation and testbed emulation, CASPER has managed to develop a proprietary SDN Controller, which implements QoE-based traffic rerouting for the challenging scenario of HTTP adaptive video streaming, leading to more stable and higher QoE scores compared to a state-of-the-art SDN Controller implementation.

IJNM Special Issue—International Journal of Network Management: QoE‐centric analysis and management of communication networks

IJNM Special Issue—International Journal of Network Management: QoE‐centric analysis and management of communication networks

A Telepresence Wheelchair with 360-Degree Vision Using WebRTC

This paper presents an innovative approach to develop an advanced 360-degree vision telepresence wheelchair for healthcare applications. The study aims at improving a wide field of view surrounding the wheelchair to provide safe wheelchair navigation and efficient assistance for wheelchair users. A dual-fisheye camera is mounted in front of the wheelchair to capture images which can be then streamed over the Internet. A web real-time communication (WebRTC) protocol was implemented to provide efficient video and data streaming. An estimation model based on artificial neural networks was developed to evaluate the quality of experience (QoE) of video streaming. Experimental results confirmed that the proposed telepresence wheelchair system was able to stream a 360-degree video surrounding the wheelchair smoothly in real-time. The average streaming rate of the entire 360-degree video was 25.83 frames per second (fps), and the average peak signal to noise ratio (PSNR) was 29.06 dB. Simulation results of the proposed QoE estimation scheme provided a prediction accuracy of 94%. Furthermore, the results showed that the designed system could be controlled remotely via the wireless Internet to follow the desired path with high accuracy. The overall results demonstrate the effectiveness of our proposed approach for the 360-degree vision telepresence wheelchair for assistive technology applications.

An Adaptive Neuro-Fuzzy Model for Quality Estimation in Wireless 2D/3D Video Streaming Systems

Delivering high Quality of Experience (QoE) is essential to the success of today’s subscription for internet video streaming services. Quality of Service (QoS) metrics are considered by the research community as the most influential factor on video QoE. Therefore, establishing QoS-QoE correlation becomes critical for improving video QoE estimation. This paper presents experimental development of effective correlation between QoE and QoS for both 2D and 3D video streaming services. This is then used to build an objective QoE estimation model for real-time streaming of both 2D and 3D video contents over wireless networks. This model is based on using Adaptive Neural Fuzzy Inference System (ANFIS) to estimate the perceived video QoE. The proposed QoE model was trained with a set of media and packet layers’ metrics, taking into account the effect of video content type, dimension, and different packet loss metrics. The performance of the proposed QoE estimation model shows a considerable estimation accuracy with a correlation coefficient of 92% and 0.167 RMSE.

A robust multimedia traffic SDN-Based management system using patterns and models of QoE estimation with BRNN

Nowadays, network infrastructures such as Software Defined Networks (SDN) achieve a huge computational power. This allows to add a high processing on the network nodes. In this paper, a multimedia traffic management system is presented. This system is based on estimation models of Quality of Experience (QoE) and also on the traffic patterns classification. In order to achieve this, a QoE estimation method has been modeled. This method allows for classifying the multimedia traffic from multimedia transmission patterns. In order to do this, the SDN controller gathers statistics from the network. The patterns used have been defined from a lineal combination of objective QoE measurements. The model has been defined by Bayesian regularized neural networks (BRNN). From this model, the system is able to classify several kind of traffic according to the quality perceived by the users. Then, a model has been developed to determine which video characteristics need to be changed to provide the user with the best possible quality in the critical moments of the transmission. The choice of these characteristics is based on the quality of service (QoS) parameters, such as delay, jitter, loss rate and bandwidth. Moreover, it is also based on subpatterns defined by clusters from the dataset and which represents network and video characteristics. When a critical network situation is given, the model selects, by using network parameters as entries, the subpattern with the most similar network condition. The minimum Euclidean distance between these entries and the network parameters of the subpatters is calculated to perform this selection. Both models work together to build a reliable multimedia traffic management system perfectly integrated into current network infrastructures, which is able to classify the traffic and solve critical situations changing the video characteristics, by using the SDN architecture.

Q2ABR: QoE‐aware adaptive video bit rate solution

SummaryIn this paper, we propose a new adaptive bit rate (ABR) streaming method. This method is based on estimating and monitoring users' video streaming experience, their quality of experience (QoE). This ensures a good user QoE and optimises bandwidth utilisation by monitoring video buffer fill rate to ensure minimal data traffic. First, we achieve a QoE evaluation model based on network bandwidth, video segment representation, and dropped video frame rate parameters. Second, following our QoE evaluation model, we formulate an ABR method using the reinforcement learning (RL) paradigm to select video representations and using a breakpoint detection mechanism to monitor end‐user QoE variation. The proposed ABR method is called “QoE‐aware adaptive bit rate (Q2ABR)” and is composed of three individual modules, one for QoE estimation using machine learning methods, one for QoE variation monitoring using the breakpoint detection mechanism, and one for video representation selection using reinforcement learning. The design objective of Q2ABR is to ensure the overall QoE of these users while maintaining a minimum variation in the standard deviation of the users' QoE values. Third, the performance of the Q2ABR method is evaluated and compared with several existing ABR approaches in the literature using real traces that we collect on different transport scenarios (such as bus and train, among others). Since this method considers the user's perception of video quality as a regulator for optimising the overall video distribution network, good results are ensured in terms of the user's experience and buffer fill rate.

Using Session Modeling to Estimate HTTP-Based Video QoE Metrics From Encrypted Network Traffic

Understanding the user-perceived quality of experience (QoE) of HTTP-based video has become critical for content providers, distributors, and network operators. For network operators, monitoring QoE is challenging due to lack of access to video streaming applications, user devices, or servers. Thus, network operators need to rely on the network traffic to infer key metrics that influence video QoE. Furthermore, with content providers increasingly encrypting the network traffic, the task of QoE inference from passive measurements has become even more challenging. In this paper, we present a methodology called eMIMIC that uses passive network measurements to estimate key video QoE metrics for encrypted HTTP-based adaptive streaming (HAS) sessions. eMIMIC uses packet headers from network traffic to model an HAS session and estimate video QoE metrics, such as average bitrate and re-buffering ratio. We evaluate our methodology using network traces from a variety of realistic conditions and ground truth collected using a lab testbed for video sessions from three popular services, two video on demand (VoD) and one Live. eMIMIC estimates re-buffering ratio within 1% point of ground truth for up to 75% sessions in VoD (80% in Live) and average bitrate with error under 100 Kb/s for up to 80% sessions in VoD (70% in Live). We also compare eMIMIC with recently proposed machine learning-based QoE estimation methodology. We show that eMIMIC can predict average bitrate with 2.8%–3.2% higher accuracy and re-buffering ratio with 9.8%–24.8% higher accuracy without requiring any training on ground truth QoE metrics. Finally, we show that eMIMIC can estimate real-time QoE metrics with at least 89.6% accuracy in identifying buffer occupancy state and at least 85.7% accuracy in identifying average bitrate class of recently downloaded chunks.

Comparison of multimedia communications QoE models by Bayesian networks and Bayesian statistics: a case study

This paper presents a comparison of Bayesian Network (BN) and Bayesian Statistics (BS) modeling for QoE (Quality of Experience) estimation and prediction in multimedia communications, with special attention to prediction. As an example of the comparison, we employ a haptic-audiovisual interactive communication system with guaranteed bandwidth. The QoE measure adopted here is subjects’ overall satisfaction (average score) of performing an interactive task under conditions specified by combinations of the video guaranteed bandwidth, video encoding bit rate, receiver’s playout buffering time and gender of each subject. For BN modeling, we utilize an R package bnlearn and create a discrete BN model of a directed acyclic graph with four nodes corresponding to the four parameters. For BS modeling, we build (1) a Bayesian hierarchical regression model with covariates of the four parameters and random effect terms reflecting users’ individualities and gender, and (2) a Bayesian regression model without the random effect terms. The two BS models are analyzed by Markov chain Monte Carlo (MCMC) simulation with the software OpenBUGS. We then find that the BN and BS models provide approximately the same estimates of the QoE measure. Regarding the prediction, however, the BS model with random effect terms outperforms the BN model and BS model without random effect terms. We thus learn that the random effect terms enhance the ability of Bayesian approaches in QoE prediction.