Quality Of Experience Monitoring Research Articles

ML-Based QoE Estimation in 5G Networks Using Different Regression Techniques

Monitoring and providing customers with a satisfying Quality of Experience (QoE) is a crucial business incentive for mobile network operators (MNOs). While the MNO is capable of monitoring a vast amount of network-related key performance indicators (KPIs), it typically does not have access to application-specific performance metrics. Among others, this is due to practical obstacles, such as missing standardized interfaces between the network and the application. Existing QoE models allow to map collected KPIs to the user-perceived quality. However, they are not dynamic, cumbersome to obtain, and often rely on application-level information, such as the stalling duration in the case of video streaming. The 5G networking architecture provides new features which can potentially overcome current limitations of in-network QoE monitoring. More specifically, the Application Function (AF) provides a standardized interface for communicating between 5G systems and third parties, such as application providers. The Network Data Analytics Function (NWDAF) is capable of collecting a vast number of network statistics from other 5G network functions and is dedicated to training and deploying Machine Learning (ML) models. This opens new possibilities, unimaginable for earlier mobile network generations, to dynamically learn the relationship between network KPIs and QoE by utilizing ML. Besides elaborating on how the new capabilities introduced with 5G can support an ML-based QoE estimation, we perform a simulation-based feasibility study which evaluates the estimation accuracy of different state-of-the-art regression techniques. In addition, we discuss them with respect to various qualitative aspects from an MNO’s point of view.

SDN and NFV for QoE-driven multimedia services delivery: The road towards 6G and beyond networks

The concept for developing the future mobile communication systems has been explored in recent years with a vision towards 5G and 6G systems. Meeting performance targets such as higher data rate transmission, lower end-to-end latency, higher capacity, lower cost, and satisfied Quality of Experience (QoE) for delivered services is the key for the success of 5G and 6G networks. Within these performance targets and associated enabling technologies, QoE and the overall management of multimedia services is the one which has been less well addressed for 5G in the past. This paper provides a comprehensive and ground-breaking discussion regarding QoE management in the context of future softwarized 5G/6G networks. We introduce a generalized QoE provisioning ecosystem for emerging multimedia streaming services in softwarized 5G/6G and beyond networks. We explore potential network softwarization and communication technologies that will enable the end-user’s QoE in 5G/6G. We further provide promising use case scenarios regarding (a) QoE-driven management of multimedia services over multi-access cloud/edge softwarized and virtualized 5G networks, (b) QoE monitoring for large traffic variations in business, residential and public areas in 5G networks, along with a Machine Learning (ML)-based softwarized controller for QoE provisioning over 5G/6G networks. We build on the achievements of 5G networks to provide the roadmap towards the development of 6G and beyond networks in terms of requirements, use case scenarios and service classes. We also present potential technologies that will be dominant through 2025–2030 in shaping the vision of 6G and beyond networks including (a) self-driving 3D network architecture, (b) network management, automation and orchestration, and (c) pervasive artificial intelligence. Moreover, we provide challenges and research directions in 6G and beyond wireless communication systems. The paper provides the first roadmap towards the management, orchestration and monitoring of multimedia 3D services in 6G and beyond networks. The comprehensive goal of this paper is to encourage researchers from the industry and academia to work together towards the realization of softwarized 6G and beyond networks while tackling the critical research challenges regarding the management of emerging 3D multimedia services and applications. The comments made by the authors in this paper are expectations on 6G, and the standardization of 6G where these aspects will be decided in 6G networks and beyond, will be started soon by different standardization bodies and fora.

Machine Learning based KPI Monitoring of Video Streaming Traffic for QoE Estimation

Quality of Experience (QoE) monitoring of video streaming traffic is crucial task for service providers. Nowadays it is challenging due to the increased usage of end-to-end encryption. In order to overcome this issue, machine learning (ML) approaches for QoE monitoring have gained popularity in the recent years. This work proposes a framework which includes a machine learning pipeline that can be used for detecting key QoE related events such as buffering events and video resolution changes for ongoing YouTube video streaming sessions in real-time. For this purpose, a ML model has been trained using YouTube streaming traffic collected from Android devices. Later on, the trained ML model is deployed in the framework's pipeline to make online predictions. The ML model uses statistical traffic information observed from the network-layer for learning and predicting the video QoE related events. It reaches 88% overall testing accuracy for predicting the video events. Although our work is yet at an early stage, the application of the ML model for online detection and prediction of video events yields quite promising results.

The CASPER user-centric approach for advanced service provisioning in mobile networks

This paper presents an overview of the project CASPER,11http://gain.di.uoa.gr/casper/. a 4-year Marie Curie Research and Innovation Staff Exchange (RISE) project running between 2016 and 2020, describing its objectives, approach, architecture, tools and key achievements. CASPER combines academic and industrial forces towards leveraging the expected benefits of Quality of Experience (QoE) exploitation in future networks. In order to achieve that, a QoE orchestrator has been proposed which implements the basic functionalities of QoE monitoring, estimation and management. With means of simulation and testbed emulation, CASPER has managed to develop a proprietary SDN Controller, which implements QoE-based traffic rerouting for the challenging scenario of HTTP adaptive video streaming, leading to more stable and higher QoE scores compared to a state-of-the-art SDN Controller implementation.

IJNM Special Issue—International Journal of Network Management: QoE‐centric analysis and management of communication networks

IJNM Special Issue—International Journal of Network Management: QoE‐centric analysis and management of communication networks

Novel QoE monitoring and management architecture with eTOM for SDN-based 5G networks

The 5th Generation (5G) network design is still under exploration and will require an innovative approach to network monitoring and management with a relentless focus on customer needs. Among the recent technologies, software-defined networking (SDN) and network functions virtualization (NFV), which are already offering network programmability and automation, have been proposed to improve resource management in 5G systems. Furthermore, the enhanced telecom operations map (eTOM) can be used to manage both End-to-End (E2E) service and end-customer experience. In this paper, we illustrate a novel SDN-based architecture for 5G networks that will be able to enhance quality-of-experience (QoE) monitoring and management by integrating SDN and eTOM. This architecture is deployed and evaluated by simulations of a service-level agreement (SLA) verification scenario. As a result, the implemented platform is capable of identifying SLA violation and can successfully enforce suitable mechanisms to dynamically adjust network parameters, thereby satisfying the SLA constraints.

Peripheral Sensing: Monitoring Quality of Experience for Video Services Based on Mobile Terminals

Quality of experience (QoE), which directly relates to both technical evolvement and profit promotion, is a vital concern for mobile video services. However, the wireless network operators have long been troubled by the problem of lacking effective QoE monitoring approaches since the traditional evaluation methods of communication quality are objective metrics oriented. Considering that mobile terminals are the network elements closest to users, it is promising to realize a real-time QoE estimation for video services by fully utilizing the sensing capabilities of mobile terminals. As the first step, we specifically develop a mobile video testing application. With support from China Unicom, one of the three major wireless network operators in China, over 80 000 data records are collected under the real-world conditions. The collected data consist of four types of subjective scores and 13 objective parameters concerning video attributes, network performance, device capability, playback events, and external factors. After preprocessing the data set through correlation analysis, we establish the two QoE estimation models based on the C4.5 method and the gradient boosting decision tree (GBDT) method, respectively. The experimental results demonstrate that the proposed models can achieve remarkable estimation performances and outperform the baseline models. Specifically, the overall estimation accuracy of the GBDT-based model is approximately 80% for a five-level scale and approaches 90% when a more practical 3-level scale is adopted. Finally, we comprehensively discuss the estimation performances based on characteristics of the data and validate the feasibility of estimating QoE based on mobile terminals-the “peripheral sensors” of the mobile networks.

Quality of experience column

Research on Quality of Experience (QoE) has advanced significantly in recent years and attracts attention from various stakeholders. Different facets have been addressed by the research community like subjective user studies to identify QoE influence factors for particular applications like video streaming, QoE models to capture the effects of those influence factors on concrete applications, QoE monitoring approaches at the end user site but also within the network to assess QoE during service consumption and to provide means for QoE management for improved QoE. However, in order to progress in the area of QoE, new research directions have to be taken. The application of QoE in practice needs to consider the entire QoE eco-system and the stakeholders along the service delivery chain to the end user.

Impact of VNF placements on QoE monitoring in the cloud

SummaryThe fast growth of video streaming is responsible for a huge amount of traffic over the past few years. Because of the variety of video content on the Internet, a potential market is emerging for video providers. However, a rapid increase in video traffic and users poses challenges for network operators maintaining user expectation. To address this problem, network operators need a mechanism to monitor the quality of experience (QoE) perceived by the user. This allows reacting to quality degradation to improve the service. With the paradigm of network function virtualization, network operators are able to deploy such a video monitoring function in the cloud. In this work, we investigate the feasibility of deploying a virtual network function (VNF) for video buffer and QoE monitoring on the Amazon Web Service cloud.To this end, we implement a VNF to analyze video flows in the network by using deep packet inspection. We investigate the influence of different points of presence (PoP) in the cloud and mobile network on the performance of the VNF for monitoring video buffer and QoE. Our findings show that the accuracy of the QoE monitoring decreases with the distance of the PoP to the client. This is not only due to the delay and bottleneck between the monitoring point and the client but also due to the client mobile access network.

Deep Learning Model for Multimedia Quality of Experience Prediction Based on Network Flow Packets

Quality of experience (QoE) is the overall acceptability of an application or service, as perceived subjectively by the end user. In particular, for video quality the QoE is dependent of video transmission parameters. To monitor and control these parameters is critical in modern network management systems, but it would be better to be able to monitor the QoE itself (in terms of both interpretation and accuracy) rather than the parameters on which it depends. In this article we present the first attempt to predict video QoE based on information directly extracted from the network packets using a deep learning model. The QoE detector is based on a binary classifier (good or bad quality) for seven common classes of anomalies when watching videos (blur, ghost, etc.). Our classifier can detect anomalies at the current time instant and predict them at the next immediate instant. This classifier faces two major challenges: first, a highly unbalanced dataset with a low proportion of samples with video anomaly, and second, a small amount of training data, since it must be obtained from individual viewers under a controlled experimental setup. The proposed classifier is based on a combination of a convolutional neural network (CNN), recurrent neural network, and Gaussian process classifier. Image processing, which is the common domain for a CNN, has been expanded to QoE detection. Based on a detailed comparison, the proposed model offers better performance metrics than alternative machine learning algorithms, and can be used as a QoE monitoring function in edge computing.

A Quality-of-Experience Database for Adaptive Video Streaming

The dynamic adaptive streaming over HTTP provides an inter-operable solution to overcome the volatile network conditions, but its complex characteristic brings new challenges to objective video quality-of-experience (QoE) measurement. To test the generalizability and to facilitate the wide usage of QoE measurement techniques in real-world applications, we establish a new database named Waterloo Streaming QoE Database III (SQoE-III). Unlike existing databases constructed with hand-crafted test sequences, the SQoE-III database, so far the largest and most realistic of its kind, consists of a total of 450 streaming videos created from diverse source content and diverse distortion patterns, with six adaptation algorithms of diverse characteristics under 13 representative network conditions. All streaming videos are assessed by 34 subjects, and a comprehensive evaluation is conducted on the performance of 15 objective QoE models from four categories with regards to their efficacy in predicting subjective QoE. Detailed correlation analysis and statistical hypothesis testing are carried out. The results of this paper shed light on the future development of adaptive bitrate streaming algorithm and video QoE monitoring system. The subjective database is available online at https://ece.uwaterloo.ca/~zduanmu/tbc2018qoe/ .

A Survey of Emerging Concepts and Challenges for QoE Management of Multimedia Services

Quality of Experience (QoE) has received much attention over the past years and has become a prominent issue for delivering services and applications. A significant amount of research has been devoted to understanding, measuring, and modelling QoE for a variety of media services. The next logical step is to actively exploit that accumulated knowledge to improve and manage the quality of multimedia services, while at the same time ensuring efficient and cost-effective network operations. Moreover, with many different players involved in the end-to-end service delivery chain, identifying the root causes of QoE impairments and finding effective solutions for meeting the end users’ requirements and expectations in terms of service quality is a challenging and complex problem. In this article, we survey state-of-the-art findings and present emerging concepts and challenges related to managing QoE for networked multimedia services. Going beyond a number of previously published survey articles addressing the topic of QoE management, we address QoE management in the context of ongoing developments, such as the move to softwarized networks, the exploitation of big data analytics and machine learning, and the steady rise of new and immersive services (e.g., augmented and virtual reality). We address the implications of such paradigm shifts in terms of new approaches in QoE modeling and the need for novel QoE monitoring and management infrastructures.

Real-Time QoE Monitoring System for Video Streaming Services with Adaptive Media Playout

Quality of Experience (QoE) of video streaming services has been attracting more and more attention recently. Therefore, in this work we designed and implemented a real-time QoE monitoring system for streaming services with Adaptive Media Playout (AMP), which was implemented into the VideoLAN Client (VLC) media player to dynamically adjust the playout rate of videos according to the buffer fullness of the client buffer. The QoE monitoring system reports the QoE of streaming services in real time so that network/content providers can monitor the qualities of their services and resolve troubles immediately whenever their subscribers encounter them. Several experiments including wired and wireless streaming were conducted to show the effectiveness of the implemented AMP and QoE monitoring system. Experimental results demonstrate that AMP significantly improves the QoE of streaming services according to the Mean Opinion Score (MOS) estimated by our developed program. Additionally, some challenging issues in wireless streaming have been easily identified using the developed QoE monitoring system.

Challenges of future multimedia QoE monitoring for internet service providers

The ever-increasing network traffic and user expectations at reduced cost make the delivery of high Quality of Experience (QoE) for multimedia services more vital than ever in the eyes of Internet Service Providers (ISPs). Real-time quality monitoring, with a focus on the user, has become essential as the first step in cost-effective provisioning of high quality services. With the recent changes in the perception of user privacy, the rising level of application-layer encryption and the introduction and deployment of virtualized networks, QoE monitoring solutions need to be adapted to the fast changing Internet landscape. In this contribution, we provide an overview of state-of-the-art quality monitoring models and probing technologies, and highlight the major challenges ISPs have to face when they want to ensure high service quality for their customers.

QoE in Video Transmission: A User Experience-Driven Strategy

The increasing popularity of video (i.e., audio-visual) applications or services over both wired and wireless links has prompted recent growing interests in the investigations of quality of experience (QoE) in online video transmission. Conventional video quality metrics, such as peak-signal-to-noise-ratio and quality of service, only focus on the reception quality from the systematic perspective. As a result, they cannot represent the true visual experience of an individual user. Instead, the QoE introduces a user experience-driven strategy which puts special emphasis on the contextual and human factors in addition to the transmission system. This advantage has raised the popularity and widespread usage of QoE in video transmission. In this paper, we present an overview of selected issues pertaining to QoE and its recent applications in video transmission, with consideration of the compelling features of QoE (i.e., context and human factors). The selected issues include QoE modeling with influence factors in the end-to-end chain of video transmission, QoE assessment (including subjective test and objective QoE monitoring) and QoE management of video transmission over different types of networks. Through the literature review, we observe that the context and human factors in QoE-aware video transmission have attracted significant attentions since the past two to three years. A vast number of high quality works were published in this area, and will be highlighted in this survey. In addition to a thorough summary of recent progresses, we also present an outlook of future developments on QoE assessment and management in video transmission, especially focusing on the context and human factors that have not been addressed yet and the technical challenges that have not been completely solved so far. We believe that our overview and findings can provide a timely perspective on the related issues and the future research directions in QoE-oriented services over video communications.

Early Detection of QoE Deterioration in HTTP Adaptive Streaming

Apart from video rate (or requested bitrate), Mean Opinion Score (MOS) has increasingly become a primary term representing Quality of Experience (QoE) in HTTP adaptive streaming (HAS). By monitoring this metric, QoE management can effectively maximize QoE for the users. However, due to various behaviors of particular commercial HAS players, deciding an appropriate monitoring interval has not been fully investigated yet. In this paper, an optimal interval is proposed to be equal to duration of a video chunk in order to aid service managers in early detecting QoE deterioration and limiting the probability of video rate deterioration. The optimal monitoring interval is evaluated by comparing with other values of interval in terms of ratio of video rate deterioration. Furthermore, MOS-based QoE monitoring method which takes into account the proposed interval is thus compared with video rate based monitoring method. The results show that with optimal interval, MOS monitoring guarantees a low ratio of video rate deterioration (around 10% for buffering state and 40% for steady state) and small average CPU Load (about 11.45%).

A Method for Evaluating QoE of Live Streaming Services

This paper presents a novel method which can provide accurate real-time evaluation for Quality of Experience (QoE) of live streaming services. First, we use the modified Analytic Hierarchy Process (AHP) to build a QoE model which is mainly based on video quality and network transmission quality in live streaming services. Then, for video quality, we take into account the unique multi-resolution feature of Internet live streaming videos and design a video quality assessment algorithm mainly based on bitrate, temporal perceptual information (TI) and spatial perceptual information (SI). This algorithm removes the interference of multi-resolution and has a good correlation with subjective perception. And for network transmission quality, we design another assessment algorithm based on buffering time and frequency. Finally, by using the QoE model and the assessment algorithms, we can build a real-time QoE monitoring and evaluation system for live streaming services.

Full-reference video quality metric assisted the development of no-reference bitstream video quality metrics for real-time network monitoring

High-quality video is being increasingly delivered over Internet Protocol networks, which means that network operators and service providers need methods to measure the quality of experience (QoE) of the video services. In this paper, we propose a method to speed up the development of no-reference bitstream objective metrics for estimating QoE. This method uses full-reference objective metrics, which makes the process significantly faster and more convenient than using subjective tests. In this process, we have evaluated six publicly available full-reference objective metrics in three different databases, the EPFL-PoliMI database, the HDTV database, and the Live Video Wireless database, all containing transmission distortions in H.264 coded video. The objective metrics could be used to speed up the development process of no-reference real-time video QoE monitoring methods that are receiving great interest from the research community. We show statistically that the full-reference metric Video Quality Metric (VQM) performs best considering all the databases. In the EPFL-PoliMI database, SPATIAL MOVIE performed best and TEMPORAL MOVIE performed worst. When transmission distortions are evaluated, using the compressed video as the reference provides greater accuracy than using the uncompressed original video as the reference, at least for the studied metrics. Further, we use VQM to train a lightweight no-reference bitstream model, which uses the packet loss rate and the interval between instantaneous decoder refresh frames, both easily accessible in a video quality monitoring system.

A Software Platform Design for Objective Video QoE Assessment

Along with the rapid development of 3G and 4G technologies, mobile video services have gained its popularity among users around the world. Consequently, Content Providers (CPs), Service Providers (SPs), and especially, the operators are paying increasing attentions to the quality of experience (QoE) of the video services which could be easily affected by the quality of network. In this paper, a novel real-time objective video QoE assessment method is proposed and a software assessment system is built to test the video service quality in the real network. Firstly, in the test terminals, the QoE measurement of the entire video services is conducted by collecting all of the customers’ experience in full-reference method, and then the QoE scores are evaluated through an accurate mathematic model. Secondly, the artifact of compression caused by video encoding should also be taken into account. Model in this part adopts no-reference method in consideration of the varied screen sizes in different terminals. What’s more, the platform also evaluates the error of network in the part of video transmission by associating no-reference PSNR with network delay, jitter, and packet loss ratio. The results of Mean Opinion Score (MOS) tests show that the proposed models estimate QoE with high quality estimation accuracy respectively. We develop a software toolkit using the test methodologies above, which can help the operators to make measurements for its network. This software toolkit is useful as a QoE monitoring tool on video streaming services and can be deployed on real network conveniently

YOUQMON

YouTube is changing the way operators manage network performance monitoring. In this paper we introduce YOUQMON, a novel on-line monitoring system for assessing the Quality of Experience (QoE) undergone by HSPA/3G customers watching YouTube videos, using network-layer measurements only. YOUQMON combines passive traffic analysis techniques to detect stalling events in YouTube video streams, with a QoE model to map stallings into a Mean Opinion Score reflecting the end-user experience. We evaluate the stalling detection performance of YOUQMON with hundreds of YouTube video streams, and present results showing the feasibility of performing real-time YouTube QoE monitoring in an operational mobile broadband network.