Quality Of Experience Metrics Research Articles

Assessment of Quality of Experience (QoE) of Web and Video Services over a Mobile Network Using a Network Emulator

This paper investigates the subjective assessment of QoE of web and video services over a mobile network. To achieve this, the authors used the network emulator (NetEm) traffic control functionality to simulate the dynamic behaviour of a mobile network. Experiments were conducted in a laboratory setting and test conditions were varied to ascertain the QoE, with a focus on QoE metrics such as delay and packet loss ratio. From the experiments conducted, it was observed that there was a negative correlation between delay and average mean opinion score (MOS), and between packet loss ratio and average MOS. The results obtained can be adopted by network operators to provide better services which would lead to improved subscriber base and profitability for the operators and better QoE for the end users.

Optimized Preference-Aware Multi-Path Video Streaming with Scalable Video Coding

Most client hosts are equipped with multiple network interfaces (e.g., WiFi and cellular networks). Simultaneous access of multiple interfaces can significantly improve the users’ quality of experience (QoE) in video streaming. An intuitive approach to achieve it is to use Multi-path TCP (MPTCP). However, the deployment of MPTCP, especially with link preference, requires OS kernel update at both the client and server side, and a vast amount of commercial content providers do not support MPTCP. Thus, in this paper, we realize a multi-path video streaming algorithm in the application layer instead, by considering Scalable Video Coding (SVC), where each layer of every chunk can be fetched from only one of the orthogonal paths. We formulate the quality decisions of video chunks subject to the available bandwidth of the different paths, chunk deadlines, and link preferences as an optimization problem. The objective is to to optimize a QoE metric that maintains a tradeoff between maximizing the playback rate of every chunk and ensuring fairness among chunks. The proposed metric prefers to use bandwidth of the links to optimize a concave utility function of the chunk quality. Even though the formulation is a non-convex discrete optimization, we provide a quadratic complexity algorithm which is shown to be optimal in some special cases. We further propose an online algorithm where several challenges including bandwidth prediction errors, are addressed. Extensive emulated experiments in a real testbed with real traces of public dataset reveal the robustness of our scheme and demonstrate its significant performance improvement compared to other multi-path algorithms.

Implementation of an Interactive Environment With Multilevel Wireless Links for Distributed Botanical Garden in University Campus

In this contribution, an end to end system to enable user interaction with a distributed botanical university campus garden is designed, implemented and tested. The proposed system employs different wireless links to collect data related to different bio physiological parameters of both the vegetation mass and the surrounding environment. Detailed analysis of these multilevel communication links is performed by using deterministic volumetric wireless channel estimation and considering underground, near ground and over ground radio propagation conditions. An in-house developed technique enables accurate wireless channel characterization for complete campus scenario considering the multiple link types and all its composing elements. Node definition and network topology is thus obtained by wireless channel analysis of over ground, near ground and underground communication for both 868 MHz and 2.4 GHz Wireless Sensor Networks in an inhomogeneous vegetation environment. Connectivity to enable user interaction as well as for telemetry and tele-control purposes within the campus is achieved by combining ZigBee and LoRaWAN transceivers with the corresponding sensor/actuator platforms. Coverage studies have been performed in order to assess communication capabilities in the set of multiple underground/near ground/over ground links, by means of deterministic channel analysis for the complete university campus location. Measurement results in lab environment as well as full system deployment are presented, showing good agreement with deterministic simulations. Moreover, system level tests have been performed over a physical campus cloud, providing adequate quality of experience metrics. The proposed solution is a scalable system that provides real time trees status monitoring by a cloud-based platform, enabling user interaction within a distributed botanical garden environment in the university campus.

EmuStream—An End-to-End Platform for Streaming Video Performance Measurement

Cord-cutting has spread like wildfire as streaming video become commonplace in the Internet. This motivated intensive research in adaptive video streaming to improve its quality-of-experience (QoE) in the presence of network quality variations. Much of the existing research either employed dummy video contents or open source videos for QoE evaluation which may not capture the full spectrum of characteristics of real-world contents. This work fills this gap by developing a novel EmuStream platform based on real-world contents to enable realistic experiments and evaluation of any adaptive streaming algorithm in any streaming platform. First, EmuStream offers the largest (700+ titles) publicly available video bitrate trace dataset derived from real-world video contents. Second, we developed a mathematical model based on the dataset which can generate bitrate trace data for arbitrary target bitrates without actual video encoding. Third, we developed a novel Virtual Video Generator (VVG) which can generate h.264/h.265-compliant virtual videos that share the same frame/segment sizes as the source videos to enable experiments mimicking the streaming of commercial video contents. Last but not least, we developed a novel Streaming Performance Meter (SPM) that can measure detailed frame-by-frame playback performance from a video recording of the streaming playback of a virtual video. By decoding the virtual video's specially-coded frames, SPM can determine the playback timing and bitrate selected for each frame for use in computing any desired QoE metric. This paper presents the EmuStream platform and validates the trace data estimation model and VVG/SPM tools via controlled experiments.

Framework towards the Process of Estimating or Predicting Perceived QoE Based on the Datasets Obtained From the Mobile Network

Nowadays, the research study community visualizes a standard shift that is going to put the focus on Quality of Experience metrics, which relate directly to complete consumer satisfaction. Yet, determining QoE coming from QoS sizes is a daunting job that powerful Software Defined Network operators are currently able to tackle through artificial intelligence strategies. In this paper, our experts pay attention to a few essential QoE factors, and we to begin with proposing a Bayesian Network design to anticipate re-buffering proportion. This paper suggested a structure for modeling mobile network QoE, making use of the vast records analytics approach. The planned platform explains the method of estimating or forecasting perceived QoE based upon the datasets obtained or collected from the mobile network to enable the mobile network operators efficiently to deal with the network functionality as well as supply the individuals an adequate mobile Internet QoE.

A Load Balancing Solution for Improving Video Quality in Loaded Wireless Network Conditions

There is an increasing user demand for high-quality content-rich multimedia services. Despite their advantages, current wireless networks in general and wireless mesh networks in particular have limitations in terms of quality of service (QoS) provisioning, especially when dealing with increased amounts of time sensitive traffic such as video. This paper presents video load balancing solution (ViLBaS), a load balancing-based mechanism which enhances delivery performance of video services over multi-hop wireless mesh networks and improves user quality of experience (QoE) levels. ViLBaS involves performance monitoring at the level of the nodes and load balancing by off-loading traffic from loaded nodes to less loaded neighboring nodes. Simulation-based results show how the proposed ViLBaS improves video delivery performance in terms of both QoS and QoE metrics [delay, throughput, packet loss, and peak signal-to-noise-ratio (PSNR)]. The comparison is performed against three other traditional approaches in different network topologies, for diverse video flow distributions, and different sizes for the video queue.

Experimental Comparison of Video Streaming Platforms and Devices with Objective Quality of Experience Metrics Towards Reliable Multimedia Applications in Education

This paper presents the result of experimental comparison of the video quality of streamed videos using streaming devices and streaming platforms. The objective metrics considered for the comparison of these media are the Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Visual Information Fidelity (VIF), Information Fidelity Criterion (IFC), and Visual Signal-to-Noise Ratio (VSNR). To determine the better platform for effective multimedia applications especially for educational purposes, videos were obtained and streamed via Data video, YouTube and Facebook. In the first two scenarios, Data video at two different settings were engaged at the encoding end and PotPlayer was used for decoding and recording of the transmitted videos. In the third and fourth scenarios, Videos were streamed via YouTube and Facebook. The aforementioned objective metrics were implemented in MATLAB R2017b for the experimental analyses and our results showed that YouTube produced streamed videos of the best quality with PNSR of 37.98, SIMM of 0.98 and VSNR of 33.12.

Design o f a Fault Tolerant Strategy f or Resource Scheduling i n Cloud Environment

Cloud computing supports the technological need of the industry supporting many other technologies. Also, the demand for computing power and storage by recent technologies is reasonably growing in a drastic way. Cloud computing, serving for these technologies are to be developed with advancements that lead to performance improvement both in support to the technologies like block-chain and big data. The allocation of cloud resources is an important strategy to be followed in a wiser manner to incorporate the needs of extra ordinary computing power. In this paper, an efficient resource allocation strategy (FTVMA) is introduced that involves the creation of effective virtual machines (VMs) and performs VM allocation in an efficient manner by considering the failure rates, previous history of failure of VM, execution efficiency as a part of effective scheduling. There exist many reasons for cloudlet failure in VMs. Some of them are overloading of VMs and non-availability of VMs. The introduced FTVMA algorithm considers the failure rate of the physical machine, load of virtual machines and the cost priority of the tasks in order to achieve Quality of Service (QoS) and Quality of Experience (QoE) of the user. The FTVMA methodology proposed in this paper works better for computation intensive VMs and is tested using CloudSim environment. The QoS metrics used to measure the performance of the proposed algorithm are Makespan and VM Utilization. The metric to measure QoE are Priority Miss Rate and Failure Rate. The proposed algorithm shows its improvement in terms of the QoS and QoE metrics. The results obtained are compared with the existing resource scheduling algorithms and it is inferred that the proposed algorithm performs better in terms of QoS and QoE.

Enhancing QOSQOE in MANETs using OLSR Protocol

Video streaming over Mobile Ad-hoc Networks (MANETs) has been on the fore as one of the most chiefly solicited web services. Within this context, the enhancement of the Quality of Experience (QoE) on such platforms, whose signature characteristic is the real-time fluctuation of their Mobile Nodes (MNs), remains a high-stakes challenge for high-fidelity transmission via the extant MANET routing protocols. Indeed, the free mobility of MNs (i.e. their real-time physical reallocation, viz. intra-network movement), renders network topology often subject to unpredictable fluctuations. It is this margin of relative unpredictability which lends itself to such instances where QoE, as perceived by the Customer/User, may be subject to varying degrees of depreciation.In this perspective, our contribution in this paper aims to optimize the MANETs' mobility system, through the re-adaptation of some extant MANET routing protocols, so as to afford safer routing courses (lower mobility thresholds equate to lower chances of noise and/or data corruption/loss); all for the capital purpose of improving subjective quality (i.e. the anticipated end-user's personal assessment of the service).For the implementation of our MANET network, our two-fold choice consisted of the NS2 version 2.9 (a powerful platform with highly reliable protocol support), supplemented by the Evalvid Framework (a field-proven tool according to many expert ratings, ideal for close monitoring of QoE metrics).We have considered various video transmission scenarios through the OLSR protocol, one of the most well-known and reliable proactive MANET routing protocols .As for QoE prediction, we expect the mean opinion scores (MOS) to provide a metric template for a rough estimation of the end-users' anticipated appraisal .The results are to demonstrate, as we shall see, that the modified heuristic algorithm of OLSR, leaning on the underlying criterion of mobility, can lead to a significant performance boost in MANETs and, by the same token, to higher returns of QoS and QoE.

A Meta-Learning Framework for Learning Multi-User Preferences in QoE Optimization of DASH

Dynamic adaptive video streaming over hypertext transfer protocol (DASH) plays a key role in video transmission over the Internet. The conventional DASH adaptation approaches mainly focus on optimizing the overall quality of experience (QoE) for all client sides, neglecting the QoE diversity of different users. In this paper, we propose a meta-learning framework for multi-user preferences (MLMP) as a new DASH adaptation approach, which is able to optimize the diverse QoE of different users. Specifically, we first design a subjective experiment to analyze the difference of QoE preferences across users, in which QoE refers to the metrics of visual quality, fluctuation, and rebuffering events. Based on our findings, we formulate the QoE optimization of multi-user preferences as a multi-task deep reinforcement learning (DRL) problem. In our formulation, the QoE preference of each user is modeled in the overall QoE calculation via assigning the weights to the three QoE metrics. Then, the MLMP framework is developed to solve the proposed multi-task DRL problem, such that the preferences regarding visual quality, fluctuation, and rebuffering events can be optimized for different users in DASH adaptation. Finally, the simulation results show that the proposed approach outperforms state-of-the-art DASH adaptation approaches in satisfying the different users' QoE preferences regarding visual quality, fluctuation, and rebuffering events.

Using Session Modeling to Estimate HTTP-Based Video QoE Metrics From Encrypted Network Traffic

Understanding the user-perceived quality of experience (QoE) of HTTP-based video has become critical for content providers, distributors, and network operators. For network operators, monitoring QoE is challenging due to lack of access to video streaming applications, user devices, or servers. Thus, network operators need to rely on the network traffic to infer key metrics that influence video QoE. Furthermore, with content providers increasingly encrypting the network traffic, the task of QoE inference from passive measurements has become even more challenging. In this paper, we present a methodology called eMIMIC that uses passive network measurements to estimate key video QoE metrics for encrypted HTTP-based adaptive streaming (HAS) sessions. eMIMIC uses packet headers from network traffic to model an HAS session and estimate video QoE metrics, such as average bitrate and re-buffering ratio. We evaluate our methodology using network traces from a variety of realistic conditions and ground truth collected using a lab testbed for video sessions from three popular services, two video on demand (VoD) and one Live. eMIMIC estimates re-buffering ratio within 1% point of ground truth for up to 75% sessions in VoD (80% in Live) and average bitrate with error under 100 Kb/s for up to 80% sessions in VoD (70% in Live). We also compare eMIMIC with recently proposed machine learning-based QoE estimation methodology. We show that eMIMIC can predict average bitrate with 2.8%–3.2% higher accuracy and re-buffering ratio with 9.8%–24.8% higher accuracy without requiring any training on ground truth QoE metrics. Finally, we show that eMIMIC can estimate real-time QoE metrics with at least 89.6% accuracy in identifying buffer occupancy state and at least 85.7% accuracy in identifying average bitrate class of recently downloaded chunks.

SmartStreamer: Preference-Aware Multipath Video Streaming Over MPTCP

Simultaneous access to multiple interfaces (e.g., WiFi and cellular networks) can significantly improve the users’ quality of experience (QoE) in video streaming. However, some interfaces could be more expensive to use and less energy efficient. Therefore, in this paper, we propose a preference-aware multipath video streaming algorithm over HTTP using multipath TCP (MPTCP). First, we formulate the quality decisions of the video chunks and the chunk's download policy subject to the chunk's deadlines, the available bandwidth of the different paths, and the link preferences as a non-convex optimization problem. The objective is to optimize a novel QoE metric that maintains a tradeoff between maximizing the quality of every video's chunk and ensuring quality fairness among all chunks for the minimum re-buffering (stall) duration, and without violating the link preference constraint. Second, we develop a polynomial time complexity algorithm to solve the proposed optimization problem, and provide guarantees for the proposed algorithm. We further propose a sliding window based online algorithm where several challenges including short bandwidth prediction with prediction errors are addressed. Extensive emulated experiments with real bandwidth traces of public datasets reveal the robustness of our scheme and demonstrate its significant performance improvement compared to the state-of-the-art multi-path streaming algorithms.

QoE awareness in progressive caching and DASH-based D2D video streaming in cellular networks

In this paper, we present an architecture to improve video streaming quality of experience (QoE) in cellular networks with high user equipment (UE) density. In the proposed architecture, video segments are progressively cached, as requested, in selected UEs called storage members (SMs). Video segments are strategically cached to be available to requesting users in the cell. Furthermore, the base-station controls the device-to-device communication between the UEs to provide collaborative peer-to-peer transmission of video segments. Dynamic adaptive streaming over HTTP is also employed to adapt the quality of video segments to network conditions. We study the improvements achieved by the proposed architecture in terms of many video streaming QoE metrics. Thereafter, we improve the operation of the proposed architecture by introducing QoE awareness to both caching and distribution of video segments. We employ QoE awareness in three aspects of the proposed architecture; cellular resource allocation, caching of video segments, and SM-assignment optimization. We analyze the improvements achieved by the prospered QoE-awareness techniques in terms of video streaming QoE metrics.

Learning QoE of Mobile Video Transmission With Deep Neural Network: A Data-Driven Approach

Quality of experience (QoE) serves as a direct evaluation of users' experiences in mobile video transmission and thus essential for network management, such as network optimization. In this paper, we propose a deep learning-based QoE prediction approach with a large-scale QoE dataset for mobile video transmission. Specifically, we develop a mobile phone application for collecting user QoE data when viewing videos transmitted over the mobile internet in a practical environment. Then, we construct a large-scale dataset by collecting over 80000 piece of data with four kinds of subjective scores and 89 network parameters. Each QoE metric is related to only some of the 89 network parameters. Therefore, we apply the feature selection method to find the feature parameters related to user scores. Additionally, the boxplot method is used to clean the raw data by removing outliers. Finally, a deep neural network (DNN) is developed to learn the relationships between the network parameters and the subjective QoE scores. The proposed DNN can also be seen as a data-driven objective QoE prediction approach for mobile video transmission, which can be used to predict the user QoE scores. The experimental results show that the proposed approach can effectively remove most features irrelevant to QoE prediction. Moreover, the performance of QoE prediction by the proposed model outperforms other state-of-the-art approaches.

Perceptual quality assessment of video using machine learning algorithm

User experience has become the most reliable and trustworthy source for service providers to assess system performance. To fulfill customer requirements, service providers require an efficient quality of experience (QoE) estimation model. QoE is a subjective metric that deals with user perception and can vary dramatically due to various factors such as emotions, the degree of annoyance, past experience, and aesthetics. Moreover, subjective QoE evaluation is expensive and time-consuming because of human participation. Therefore, a model is required to objectively measure QoE with reasonable accuracy. In the context of service and network providers fulfilling user requirements, with a reasonable quality of service (QoS), needs to be provided to the relevant services/applications. However, QoS parameters do not reflect subjective opinion of the user accurately. Therefore, it is necessary to compute the mapping or correlation between QoS and QoE. The mapping may help service providers to understand the behavior of the overall network on user experience and efficiently manage the network resources. In this paper, a new feature number of displayed frames impacted (NoDFI) along with a machine learning based model is presented to compute a correlation between QoS and QoE. A publically available dataset is used to represent the correlation between objective QoS parameters and subjective QoE metric. The result of experiments showed that proposed feature NoDFI proved to be a valuable addition when compared with previously proposed models.

Measuring Web Latency and Rendering Performance: Method, Tools, and Longitudinal Dataset

This paper presents Webget , a measurement tool that measures Web quality of service (QoS) metrics, including the DNS lookup time, time to first byte (TTFB), and the download time. Webget also captures Web complexity metrics, such as the number and the size of objects that make up the website. We deploy the Webget test to measure the Web performance of Google, YouTube, and Facebook from 182 SamKnows probes. Using a 3.5-year-long (January 2014–July 2017) dataset, we show that the DNS lookup time of these popular content delivery networks (CDNs) and the download time of Google have improved over time. We also show that the TTFB toward Facebook exhibits worse performance than the Google CDN. Moreover, we show that the number and the size of objects are not the only factors that affect the Web download time. We observe that these webpages perform differently across regions and service providers. We also developed a Web measurement system, Web performance and rendering (WePR) that measures the same Web QoS and complexity metrics as Webget , but it also captures the Web quality of experience metrics, such as rendering time. WePR has a distributed architecture where the component that measures the Web QoS and complexity metrics is deployed on the SamKnows probe, while the rendering time is calculated on a central server. We measured the rendering performance of four websites. We show that in 80% of the cases, the rendering time of the websites is faster than the downloading time. The source code of the WePR system and the dataset is made publicly available.

Q-SQUARE: A Q-learning approach to provide a QoE aware UAV flight path in cellular networks

This paper deals with the adoption of Unmanned Aerial Vehicles (UAVs) as mobile Base Stations providing video streaming services within a cellular macro area. We devise a Q-learning based UAV flight planning algorithm aimed at improving the Quality of Experience (QoE) of video users. Specifically, the proposed algorithm, herein denoted as Q-SQUARE, leverages the well-established Q-learning algorithm by introducing a reward related to a key QoE metric that is the video segment delay. The Q-SQUARE algorithm also accounts for different UAV recharging stations being available in the covered area. The performance analysis, as a function of the number of UAVs and recharging stations, show that Q-SQUARE identifies the UAV flight paths, i.e. specific space-time allocation of the available bandwidth resources, that definitely improve the QoE of the streaming services.

Begin with the End in Mind: A Unified End-to-End Quality-of-Experience Monitoring, Optimization, and Management Framework

There has been an increasing consensus in the video distribution industry that the design and operation of the full video delivery chain need to be driven by the quality of experience (QoE) appropriate to the end users. Here, we propose a new framework that uses a unified end-to-end solution to produce consistent QoE scores at all points along the delivery chain under the same evaluation criterion. This is a framework that produces a clear picture instantaneously to operation engineers, managing executives, and content creators about how video QoE degrades along the chain; a framework that allows immediate issue identification, localization, and resolution; a framework that enables quality and resource usage optimization; and a framework that provides reliable predictive metrics for long-term strategic resource and infrastructure allocations. The main challenge in the implementation of such a framework is to create a unified QoE metric that not only accurately predicts human QoE but is also lightweight and versatile, readily plugged into multiple points in the video delivery chain. We show that the SSIMPLUS metric offers the best promise, and we demonstrate the benefits of the proposed framework and QoE metric using bandwidth optimization as an example.

A Cache-Level Quality of Experience Metric to Characterize ICNs for Adaptive Streaming

Adaptive streaming has motivated information-centric network designs to improve end-user quality of experience (QoE). However, their management and evaluation rely either on conventional cache-level metrics that are poor representations of QoE, or consumer-side indicators that are opaque to network services. This letter proposes a measure to bridge the gap between cache performance and consumer QoE. We introduce maximal sustainable bitrate (MSB), defined as the highest bitrate deliverable in time to be in time to meet a given request without buffering. Based on our observations, we posit that QoE is maximal when requested bitrates match a cache's MSB for that content. We design a cache-level reward function as a benchmark metric that measures the difference between requested bitrates and MSB. We hypothesize that aggregated rewards are an indicator of overall system performance. Performance evaluations show high correlation between the sum of cache rewards and consumer QoE.

A Fuzzy Logic System for Vertical Handover and Maximizing Battery Lifetime in Heterogeneous Wireless Multimedia Networks

Bandwidth and power hungry applications are proliferating in mobile networks at a rapid pace. However, mobile devices have been suffering from a lack of sufficient battery capacity for the intensive/continuous use of these applications. In addition, the mobile ecosystem is currently heterogeneous and comprises a plethora of networks with different technologies such as LTE, Wi-Fi, and WiMaX. Hence, an issue must be addressed to ensure that quality of experience (QoE) is provided for the users in this scenario: an energy-efficient strategy that is designed to extend the battery lifetime of mobile devices. This paper proposes an architecture which provides an intelligent decision-making support system based on Fuzzy Logic for saving the energy of mobile devices within an integrated LTE and Wi-Fi network. The simulated experiments show the benefits of the solution this architecture can provide by using QoE metrics.