Subjective Quality Of Experience Research Articles

Usefulness of QoS in Multidimensional QoE Prediction for Haptic–Audiovisual Communications

This article investigates prediction of QoE (Quality of Experience) by comparing borrowing–from–neighbor situations and isolated ones. We demonstrate that joint utilization of multiple QoE measures enhances the accuracy of QoE prediction compared to that by a collection of individual QoE measures each regressed on QoS (Quality of Service) parameters, while the accuracy improvement with additional usage of QoS information in the former is limited. As an example of system which needs multidimensional QoE representation, the article gives haptic audiovisual interactive communications. We employ QoE and QoS data taken previously in an experiment, where 13 QoE measures (a five–point score each) and 12 QoS parameters (nonnegative continuous values each) are available at three average rates of load traffic. We build two kinds of Bayesian models for QoE prediction; one is a logistic regression model of a single QoE measure as the response variable and QoS parameters as predictors, which is a typical traditional method of discrete QoE prediction and isolated in a sense. The other is a structural equation model (SEM) with latent constructs (i.e., factors) of audiovisual quality, haptic quality and user experience quality; the original SEM, which contains only QoE indicators of three constructs (AVQ, HQ and UXQ), was proposed in one of the author’s previous studies. This article extends the SEM in order to accommodate QoS parameters. We develop two kinds of new SEMs with QoS parameters: One has three extended constructs referred to as eAVQ, eHQ and eUXQ, each of which has both QoE and QoS indicators, and the other has separate constructs for QoE and QoS, which lead to totally six constructs (AVqoe, Hqoe, UXqoe, AVqos, Hqos and UXqos). We performed Markov chain Monte Carlo (MCMC) simulation of the Bayesian models with the JAGS software in an R environment. For comparison of QoE prediction accuracy, we adopt the 10–fold cross validation method and in part WAIC (widely applicable information criterion). We then found that the three–construct models outperform the logistic regression models with respect to all subjective QoE measures and that the two kinds of the models are comparable as for the objective measure. The six–construct model exhibits almost the same accuracy as that of the three-construct one unless the number of QoE measures ( n qoe ) in the model is small. When the number n qoe is small, single–construct models may be a better choice. We have thus learned that multiple QoE measures should be utilized jointly (i.e., borrowing from neighbor) in QoE prediction rather than resorting to QoS information only.

Improving QoE of Deep Neural Network Inference on Edge Devices: A Bandit Approach

Edge devices, including, in particular, mobile devices, have been emerging as an increasingly more important platform for deep neural network (DNN) inference. Typically, multiple lightweight DNN models generated using different architectures and/or compression schemes can fit into a device, thus selecting an optimal one is crucial in order to maximize the users’ Quality of Experience (QoE) for edge inference. The existing approaches to device-aware DNN optimization are usually time consuming and not scalable in view of extremely diverse edge devices. More importantly, they focus on optimizing standard performance metrics (e.g., accuracy and latency), which may not translate into improvement of the users’ actual subjective QoE. In this article, we propose a novel automated and user-centric DNN selection engine, called <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathsf {Aquaman}$ </tex-math></inline-formula> , which keeps users into a closed loop and leverages their QoE feedback to guide DNN selection decisions. The core of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathsf {Aquaman}$ </tex-math></inline-formula> is a neural network-based QoE predictor, which is continuously updated online. Additionally, we use neural bandit learning to balance exploitation and exploration, with a provably efficient QoE performance. Finally, we evaluate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathsf {Aquaman}$ </tex-math></inline-formula> on a 15-user experimental study as well as synthetic simulations, demonstrating the effectiveness of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathsf {Aquaman}$ </tex-math></inline-formula> .

360-Degree Video Bandwidth Reduction: Technique and Approaches Comprehensive Review

Recently, the usage of 360-degree videos has prevailed in various sectors such as education, real estate, medical, entertainment and more. The development of the Virtual World “Metaverse” demanded a Virtual Reality (VR) environment with high immersion and a smooth user experience. However, various challenges are faced to provide real-time streaming due to the nature of high-resolution 360-degree videos such as high bandwidth requirement, high computing power and low delay tolerance. To overcome these challenges, streaming methods such as Dynamic Adaptive Streaming over HTTP (DASH), Tiling, Viewport-Adaptive and Machine Learning (ML) are discussed. Moreover, the superiorities of the development of 5G and 6G networks, Mobile Edge Computing (MEC) and Caching and the Information-Centric Network (ICN) approaches to optimize the 360-degree video streaming are elaborated. All of these methods strike to improve the Quality of Experience (QoE) and Quality of Service (QoS) of VR services. Next, the challenges faced in QoE modeling and the existing objective and subjective QoE assessment methods of 360-degree video are presented. Lastly, potential future research that utilizes and further improves the existing methods substantially is discussed. With the efforts of various research studies and industries and the gradual development of the network in recent years, a deep fake virtual world, “Metaverse” with high immersion and conducive for daily life working, learning and socializing are around the corner.

Stimuli Generation for Quality-of-Experience Evaluation of Mobile Applications

The assessment of Quality-of-Experience (QoE) corresponding to a given set of Quality-of-Service (QoS) parameters is of high importance for any mobile network operator. Methods for subjective QoE assessment such as MUlti Stimulus test with Hidden Reference and Anchors (MUSHRA) or Subjective Assessment Methodology for Video Quality (SAMVIQ) are based on collecting user feedback for one or several stimuli, i.e. differently processed versions of a source. &#x0D; Generating these stimuli corresponding to a given set of QoS parameters for a QoE assessment of a mobile application is a time-consuming, non-trivial task. Therefore, in this paper we propose a novel publicly available open-source framework for stimuli generation for QoE assessment of mobile applications. It is based on the combination of network emulation and automatic user-interface control of a real mobile device, whose behavior is recorded and post-processed to generate suitable stimuli. &#x0D; The article presents the basic concept of the framework, describes its open-source implementation, and concludes with an initial evaluation of the framework based on typical types of mobile applications.

Crowdsourcing platform for QoE evaluation for cloud multimedia services

This paper presents a novel web-based crowdsourcing platform for the assessment of the subjective and objective quality of experience (QoE) of the video service in the cloud-server environment. The user has the option to enter subjective QoE data for video service by filling out a web questionnaire. The objective QoE data of the cloud-server, network condition, and the user device is automatically captured by the crowdsourcing platform. Our proposed system collects both objective and subjective QoE simultaneously in real-time. The paper presents the key technologies used in the development of the platform and describes the functional requirements and design ideas of the system in detail. The system collects real-time comprehensive data to enhance the quality of the user experience to provide a valuable reference. The system is tested in a real-time environment and the test results are given in terms of the system performance. The crowdsourcing platform has new features of real-time network monitoring, the client device, and cloud monitoring, which currently has not been provided by existing web platforms and crowdsourcing frameworks. The results show that 1MB buffer is filled 100% very soon after starting watching videos from the crowdsourcing platform.

An automated model for the assessment of QoE of adaptive video streaming over wireless networks

Nowadays, heterogeneous devices are widely utilizing Hypertext Transfer Protocol (HTTP) to transfer the data. Furthermore, HTTP adaptive video streaming (HAS) technology transmits the video data over wired and wireless networks. In adaptive technology services, a client’s application receives a streaming video through the adaptation of its quality to the network condition. However, such a technology has increased the demand for Quality of Experience (QoE) in terms of prediction and assessment. It can also cause a challenging behavior regarding subjective and objective QoE evaluations of HTTP adaptive video over time since each Quality of Service (QoS) parameter affects the QoE of end-users separately. This paper introduces a methodology design for the evaluation of subjective QoE in adaptive video streaming over wireless networks. Besides, some parameters are considered such as video characteristics, segment length, initial delay, switch strategy, stalls, as well as QoS parameters. The experiment’s evaluation demonstrated that objective metrics can be mapped to the most significant subjective parameters for user’s experience. The automated model could function to demonstrate the importance of correlation for network behaviors’ parameters. Consequently, it directly influences the satisfaction of the end-user’s perceptual quality. In comparison with other recent related works, the model provided a positive Pearson Correlation value. Simulated results give a better performance between objective Structural Similarity (SSIM) and subjective Mean Opinion Score (MOS) evaluation metrics for all video test samples.

A QoE adaptive management system for high definition video streaming over wireless networks

The development of the smart devices had led to demanding high-quality streaming videos over wireless communications. In Multimedia technology, the Ultra-High Definition (UHD) video quality has an important role due to the smart devices that are capable of capturing and processing high-quality video content. Since delivery of the high-quality video stream over the wireless networks adds challenges to the end-users, the network behaviors ‘factors such as delay of arriving packets, delay variation between packets, and packet loss, are impacted on the Quality of Experience (QoE). Moreover, the characteristics of the video and the devices are other impacts, which influenced by the QoE. In this research work, the influence of the involved parameters is studied based on characteristics of the video, wireless channel capacity, and receivers’ aspects, which collapse the QoE. Then, the impact of the aforementioned parameters on both subjective and objective QoE is studied. A smart algorithm for video stream services is proposed to optimize assessing and managing the QoE of clients (end-users). The proposed algorithm includes two approaches: first, using the machine-learning model to predict QoE. Second, according to the QoE prediction, the algorithm manages the video quality of the end-users by offering better video quality. As a result, the proposed algorithm which based on the least absolute shrinkage and selection operator (LASSO) regression is outperformed previously proposed methods for predicting and managing QoE of streaming video over wireless networks.

Towards Perceptually Optimized Adaptive Video Streaming-A Realistic Quality of Experience Database

Measuring Quality of Experience (QoE) and integrating these measurements into video streaming algorithms is a multi-faceted problem that fundamentally requires the design of comprehensive subjective QoE databases and objective QoE prediction models. To achieve this goal, we have recently designed the LIVE-NFLX-II database, a highly-realistic database which contains subjective QoE responses to various design dimensions, such as bitrate adaptation algorithms, network conditions and video content. Our database builds on recent advancements in content-adaptive encoding and incorporates actual network traces to capture realistic network variations on the client device. The new database focuses on low bandwidth conditions which are more challenging for bitrate adaptation algorithms, which often must navigate tradeoffs between rebuffering and video quality. Using our database, we study the effects of multiple streaming dimensions on user experience and evaluate video quality and quality of experience models and analyze their strengths and weaknesses. We believe that the tools introduced here will help inspire further progress on the development of perceptually-optimized client adaptation and video streaming strategies. The database is publicly available at http://live.ece.utexas.edu/research/LIVE NFLX II/live nflx plus.html.

Statistical methods and models based on quality of experience distributions

Due to biased assumptions on the underlying ordinal rating scale in subjective Quality of Experience (QoE) studies, Mean Opinion Score (MOS)-based evaluations provide results, which are hard to interpret and can be misleading. This paper proposes to consider the full QoE distribution for evaluating, reporting, and modeling QoE results instead of relying on MOS-based metrics derived from results based on ordinal rating scales. The QoE distribution can be represented in a concise way by using the parameters of a multinomial distribution without losing any information about the underlying QoE ratings, and even keeps backward compatibility with previous, biased MOS-based results. Considering QoE results as a realization of a multinomial distribution allows to rely on a well-established theoretical background, which enables meaningful evaluations also for ordinal rating scales. Moreover, QoE models based on QoE distributions keep detailed information from the results of a QoE study of a technical system, and thus, give an unprecedented richness of insights into the end users’ experience with the technical system. In this work, existing and novel statistical methods for QoE distributions are summarized and exemplary evaluations are outlined. Furthermore, using the novel concept of quality steps, simulative and analytical QoE models based on QoE distributions are presented and showcased. The goal is to demonstrate the fundamental advantages of considering QoE distributions over MOS-based evaluations if the underlying rating data is ordinal in nature.

Less is (Just as Good as) More - an Investigation of Odor Intensity and Hedonic Valence in Mulsemedia QoE using Heart Rate and Eye Tracking

Using olfactory media to enhance traditional multimedia content opens up novel opportunities for user interactions. Whilst the influence of olfaction on user experience in mulsemedia (multiple sensorial media) environments has been previously studied, the impact of the fundamental dimensions of scent intensity and valence (odor hedonic dimension or pleasantness) have been largely unexplored. This is precisely what we target in this paper, which reports the results of an empirical investigation examining how scent intensity and valence impact mulsemedia Quality of Experience (QoE). Accordingly, 54 participants were exposed to different odor valences and scent intensity levels when viewing three short multimedia clips. In particular, we examine both subjective (self-reported) as well as objective QoE metrics, as evidenced by user heart rates and eye gaze patterns. Results show that whilst eye gaze patterns are largely unaffected by the experimental conditions, valence does have a statistically significant impact upon user heart rates, as does intensity for two of the three clips employed in our study. In terms of subjective QoE, results indicate that hedonic valence impacts on the sense of reality and enjoyment; however varying odor intensity levels do not seem to differentially impact on user experience, bringing into question the need for strong scent intensities.

The evaluation video quality in social clouds

Social clouds are nowadays very popular among users around the world, and users post videos and images on their wall to share with family and friends. Social cloud automatically compressed the videos during the upload process for fast load in webpages and save storage, which decreases the Quality of Experience (QoE) of the end-user. Previously subjective QoE of social cloud videos was conducted to measure the user satisfaction level of social cloud's compression, but objective QoE is never considered in research. Subjective QoE based on user perception, but the user did not know how much noise is added or blurred videos during compress and unable to provide accurate ratings. In this regards, technical details such as noise level, blurring, blocking and data frame metric (DFM) are required to know the comparison of original videos and compressed videos of social clouds. This paper is based on objective QoE of assessment of video quality of original and compressed videos of social clouds, such as Facebook, Qzone, and Tumblr. The results show Facebook adds more noise, blurring, blocking and DFM effects as compared to Qzone and Tumblr social clouds.

QoE Assessment of Mobile Multiparty Audiovisual Telemeetings

The expectations of modern mobile users are increasingly moving towards being able to access demanding services regardless of context or system influence factors, such as network conditions, service topology, and device processing capabilities. Multiparty audiovisual telemeetings are an example of a real-time, delay sensitive, and heavy load service, demanding to run on smartphones that are limited in display size, processing power, and battery capacity. In this paper, we first provide an overview of multiparty audiovisual calls established via mobile devices and key aspects influencing Quality of Experience (QoE). We then report on the results of five user studies conducted over the course of the past 4 years, focused on investigating the impact of video quality in terms of different video encoding parameter configurations (namely bitrate, frame rate, and resolution) on subjective QoE scores for WebRTC-based video calls. We identify lower and upper bounds on video configuration parameters when used in the context of three-party calls. Results have shown that in certain cases it is better to provide constant lower objective video quality than to switch between higher and lower qualities, since participants start to perceive impairments. Finally, we investigate the relationship between objectively measured video quality impairments (blurriness and blockiness) and subjective user scores. Obtained results indicate that the Birnbaum-Saunders distribution for blockiness and the Burr and Gamma distributions for blurriness provide good fits for quality ratings. Gathered results aim to provide input for deriving QoE-aware service adaptation strategies, enabling increased resource allocation efficiency while maintaining acceptable end-user QoE.

A Longitudinal Study on Quality of Experience (QoE) measures to predict customer’s Likelihood to Recommend (L2R) a service

Models that predict satisfaction with a service over time need to consider the impact of emotions and remembered quality of experience in predicting overall attitudes towards a service. However, prior research on subjective quality of experience has typically focused on experiments conducted in a single session or over a short period of time. Thus, there is a gap between our understanding of instantaneous quality of experience and long-term judgments, such as overall satisfaction, and likelihood to recommend and likelihood to churn. The goal of the study reported here was to carry out a longitudinal study that would provide initial insights into how experiences of service quality over time are accumulated into memories that then drive longer term attitudes about the service. Our longitudinal study was carried out over a period of roughly 4 weeks with around 3 sessions per week. To facilitate the study, an online service was constructed that would let participants search through YouTube videos, and that added impairments (specified according to an overall experimental design) to the videos before they were played. Participants were asked to rate several measures, including Technical Quality, after each video was viewed. They were also asked to give overall impressions after each session of five videos had been viewed. The results were analyzed in terms of both sequencing effects within sessions. and memory effects that carried over between sessions.

Seeing Isn’t Believing: QoE Evaluation for Privacy-Aware Users

More and more network media users concern about their privacy issues since they know that their network behaviors are being observed, and thus the observable users’ data are not reliable and sufficient in this case. How to evaluate the true quality of experience (QoE) of the privacy-aware users has become a significant technical challenge because of the most majority of existing data-driven QoE evaluation schemes based on the premise of the true and adequate users’ observations. To get over this dilemma, this paper proposes a systematic and robust QoE evaluation scheme with unreliable and insufficient observation data. Specifically, we first translate the subjective privacy-aware QoE evaluation problem into an objective rational user analysis procedure. Then, a semantics-based similarity measurement for multidimensional correlation analysis is constructed to classify the observable data. Subsequently, the highlight of this paper lies in proposing a class-level joint user classification and data cleaning strategy by frequently updating the training processes. Through elaborately designing an iterative framework, it can effectively resolve the data sparsity and inconsistency problems due to the user privacy-aware preferences. Importantly, we also introduce an efficient QoE model construction method for online implementation, and numerical results validate its efficiency for different kinds of privacy-aware users.

Learning QoE of Mobile Video Transmission With Deep Neural Network: A Data-Driven Approach

Quality of experience (QoE) serves as a direct evaluation of users' experiences in mobile video transmission and thus essential for network management, such as network optimization. In this paper, we propose a deep learning-based QoE prediction approach with a large-scale QoE dataset for mobile video transmission. Specifically, we develop a mobile phone application for collecting user QoE data when viewing videos transmitted over the mobile internet in a practical environment. Then, we construct a large-scale dataset by collecting over 80000 piece of data with four kinds of subjective scores and 89 network parameters. Each QoE metric is related to only some of the 89 network parameters. Therefore, we apply the feature selection method to find the feature parameters related to user scores. Additionally, the boxplot method is used to clean the raw data by removing outliers. Finally, a deep neural network (DNN) is developed to learn the relationships between the network parameters and the subjective QoE scores. The proposed DNN can also be seen as a data-driven objective QoE prediction approach for mobile video transmission, which can be used to predict the user QoE scores. The experimental results show that the proposed approach can effectively remove most features irrelevant to QoE prediction. Moreover, the performance of QoE prediction by the proposed model outperforms other state-of-the-art approaches.

Perceptual quality assessment of video using machine learning algorithm

User experience has become the most reliable and trustworthy source for service providers to assess system performance. To fulfill customer requirements, service providers require an efficient quality of experience (QoE) estimation model. QoE is a subjective metric that deals with user perception and can vary dramatically due to various factors such as emotions, the degree of annoyance, past experience, and aesthetics. Moreover, subjective QoE evaluation is expensive and time-consuming because of human participation. Therefore, a model is required to objectively measure QoE with reasonable accuracy. In the context of service and network providers fulfilling user requirements, with a reasonable quality of service (QoS), needs to be provided to the relevant services/applications. However, QoS parameters do not reflect subjective opinion of the user accurately. Therefore, it is necessary to compute the mapping or correlation between QoS and QoE. The mapping may help service providers to understand the behavior of the overall network on user experience and efficiently manage the network resources. In this paper, a new feature number of displayed frames impacted (NoDFI) along with a machine learning based model is presented to compute a correlation between QoS and QoE. A publically available dataset is used to represent the correlation between objective QoS parameters and subjective QoE metric. The result of experiments showed that proposed feature NoDFI proved to be a valuable addition when compared with previously proposed models.

Optimal inter-view rate allocation for multi-view video plus depth over MPEG-DASH using QoE measures and paired comparison

Rapid advances in compression and transmission technologies enabled multi-view video streaming over the Internet through state-of-the-art MPEG standards such as dynamic adaptive streaming over HTTP (DASH) and high-efficiency video coding. Yet, standards for subjective and objective measurement of quality of experience (QoE) are lagging in the area. The MPEG-DASH system requires constant bitrate (CBR) coding and an efficient rate control strategy in order to increase QoE. Multi-view video plus depth format is preferable due to high compression gain but quality of intermediate view is crucial, strictly depending on performance of inter/intra-view rate allocation as well as rendering algorithms. We propose to find optimal ratio for inter-view rate allocation in CBR using our previously developed objective QoE measures using depth maps and structural similarities. To assess small differences and discriminate similar quality levels, a new subjective assessment methodology is also proposed as a multiple stimuli plus simultaneous presentation for paired comparison. The results show that the unequal rate allocation is superior to the equal rate allocation in terms of objective QoE as 0.012–0.04 (which translates into a bit rate saving of 15–24%) and subjective QoE as 0.3–0.5. Furthermore, a very high correlation is achieved between the new QoE measures and paired comparison results.

Visual expectations change subjective experience without changing performance.

It is widely believed that visual expectations can change the subjective experiences of humans. We investigated how visual expectations in a recognition task affected objective performance and subjective perception. Using a 2-alternative-forced-choice task based on digit recognition of briefly presented and visually masked digits, we found over two experiments that expectations changed the quality of the experiences without changing the performance capabilities associated with the quality of experience. Expectations were manipulated by providing a cue indicating the set of possible digits that might appear on each trial. The results also inform the debate about whether subjective experiences can be categorized in a dichotomous manner or in a graded manner. We found that subjective experiences were graded near the objective threshold and more dichotomous away from the threshold. Furthermore, distinct expectations resulted in a more dichotomous distribution of subjective experience. We also provide evidence of an interesting relationship between stimulus duration, objective performance and subjective ratings. Only experiences that were rated as evoking some degree of perception showed systematic improvements in objective performance as a function of stimulus duration. These findings suggest that subjective experience cannot be understood without considering the broader cognitive context, namely that the quality of subjective experiences is dependent on a multitude of factors such as attention, task requirements and cognitive expectations.

Seamless handover for video streaming over an SDN-based Aeronautical Communications Network

There have been increasing interests in applying Software Defined Networking (SDN) to aeronautical communications primarily for air traffic management purposes. From the service passenger communications' point of view, a major goal is to improve passengers' perception of quality of experience on the infotainment services being provided for them. Due to the high speed of aircrafts and the use of multiple radio technologies during different flight phases and across different areas, vertical handovers between these different radio technologies are envisaged. This poses a challenge to maintain the quality of service during such handovers, especially for high bandwidth applications such as video streaming. This paper proposes an SDN-based aeronautical communications architecture consisting of both satellite and terrestrial-based radio technology. In addition, an experimental implementation of the Locator ID Separation Protocol (LISP) protocol with built-in multi-homing capability over the SDN-based architecture was proposed to handle vertical handovers between the satellite and other radio technologies onboard the aircraft. By using both objective and subjective Quality of Experience (QoE) metrics, the simulation experiments show the benefit of combining LISP with SDN to improve the video streaming quality during the handover in the aeronautical communication environment.

QoE-Oriented Rate Adaptation for DASH With Enhanced Deep Q-Learning

With the popularity of handheld devices, the development of wireless communication technology and the proliferation of multimedia resources, mobile video has become the main business in LTE networks with explosive traffic demands. How to improve the quality of experience (QoE) of mobile video in the dynamic and complex network environment has become a research focus. Dynamic adaptive streaming over HTTP technology introduces adaptive bitrate (ABR) requests at the client side to improve video QoE and various rate adaptation algorithms are also constantly proposed. In view of the limitations of the existing heuristic or learning-based ABR methods, we propose redirecting enhanced Deep Q-learning toward DASH video QoE (RDQ), a QoE-oriented rate adaptation framework based on enhanced deep Q-learning. First, we establish a chunkwise subjective QoE model and utilize it as the reward function in reinforcement learning so that the strategy can converge toward the direction of maximizing the subjective QoE score. Then, we apply several effective improvements of deep Q-learning to the RDQ agent’s neural network architecture and learning mechanism to achieve faster convergence and higher average reward than other learning-based methods. The proposed RDQ agent has been thoroughly evaluated using trace-based simulation on the real-time LTE network data. For disparate network scenarios and different video contents, the RDQ agent can outperform the existing methods in terms of the QoE score. The breakdown analysis shows that RDQ can suppress the number and the duration of the stalling events to the minimum while maintaining high video bitrate, thus achieving better QoE performance than other methods.