Quality Of Experience Evaluation Research Articles

A Tutorial for Olfaction-Based Multisensorial Media Application Design and Evaluation

Recently, multimedia researchers have added several so-called new media to the traditional multimedia components (e.g., olfaction, haptic, and gustation). Evaluating multimedia user-perceived Quality of Experience (QoE) is already non-trivial and the addition of multisensorial media components increases this challenge. No standardized methodology exists to conduct subjective quality assessments of multisensorial media applications. To date, researchers have employed different aspects of audiovisual standards to assess user QoE of multisensorial media applications and thus, a fragmented approach exists. In this article, the authors highlight issues researchers face from numerous perspectives including applicability (or lack of) existing audiovisual standards to evaluate user QoE and lack of result comparability due to varying approaches, specific requirements of olfactory-based multisensorial media applications, and novelty associated with these applications. Finally, based on the diverse approaches in the literature and the collective experience of authors, this article provides a tutorial and recommendations on the key steps to conduct olfactory-based multisensorial media QoE evaluation.

Investigation of Quality of Experience Using ECG Features during Consumption of Audiovisual Content

Development of new multimedia technologies allows us to receive better quality of audio and video content. Quality of Experience (QoE) evaluates given content from the consumer’s perspective. This measurement allows to evaluate not only visual and audible quality, but also general acceptability of provided service. QoE evaluation is getting popular between engineers, designers, retailers who wants to provide high quality content for consumers. QoE is generally evaluated subjectively by surveys. It is possible to find relationship between physiological signals measured while user is consuming audiovisual content and make the subjective evaluation of this experience. This paper investigates relationship between heart rate and QoE while user is watching 1 min duration video recordings on three different devices. Heart rate was calculated as mean RR interval for each recording. Mean RR intervals of 0.848 s, 0.869 s and 0.884 s were calculated for low, medium and high QoE device configurations, respectively. ANOVA analysis results indicates a relation between heart rate and QoE level. The results can help to develop further the investigations of QoE level and heart rate relationship for various subjective assessment, device configurations and content provided.

Quality of experience evaluation of HTTP video streaming based on user interactive behaviors

User interactive behaviors play a dual role during the hypertext transfer protocol (HTTP) video service: reflection and influence. However, they are seldom taken into account in practices. To this end, this paper puts forward the user interactive behaviors, as subjective factors of quality of experience (QoE) from viewer level, to structure a comprehensive multilayer evaluation model based on classic network quality of service (QoS) and application QoS. First, dual roles of user behaviors are studied and the characteristics are extracted where the user experience is correlated with user interactive behaviors. Furthermore, we categorize QoE factors into three dimensions and build the metric system. Then we perform the subjective tests and investigate the relationships among network path quality, user behaviors, and QoE. Ultimately, we employ the back propagation neural network (BPNN) to validate our analysis and model. Through the simulation experiment of mathematical and BPNN, the dual effects of user interaction behaviors on the reflection and influence of QoE in the video stream are analyzed, and the QoE metric system and evaluation model are established.

HAS QoE prediction based on dynamic video features with data mining in LTE network

Evaluation of HTTP adaptive streaming (HAS) quality of experience (QoE) over LTE network is a challenging topic because of multi-segment and multi-rate features of dynamic video sequences. Different from the traditional QoE evaluation methods based on network parameters, this paper proposes the HAS QoE prediction methods based on its dynamic video segment features with data mining. Considering the application requirement of the trade-off between accuracy and complexity, two sets of methodologies are designed to evaluate the HAS QoE including regression and classification. In regression method, we propose the evolved PSNR (ePSNR) model using differential peak signal to noise ratio (dPSNR) statistics as the segment features to evaluate HAS QoE. In classification method, we propose the improved weighted k-nearest neighbors (WkNN) by using dynamic weighted mapping according to the position of video chunk to meet the dynamic segment and rate features of HAS. In order to train and test these methods, we build a real-time HAS video-on-demand (VOD) system in LTE network and do subjective test in different video scenes. With the mean opinion score (MOS), the regression and classification methods are trained to predict the HAS QoE. The validated results show that the proposed ePSNR and WkNN methods outperform other evaluation methods.

QoE collaborative evaluation method based on fuzzy clustering heuristic algorithm.

At present, to realize or improve the quality of experience (QoE) is a major goal for network media transmission service, and QoE evaluation is the basis for adjusting the transmission control mechanism. Therefore, a kind of QoE collaborative evaluation method based on fuzzy clustering heuristic algorithm is proposed in this paper, which is concentrated on service score calculation at the server side. The server side collects network transmission quality of service (QoS) parameter, node location data, and user expectation value from client feedback information. Then it manages the historical data in database through the “big data” process mode, and predicts user score according to heuristic rules. On this basis, it completes fuzzy clustering analysis, and generates service QoE score and management message, which will be finally fed back to clients. Besides, this paper mainly discussed service evaluation generative rules, heuristic evaluation rules and fuzzy clustering analysis methods, and presents service-based QoE evaluation processes. The simulation experiments have verified the effectiveness of QoE collaborative evaluation method based on fuzzy clustering heuristic rules.

QoE Evaluation of Multimedia Services Based on Audiovisual Quality and User Interest

Quality of experience (QoE) has significant influence on whether or not a user will choose a service or product in the competitive era. For multimedia services, there are various factors in a communication ecosystem working together on users, which stimulate their different senses inducing multidimensional perceptions of the services, and inevitably increase the difficulty in measurement and estimation of the user's QoE. In this paper, a user-centric objective QoE evaluation model (QAVIC model for short) is proposed to estimate the user's overall QoE for audiovisual services, which takes account of perceptual audiovisual quality (QAV) and user interest in audiovisual content (IC) amongst influencing factors on QoE such as technology, content, context, and user in the communication ecosystem. To predict the user interest, a number of general viewing behaviors are considered to formulate the IC evaluation model. Subjective tests have been conducted for training and validation of the QAVIC model. The experimental results show that the proposed QAVIC model can estimate the user's QoE reasonably accurately using a 5-point scale absolute category rating scheme.

A QoE centric distributed caching approach for vehicular video streaming in cellular networks

AbstractDistributed caching‐empowered wireless networks can greatly improve the efficiency of data storage and transmission and thereby the users' quality of experience (QoE). However, how this technology can alleviate the network access pressure while ensuring the consistency of content delivery is still an open question, especially in the case where the users are in fast motion. Therefore, in this paper, we investigate the caching issue emerging from a forthcoming scenario where vehicular video streaming is performed under cellular networks. Specifically, a QoE centric distributed caching approach is proposed to fulfill as many users' requests as possible, considering the limited caching space of base stations and basic user experience guarantee. Firstly, a QoE evaluation model is established using verified empirical data. Also, the mathematic relationship between the streaming bit rate and actual storage space is developed. Then, the distributed caching management for vehicular video streaming is formulated as a constrained optimization problem and solved with the generalized–reduced gradient method. Simulation results indicate that our approach can improve the users' satisfaction ratio by up to 40%. Copyright © 2015 John Wiley & Sons, Ltd.

Recommend trustworthy services using interval numbers of four parameters via cloud model for potential users

How to discover the trustworthy services is a challenge for potential users because of the deficiency of usage experiences and the information overload of QoE (quality of experience) evaluations from consumers. Aiming to the limitations of traditional interval numbers in measuring the trustworthiness of service, this paper proposed a novel service recommendation approach using the interval numbers of four parameters (INF) for potential users. In this approach, a trustworthiness cloud model was established to identify the eigenvalue of INF via backward cloud generator, and a new formula of INF possibility degree based on geometrical analysis was presented to ensure the high calculation precision. In order to select the highly valuable QoE evaluations, the similarity of client-side feature between potential user and consumers was calculated, and the multi-attributes trustworthiness values were aggregated into INF by the fuzzy analytic hierarchy process method. On the basis of ranking INF, the sort values of trustworthiness of candidate services were obtained, and the trustworthy services were chosen to recommend to potential user. The experiments based on a realworld dataset showed that it can improve the recommendation accuracy of trustworthy services compared to other approaches, which contributes to solving cold start and information overload problem in service recommendation.

Crowdsourcing vs. laboratory experiments – QoE evaluation of binaural playback in a teleconference scenario

Experiments for the subjective evaluation of multimedia presentations and content are traditionally conducted in a laboratory environment. In this respect common procedures for the evaluation of teleconference systems are no different. The strictly controlled laboratory environment, however, often gives a rather poor representation of the actual use case. Therefore in this study we crowdsourced the evaluation of a teleconference system to perform the evaluation in a real-life environment. Moreover, we used the unique possibilities of crowdsourcing to employ two different demographics by hiring workers from Germany on the one hand and the US and Great Britain on the other hand. The goal of this experiment was to assess the perceived Quality of Experience (QoE) during a listening test and compare the results to results from a similar listening test conducted in the controlled laboratory environment. In doing so, we observed not only intriguing differences in the collected QoE ratings between the results of laboratory and crowdsourcing experiments, but also between the different worker demographics in terms of reliability, availability and efficiency.

A practical model for ensemble estimation of QoS and QoE in VoIP services via fuzzy inference systems and fuzzy evidence theory

Nowadays, there is an increasing number of Voices over IP (VoIP) services offered in telecommunication networks with specific quality requirements. Such requirements impact upon both the objective quality of service (QoS) of the end-to end connection as well as the subjective quality of experience (QoE) as perceived by the end user. In this study, we have proposed an integrated QoS and QoE evaluation system based on the combination of fuzzy inference systems and fuzzy evidence theory. For quality criteria, we have used QoS as a technical source from the viewpoint of service provider and QoE as the end user approach in front of service. We have divided the positive and negative variables of QoE into two distinct systems. At the same time, a parameter has been proposed as quality, which is the total quality of a service according to the objective and subjective viewpoints. To calculate the parameters mentioned above, a fuzzy inference system has been utilized. In addition, to obtain all data sources as an evidence of quality, Dempster---Shafer evidence theory has been employed. Finally, the proposed approach evaluated through the quality of VoIP services in three real cases and the results are discussed.

Network Performance Testing System Integrating Models for Automatic QoE Evaluation of Popular Services: YouTube and Facebook

Smartphones have significantly changed the traffic pattern usage of mobile subscribers. In addition to the classical file sharing and web browsing usage, social networking applications and access to video content are now a significant percentage of the typical user data traffic consumption. As the change in usage pattern affects the performance of mobile networks, operators need methods to estimate how well the network behaves. The metrics to use for the evaluation of the quality of experience (QoE) are only well documented for traditional applications. In this paper a novel QoE system that automatically captures and processes the metrics needed to evaluate two commonly used applications, the social networking application Facebook and the video download application YouTube, is presented. The system uses a test application developed for Android phones, which report the results to a centralized platform capable of processing and presenting the results in different formats. The system has been used to test those two services over 3G/LTE network technologies. In the case of the YouTube services, performance results show a subjective score very similar for the 3G and LTE because of the automatic adaptation of the video coding rate to the network conditions. Testing of the Facebook application has led to the proposal of a new MOS expression that takes into account the content type being posted, which provides more reasonable MOS estimations than the generic web browsing models available in the literature and commonly used to estimate Facebook QoE.

Statistical user behavior detection and QoE evaluation for thin client services

Remote desktop connection (RDC) services offer clients the ability to access remote content and services, commonly in the context of accessing their working environment. With the advent of cloud-based services, an example use case is that of delivering virtual PCs to users in WAN environments. In this paper, we aim to detect and analyze common user behavior when accessing RDC services, and use this as input for making Quality of Experience (QoE) estimations and subsequently providing input for effective QoE management mechanisms. We first identify different behavioral categories, and conduct traffic analysis to determine a feature set to be used for classification purposes. We propose a machine learning approach to be used for classifying behavior, and use this approach to classify a large number of real-world RDCs. We further conduct QoE evaluation studies to determine the relationship between different network conditions and subjective end user QoE for all identified behavioral categories. Results show an exponential relationship between QoE and delay and loss degradations, and a logarithmic relationship between QoE and bandwidth limitations. Obtained results may be applied in the context of network resource planning, as well as in making QoE-driven resource allocation decisions.

A Method for Evaluating QoE of Live Streaming Services

This paper presents a novel method which can provide accurate real-time evaluation for Quality of Experience (QoE) of live streaming services. First, we use the modified Analytic Hierarchy Process (AHP) to build a QoE model which is mainly based on video quality and network transmission quality in live streaming services. Then, for video quality, we take into account the unique multi-resolution feature of Internet live streaming videos and design a video quality assessment algorithm mainly based on bitrate, temporal perceptual information (TI) and spatial perceptual information (SI). This algorithm removes the interference of multi-resolution and has a good correlation with subjective perception. And for network transmission quality, we design another assessment algorithm based on buffering time and frequency. Finally, by using the QoE model and the assessment algorithms, we can build a real-time QoE monitoring and evaluation system for live streaming services.

A cost-effective and real-time QoE evaluation method for multimedia streaming services

Traditionally, subjective tests are conducted to obtain the Mean Opinion Score (MOS) or Quality of Experience (QoE) of a video streaming service. However, subjective tests are time-consuming and rather costly. Thus, a cost-effective and real-time QoE evaluation method for video streaming services is presented in this paper. The proposed scheme first adopts subjective tests to find the relationships between the QoE and individual QoS parameters. Next, based on subjective test results, the corresponding QoS---QoE mapping functions for individual QoS metrics are derived using the regression approach. Finally, the integrated QoE function, which is a multi-variate function of multiple QoS metrics, for assessing the overall QoE of video streaming services is proposed. Hence, network/service providers only need to measure related objective QoS parameters while the corresponding MOS/QoE can be easily derived in real time based on the integrated QoE function. Since subjective tests only need to be conducted occasionally for updating the parameters in the QoS---QoE mapping functions, the cost of QoE assessment decreases significantly.

User feature-aware trustworthiness measurement of cloud services via evidence synthesis for potential users

Cloud computing can provide elastic and dynamic resources on demand, which facilitates service providers to make profits resulting from the long tail effect. It becomes vitally important to ensure that cloud services can be acceptable to more potential users. However, it is challenging for potential users to discover the trustworthy cloud services due to the deficiency of usage experiences and the information overload of QoE (quality of experience) evaluations from consumers. This paper presents a user feature-aware trustworthiness measurement approach for potential users. In this approach, the influence factors of QoE are systematically analyzed based on the user feature model and the quantitative computation methods are designed to measure the user feature similarity. In addition, employing FAHP (fuzzy analytic hierarchy process) method identifies the user feature community. To enhance the accuracy of trustworthiness measurement, the false evidences in QoE evaluations are iteratively filtered out with dynamic mean distance threshold. Finally, the trustworthiness of service is measured via evidence synthesis combining user feature similarity. The experiments show that this approach is effective to improve the quality of trustworthiness measurement, which is helpful to solve information overload problem and cold start problem of trusted service recommendation for potential users.

Towards Evaluating the Quality of Experience of Remote Patient Monitoring Services

The applicability of advanced mobile technologies in the m-Health domain has led to a number of studies and (limited) commercial products supporting delivery of health services to remote users. A key issue regarding successful delivery and acceptance of such services is meeting their Quality of Service (QoS) and Quality of Experience (QoE) requirements, focusing on technical aspects and end user perceived quality, respectively. In this paper, the authors address the topic of evaluating QoE for non-emergency remote patient monitoring services. They identify relevant QoE influence factors and metrics, and present the results of a QoE evaluation study, whereby they focus on usability aspects. The study involves 26 users testing a prototype version of the Ericsson Mobile Health service, which is based on a smartphone application and measurement of vital signs via medical sensors. The results show a strong correlation between QoE and: perceived effectiveness of the mobile interface (regarding both adequacy of smartphone screen size and smartphone application navigational support), perceived ease of conducting a blood pressure measurement task, and user motivation for service usage.

A Quality of Experience Model for Haptic Virtual Environments

Haptic-based Virtual Reality (VR) applications have many merits. What is still obscure, from the designer's perspective of these applications, is the experience the users will undergo when they use the VR system. Quality of Experience (QoE) is an evaluation metric from the user's perspective that unfortunately has received limited attention from the research community. Assessing the QoE of VR applications reflects the amount of overall satisfaction and benefits gained from the application in addition to laying the foundation for ideal user-centric design in the future. In this article, we propose a taxonomy for the evaluation of QoE for multimedia applications and in particular VR applications. We model this taxonomy using a Fuzzy logic Inference System (FIS) to quantitatively measure the QoE of haptic virtual environments. We build and test our FIS by conducting a users' study analysis to evaluate the QoE of a haptic game application. Our results demonstrate that the proposed FIS model reflects the user's estimation of the application's quality significantly with low error and hence is suited for QoE evaluation.

Toward a Mathematical Model for Quality of Experience Evaluation of Haptic Applications

There is rapid progress in the advancement of user interfaces. One such advancement is enabling the sense of touch, or haptics, as part of the interface. Haptic devices are seeing growth in many types of applications such as gaming and medical simulation. Assessing the quality of experience (QoE) of the user is necessary to evaluate how the user perceives such interfaces. The QoE is a user-centric parameter that shifts the paradigm of evaluation from the technology itself to the user. This paper proposes a mathematical-based QoE evaluation of haptic-based applications. A mathematical model that is able to quantify the QoE of the user is described. By conducting a user study in which users evaluate a haptic-based game application, we were able to test and validate the mathematical model. There are several approaches in determining the weights to be used with the mathematical model. This paper presents and compares different approaches for weight determination, namely even weight distribution, correlation-based weights, even weights–correlation combination, linear regression analysis, and principal component analysis (PCA). Our results show that PCA weight determination performs slightly better than the rest of the approaches.

USING FINITE STATE MODELS FOR QUALITY EVALUATION AT WEB SERVICE DEVELOPMENT STEPS

This paper presents a short study on evaluating the web service quality based on finite state models using the well‐known metrics of Quality of Service (QoS) and Quality of Experience (QoE). These metrics represent objective and subjective assessments. The development of any web service involves several steps and the final values of the service QoS and QoE can be essentially improved/deteriorated based on the quality reached at each development step. In this paper, five service development steps are considered: those are requirements specification, provisioning, composition, implementation, and usage and management. At each step different finite state models for refining the service quality estimation are utilized. A running example of a vacation planner service illustrates the application of finite state models to improve the accuracy of the QoS and QoE evaluation

Crowdsourcing Multimedia QoE Evaluation: A Trusted Framework

Crowdsourcing has emerged in recent years as a potential strategy to enlist the general public to solve a wide variety of tasks. With the advent of ubiquitous Internet access, it is now feasible to ask an Internet crowd to conduct QoE (Quality of Experience) experiments on their personal computers in their own residences rather than in a laboratory. The considerable size of the Internet crowd allows researchers to crowdsource their experiments to a more diverse set of participant pool at a relatively low economic cost. However, as participants carry out experiments without supervision, the uncertainty of the quality of their experiment results is a challenging problem. In this paper, we propose a crowdsourceable framework to quantify the QoE of multimedia content. To overcome the aforementioned quality problem, we employ a paired comparison method in our framework. The advantages of our framework are: 1) trustworthiness due to the support for cheat detection; 2) a simpler rating procedure than that of the commonly-used but more difficult mean opinion score (MOS), which places less burden on participants; 3) economic feasibility since reliable QoE measures can be acquired with less effort compared with MOS; and 4) generalizability across a variety of multimedia content. We demonstrate the effectiveness and efficiency of the proposed framework by a comparison with MOS. Moreover, the results of four case studies support our assertion that the framework can provide reliable QoE evaluation at a lower cost.