User Experience Metrics Research Articles

Using Vision and Speech Features for Automated Prediction of Performance Metrics in Multimodal Dialogs

Predicting and analyzing multimodal dialog user experience (UX) metrics, such as overall call experience, caller engagement, and latency, among other metrics, in an ongoing manner is important for evaluating such systems. We investigate automated prediction of multiple such metrics collected from crowdsourced interactions with an open-source, cloud-based multimodal dialog system in the educational domain. We extract features from both the audio and video signals and examine the efficacy of multiple machine learning algorithms in predicting these performance metrics. The best performing audio features consist of multiple low-level audio descriptors—intensity, loudness, cepstra, pitch, and so on—and their functionals, extracted using the OpenSMILE toolkit, while the video features are bags of visual words that use 3D Scale-Invariant Feature Transform descriptors. We find that our proposed methods outperform the majority vote classification baseline in predicting various UX metrics rated by both the user and experts. Our results suggest that such automated prediction of performance metrics can not only inform the qualitative and quantitative analysis of dialogs but also be potentially incorporated into dialog management routines for positively impacting UX and other metrics during the course of the interaction.

Priority-based content processing with Q-routing in information-centric networking (ICN)

With exponential increase in the number of users and available data, service providers are facing hard times to satisfy and improve end user experience. Researchers have come up with the idea of exploiting increasing number of routers in a network, and it leads to the development of information-centric networking (ICN). Efficient usage of the in-network caches and content forwarding methodology are the key issues in an ICN architecture. ICN reduces average hop count and correspondingly average content download delay because the intra-domain routers in ICN have storage capacity and they can act as temporary content provider. In this paper, we address the content management issue in a cache with finite storage capability and propose an efficient content management policy that changes a router to a self-sustained cache. We propose a novel methodology to process content packets in the buffer of a cache and correspondingly reduce the propagation delay through a cache. We simulate our proposed algorithm over real-life network environment and evaluate the performance of different user experience metrics, e.g. average latency, throughput, goodput, and link load. Simulation results suggest that our proposed model outperforms the existing state-of-the-art on-path caching strategies.

Throughput and Delay Analysis in Video Streaming Over Block-Fading Channels

We study video streaming over a slow-fading wireless channel. In a streaming application, video packets are required to be decoded and displayed in the order they are transmitted as the transmission goes on. This results in per-packet delay constraints, and the resulting channel can be modeled as a physically degraded fading broadcast channel with as many virtual users as the number of packets. In this paper, we study two important quality of user experience (QoE) metrics, namely throughput and interdecoding delay. We introduce several transmission schemes, and compare their throughput and maximum interdecoding delay performances. We also introduce a genie-aided scheme, which provides theoretical bounds on the achievable performance. We observe that adapting the transmission rate at the packet level, i.e., periodically dropping a subset of the packets, leads to a good tradeoff between the throughput and the maximum interdecoding delay. We also show that an approach based on initial buffering leads to an asymptotically vanishing packet loss rate at the expense of a relatively large initial delay. For this scheme, we derive a condition on the buffering time that leads to throughput maximization.

Avoiding Interruptions — A QoE Reliability Function for Streaming Media Applications

We take an analytical approach to study fundamental rate-delay-reliability trade-offs in the context of media streaming. We consider the probability of interruption in media playback (buffer underflow) as well as the number of initially buffered packets (initial waiting time) as the Quality of user Experience (QoE) metrics. We characterize the optimal trade-off between these metrics as a function of system parameters such as the packet arrival rate and file size, for different channel models. In the first model, we assume packets arrive according to independent Poisson processes from multiple servers or peers. We use random linear network coding to simplify the packet requests at the network layer and avoid duplicate packet reception. This allows us to model the receiver's buffer as a queue with Poisson arrivals and deterministic departures. For this model, we show that for arrival rates slightly larger than the play rate, the minimum initial buffering required to achieve certain level of interruption probability remains bounded as the file size grows. This is not the case when the arrival rate and the play rate match. In the second model, we consider channels with memory, which can be modeled using Markovian arrival processes. We characterize the optimal trade-off curves for the infinite file size case, in such Markovian environments.

A Strategic Approach to Metrics for User Experience Designers

User experience (UX) designers asked to justify return-on-investment (ROI) for UX activities often rely on published ROI studies and UX metrics that do not address decision makers' concerns. With a little knowledge of business strategy and metrics and an understanding of their own value to an organization, UX practitioners can (a) identify the financial and non-financial metrics and goals that drive change in their organizations, (b) draw a clear picture for decision makers of the connection between their value and the company’s goals, and (c) demonstrate a positive return on investment in UX activities.