SPQER: Speech Quality Evaluation Using Word Recognition for VoIP Communication in Lossy and Mobile Networks

Bertram Schuetz,Nils Aschenbruck

doi:10.1109/ojcs.2020.3011392

Bertram Schuetz, Nils Aschenbruck

Open Access

https://doi.org/10.1109/ojcs.2020.3011392

Copy DOI

Abstract

In this paper, we introduce SPQER (pronounced speaker), a novel approach to evaluate the quality of experience for real-time Voice over IP (VoIP) communication in mobile and lossy networks. Traditional speech quality metrics, e.g., Perceptual Evaluation of Speech Quality (PESQ) or the Hearing-Aid Speech Quality Index (HASQI), directly compare frequencies and amplitudes to calculate the received signal distortions. SPQER instead uses machine learning classification to evaluate the percentage of recognizable words in conjunction with a time-based decay function to penalize delay and cross-talking. So instead of evaluating noise, SPQER directly answers the question: What percentage of words is the recipient able to understand? We presented a sensitivity analysis, which is based on testbed experiments for different packet loss rates and simulated delays, to asses the impact of challenging link conditions. A final correlation analysis to a short user study shows that SPQER can better evaluate the amount of understandable words than PESQ and HASQI, while still giving a more precise indication about the voice quality than the Word Error Rate (WER) metric.

Highlights

Voice over IP (VoIP) telephony is a central pillar of modern communication
To quantify the quality of a VoIP call from a user perspective, traditional Quality of Experience (QoE) metrics are based on signal-comparison, e.g, the Perceptual Evaluation of Speech Quality (PESQ) [8] or the Hearing-Aid Speech Quality Index (HASQI) [11]
The recordings are used to calculate the values for PESQ, HASQI, Word Error Rate (WER) and Speech Quality Evaluation using Word Recognition (SPQER)

Summary

Introduction

Voice over IP (VoIP) telephony is a central pillar of modern communication. With the growing mobile infrastructure, more available bandwidth, and cheaper data plans, an increasing number of VoIP calls are made from mobile devices. Since most VoIP applications are based on RTP over UDP, there is no transport layer reliability to recover lost packets Such loss can occur due to interferences or signal obstruction, which is especially common in mobile scenarios. To quantify the quality of a VoIP call from a user perspective, traditional Quality of Experience (QoE) metrics are based on signal-comparison, e.g, the Perceptual Evaluation of Speech Quality (PESQ) [8] or the Hearing-Aid Speech Quality Index (HASQI) [11]. These metrics were developed to evaluate the artifacts and distortions, which occur due to compression by the voice encoding. Instead of performing a straight signal comparison, SPQER uses machine learning-based word recognition to evaluate

Objectives

Results

Conclusion