Subjective speech quality measurement with and without parallel task: Laboratory test results comparison

Hakob Avetisyan,Jan Holub,Gavin Kearney

doi:10.1371/journal.pone.0199787

Abstract

This paper focuses on a novel methodology of subjective speech quality measurement and repeatability of its results between laboratory conditions and simulated environmental conditions. A single set of speech samples was distorted by various background noises and low bit-rate coding techniques. This study aimed to compare results of subjective speech quality tests with and without a parallel task deploying the ITU-T P.835 methodology. Afterward, tests results performed with and without a parallel task were compared using Pearson correlation, CI95, and numbers of opposite pair-wise comparisons. The tests show differences in results in the case of a parallel task.

Highlights

Each generation of mobile phones has different advanced features and characteristics designed to have a better quality of voice processing and noise suppression
The principle of P.835 is to repeat the assessment of each speech sample three times, requiring the subjects to focus on a different aspect of the sample quality during each assessment
Speech Mean Opinion Score (MOS) (S-MOS) comparison between A and B tests are shown in S1 Fig. Its Pearson correlation coefficient value is 0.971

Summary

Introduction

Each generation of mobile phones has different advanced features and characteristics designed to have a better quality of voice processing and noise suppression. Objective methods [2] are used to replace test subjects using psycho-acoustic modeling, comparing clean and distorted speech samples algorithmically Outputs from these two method groups are often mapped to the subjective quality scale Mean Opinion Score (MOS) [2]. Listeners evaluate tested samples on a five-point scale This procedure is suitable for samples processed by noise canceling algorithms that remove certain part of background noise and corrupt the speech itself. This means that some subjects decreased the speech quality score due to background noise even for non-distorted speech and considered speech distorted by artificial coding artifact as noisy It indicates that the P.835 methodology is too complex if used with the parallel task of the described type. Not all subjects can correctly assess speech distortion (only) and background noise annoyance (only) in different playouts as required by the P.835, as they are distracted by another task in parallel

Objectives

Methods

Results

Conclusion