Effect of Environmental Noise in Speech Quality Assessment Studies using Crowdsourcing

Abstract

Crowdsourcing is a valid approach to collect and annotate data efficiently and cost-effectively. This approach permits us to reach a large and diverse pool of users that usually work from home employing their computers and headphones. Still, there is insufficient information about the users' surroundings. Specifically, little knowledge about the background noise to which users might be exposed to when executing crowd-work. The validity of the data gathered in a disturbed environment is questionable, especially in speech quality assessment and other audio-related tasks. This work presents the results of a simulated crowdsourcing study conducted in the laboratory. We investigate the influence of environmental background noise in speech quality assessment tests. Three groups of listeners were recruited to rate the quality of speech files under the influence of background noise at different levels. Two types of noise were tested, i.e., street noises and TV-Show. Our findings suggest that the threshold at which an environmental background noise would significantly affect the speech quality ratings in crowdsourcing is between 43dB(A) and 50dB(A). Additionally, listeners tolerated more the TV-Show noise. They provided more accurate ratings while conducting the test under the influence of higher levels of the TV-Show noise, than at lower levels of the street noise. We also found that the presence of background noise does not cause a constant bias of the quality scores; instead, its impact depends on the speech degradation condition under test.