Speaker Verification by Human Listeners: Experiments Comparing Human and Machine Performance Using the NIST 1998 Speaker Evaluation Data

Abstract

Schmidt-Nielsen, Astrid, and Crystal, Thomas H., Speaker Verification by Human Listeners: Experiments Comparing Human and Machine Performance Using the NIST 1998 Speaker Evaluation Data, Digital Signal Processing10(2000), 249–266.The speaker verification performance of human listeners was compared to that of computer algorithms/systems. Listening protocols were developed to emulate as closely as possible the 1998 algorithm evaluation run by the U.S. National Institute of Standards and Technology (NIST), while taking into account human memory limitations. A subset of the target speakers and test samples from the same telephone conversation data was used. Ways of combining listener data to arrive at a group decision were explored, and the group mean worked well. The human results were very competitive with the best computer algorithms in the same handset condition. For same numbertesting, with 3-s samples, listener panels and the best algorithm had the same equal-error rate (EER) of 8%. Listeners were better than typical algorithms. For different numbertesting, EERs increased but humans had a 40% lower equal-error rate. Human performance in general seemed relatively robust to degradation.