Preliminary Evaluation of Automated Speech Recognition Apps for the Hearing Impaired and Deaf.

Leontien Pragt,Jan-Willem A Wasmann,Dagmar Grob,Peter Van Hengel

doi:10.3389/fdgth.2022.806076

Abstract

ObjectiveAutomated speech recognition (ASR) systems have become increasingly sophisticated, accurate, and deployable on many digital devices, including on a smartphone. This pilot study aims to examine the speech recognition performance of ASR apps using audiological speech tests. In addition, we compare ASR speech recognition performance to normal hearing and hearing impaired listeners and evaluate if standard clinical audiological tests are a meaningful and quick measure of the performance of ASR apps.MethodsFour apps have been tested on a smartphone, respectively AVA, Earfy, Live Transcribe, and Speechy. The Dutch audiological speech tests performed were speech audiometry in quiet (Dutch CNC-test), Digits-in-Noise (DIN)-test with steady-state speech-shaped noise, sentences in quiet and in averaged long-term speech-shaped spectrum noise (Plomp-test). For comparison, the app's ability to transcribe a spoken dialogue (Dutch and English) was tested.ResultsAll apps scored at least 50% phonemes correct on the Dutch CNC-test for a conversational speech intensity level (65 dB SPL) and achieved 90–100% phoneme recognition at higher intensity levels. On the DIN-test, AVA and Live Transcribe had the lowest (best) signal-to-noise ratio +8 dB. The lowest signal-to-noise measured with the Plomp-test was +8 to 9 dB for Earfy (Android) and Live Transcribe (Android). Overall, the word error rate for the dialogue in English (19–34%) was lower (better) than for the Dutch dialogue (25–66%).ConclusionThe performance of the apps was limited on audiological tests that provide little linguistic context or use low signal to noise levels. For Dutch audiological speech tests in quiet, ASR apps performed similarly to a person with a moderate hearing loss. In noise, the ASR apps performed more poorly than most profoundly deaf people using a hearing aid or cochlear implant. Adding new performance metrics including the semantic difference as a function of SNR and reverberation time could help to monitor and further improve ASR performance.

Highlights

Since 2017, several ASR systems have claimed speech recognition performance close to that of normally hearing humans [1, 2]
The SwitchBoard and CallHome were collected under low noise and low reverberation conditions [9], and a large portion of the Librispeech corpus has undergone noise removal and volume normalization [10]
When transcribing speech-in-noise, the ASR apps performed in the performance range of CI recipients

Summary

Introduction

Since 2017, several ASR systems have claimed speech recognition performance close to that of normally hearing humans [1, 2]. ASRs are evaluated on well-studied large (>100 h) collections of speech, referred to as a corpus. The SwitchBoard corpus and CallHome corpus are well-known collections of conversational phone calls [8], whereas Librispeech is a corpus comprising speech from public domain audiobooks. The CallHome corpus consists of more informal conversations between friends and family [8]. None of these corpora are ideal for use in acoustically challenging environments. The SwitchBoard and CallHome were collected under low noise and low reverberation conditions [9], and a large portion of the Librispeech corpus has undergone noise removal and volume normalization [10]

Methods

Results

Discussion

Conclusion