ALBAYZIN 2018 spoken term detection evaluation: a multi-domain international evaluation in Spanish

Javier Tejedor,Doroteo T Toledano,Mikel Peñagarikano,Jose M Ramirez,Laura Docio-Fernandez,Ana R Montalvo,Paula Lopez-Otero,Luis Javier Rodriguez-Fuentes

doi:10.1186/s13636-019-0159-7

Abstract

Search on speech (SoS) is a challenging area due to the huge amount of information stored in audio and video repositories. Spoken term detection (STD) is an SoS-related task aiming to retrieve data from a speech repository given a textual representation of a search term (which can include one or more words). This paper presents a multi-domain internationally open evaluation for STD in Spanish. The evaluation has been designed carefully so that several analyses of the main results can be carried out. The evaluation task aims at retrieving the speech files that contain the terms, providing their start and end times, and a score that reflects the confidence given to the detection. Three different Spanish speech databases that encompass different domains have been employed in the evaluation: the MAVIR database, which comprises a set of talks from workshops; the RTVE database, which includes broadcast news programs; and the COREMAH database, which contains 2-people spontaneous speech conversations about different topics. We present the evaluation itself, the three databases, the evaluation metric, the systems submitted to the evaluation, the results, and detailed post-evaluation analyses based on some term properties (within-vocabulary/out-of-vocabulary terms, single-word/multi-word terms, and native/foreign terms). Fusion results of the primary systems submitted to the evaluation are also presented. Three different research groups took part in the evaluation, and 11 different systems were submitted. The obtained results suggest that the STD task is still in progress and performance is highly sensitive to changes in the data domain.

Highlights

Search on speech (SoS) has become an interesting research area due to the huge amount of information stored in audio and video repositories
2.2 Evaluation metric In spoken term detection (STD), a hypothesized occurrence is called a detection; if the detection corresponds to an actual occurrence, it is called a hit, otherwise it is called a false alarm
The best performance is obtained with the Kaldi-deep neural network (DNN) system, for which the small performance gap between Maximum Term-Weighted Value (MTWV) and Actual Term-Weighted Value (ATWV) suggests that the threshold has been well-calibrated

Summary

Introduction

Search on speech (SoS) has become an interesting research area due to the huge amount of information stored in audio and video repositories. Significant research has been carried out in SoS for spoken document retrieval (SDR) [2,3,4,5,6,7], keyword spotting (KWS) [8,9,10,11,12,13], spoken term detection (STD) [14,15,16,17,18,19], and query-by-example (QbE) STD and SDR [20,21,22,23,24,25]. The detection subsystem integrates a term detector and a decision maker. The term detector searches for putative detections of the terms in the index, and the decision maker decides whether each putative detection is a hit or a false alarm (FA) based on certain confidence measures

Methods

Results

Conclusion