Search on speech from spoken queries: the multi-domain International ALBAYZIN 2018 query-by-example spoken term detection evaluation

Abstract

The huge amount of information stored in audio and video repositories makes search on speech (SoS) a priority areanowadays. Within SoS, Query-by-Example Spoken Term Detection (QbE STD) aims to retrieve data from a speechrepository given a spoken query. Research on this area is continuously fostered with the organization of QbE STDevaluations. This paper presents a multi-domain internationally open evaluation for QbE STD in Spanish. Theevaluation aims at retrieving the speech files that contain the queries, providing their start and end times, and a scorethat reflects the confidence given to the detection. Three different Spanish speech databases that encompassdifferent domains have been employed in the evaluation: MAVIR database, which comprises a set of talks fromworkshops; RTVE database, which includes broadcast television (TV) shows; and COREMAH database, which contains2-people spontaneous speech conversations about different topics. The evaluation has been designed carefully sothat several analyses of the main results can be carried out. We present the evaluation itself, the three databases, theevaluation metrics, the systems submitted to the evaluation, the results, and the detailed post-evaluation analysesbased on some query properties (within-vocabulary/out-of-vocabulary queries, single-word/multi-word queries, andnative/foreign queries). Fusion results of the primary systems submitted to the evaluation are also presented. Threedifferent teams took part in the evaluation, and ten different systems were submitted. The results suggest that theQbE STD task is still in progress, and the performance of these systems is highly sensitive to changes in the datadomain. Nevertheless, QbE STD strategies are able to outperform text-based STD in unseen data domains.