RT Journal Article
T1 Next generation community assessment of biomedical entity recognition web servers: metrics, performance, interoperability aspects of BeCalm
A1 Pérez Pérez, Martín
A1 Pérez Rodríguez, Gael
A1 Blanco Míguez, Aitor
A1 Fernández Riverola, Florentino
A1 Valencia, Alfonso
A1 Krallinger, Martin
A1 GARCIA LOURENÇO, Analia Maria
K1 1203.12 Bancos de Datos
K1 1203.17 Informática
K1 2499 Otras Especialidades Biológicas
AB Background: Shared tasks and community challenges represent key instruments to promote research, collaborationand determine the state of the art of biomedical and chemical text mining technologies. Traditionally, such tasksrelied on the comparison of automatically generated results against a so-called Gold Standard dataset of manuallylabelled textual data, regardless of efficiency and robustness of the underlying implementations. Due to the rapidgrowth of unstructured data collections, including patent databases and particularly the scientific literature, there is apressing need to generate, assess and expose robust big data text mining solutions to semantically enrich documentsin real time. To address this pressing need, a novel track called “Technical interoperability and performance of annotationservers” was launched under the umbrella of the BioCreative text mining evaluation effort. The aim of this trackwas to enable the continuous assessment of technical aspects of text annotation web servers, specifically of onlinebiomedical named entity recognition systems of interest for medicinal chemistry applications.Results: A total of 15 out of 26 registered teams successfully implemented online annotation servers. They returnedpredictions during a two-month period in predefined formats and were evaluated through the BeCalm evaluationplatform, specifically developed for this track. The track encompassed three levels of evaluation, i.e. data formatconsiderations, technical metrics and functional specifications. Participating annotation servers were implementedin seven different programming languages and covered 12 general entity types. The continuous evaluation of serverresponses accounted for testing periods of low activity and moderate to high activity, encompassing overall 4,092,502requests from three different document provider settings. The median response time was below 3.74 s, with a medianof 10 annotations/document. Most of the servers showed great reliability and stability, being able to process over100,000 requests in a 5-day period.Conclusions: The presented track was a novel experimental task that systematically evaluated the technical performanceaspects of online entity recognition systems. It raised the interest of a significant number of participants.Future editions of the competition will address the ability to process documents in bulk as well as to annotate full-textdocuments.
PB Journal of Cheminformatics
SN 17582946
YR 2019
FD 2019-06-24
LK http://hdl.handle.net/11093/4072
UL http://hdl.handle.net/11093/4072
LA eng
NO Journal of Cheminformatics, 11(1): 42 (2019)
NO Portuguese Foundation for Science and Technology | Ref. UID/BIO/04469/2013
DS Investigo
RD 13-dic-2024