BenchIE^FL: A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark
Fabrice Lamarche, Philippe Langlais
Abstract
Open Information Extraction (OIE) is a field of natural language processing that aims to present textual information in a format that allows it to be organized, analyzed and reflected upon. Numerous OIE systems are developed, claiming ever-increasing performance, marking the need for objective benchmarks. BenchIE is the latest reference we know of. Despite being very well thought out, we noticed a number of issues we believe are limiting. Therefore, we propose BenchIE^FL, a new OIE benchmark which fully enforces the principles of BenchIE while containing fewer errors, omissions and shortcomings when candidate facts are matched towards reference ones. BenchIE^FL allows insightful conclusions to be drawn on the actual performance of OIE extractors.
BibTeX
@inproceedings{lamarche-langlais-2024-benchie,
title = "{B}ench{IE}{\textasciicircum}{FL}: A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark",
author = "Lamarche, Fabrice and
Langlais, Philippe",
editor = "Ku, Lun-Wei and
Martins, Andre and
Srikumar, Vivek",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-acl.496/",
doi = "10.18653/v1/2024.findings-acl.496",
pages = "8372--8394"
}