Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Dataset
Satanu Ghosh, Neal Brodnik, Carolina Frey, Collin Holgate, Tresa Pollock, Samantha Daly, Samuel Carton
Abstract
We explore the ability of GPT-4 to perform ad-hoc schema-based information extraction from scientific literature. We assess specifically whether it can, with a basic one-shot prompting approach over the full text of the included manusciprts, replicate two existing material science datasets, one pertaining to multi-principal element alloys (MPEAs), and one to silicate diffusion. We collaborate with materials scientists to perform a detailed manual error analysis to assess where and why the model struggles to faithfully extract the desired information, and draw on their insights to suggest research directions to address this broadly important task.
BibTeX
@inproceedings{ghosh-etal-2024-toward,
title = "Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Dataset",
author = "Ghosh, Satanu and
Brodnik, Neal and
Frey, Carolina and
Holgate, Collin and
Pollock, Tresa and
Daly, Samantha and
Carton, Samuel",
editor = "Ku, Lun-Wei and
Martins, Andre and
Srikumar, Vivek",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-acl.897/",
doi = "10.18653/v1/2024.findings-acl.897",
pages = "15109--15123"
}