ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting
Steven H Wang, Maksim Zubkov, Kexin Fan, Sarah Harrell, Yuyang Sun, Wei Chen, Andreas Plesner, Roger Wattenhofer
Abstract
Contract clause retrieval is foundational to contract drafting because lawyers rarely draft contracts from scratch; instead, they locate and revise the most relevant precedent clauses. We introduce the Atticus Clause Retrieval Dataset (ACORD), the first expert-annotated benchmark specifically designed for contract clause retrieval to support contract drafting tasks. ACORD focuses on complex contract clauses such as Limitation of Liability, Indemnification, Change of Control, and Most Favored Nation. It includes 114 queries and over 126,000 query-clause pairs, each ranked on a scale from 1 to 5 stars. The task is to find the most relevant precedent clauses to a query. The bi-encoder retriever paired with pointwise LLMs re-rankers shows promising results. However, substantial improvements are still needed to manage the complex legal work typically undertaken by lawyers effectively. As the first expert-annotated benchmark for contract clause retrieval, ACORD can serve as a valuable IR benchmark for the NLP community.
BibTeX
@inproceedings{wang-etal-2025-acord,
title = "{ACORD}: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting",
author = "Wang, Steven H and
Zubkov, Maksim and
Fan, Kexin and
Harrell, Sarah and
Sun, Yuyang and
Chen, Wei and
Plesner, Andreas and
Wattenhofer, Roger",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.1206/",
doi = "10.18653/v1/2025.acl-long.1206",
pages = "24739--24762",
ISBN = "979-8-89176-251-0"
}