ScEdit: Script-based Assessment of Knowledge Editing
Xinye Li, Zunwen Zheng, Qian Zhang, Dekai Zhuang, Jiabao Kang, Liyan Xu, Qingbin Liu, Xi Chen
Abstract
Knowledge Editing (KE) has gained increasing attention, yet current KE tasks remain relatively simple. Under current evaluation frameworks, many editing methods achieve exceptionally high scores, sometimes nearing perfection. However, few studies integrate KE into real-world application scenarios (e.g., recent interest in LLM-as-agent). To support our analysis, we introduce a novel script-based benchmark – ScEdit (Script-based Knowledge Editing Benchmark) – which encompasses both counterfactual and temporal edits. We integrate token-level and text-level evaluation methods, comprehensively analyzing existing KE techniques. The benchmark extends traditional fact-based (“What”-type question) evaluation to action-based (“How”-type question) evaluation. We observe that all KE methods exhibit a drop in performance on established metrics and face challenges on text-level metrics, indicating a challenging task. Our benchmark is available at https://github.com/asdfo123/ScEdit.
BibTeX
@inproceedings{li-etal-2025-scedit,
title = "{S}c{E}dit: Script-based Assessment of Knowledge Editing",
author = "Li, Xinye and
Zheng, Zunwen and
Zhang, Qian and
Zhuang, Dekai and
Kang, Jiabao and
Xu, Liyan and
Liu, Qingbin and
Chen, Xi and
Tu, Zhiying and
Chu, Dianhui and
Sui, Dianbo",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.104/",
doi = "10.18653/v1/2025.findings-acl.104",
pages = "2032--2052",
ISBN = "979-8-89176-256-5"
}