LUME: LLM Unlearning with Multitask Evaluations
Anil Ramakrishna, Yixin Wan, Xiaomeng Jin, Kai-Wei Chang, Zhiqi Bu, Bhanukiran Vinzamuri, Volkan Cevher, Mingyi Hong
Abstract
Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining. In this work, we develop a multi-task unlearning benchmark LUME that features three tasks: (1) unlearn synthetically generated creative short novels, (2) unlearn synthetic biographies with sensitive information, and (3) unlearn a collection of public biographies. We further release two fine-tuned LLMs of 1B and 7B parameter sizes as the target models. We conduct detailed evaluations of several recently-proposed algorithms and present results on carefully crafted metrics to understand their behavior and limitations.
BibTeX
@inproceedings{emnlp2025_lumellmunlearnin,
title = {LUME: LLM Unlearning with Multitask Evaluations},
author = {Anil Ramakrishna and Yixin Wan and Xiaomeng Jin and Kai-Wei Chang and Zhiqi Bu and Bhanukiran Vinzamuri and Volkan Cevher and Mingyi Hong and Rahul Gupta},
booktitle = {EMNLP 2025},
year = {2025}
}