Token-wise Influential Training Data Retrieval for Large Language Models
Huawei Lin, Jikai Long, Zhaozhuo Xu, Weijie Zhao
Abstract
Given a Large Language Model (LLM) generation, how can we identify which training data led to this generation? In this paper, we proposed RapidIn, a scalable framework adapting to LLMs for estimating the influence of each training data. The proposed framework consists of two stages: caching and retrieval. First, we compress the gradient vectors by over 200,000x, allowing them to be cached on disk or in GPU/CPU memory. Then, given a generation, RapidIn efficiently traverses the cached gradients to estimate the influence within minutes, achieving over a 6,326x speedup. Moreover, RapidIn supports multi-GPU parallelization to substantially accelerate caching and retrieval. Our empirical result confirms the efficiency and effectiveness of RapidIn.
BibTeX
@inproceedings{lin-etal-2024-token,
title = "Token-wise Influential Training Data Retrieval for Large Language Models",
author = "Lin, Huawei and
Long, Jikai and
Xu, Zhaozhuo and
Zhao, Weijie",
editor = "Ku, Lun-Wei and
Martins, Andre and
Srikumar, Vivek",
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.acl-long.48/",
doi = "10.18653/v1/2024.acl-long.48",
pages = "841--860"
}