EMNLP 2024main22 citations
Extracting Prompts by Inverting LLM Outputs
Collin Zhang, John Xavier Morris, Vitaly Shmatikov
Abstract
We consider the problem of language model inversion: given outputs of a language model, we seek to extract the prompt that generated these outputs. We develop a new black-box method, output2prompt, that extracts prompts without access to the model’s logits and without adversarial or jailbreaking queries. Unlike previous methods, output2prompt only needs outputs of normal user queries. To improve memory efficiency, output2prompt employs a new sparse encoding techique. We measure the efficacy of output2prompt on a variety of user and system prompts and demonstrate zero-shot transferability across different LLMs.
BibTeX
@inproceedings{zhang-etal-2024-extracting,
title = "Extracting Prompts by Inverting {LLM} Outputs",
author = "Zhang, Collin and
Morris, John Xavier and
Shmatikov, Vitaly",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-main.819/",
doi = "10.18653/v1/2024.emnlp-main.819",
pages = "14753--14777"
}