NeurIPS 2024poster121 citations
Many-shot Jailbreaking
Cem Anil, Esin DURMUS, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong
Abstract
We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This attack is newly feasible with the larger context windows recently deployed by language model providers like Google DeepMind, OpenAI and Anthropic. We find that in diverse, realistic circumstances, the effectiveness of this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closed-weight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.
large language modelslong contextrobustnessjailbreaksin-context learning
BibTeX
@inproceedings{
anil2024manyshot,
title={Many-shot Jailbreaking},
author={Cem Anil and Esin DURMUS and Nina Rimsky and Mrinank Sharma and Joe Benton and Sandipan Kundu and Joshua Batson and Meg Tong and Jesse Mu and Daniel J Ford and Francesco Mosconi and Rajashree Agrawal and Rylan Schaeffer and Naomi Bashkansky and Samuel Svenningsen and Mike Lambert and Ansh Radhakrishnan and Carson Denison and Evan J Hubinger and Yuntao Bai and Trenton Bricken and Timothy Maxwell and Nicholas Schiefer and James Sully and Alex Tamkin and Tamera Lanham and Karina Nguyen and Tomasz Korbak and Jared Kaplan and Deep Ganguli and Samuel R. Bowman and Ethan Perez and Roger Baker Grosse and David Duvenaud},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=cw5mgd71jW}
}