The Threat of PROMPTS in Large Language Models: A System and User Prompt Perspective
Zixuan Xia, Haifeng Sun, Jingyu Wang, Qi Qi, Huazheng Wang, Xiaoyuan Fu, Jianxin Liao
Abstract
Prompts, especially high-quality ones, play an invaluable role in assisting large language models (LLMs) to accomplish various natural language processing tasks. However, carefully crafted prompts can also manipulate model behavior. Therefore, the security risks that “prompts themselves face” and those “arising from harmful prompts” cannot be overlooked and we define the Prompt Threat (PT) issues. In this paper, we review the latest attack methods related to prompt threats, focusing on prompt leakage attacks and prompt jailbreak attacks. Additionally, we summarize the experimental setups of these methods and explore the relationship between prompt threats and prompt injection attacks.
BibTeX
@inproceedings{xia-etal-2025-threat,
title = "The Threat of {PROMPTS} in Large Language Models: A System and User Prompt Perspective",
author = "Xia, Zixuan and
Sun, Haifeng and
Wang, Jingyu and
Qi, Qi and
Wang, Huazheng and
Fu, Xiaoyuan and
Liao, Jianxin",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.675/",
doi = "10.18653/v1/2025.findings-acl.675",
pages = "12994--13035",
ISBN = "979-8-89176-256-5"
}