Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements
Shu Yang, Shenzhe Zhu, Zeyu Wu, Keyu Wang, Junchi Yao, Junchao Wu, Lijie Hu, Mengdi Li
Abstract
With the increasing integration of large language models (LLMs) into real-world applications such as finance, e-commerce, and recommendation systems, their susceptibility to misinformation and adversarial manipulation poses significant risks. Existing fraud detection benchmarks primarily focus on single-turn classification tasks, failing to capture the dynamic nature of real-world fraud attempts. To address this gap, we introduce Fraud-R1, a challenging bilingual benchmark designed to assess LLMs’ ability to resist fraud and phishing attacks across five key fraud categories: Fraudulent Services, Impersonation, Phishing Scams, Fake Job Postings, and Online Relationships, covering subclasses. Our dataset comprises manually curated fraud cases from social media, news, phishing scam records, and prior fraud datasets.
BibTeX
@inproceedings{yang-etal-2025-fraud,
title = "Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of {LLM} Against Augmented Fraud and Phishing Inducements",
author = "Yang, Shu and
Zhu, Shenzhe and
Wu, Zeyu and
Wang, Keyu and
Yao, Junchi and
Wu, Junchao and
Hu, Lijie and
Li, Mengdi and
Wong, Derek F. and
Wang, Di",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.226/",
doi = "10.18653/v1/2025.findings-acl.226",
pages = "4374--4420",
ISBN = "979-8-89176-256-5"
}