Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties

Fahim Faisal; Md Mushfiqur Rahman; Antonios Anastasopoulos

EMNLP 20250 citations

Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties

Fahim Faisal, Md Mushfiqur Rahman, Antonios Anastasopoulos

Abstract

There has been little systematic study on how dialectal differences affect toxicity detection by modern LLMs. Furthermore, although using LLMs as evaluators (“LLM-as-a-judge”) is a growing research area, their sensitivity to dialectal nuances is still underexplored and requires more focused attention. In this paper, we address these gaps through a comprehensive toxicity evaluation of LLMs across diverse dialects. We create a multi-dialect dataset through synthetic transformations and human-assisted translations, covering 10 language clusters and 60 varieties. We then evaluate five LLMs on their ability to assess toxicity, measuring multilingual, dialectal, and LLM-human consistency. Our findings show that LLMs are sensitive to both dialectal shifts and low-resource multilingual variation, though the most persistent challenge remains aligning their predictions with human judgments.

BibTeX

@inproceedings{emnlp2025_dialectaltoxicit,
  title = {Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties},
  author = {Fahim Faisal and Md Mushfiqur Rahman and Antonios Anastasopoulos},
  booktitle = {EMNLP 2025},
  year = {2025}
}