CAVA: A Tool for Cultural Alignment Visualization & Analysis
Nevan Giuliani, Cheng Charles Ma, Prakruthi Pradeep, Daphne Ippolito
Abstract
It is well-known that language models are biased; they have patchy knowledge of countries and cultures that are poorly represented in their training data. We introduce CAVA, a visualization tool for identifying and analyzing country-specific biases in language models.Our tool allows users to identify whether a language model successful captures the perspectives of people of different nationalities. The tool supports analysis of both longform and multiple-choice models responses and comparisons between models.Our open-source code easily allows users to upload any country-based language model generations they wish to analyze.To showcase CAVA’s efficacy, we present a case study analyzing how several popular language models answer survey questions from the World Values Survey.
BibTeX
@inproceedings{giuliani-etal-2024-cava,
title = "{CAVA}: A Tool for Cultural Alignment Visualization {\&} Analysis",
author = "Giuliani, Nevan and
Ma, Cheng Charles and
Pradeep, Prakruthi and
Ippolito, Daphne",
editor = "Hernandez Farias, Delia Irazu and
Hope, Tom and
Li, Manling",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-demo.16/",
doi = "10.18653/v1/2024.emnlp-demo.16",
pages = "153--161"
}