Karolina Stanczak — accepted AI-conference papers

2025

CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics

Shravan Nayak, Mehar Bhatia, Xiaofeng Zhang, Verena Rieser, Lisa Anne Hendricks, Sjoerd Van Steenkiste +3

EMNLP 2025

The increasing ubiquity of text-to-image (T2I) models as tools for visual content generation raises concerns about their ability to accurately represent diverse cultural contexts - where missed cues can stereotype communities and undermine usability. In this work, we present the first study to syste

Cited by 0 Source ↗PDF ↗Scholar ↗

2025

SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models

Margaret Mitchell, Giuseppe Attanasio, Ioana Baldini, Miruna Clinciu, Jordan Clive, Pieter Delobelle +48

NAACL 2025long

Large Language Models (LLMs) reproduce and exacerbate the social biases present in their training data, and resources to quantify this issue are limited. While research has attempted to identify and mitigate such biases, most efforts have been concentrated around English, lagging the rapid advanceme…

Cited by 1 Source ↗PDF ↗Scholar ↗

2025

SafeArena: Evaluating the Safety of Autonomous Web Agents

Ada Defne Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Patel, Esin DURMUS +3

ICML 2025poster

LLM-based agents are becoming increasingly proficient at solving web-based tasks. With this capability comes a greater risk of misuse for malicious purposes, such as posting misinformation in an online forum or selling illicit substances on a website. To evaluate these risks, we propose SafeArena, a…

Cited by 4 Source ↗PDF ↗Scholar ↗

2024

Benchmarking Vision Language Models for Cultural Understanding

Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks +2

EMNLP 2024main

Foundation models and vision-language pre-training have notably advanced Vision Language Models (VLMs), enabling multimodal processing of visual and linguistic data. However, their performance has been typically assessed on general scene understanding - recognizing objects, attributes, and actions -…

Cited by 24 Source ↗PDF ↗Scholar ↗

2024

Social Bias Probing: Fairness Benchmarking for Language Models

Marta Marchiori Manerba, Karolina Stanczak, Riccardo Guidotti, Isabelle Augenstein

EMNLP 2024main

While the impact of social biases in language models has been recognized, prior methods for bias evaluation have been limited to binary association tests on small datasets, limiting our understanding of bias complexities. This paper proposes a novel framework for probing language models for social b…

Cited by 21 Source ↗PDF ↗Scholar ↗

2023

Measuring Intersectional Biases in Historical Documents

Nadav Borenstein, Karolina Stanczak, Thea Rolskov, Natacha Klein Käfer, Natália da Silva Perez, Isabelle Augenstein

ACL 2023findings

Data-driven analyses of biases in historical texts can help illuminate the origin and development of biases prevailing in modern society. However, digitised historical documents pose a challenge for NLP practitioners as these corpora suffer from errors introduced by optical character recognition (OC…

Cited by 9 Source ↗PDF ↗code ↗Scholar ↗

2022

Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models

Karolina Stanczak, Edoardo Ponti, Lucas Torroba Hennigen, Ryan Cotterell, Isabelle Augenstein

NAACL 2022long

The success of multilingual pre-trained models is underpinned by their ability to learn representations shared by multiple languages even in absence of any explicit supervision. However, it remains unclear how these models learn to generalise across languages. In this work, we conjecture that multil…

Cited by 32 Source ↗PDF ↗code ↗Scholar ↗