Jesse Roberts — accepted AI-conference papers

2024

The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performance

Kyle Moore, Jesse Roberts, Thao Pham, Oseremhen Ewaleifoh, Douglas Fisher

EMNLP 2024finding

Cloze testing is a common method for measuring the behavior of large language models on a number of benchmark tasks. Using the MMLU dataset, we show that the base-rate probability (BRP) differences across answer tokens are significant and affect task performance ie. guess A if uncertain. We find tha…

Cited by 5 Source ↗PDF ↗code ↗Scholar ↗

2024

Using Artificial Populations to Study Psychological Phenomena in Neural Models

Jesse Roberts, Kyle Moore, Drew Wilenzick, Douglas Fisher

AAAI 2024technical

The recent proliferation of research into transformer based natural language processing has led to a number of studies which attempt to detect the presence of human-like cognitive behavior in the models. We contend that, as is true of human psychology, the investigation of cognitive behavior in lang…

Cited by 8 Source ↗PDF ↗code ↗Scholar ↗