Context Limitations Make Neural Language Models More Human-Like
Tatsuki Kuribayashi, Yohei Oseki, Ana Brassard, Kentaro Inui
Abstract
Language models (LMs) have been used in cognitive modeling as well as engineering studies—they compute information-theoretic complexity metrics that simulate humans’ cognitive load during reading.This study highlights a limitation of modern neural LMs as the model of choice for this purpose: there is a discrepancy between their context access capacities and that of humans.Our results showed that constraining the LMs’ context access improved their simulation of human reading behavior.We also showed that LM-human gaps in context access were associated with specific syntactic constructions; incorporating syntactic biases into LMs’ context access might enhance their cognitive plausibility.
BibTeX
@inproceedings{kuribayashi-etal-2022-context,
title = "Context Limitations Make Neural Language Models More Human-Like",
author = "Kuribayashi, Tatsuki and
Oseki, Yohei and
Brassard, Ana and
Inui, Kentaro",
editor = "Goldberg, Yoav and
Kozareva, Zornitsa and
Zhang, Yue",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.emnlp-main.712/",
doi = "10.18653/v1/2022.emnlp-main.712",
pages = "10421--10436"
}