Insights into LLM Long-Context Failures: When Transformers Know but Don’t Tell

Muhan Gao; TaiMing Lu; Kuai Yu; Adam Byerly; Daniel Khashabi

EMNLP 2024finding7 citations

Insights into LLM Long-Context Failures: When Transformers Know but Don’t Tell

Muhan Gao, TaiMing Lu, Kuai Yu, Adam Byerly, Daniel Khashabi

Abstract

Large Language Models (LLMs) exhibit positional bias, struggling to utilize information from the middle or end of long contexts. Our study explores LLMs’ long-context reasoning by probing their hidden representations. We find that while LLMs encode the position of target information, they often fail to leverage this in generating accurate responses. This reveals a disconnect between information retrieval and utilization, a “know but don’t tell” phenomenon. We further analyze the relationship between extraction time and final accuracy, offering insights into the underlying mechanics of transformer models.

BibTeX

@inproceedings{gao-etal-2024-insights,
    title = "Insights into {LLM} Long-Context Failures: When Transformers Know but Don`t Tell",
    author = "Gao, Muhan  and
      Lu, TaiMing  and
      Yu, Kuai  and
      Byerly, Adam  and
      Khashabi, Daniel",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.447/",
    doi = "10.18653/v1/2024.findings-emnlp.447",
    pages = "7611--7625"
}