Pavan Tankala

PUB: A Pragmatics Understanding Benchmark for Assessing LLMs’ Pragmatics Capabilities

LLMs have demonstrated remarkable capability for understanding semantics, but their understanding of pragmatics is not well studied. To this end, we release a Pragmatics Understanding Benchmark (PUB) dataset consisting of fourteen tasks in four pragmatics phenomena, namely; Implicature, Presuppositi…