Lukas Kinder

Positional Overload: Positional Debiasing and Context Window Extension for Large Language Models using Set Encoding

Large Language Models (LLMs) typically track the order of tokens using positional encoding, which causes the following problems: positional bias, where the model is influenced by an ordering within the prompt, and a fixed context window, as models struggle to generalize to positions beyond those enc…