Jikun Hu

PEPE: Long-context Extension for Large Language Models via Periodic Extrapolation Positional Encodings

Long-context extension seeks to expand the contextual window in pre-trained large language models (LLMs), allowing them to handle several multiples of their original training context lengths. The primary method for extending the window length involves expanding the initial positional encodings, such