2024
Extending Input Contexts of Language Models through Training on Segmented Sequences
NAACL 2024findings
Effectively training language models on longinputs poses many technical challenges. As acost consideration, languages models are pre-trained on a fixed sequence length before beingadapted to longer sequences. We explore var-ious methods for adapting models to longerinputs by training on segmented se…