Santiago Akle Serano

SWAN: An Efficient and Scalable Approach for Long-Context Language Modeling

We present SWAN, a causal Transformer architecture in the decoder-only style that generalizes robustly to sequence lengths substantially longer than those seen during training. SWAN interleaves layers without positional encodings (NoPE) and sliding-window attention layers equipped with rotary positi