2026
MeSH: Memory-as-State-Highways for Recursive Transformers
ICLR 2026poster
Recursive transformers reuse parameters and iterate over hidden states multiple times, decoupling compute depth from parameter depth. However, under matched compute, recursive models with fewer parameters often lag behind non-recursive counterparts. By probing hidden states, we trace this performanc…