2026
Separating representation from reconstruction enables scalable text encoders
ICML 2026poster
While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We examine this disparity by revisiting evaluation through the lens of finetuning under frozen backbone and linear probing. As models scale, their representations become increasingly unexploitable by frozen prob…