Michaël Cadilhac

Knee-Deep in C-RASP: A Transformer Depth Hierarchy

It has been observed that transformers with greater depth (that is, more layers) have more capabilities, but can we establish formally which capabilities are gained? We answer this question with a theoretical proof followed by an empirical study. First, we consider transformers that round to fixed p…