Amanuel Tesfaye

Two (narrow) heads are better than (an arbitrarily wide) one

In this paper, we establish a dimension- and precision-independent impossibility result for a simplified transformer model. Due to their size, a comprehensive understanding of the internal operations of frontier large language models (LLMs) is beyond the reach of current methods, but research into s…