Kyogo Kawaguchi

A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language

Increase in data, size, or compute can lead to sudden learning of specific capabilities by a neural network---a phenomenon often called "emergence". Beyond scientific understanding, establishing the causal factors underlying such emergent capabilities is crucial to enable risk regulation frameworks…