Mana Sakai

Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs

In modern theoretical analyses of neural networks, the infinite-width limit is often invoked to justify Gaussian approximations of neuron preactivations (e.g., via neural network Gaussian processes or Tensor Programs). However, these Gaussian-based asymptotic theories have so far been unable to capt…