AIConfPaper
← Search

Imanol Schlag

13 accepted papers

2025

INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge

Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary +51
ICLR 2025spotlight

The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (i.e., mult…

Cited by 9Source ↗PDF ↗Scholar ↗
2025

Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks

Yixuan Xu, Antoine Bosselut, Imanol Schlag
NeurIPS 2025poster

Large language models are known to memorize parts of their training data, posing risk of copyright violations. To systematically examine this risk, we pretrain language models (1B/3B/8B) from scratch on 83B tokens, mixing web-scale data with public domain books used to simulate copyrighted content a…

Cited by 0Source ↗Scholar ↗
2024

Navigating Scaling Laws: Compute Optimality in Adaptive Model Training

Sotiris Anagnostidis, Gregor Bachmann, Imanol Schlag, Thomas Hofmann
ICML 2024spotlight

In recent years, the state-of-the-art in deep learning has been dominated by very large models that have been pre-trained on vast amounts of data. The paradigm is very simple: investing more computational resources (optimally) leads to better performance, and even predictably so; neural scaling laws…

Cited by 1Source ↗PDF ↗Scholar ↗
2024

On the Effect of (Near) Duplicate Subwords in Language Modelling

Anton Schäfer, Thomas Hofmann, Imanol Schlag, Tiago Pimentel
ACL 2024findings

Tokenisation is a core part of language models (LMs). It involves splitting a character sequence into subwords which are assigned random indices before being served to the LM. However, this process—while typically lossless—may lead to less efficient LM training, because it removes character-level in…

Cited by 1Source ↗PDF ↗code ↗Scholar ↗
2024

Understanding and Minimising Outlier Features in Transformer Training

Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, Thomas Hofmann
NeurIPS 2024poster

Outlier Features (OFs) are neurons whose activation magnitudes significantly exceed the average over a neural network's (NN) width. They are well known to emerge during standard transformer training and have the undesirable effect of hindering quantisation in afflicted models. Despite their practica…

Cited by 2Source ↗PDF ↗Scholar ↗
2022

A Modern Self-Referential Weight Matrix That Learns to Modify Itself

Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber
ICML 2022spotlight

The weight matrix (WM) of a neural network (NN) is its program. The programs of many traditional NNs are learned through gradient descent in some error function, then remain fixed. The WM of a self-referential NN, however, can keep rapidly modifying all of itself during runtime. In principle, such N…

Cited by 43Source ↗PDF ↗code ↗Scholar ↗
2022

Block-Recurrent Transformers

DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, Behnam Neyshabur
NeurIPS 2022accept

We introduce the Block-Recurrent Transformer, which applies a transformer layer in a recurrent fashion along a sequence, and has linear complexity with respect to sequence length. Our recurrent cell operates on blocks of tokens rather than single tokens during training, and leverages parallel comput…

Cited by 131Source ↗PDF ↗code ↗Scholar ↗
2022

Solving Quantitative Reasoning Problems with Language Models

Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh +8
NeurIPS 2022accept

Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering questions at…

Cited by 815Source ↗PDF ↗Scholar ↗
2021

Going Beyond Linear Transformers with Recurrent Fast Weight Programmers

Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber
NeurIPS 2021poster

Transformers with linearised attention (''linear Transformers'') have demonstrated the practical scalability and effectiveness of outer product-based Fast Weight Programmers (FWPs) from the '90s. However, the original FWP formulation is more general than the one of linear Transformers: a slow neural…

Cited by 81Source ↗PDF ↗code ↗Scholar ↗
2021

Learning Associative Inference Using Fast Weight Memory

Imanol Schlag, Tsendsuren Munkhdalai, Jürgen Schmidhuber
ICLR 2021poster

Humans can quickly associate stimuli to solve problems in novel contexts. Our novel neural network model learns state representations of facts that can be composed to perform such associative inference. To this end, we augment the LSTM model with an associative memory, dubbed \textit{Fast Weight Mem…

Cited by 52Source ↗PDF ↗code ↗Scholar ↗
2021

Linear Transformers Are Secretly Fast Weight Programmers

Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber
ICML 2021spotlight

We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early ’90s, where a slow neural net learns by gradient descent to program the fast weights of another net through sequences of elementary programming instructions which are additive outer prod…

Cited by 265Source ↗PDF ↗code ↗Scholar ↗