What Layers When: Learning to Skip Compute in LLMs with Residual Gates
Filipe Laitenberger, Dawid Jan Kopiczko, Cees G. M. Snoek, Yuki M Asano
Abstract
We introduce GateSkip, a simple residual-stream gating mechanism that enables token-wise layer skipping in decoder-only LMs. Each Attention/MLP branch is equipped with a sigmoid-linear gate that compresses the branch’s output before it re-enters the residual stream. During inference we rank tokens by the gate and skip low-importance ones using a per-layer budget. While early-exit or router-based Mixture-of-Depths models are known to be unstable and need extensive retraining, our smooth, differentiable gates fine-tune stably on top of pretrained models. On long-form reasoning, we save up to 15% compute while retaining >90% of baseline accuracy. On instruction-tuned models we see accuracy gains at full compute and match baseline quality near 50% savings. The learned gates give insight into transformer information flow (e.g., BOS tokens act as anchors), and the method combines easily with quantization, pruning, and self-speculative decoding.
BibTeX
@inproceedings{
laitenberger2026what,
title={What Layers When: Learning to Skip Compute in {LLM}s with Residual Gates},
author={Filipe Laitenberger and Dawid Jan Kopiczko and Cees G. M. Snoek and Yuki M Asano},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=aiP6XfaYZR}
}