2026
Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
ICML 2026poster
Layer dropout (a.k.a.\ stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformers. However, as models and datasets have scaled, dropout---particularly layer dropout---has largely disappeared from LLM…