2026
Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining
ICLR 2026poster
Incorporating metadata in Large Language Models (LLMs) pretraining has recently emerged as a promising approach to accelerate training. However prior work highlighted only one useful signal—URLs, leaving open the question of whether other forms of metadata could yield greater benefits. In this study…