2026
Joint Selection for Large-Scale Pre-Training Data via Policy Gradient-based Mask Learning
ICLR 2026poster
A fine-grained data recipe is crucial for pre-training large language models (LLMs), as it can significantly enhance training efficiency and model performance. One important ingredient in the recipe is to select samples based on scores produced by defined rules, LLM judgment, or statistical informat…