Zhengqi Sun

An Information-Theoretic Criterion for Efficient Data Synthesis

Synthetic data becomes crucial for large language model training, but its effectiveness is highly inconsistent. Here we provide an information-theoretic understanding of such inconsistency. Synthetic data is effective only when the generation-training loop is information-open: it has continuous info…