Xiayu Sun

Regret Pre-training: Bridging Prior and Posterior Views for Enhanced Knowledge Grounding

Causal language models factorize sequence probabilities using only preceding context, leaving future information unexploited during training despite its availability in the training data. This paper introduces Regret Pre-training, a self-supervised framework grounded in the Learning Using Privileged…