3 accepted papers
As data scaling approaches its natural limits, recent work has explored the potential of data reuse, whose impact on optimization remains poorly understood. This work investigates how data repetition affects learning efficiency in terms of compute. We investigate an inverse-scaling effect that using…
Inference-time computation has emerged as a promising scaling axis for improving large language model reasoning. However, despite yielding impressive performance, the optimal allocation of inference-time computation remains poorly understood. A central question is whether to prioritize sequential sc…
Large language models have the ability to generate text that mimics patterns in their inputs. We introduce a simple Markov Chain sequence modeling task in order to study how this in-context learning capability emerges. In our setting, each example is sampled from a Markov chain drawn from a prior di…