MANTA: A Scalable Pipeline for Transmuting Massive Web Corpora into Instruction Datasets
Heuiyeen Yeen, Seokhee Hong, Hyeongu Yun, Jinsik Lee
Abstract
We introduce MANTA, an automated pipeline that generates high-quality large-scale instruction fine-tuning datasets from massive web corpora while preserving their diversity and scalability. By extracting structured syllabi from web documents and leveraging high-performance LLMs, our approach enables highly effective query-response generation with minimal human intervention. Extensive experiments on 8B-scale LLMs demonstrate that fine-tuning on the MANTA-1M dataset significantly outperforms other massive dataset generation methodologies, particularly in knowledge-intensive tasks such as MMLU and MMLU-Pro, while also delivering superior performance across a broad spectrum of tasks. Moreover, MANTA supports seamless scalability by allowing the continuous integration of web corpus data, enabling expansion into domains requiring intensive knowledge.
BibTeX
@inproceedings{emnlp2025_mantaascalablepi,
title = {MANTA: A Scalable Pipeline for Transmuting Massive Web Corpora into Instruction Datasets},
author = {Heuiyeen Yeen and Seokhee Hong and Hyeongu Yun and Jinsik Lee},
booktitle = {EMNLP 2025},
year = {2025}
}