RAGthoven: A Configurable Toolkit for RAG-enabled LLM Experimentation
Gregor Karetka, Demetris Skottis, Lucia Dutková, Peter Hraška, Marek Suppa
Abstract
Large Language Models (LLMs) have significantly altered the landscape of Natural Language Processing (NLP), having topped the benchmarks of many standard tasks and problems, particularly when used in combination with Retrieval Augmented Generation (RAG). Despite their impressive performance and relative simplicity, its use as a baseline method has not been extensive. One of the reasons might be that adapting and optimizing RAG-based pipelines for specific NLP tasks generally requires custom development which is difficult to scale. In this work we introduce RAGthoven, a tool for automatic evaluation of RAG-based pipelines. It provides a simple yet powerful abstraction, which allows the user to start the evaluation process with nothing more than a single configuration file. To demonstrate its usefulness we conduct three case studies spanning text classification, question answering and code generation usecases. We release the code, as well as the documentation and tutorials, at https://github.com/ragthoven-dev/ragthoven
BibTeX
@inproceedings{karetka-etal-2025-ragthoven,
title = "{RAG}thoven: A Configurable Toolkit for {RAG}-enabled {LLM} Experimentation",
author = "Karetka, Gregor and
Skottis, Demetris and
Dutkov{\'a}, Lucia and
Hra{\v{s}}ka, Peter and
Suppa, Marek",
editor = "Rambow, Owen and
Wanner, Leo and
Apidianaki, Marianna and
Al-Khalifa, Hend and
Eugenio, Barbara Di and
Schockaert, Steven and
Mather, Brodie and
Dras, Mark",
booktitle = "Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations",
month = jan,
year = "2025",
address = "Abu Dhabi, UAE",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.coling-demos.12/",
pages = "117--125"
}