GAICo: A Deployed and Extensible Framework for Evaluating Diverse and Multimodal Generative AI Outputs
Nitin Gupta, Pallav Koppisetti, Kausik Lakkaraju, Biplav Srivastava
Abstract
The rapid proliferation of Generative AI (GenAI) into diverse, high-stakes domains necessitates robust and reproducible evaluation methods. However, practitioners often resort to ad-hoc, non-standardized scripts, as common metrics are often unsuitable for specialized, structured outputs (e.g., automated plans, time-series) or holistic comparison across modalities (e.g., text, audio, and image). This fragmentation hinders comparability and slows AI system development. To address this challenge, we present GAICo (Generative AI Comparator): a deployed, open-source Python library that streamlines and standardizes GenAI output comparison. GAICo provides a unified, extensible framework supporting a comprehensive suite of reference-based metrics for unstructured text, specialized structured data formats, and multimedia (images, audio). Its architecture features a high-level API for rapid, end-to-end analysis, from multi-model comparison to visualization and reporting, alongside direct metric access for granular control. We demonstrate GAICo
BibTeX
@inproceedings{aaai2026_gaicoadeployedan,
title = {GAICo: A Deployed and Extensible Framework for Evaluating Diverse and Multimodal Generative AI Outputs},
author = {Nitin Gupta and Pallav Koppisetti and Kausik Lakkaraju and Biplav Srivastava},
booktitle = {AAAI 2026},
year = {2026}
}