TelBench: A Benchmark for Evaluating Telco-Specific Large Language Models
Sunwoo Lee, Dhammiko Arya, Seung-Mo Cho, Gyoung-eun Han, Seokyoung Hong, Wonbeom Jang, Seojin Lee, Sohee Park
Abstract
The telecommunications industry, characterized by its vast customer base and complex service offerings, necessitates a high level of domain expertise and proficiency in customer service center operations. Consequently, there is a growing demand for Large Language Models (LLMs) to augment the capabilities of customer service representatives. This paper introduces a methodology for developing a specialized Telecommunications LLM (Telco LLM) designed to enhance the efficiency of customer service agents and promote consistency in service quality across representatives. We present the construction process of TelBench, a novel dataset created for performance evaluation of customer service expertise in the telecommunications domain. We also evaluate various LLMs and demonstrate the ability to benchmark both proprietary and open-source LLMs on predefined telecommunications-related tasks, thereby establishing metrics that define telcommunications performance.
BibTeX
@inproceedings{lee-etal-2024-telbench,
title = "{T}el{B}ench: A Benchmark for Evaluating Telco-Specific Large Language Models",
author = "Lee, Sunwoo and
Arya, Dhammiko and
Cho, Seung-Mo and
Han, Gyoung-eun and
Hong, Seokyoung and
Jang, Wonbeom and
Lee, Seojin and
Park, Sohee and
Sek, Sereimony and
Song, Injee and
Yoon, Sungbin and
Davis, Eric",
editor = "Dernoncourt, Franck and
Preo{\c{t}}iuc-Pietro, Daniel and
Shimorina, Anastasia",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
month = nov,
year = "2024",
address = "Miami, Florida, US",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-industry.45/",
doi = "10.18653/v1/2024.emnlp-industry.45",
pages = "609--626"
}