Advancing Sequential Numerical Prediction in Autoregressive Models
Xiang Fei, Jinghui Lu, Qi Sun, Hao Feng, Yanjie Wang, Wei Shi, An-Lan Wang, Jingqun Tang
Abstract
Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces Numerical Token Integrity Loss(NTIL) to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover’s Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL.
BibTeX
@inproceedings{fei-etal-2025-advancing,
title = "Advancing Sequential Numerical Prediction in Autoregressive Models",
author = "Fei, Xiang and
Lu, Jinghui and
Sun, Qi and
Feng, Hao and
Wang, Yanjie and
Shi, Wei and
Wang, An-Lan and
Tang, Jingqun and
Huang, Can",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-short.44/",
doi = "10.18653/v1/2025.acl-short.44",
pages = "562--574",
ISBN = "979-8-89176-252-7"
}