Anti-LM Decoding for Zero-shot In-context Machine Translation
Suzanna Sia, Alexandra DeLucia, Kevin Duh
Abstract
Zero-shot In-context learning is the phenomenon where models can perform a task given only the instructions. However, pre-trained large language models are known to be poorly calibrated for zero-shot tasks. One of the most effective approaches to handling this bias is to adopt a contrastive decoding objective, which accounts for the prior probability of generating the next token by conditioning on a context. This work introduces an Anti-Language Model objective with a decay factor designed to address the weaknesses of In-context Machine Translation. We conduct our experiments across 3 model types and sizes, 3 language directions, and for both greedy decoding and beam search. The proposed method outperforms other state-of-the-art decoding objectives, with up to 20 BLEU point improvement from the default objective in some settings.
BibTeX
@inproceedings{sia-etal-2024-anti,
title = "Anti-{LM} Decoding for Zero-shot In-context Machine Translation",
author = "Sia, Suzanna and
DeLucia, Alexandra and
Duh, Kevin",
editor = "Duh, Kevin and
Gomez, Helena and
Bethard, Steven",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024",
month = jun,
year = "2024",
address = "Mexico City, Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-naacl.216/",
doi = "10.18653/v1/2024.findings-naacl.216",
pages = "3403--3420"
}