Conditional Language Policy: A General Framework For Steerable Multi-Objective Finetuning
Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li
Abstract
Reward-based finetuning is crucial for aligning language policies with intended behaviors (*e.g.*, creativity and safety). A key challenge is to develop steerable language models that trade-off multiple (conflicting) objectives in a flexible and efficient manner. This paper presents Conditional Language Policy (CLP), a general framework for finetuning language models on multiple objectives. Building on techniques from multi-task training and parameter-efficient finetuning, CLP learn steerable models that effectively trade-off conflicting objectives at *inference time*. Notably, this does not require training or maintaining multiple models to achieve different trade-offs between the objectives. Through extensive experiments and ablations on two summarization datasets, we show that CLP learns steerable language models that outperform and Pareto-dominate the existing approaches for multi-objective
BibTeX
@inproceedings{wang-etal-2024-conditional,
title = "Conditional Language Policy: A General Framework For Steerable Multi-Objective Finetuning",
author = "Wang, Kaiwen and
Kidambi, Rahul and
Sullivan, Ryan and
Agarwal, Alekh and
Dann, Christoph and
Michi, Andrea and
Gelmi, Marco and
Li, Yunxuan and
Gupta, Raghav and
Dubey, Kumar Avinava and
Rame, Alexandre and
Ferret, Johan and
Cideron, Geoffrey and
Hou, Le and
Yu, Hongkun and
Ahmed, Amr and
Mehta, Aranyak and
Hussenot, Leonard and
Bachem, Olivier and
Leurent, Edouard",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-emnlp.118/",
doi = "10.18653/v1/2024.findings-emnlp.118",
pages = "2153--2186"
}