Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
Shokichi Takakura, Akifumi Wachi, Rei Higuchi, Kohei Miyaguchi, Taiji Suzuki
Abstract
Aligning large language models (LLMs) to diverse human preferences is fundamentally challenging since criteria can often conflict with each other. Inference-time alignment methods have recently gained popularity as they allow LLMs to be aligned to multiple criteria via different alignment algorithms at inference time. However, inference-time alignment is computationally expensive since it often requires multiple forward passes of the base model. In this work, we propose *inference-aware meta-alignment* (IAMA), a novel approach that enables LLMs to be aligned to multiple criteria with minimal computational overhead at inference time. IAMA trains a base model such that it can be effectively aligned to multiple task optima via different inference-time alignment algorithms. To solve the non-linear optimization problems involved in IAMA, we propose *non-linear GRPO*, which provably converges to the optimal solution in the space of probability measures.
BibTeX
@inproceedings{
takakura2026inferenceaware,
title={Inference-Aware Meta-Alignment of {LLM}s via Non-Linear {GRPO}},
author={Shokichi Takakura and Akifumi Wachi and Rei Higuchi and Kohei Miyaguchi and Taiji Suzuki},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=Km8yqb7SfN}
}