SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
Yewei Liu, Xiyuan Wang, Yansheng Mao, Yoav Gelberg, Haggai Maron, Muhan Zhang
Abstract
We propose SHINE (Scalable Hyper In-context NEtwork), a scalable hypernetwork that can map diverse meaningful contexts into high-quality LoRA adapters for large language models (LLM). By reusing the frozen LLM's own parameters in an in-context hypernetwork design and introducing architectural innovations, SHINE overcomes key limitations of prior hypernetworks and achieves strong expressive power with a relatively small number of parameters. We introduce a pretraining and instruction fine-tuning pipeline, and train our hypernetwork to generate high quality LoRA adapters from diverse meaningful contexts in a single forward pass. It updates LLM parameters without any fine-tuning, and immediately enables complex question answering tasks related to the context without directly accessing the context, effectively transforming in-context knowledge to in-parameter knowledge in one pass. Our work achieves outstanding results on various tasks, greatly saves time, computation and memory costs compared to SFT-based LLM adaptation, and shows great potential for scaling. Our code is available at https://anonymous.4open.science/r/metalora-734E
BibTeX
@inproceedings{
liu2026shine,
title={{SHINE}: A Scalable In-Context Hypernetwork for Mapping Context to Lo{RA} in a Single Pass},
author={Yewei Liu and Xiyuan Wang and Yansheng Mao and Yoav Gelberg and Haggai Maron and Muhan Zhang},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=ZMexYcAibv}
}