2026
LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
ICLR 2026poster
Recent attempts to combine low-rank adaptation (LoRA) with mixture-of-experts (MoE) for multi-task adaptation of Large Language Models (LLMs) often replace whole attention/FFN layers with switch experts or append parallel expert branches, undermining parameter efficiency and limiting task specializa…