Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature
Task Arithmetic yields a modular, scalable way to adapt foundation models. Combining multiple task vectors, however, can lead to cross-task interference, causing representation drift and degraded performance. Representation drift regularization provides a natural remedy to disentangle task vectors;…