Aldi Fahrezi

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models

Foundation models (FMs) adapt surprisingly well to downstream tasks with fine-tuning. However, their colossal parameter space prohibits their training on resource-constrained edge-devices. For federated fine-tuning, we need to consider the smaller FMs of few billion parameters at most, namely on-dev…