Darius Catrina

Reverse Distillation: Disentangling and Scaling Protein Language Model Representations

Unlike the foundation model scaling laws seen in natural language processing and computer vision, biological foundation models scale relatively poorly. For example, the ESM-2 family of protein language models plateaus at 650M-3B parameters on ProteinGym benchmarks. We address this limitation by intr…