Dongqi Li

Distilling and Scaling Hierarchical Vision Transformer to 30B Parameters

Recent efforts have successfully scaled plain Vision Transformers (ViTs) to unprecedented sizes, ranging from 6B to 22B parameters. In contrast, their hierarchical counterparts have remained largely constrained to less than 1B parameters. To bridge this gap, we propose EHV, an efficient hierarchical